Numerical Utilities - Num-Util

General Purpose Vector and Matrix Math Utilities
Version 1.2

The Numerical Utilities perform common mathematical operations on files containing lists of numbers. Files may represent vectors or matrices. The files may be plotted by various tools, such as XGRAPH, and the utilities may be considered a means to manipulate data to be plotted.

The Num-Utils are useful for anyone studying Digital Signal Processing (DSP), linear algebra, statistics, or numerical analysis in general, as well any field dealing with numerical data. The utilities form a simple and inexpensive alternative to-, or can be used to complement-, more extensive environments such as Octave, Matlab, or programming libraries such as LAPACK by leveraging the familiar capabilities of the operating system's command-line and/or script files. Num-Util enables file-names to be treated as variable-names.

List of Num-Utils:

 Scaling / Shifting
  scale		- Scales a real or complex vector file by a scalar.
  shift		- Adds a scalar to a real or complex vector file.
  normalize	- Normalize a vector.  Makes sum (or max) of terms equal 1.0.

 Generating Functions
  sine_gen	- Generates sinusoidal vectors.
  rand_gen	- Generates random vectors, noise.
  enter_vec	- Creates new vector or matrix by keyboard prompts/input.
  identity_matrix - Generates identity matrix.

 Conversions
  abs		- Computes absolute value, or complex magnitude.
  cmag		- Converts a complex vector file into real (magnitude) vector.
  db		- Converts a real vector file from linear to deciBells (dB).
  phase		- Converts a complex vector file into magnitude and phase.
  real2cmplx	- Converts a real vector to a complex vector.
  extract_real      - Extracts real part of complex vector to real vector.
  extract_imaginary - Extracts imaginary part of complex vector to real vector.
  barchart 	- Converts real vector to bar-chart graph.
  image2matrix	- Convert an image file to a matrix file.
  mat2image	- Convert matrix to an image file.
  num2vec	- Converts simple number list to NumUtils Vector format.
  vec2nums	- Converts real or complex vector files to simple number list files.

 Operations / Transforms
  add		- Adds two real or complex vectors or matrices.
  sub		- Subtracts one vector or matrix from another.
  conjg		- Conjugates a complex vector.
  dotpr		- Computes the dot-product (inner-product) of two vectors.
  cdotpr	- Complex-conjugate dot-product of two complex vectors.

  matmul	- Matrix-matrix or Matrix-vector multiply, real or complex.  
  multpbp	- Multiplies real or complex vector by vector (point by point).
  outpr		- Outer-product multiply. Multiplies two vectors, produces matrix.

  zfill		- zero-fills a vector to arbitrary length.
  convolve	- Convolves two complex or real vectors. (FIR filter)
  correlate	- Computes the correlation of two complex or real vectors.
  fft   		- Computes FFT on a complex or real vector (of any length).
  window	- Computes windowing functions, hamming, hanning, bartlet, blackman.
  transpose	- Transposes real matrix, or conjugate-transposes complex matrices.
  ucirc		- Produces Unit Circle points for Z-plane plots.
  fft2D		- Computes the 2D-FFT on a complex or real matrix.
  eigen		- Computes Eigen vectors and values.
  svd		- Computes Singular Value Decomposition.
  z_trans	- Z-transform, finds roots of arbitrary polynomials.
  matinvert	- Invert a real or complex matrix.
  matXvec	- Extract matrix row or column to a vector.
  mdeterm	- Compute the matrix determinant, real or complex.
  augmatvec	- Augment matrix by vector.  Adds row or column to matrix.
  join2xy	- Joins 2 Real Vectors into one Vector, becoming x and y.
                  Useful for plotting one file against another.            
  differentiate - Differentiates a real vector.
  integrate 	- Integrates a real vector.
  decimate 	- Decimates (sub-samples) a real or complex vector.



 Statistics
  statistics	- Computes mean, variance, min, max, and
			optionally the median, of a real vector.
  histogram	- Plots the histogram of real vectors.


TBD:  convolve2d, more examples, tutorials, etc.. Any volunteers?

Background:

The Num-Util are provided as source-code, and are conveniently compiled on most any platform with a C compiler. The utilities are simple and self-contained. No additional tools or libraries are needed. A simple Build.com script is provided to automatically compile the utilities, or to show the suggested compilation commands. Most of the Num-Util programs are under 100 lines of code. (grep ";" xx.c | wc)

Num-Util is consistent with the convention of other file utilities, such as copy, diff, cat, sort, zip, grep, awk, xgraph, latex, etc.. All data files are ASCII text files, (not binary) which are readable, printable, and convenient to produce, edit, or read with other programs. Being file-based utilities, data is not limited by memory (RAM size).

Because each Num-Util tool is an independent utility, they can be called from programs without risk of library or symbol conflicts. (eg. system() call.) They can also be called directly from the command-line, -interactively-, without need of writing a program. Although many of the functions are trivial, the collection has value by virtue of it's consistency and completeness.

Num-Util presently supports four formats for data files:

	(1) Real Vector, (2) Complex Vector,
	(3) Real Matrix, (4) Complex Matrix.

A simple XML-like convention in the data-file's first line indicates the file's type, and enables the tools to operate consistently on either kind of data-file. The header also specifies the file's dimension(s), as either vector or 2D matrix. See below for format specification. The utilities include extensive checking and are fully documented.

The venerable Num-Util have been in use by several groups of researchers for more than a decade. We noticed others often re-inventing common functions on projects without knowledge of the Num-Util, and realized they may be beneficial to share with others. Therefore, we hope these utilities and conventions will be helpful, and that contributors might combine efforts by adding additional functions and utilities consistent with the initial set, as well as to help in improving and enhancing them, or that it serves as the basis or template for other endeavors. Num-Util are released under GPL, included in gpl.txt. So that everyone may share in combined efforts, we request that improvements/comments be directed back to xgraphadmin@xgraph.org for centralized re-distribution with credit.

Download:

Click (or right-click) Download NumUtil.tar.gz . Vers-1.2 (44 KB)

Click (or right-click) Download NumUtil.zip . Vers-1.2 (122 KB)

Build/Install:

Unpack: (Unpacks into a sub-directory called numutils.)
cd to a directory above where you wish to install/update numutils
gunzip numutil.tar
tar xvf numutil.tar
Go down into the new numutil directory.
cd numutil
Source Build.com
This makes executables in numutils/bin.
Place numutils/bin in your path. (Use full absolute location.)
For example, assuming you unpacked into /home/bart/numutils, and you are in csh or tcsh:
set path = ( /home/bart/numutils/bin $path )
For bash shell, it would be:
export PATH=.:/home/bart/numutils/bin:$PATH

Test by trying some examples. Some example data files and run-scripts are provided in the examples directory.
Try: source script_fft.com
(
sine_gen -f 3.0 -n 100 swA.dat
sine_gen -f 11.0 -n 100 swB.dat
add swA.dat swB.dat composite.dat
xgraph -a -pl composite.dat time.title &
fft composite.dat composite.fft
cmag composite.fft composite.spectrum
xgraph composite.spectrum spectrum.title &
)
Try: source script_convolve.com
(
convolve sequenceA.dat sequenceB.dat sequenceC.dat
xgraph sequenceC.dat
)
Try: source script_image.com
(
sine_gen -n 100 -f 2 -realvec v.dat
outpr v.dat v.dat m.dat
mat2image m.dat m.ppm
gimp m.ppm
)
Try: source script_color_image.com




Data File Formats

The first line indicates the file type.  It must start with 
an exclamation mark, followed by dimensional and typing
information of the form:
	! <Type> ? </Type>

Where "?" is one of:
	- Real Vector
	- Real Matrix[,]
	- Complex Vector
	- Complex Matrix[,]

Specifically:

 - Real Vector - Data stored as:  (index) (value)
	Example:
		!<Type> Real Vector </Type>
		! indx  value
		  0	45.6
		  1	23.2	
		  2	10.9

 - Real Matrix[rows,cols] - Data stored as:  (row) (col) (value)
	The column is the more rapidly changing index.
	Example:
		!<Type> Real Matrix[2,2] </Type>
		! r c   value
		  1 1	52.1	! M[1,1]
		  1 2	34.5	! M[1,2]
		  2 1	29.4	! M[2,1]
		  2 2	91.8	! M[2,2]

 - Complex Vector - Data stored as:  (real) (imaginary)
	Example:
		!<Type> Complex Vector </Type>
		! real  imag
		  0.89	0.34
		  0.56	0.77
		  0.43	0.18

 - Complex Matrix[rows,cols] - Data stored as:  (real) (imag)
	Example:
		!<Type> Complex Matrix[2,2] </Type>
		! real  imag
		  0.89   0.34	! M[1,1]
		  0.56   0.77	! M[1,2]
		  0.43   0.18	! M[2,1]
		  0.67	 0.25	! M[2,2]

With this scheme, other formats could be defined, such as integer vectors, or higher dimensional matrices, but no such formats are presently used.

Optional comment fields can be added by starting with "!". Everything after the "!" until end of line is considered a comment.

Comment fields can also be added by using the C-style comment designators, "/*" and "*/". These may span multiple lines.