Testing and Specifications

“Quality is never an accident.  It is always the result of intelligent effort.”- John Ruskin

Testing the performance and robustness of an algorithm requires specifications (aims and tolerances) for all test parameters to be set.

 

Test Capability Index (TCI)

The Test Capability Index (TCI) is used to evaluate the noise:

TCI

where USL is the Upper Specification Limit, LSL is the Lower Specification Limit and 6σ is the variability of the process.  We work with customers to establish the TCI required to accept a test algorithm or systems.  This measure is particularly important when multiple capture devices are required to provide identical output data.

Correlation to Visual

Algorithm output is often required to match the assessment of human observers.  For instance, the amount of noise in a cell phone image can be measured using a simple flat field and a standard deviation algorithm.  To determine if the algorithm works to predict when an image is objectionable, images need to be ranked by humans and the algorithm is correlated to the ranking.  An objectionable (fail) level is established by the customer after a correlation is established.

 

Just-Noticeable Difference (JND)

Metrics that are correlated to visual can also be analyzed to calculate a JND.  For instance, a metric may have a JND of 1.0 to indicate that a human observer would notice a difference in the image when the metric increases or decreases by 1.  A study using human observers is conducted to determine JND.

 

Cumulative Match Characteristic (CMC)

The CMC curve shows how well a matching function performs against a known “ground truth” set.  Well-performing algorithms can detect a true match (according to the ground truth), without indicating any false positive matches.  These curves approach 100% quickly.   Poor algorithms are not able to find the correct matches and get to 100% with a lower slope.  We use the CMC curve to evaluate algorithms and to determine algorithm acceptability with customers.