Types of invariance

'Invariance' is when different patterns are mapped onto a single output dimension. For example, many different images of dogs might all activate one 'dog-detector' (one of several output dimensions), while still activating other output dimensions of size, furriness, or position.

But there can be many reasons to map different patterns onto a single dimension. Those different reasons correspond to different types of invariance, as discussed below.

1. Tolerance (to jitter, fluctuations, & noise)

   If one pattern is mapped to a certain output dimension, then a nearly identical pattern should clearly be mapped to the same output. For example, if one picture of a dog is mapped to 'dog,' then the same picture with a single pixel changed should be mapped to 'dog' as well. Somehow, the system needs a minimal way of blurring very similar patterns together; at the very least, that can be done by the natural graininess of the input sensors and filters, and may also be done more explicitly at higher processing stages.

2. Hard-wired algorithms

   Sometimes, very different patterns should be mapped to a common dimension, just because we know a priori when designing the system that this should happen. Because the patterns are quite different, the system needs an explicit, algorithmic mapping to discover the invariance; and because the the designers know of this algorithm in advance, they (or Nature) can hard-wire it into the sytem. As a result, the system can discover invariances among patterns unlike any it has seen before, because the mapping algorithm is applied to all patterns, without waiting for any experience or learning. For example, all of the following could be accomplished by hard-wired invariance:

   • Tempo invariance. Suppose a system has learned to map one temporal pattern onto a certain output dimension (say mapping the sequence of notes in a certain tune onto its 'name,' or mapping the pixel-image of a moving contour onto some label for it). If tempo invariance is in use, then hearing the same tune speeded up or slowed down should recall the original name, and seeing the same contour moving faster or slower should recall the original contour label. (There are obvious limits to how much speeding or slowing is allowed before the invariance should be considered broken; perhaps +/- 50% would be a reasonable start).
   • Sequence invariance. Tempo invariance applies to spatio-temporal sequences which are sped up or slowed down uniformly. Sequence invariance applies to the same kind of temporally-extended patterns, but relaxes the uniformity requirement, so that any other input patterns having the same components activated in the same order will map to the common output dimension. In some cases, this seems to be a good idea, for example in understanding words or sentences even if their component phonemes or words are spoken irregularly. But in other cases it makes less sense....if you heard the opening phrase of Beethoven's Fifth Symphony, but with the first three notes unevenly spaced ('daa-di-daaaa, duuuuhhh'), you probably wouldn't recognize it.
   • Echos When we converse indoors, most of the sound energy falling on our ears comes not directly from the speaker, but from reverberation off the hard walls. Yet we are usually completely unaware of those echos, and cancel them unconsciously. An echo-cancelling algorithm--in general, one which discovered consistently delayed correlations/echos across a particular set of inputs--could implement this kind of perceptual invariance, and could be hard-wired to function immediately, without learning.
   • Contrast/intensity We naturally recognize images as the same regardless of their overall illumination, and recognize tunes regardless of how loud or soft they sound. So at any level in the system in which overall 'intensity' or 'contrast' is coded, it would probably be easy and sensible to ignore that dimension when mapping inputs to invariant output dimensions.

   Note that the four examples above are rather restricted: the first three involve transformations of time alone, and the final one only works for dimensions coding a very specific kind of value (intensity, which is distinct from other common scalars like probability or position). The more general case--hard-wiring invariance among different 'spatial' patterns--is nearly impossible, because the patterns themselves have to be learned, and the system does not know a priori which dimensions go into which patterns. Time is the main exception, since it is the only 'dimension' which the system can access directly and separately everywhere; intensity is an exception too, but only at the outermost sensory layers.

   So in order to find the most general kind of invariances (like 'dogs'), we need to explore arbitary mappings.

3. Learned, arbitrary mappings

   This is what people usually think of by 'invariance': utterly dissimilar images (e.g. various views of various kinds of dogs) all get mapped to a single category. Arbitrary mappings like this can be defined easily enough if there is a supervisor (or 'teacher') to indicate which pattern goes into which category... but a natural learning system needs to figure out the categories on its own, without a teacher. What are the naturally available signals from which it might learn these invariances?

   Here are some possible natural clues for learning invariances. They are somewhat vague, and each is doubtless used somewhere yet probably not used everywhere.

   • Similarity. If two patterns are statistically nearly identical (as discussed above), they should be mapped onto the same category. But the same principle can work for patterns which are further apart but still reasonably similar, such as the same dog viewed from slightly different angles, or a banana which is green rather than yello. But it is an open question how similar is similar enough, and such similarity will be measured not in the inputs space (like pixels) but at an intermediate representation.
   • Context. If one pattern shows up in the same 'context' as another, it can be put into the same category. For example, if you see a pink piece of paper being passed as payment for something, you can deduce it is money even if the only money you have seen is green. So this invariance-cue has nothing to do with the content of the pattern itself, not with any aspect of time; it relies purely on whatever other patterns are co-active with it.
   • Temporal contiguity. Perhaps the most generic cue from the natural world about 'what is the same as what' comes from temporal contiguity: most things in the world ('objects') stay as they are, at least over short periods of time, so changes in our perception of them can be assumed to be different views of a single invariant thing. For example, if a baby looks at a dog for awhile, her brain might infer that each view belongs to the same category as the view immediately preceding it, and thus all the view can be chained together into a single category.

     This is an extremely general approach, potentially useful for all sensory modalities and levels of abstraction. It could be used to group together similarly-oriented contours in early vision (as occurs in 'complex cells'), yet also to group together utterances from a single speaker, or life-experiences in a single country. The key is that time is a unique sensory dimesion, and thus provides unique cues to grouping and organizing the myriad other 'spatial' dimensions.

     While this works well at discovering objects whose transformations vary quickly, it cannot work when a tranformation is fixed for awhile. For example, you cannot use temporal contiguity to learn pitch-invariance--that a tune in one key is 'the same' as in a different key--because you never get to hear the two keys right after each other. Nor can you use this trick to discover that a word spoken by one voice is the same as that word spoken by someone else, since you don't get two different people to say the same word in quick succession. This specific failure occurs because temporal contiguity cannot be used as a cue for patterns which stretch across time; if the pattern takes up time, the cue cannot do so simultaneously.

   • Learned transformations. Depending on its design, the system may be capable of learning very general rules about how to categorize patterns, rather than needing exposure to each possible pattern first. For example, the system might learn how to relate arbitrary shapes on the left and right side of the visual field: it learns that a triangle on the left and a triangle on the right belong to the same category, and likewise for a rectangle, circle, etc... so when it sees a dog on the right for the very first time, it can assign it to the same category to which it already assigned the dog on the left. (This is a generalization of the "shifter" hypothesis.)