Visual Representation

Representations consist of content and format. The content is what is being represented, and the format is nature of the representation. Format can be thought of at two levels: physical and informational. For example, representing temperature with the number 8 can be thought of as ink on paper in the physical sense, or as a number in an informational sense. The informational sense is more appropriate for cognitive representations. Physically, mental representation consists of neural anatomy and the chemical and electrical states of neurons. Debates about mental representations are not at this level of abstraction. The representational debates occur over informational notions: propositions, images, bitmaps, etc.

Thus a representation can be considered propositional or image-like as a result of how it interacts informationally with the representing agent. What, then, is a visual representation?

I distinguish two types of visual representation. The first is a representation that is interpreted by a perceptual visual system, but need not have visual information as its content. For example, the level of mercury in a thermometer is a visual representation of temperature, even though temperature is not inherently visual. This is not the sense of visual representation I am using in this dissertation. Mental representations are neurons in the dark; no agent is using light to see them.

The sense I use in this dissertation is of visual representations as representing visual information, which is, roughly speaking, the information a visual perceptual system extracts from a scene. More formally, visual information consists of shapes, their sizes, locations, motions, and spatial relationships between shapes (e.g. connections, overlaps).

This sense of visual representation includes all represented visual information regardless of format (propositions, bitmaps, neural network weights, etc.) (Glasgow et al., 1995). Since format differences are informationally distinct rather than physically distinct, the format of a representation is determined by how the agent interprets that information. For example, there are no physical propositions in the head, but the way the mind accesses some information may reveal the representational format as propositional.

One of Covlan's primitive elements is the point. Entire images could be created out of points, which would result in a bitmap image. This dissertation does not explore the use of bitmaps, though they can be used to resolve symbolic mismatches at the visual symbol level (such as the letter ``o'' and and ellipse).