Towards a Synthesis of Text and Image

Multimedia technologies offer unprecedented opportunities for combining textual, auditory and visual content, enabling artists to develop more or less complex interrelations between diverse media, and it is sometimes argued that since digital text, sound and image are all ultimately represented in the same way (as sequences of bits in computer memory) they are therefore freely interchangeable. In principle, it is not difficult to output any kind of data in any desired form -for example, a stream of text bytes can be interpreted as image data, or output as a sequence of sounds. However, the results of such transcriptions are very often meaningless because of syntactic disparity or dimensional mismatch (text is essentially one-dimensional, whereas images exist in two or more dimensions), or because things expressed in one medium might be difficult or impossible to express in another. This research attempts to forge meaningful connections between text and image through the use of generative grammars. The grammars, implemented in software, are used for the origination of visually descriptive or evocative texts; the same or similar grammars are used also for picture generation, enabling examination of the relationship of text to image, and of the potential of computer-generated texts as aids to visual creativity.