Embodied Schemas for Cross-Modal Mapping in the Design of Gestural Controllers
Symposium:
- ISEA2011: 17th International Symposium on Electronic Art
-
More presentations from ISEA2011:
Session Title:
- Media Control Interfaces
Presentation Title:
- Embodied Schemas for Cross-Modal Mapping in the Design of Gestural Controllers
Presenter(s):
Venue(s):
Abstract:
Alongside changing processes and theoretical frameworks for making art, artist-researchers working with digital technology are extending the physical interface by incorporating gestural control into their work, looking at ways to read human actions and ways to exploit the ways in which people understand the world through their body. Applications of gestural controllers, such as these, can be seen in a wide variety of presentation formats, from interactive installation and performance to gaming. Such technologies allow for the design of novel interactive experiences but challenges still remain in designing controllers that support expressivity, meaningful interaction and intuitive control.
Embodied Cognition is an area of cognitive science that focusses on perception, cognition and action as profoundly shaped by the human experience of having a body and living in a physical environment and in a culture. It offers a new way and an alternative approach that complements current initiatives in the design of interactive technologies and gestural controllers. Embodied Cognition offers a novel way to analyse the complex interactions between user and technology in terms of the fundamental categories of embodied existence.
Image Schemata are fundamental categories that are neurally encoded, pre-conceptual symbols that are recruited from experiences of bodily movement and perceptual interaction, including the body’s interaction with its environment in terms of spatial relations, perception of force and magnitude. These categories are involved in organizing mental representations into meaningful coherent units and are implicated in the formation of new concepts. This paper looks at the design of meaningful controllers that depend upon empirical knowledge of fundamental categories in terms of how the body interacts in and understands its environment. It proposes that these structures may be used in order to identify correlations between gesture and other phenomena such as sound and colour. Such correlations are the necessary building blocks of a grounded cross-modal mapping schema on which to base the design of controllers that allow for meaningful gestural interaction with music and image.
In doing so, it presents an interdisciplinary approach to the design of interfaces to digital technology, one that can have considerable impact in the arts and technology domain.
