Human Computer Interaction
ERCIM News No.46, July 2001 [contents]

Multimodal Interfaces for Multimedia Applications: Consequences on the Interface Design

by Laurent Romary and Nadia Bellalem

Under the auspices of the European project MIAMM, two teams of INRIA collaborate to define new types of multimodal interactions involving strong language recognition and understanding capacities.

Considering the ever growing size of information available on line, coupled with the wider variety of potential users which are likely to demand some kind of access to it without necessarily being computer experts, it is mandatory to provide access means which are basically based upon human spontaneous communication abilities. Such interfaces must obviously incorporate dialogue component based on speech and gesture, without losing fundamental qualities like simplicity, efficiency, speed and robustness. Conversely, the system must be able to answer either in natural language or by means of an adequate presentation of available information in a graphical way, probably coupled with force feedback devices to get a better grasp on the actual organisation of the information which is being presented.

The aim is thus to design a human-computer flexible interface which is able to adapt to the communicative capabilities of the user by interpreting the user’s requests in the application context. The development of such interfaces may have strong consequences on more fundamental research on man-machine communication:

In the context of a multimodal interface comprising a haptic device, it is essential to analyse how such a presentation can integrate the third dimension (depth). It should be both more natural from a perceptual point of view, and provide situations where direct action on the scene is preferable to communication with the virtual assistant (situations of emergency, re-organization of the scene etc.). This opens a wide range of possible research that has been barely tackled so far.

These various aspects have been for long part of the core activities of the ‘Langue et Dialogue’ team at Loria, mainly in the context of multimodal reference interaction with ‘classical’ designation device, but our strong involment in the future MIAMM European project (Multidimensional Information Access using Multiple Modalities). This project aims at providing a multimodal interface based on a haptic device to provide access to multimedia data. In collaboration with the Parole team at Loria, we will develop the following aspects:

The MIAMM project will start in September 2001 and also involves DFKI, Sarrebruck; TNO, Netherlands Organisation for Applied Scientific Research, Delft; Sony Int. Europe GmbH, Stuttgart and Canon Research Center Europe Ltd., Guilford.


Please contact:
Nadia Bellalem — LORIA
Tel: +33 3 83 50 20 33