Human Computer Interaction
ERCIM News No.46, July 2001 [contents]

Designing for Tangible Interaction: The BUILD-IT Project

by Morten Fjeld and Matthias Rauterberg

One of the most challenging research questions in human-computer interaction is: what will be the next generation of user interfaces? How can we interact with computers without a keyboard, monitor, and mouse? To find possible answers to these questions, scientists at the Swiss Federal Institute of Technology (ETH) in Zurich and at the Technical University Eindhoven investigate the theoretical grounding, the practical design, and usability aspects of Tangible User Interfaces (TUIs). As a result, the BUILD-IT Project has been developing a planning tool, based on computer-vision technology, to support complex planning and composition tasks.

In the year 1995, inspired by the work of Pierre Wellner in the context of the augmented reality design paradigm (Interacting with paper on the DigitalDesk. Communications of the ACM. 36(7), pp. 86-96), a research line for further investigations of TUIs was established at the Swiss Federal Institute of Technology (ETH) in Zurich.

To compare the advantages and disadvantages of a computer vision based TUI a field study was carried out. During five days of the largest computer fair in Switzerland, four different interaction styles were realized and presented for public use: (1) a command language interface, (2) a direct manipulative mouse interface, (3) a touch screen interface, and (4) our new computer-vision based TUI interface. The aim of the field study was to establish which was the best interaction style. The statistical analysis of all empirically recorded user actions showed clearly that the most appropriate interaction style for complex problem solving tasks is the TUI. We conclude that TUI’s are promising candidates for the next generation of interaction styles.

Encouraged by these results, we started a project with a TUI called the BUILD-IT system. This is a planning tool based on computer-vision technology to support complex planning and composition tasks. The system enables users, grouped around a table, to interact in a virtual scene, using physical bricks to select and manipulate virtual models. A plan view of the scene is projected onto the table. A perspective view of the scene, called side view, is projected on the wall (see Figure 1). The plan view contains a storage space with originals, allowing users to create new models and to activate tools (eg, navigation and height tools). Model selection is done by putting a representative ‘brick’ at the model position (see Figure 2). Once selected, models can be positioned, rotated and fixed by simple brick manipulation (see Figure 3).

Figure 1: A design team using BUILD-IT. Figure 2: Model selection with a ‘brick’. Figure 3: Multiple bricks for two-handed interaction.

Our design practice is grounded on a work-psychological tradition called activity theory. This theory is based on the concept of tools mediating between subjects and objects. Our design philosophy and practice is exemplified by the realization of the BUILD-IT system. Guided by task analysis, a set of specific tools for different 3D planning and configuration tasks was implemented as part of the actual system. We investigate both physical and virtual tools. These tools allow users to adjust model height, viewpoint, and scale of the virtual setting. Finally, our design practice is summarised in a set of design guidelines.

Using the BUILD-IT system as a research platform for tangible interaction, our exploration takes the following path: We first introduce some of the problems related to working in physical and virtual environments, then indicate a few guidelines to achieve what we call natural interaction. Then we give more details about the interaction content, which are configuration and planning tasks. We then come up with new implementations for three-dimensional (3D) navigation. As a particular use of hand-held tools, we introduce alternative ways to control model height in the BUILD-IT system. We also discuss the outcome of our design activity and suggest ways to advance the issues presented before. One continuation of this research line is established with the Visual-Interaction-Platform (VIP) at the Technical University Eindhoven.

The BUILD-IT project was financially supported by the Institute for Construction and Design, the Institute of Hygiene and Applied Physiology, the Institute for Work Psychology of the ETH Zurich, and by the Swiss Commission for Technology and Innovation (KTI project) together with several Swiss and German Companies.

Man Machine Interaction at ETH Zurich:
BUILD-IT Project:
BUILD-IT Publications:
VIP Project:
HCI Publications:

Please contact:
Morten Fjeld — ETH Zurich
Tel: +41 1 632 39 83,

Matthias Rauterberg — IPO-Center for User System Interaction, The Netherlands
Tel: +31 40 247 5215