Development Tools for Dynamic Environment-Adapted Mobile Interfaces

by Anthony Savidis and Constantine Stephanidis

One of the main challenges in mobile computing is to maintain continuous high-quality interaction between users and interactive applications. The Voyager toolkit, developed at the Human-Computer Interaction Lab of ICS-FORTH, provides all the necessary implementation components to build interactive applications that are capable of utilizing different types of User Interface input-output resources, as they are dynamically discovered or lost during mobile interaction.

In mobile computing, users are faced with a dynamically changing environment, and are offered a large and varying range of remote wireless devices. In this situation, users may carry small 'pocket style' processing units, emphasising mainly wireless networking (such as BlueTooth), an embedded OS, and some locally running applications, while all interaction may evolve dynamically by utilising such remotely available devices (ie the small machine offers no UI equipment). Dynamic configuration and continuity of interaction are key ingredients in this world of mobile applications, supporting distributed user interfaces where the input and output (I/O) devices (such as buttons, text lines, raster displays, loudspeakers etc) are discovered 'on the fly' during user mobility.

The dynamism and continuity of interaction inherent in these scenarios are at the root of the 2WEAR project, for which FORTH (coordinator), ETHZ (Switzerland), MA Systems & Control (UK), and NOKIA/NRC (Finland) have joined forces to explore mobile computing as an agglomerate of small, wearable and physically distributed devices. Funded by the EC, 2WEAR took off on January 1, 2001 with the objective to build such a system, in terms of both software architecture and devices that communicate using short-range radio technology, and deliver it by December 2003.

In dynamic, transient environments, the main architectural challenge is to maintain high-quality interaction between users and applications. This principle of continuity emphasises the uninterrupted sequence of dialogue activities and ensures that the human-computer interaction dialogues transform gracefully from the user’s perspective. Developed at the Human-Computer Interaction Lab of ICS-FORTH, the Voyager Toolkit is a User Interface (UI) development framework (delivered as a C++ software library) which provides interface programmers with all the necessary implementation components to build interactive applications that utilise different types of UI I/O resources, as they are discovered or lost during interaction. As shown in Figure 1, a 2WEAR UI is implemented as a typical client UI, while the UI I/O resource distribution is handled underneath, through the 2WEAR communication technology.

The Voyager toolkit places abstract interactive objects (the Selector and TextEntry) at the service of the programmer, hiding run-time coordination and control details, and revealing only semantic characteristics at the API level. These objects are internally linked to the lower-level management of remote UI resources to cater for I/O control through various levels of specialisation that are completely hidden to the application programmer. Programmers thus have an effective instrument to easily craft mobile dialogues with diverse dynamic UI I/O resources. Furthermore, the single-focus policy for objects and applications has been enforced, meaning that a single application can have the focus in a mobile system and that only the focus object will actually allocate resources for interaction purposes. This modality, which is concurrently supported in Windowing systems, turned out to be a necessary decision in mobile interactions, since the physical display 'breadth' of the devices which are dynamically engaged does not allow concurrent dialogues to be practically supported.

When new resources appear, if dialogue is 'suspended', no action is taken, since the application does not own the focus. Otherwise, the Voyager UI framework substitutes the instantiation of the current focus abstract object with the best possible I/O instantiation without affecting the state of the focus abstract object. When new UI resources are discovered, the UI framework finds the best possible I/O instantiations by recycling the resources currently used by the focus, plus those just discovered, as well as those that are not granted and are thus available. If the best instantiations found are the ones currently in use, no actions are taken. Otherwise, the current I/O instantiations are released (assuming dialogue was not 'stalled', in which case those instantiations are null), while the current best-found input and output instantiations are activated (when output instantiations are activated, or when they are associated with a different abstract object, they call their display method automatically). In the case where the dialogue was previously in a 'stalled' state, and is now in a 'working' state, dialogue revival is automatically signalled.

When UI resources are lost (either due to network failure, or to lack of responsiveness, or because the services are gracefully removed), if the dialogue is already 'stalled', no further action is taken. Otherwise, the UI framework asks the current focus object if it happens to use any of the lost resources. If not, no action is taken, but if this is the case, the current input and output instantiations are released, and a configuration round is entered, seeking for the best possible input and output instantiations. If no such instantiations can be found with the currently available resources, the dialogue is signalled as 'stalled'. Otherwise, the input and output instantiations that have been found are activated.

To conclude, the challenges in implementing mobile interactive applications capable of dynamically utilising the UI resources discovered in the surrounding environment are: (a) handling the dynamic resource coordination and control requirements at runtime, and (b) offering a compact and invariant programming model which is intact from the variety of dynamic UI resources. The Voyager UI development framework has been expressly designed and implemented to address these two challenges.

Please contact:
Constantine Stephanidis, ICS-FORTH
Tel: +30 2810 39 1741
E-mail: cs@ics.forth.gr