WWW Interface Design, Driven by Heuristic Evaluation:

The EINS-Web Project


S. Mangiaracina

P.G. Marchetti


Consiglio Nazionale delle Ricerche

Area della Ricerca di Bologna

European Space Agency — ESRIN

Informatics Department







This paper describes the experience of the evaluation and design of the EINS-Web user interface. EINS-Web allows the access to distributed collections of bibliographic and textual databases, together with a seamless interaction with the whole World Wide Web on Internet. The heuristic evaluation of the Graphical User Interface (GUI) was run in two steps. The results of the evaluation of the first version of the GUI were used to drive the design of the Web version. This paper describes the methodology used and the lessons learned. The interaction among evaluators and designers proved to improve the success of a spiral design methodology, which is needed to cope with the requirements of designing interfaces targeted at the rapidly evolving Internet world.

1. Introduction

Four different approaches can be used to perform GUI evaluation: formal (by means of technical analysis), automatic (by means of ad hoc software tools), empirical (by means of experiments together with user) and heuristic (judgments and opinions stated after the interaction with the GUI) [1]. We did select the latter methodology, because the heuristic evaluation is largely independent of the software and it is proactive, allowing picking up easily the suggestions and adopting them during the design and its review. During the EINS Web project we decided to drive the interface design by means of the heuristic evaluation, using a spiral design approach.

Heuristic evaluation is performed by looking at an interface and trying to come up with an opinion about what is good and bad about the interface. Ideally, people would conduct such evaluations according to certain rules, such as those listed in typical guideline documents. Most people perform their own "heuristic evaluation" on the basis of their intuition and common sense instead.

In order to evaluate a GUI that allows access to collections of bibliographical and textual databases, an information problem has to be formulated and opinions on how the interface supports the user throughout the satisfaction of his/her information needs have to be assessed.


2. The evaluation process

EINS-Web allows accessing distributed collections of bibliographic and textual databases, together with a seamless interaction with the whole World Wide Web on Internet (see Figure 1 for an overview of the architecture).

Figure 1 EINS Web Architecture

Figure 2 EINS-Web Advanced Search Form




Figure 3 EINS-Web Title List Form

The evaluation of EINS-Web has been carried on in the Library of the National Research Council (CNR) in Bologna (Italy), which was selected as the test site, involving several researchers of the CNR campus with their real information problems (in various fields, such as chemistry, material science, electronics, physics, geology, environment, etc...) and a mixed group of evaluators, composed by information specialists and experts in user interfaces.

Figure 2 and 3 give an idea of the look and feel of EINS-Web GUI.

In the first step towards the construction of the EINS Web interface, we re used the design efforts for the development of the previous version of the interface (BRAQUE PC) (BRAQUE = BRowse And QUEry), developed for the Windows environment.

The design of the BRAQUE PC interface was initially based on an analysis of users’ information seeking behavior and Cognitive Task Analysis [2]. As a matter of fact, the information seeking strategies can identify a multiple dimensional space. This space is characterized by the information problem, by the nature of the information itself (e.g. information, meta-information), by the user’s goal (e.g. learn, select), by the information access method (e.g. browse, search) or by the information access mode (for example: recognize, specify). During the design process we understood that different information seeking behaviors were relying on common functional elements. We used the two basic functional elements: the browser and the searcher to cast the multidimensional space defined by the identified information seeking strategy [3]. The heuristic evaluation method was then evaluated and used to assess user satisfaction, ease of learning, ease of use, error prevention and efficiency of the interface and as a feedback tool to drive a spiral design process. Evaluator’s judgment was based on the nine heuristics from Nielsen (see Table 1) [4].

Any usability problem was fitted and analyzed against one of the nine criteria (see Table 2). A number of issues were detected, mainly of aesthetic nature. Some were serious, influencing heavily the interaction. Where possible, suitable solutions were identified. Sometimes solutions could be drawn from the nature of the problem itself.

The designers then revised the report produced by the evaluation team [5]. It came up that in some cases they were already aware of the presence of an interface problem, but did show that the implementation approach was lead by precise constraints from the underlying architecture. In most of the cases the developers agreed with the evaluators recognizing the usability problems. In one case only the disagreement could not be resolved. For BRAQUE PC the designers used the classic desktop metaphor. A good management of buttons, windows and pull down menus together with a simple use of colors, makes simple and effective the objects' manipulation. The designers selected a task-oriented approach, associating different tasks to different environments (windows). In BRAQUE PC it is possible to interact with the databases either via the query language or via graphical objects. This possibility was considered very important, in fact it allows either to enter the commands in the query language or to interact with graphical tools (and eventually verifying the correspondence with the query language commands). Good also was considered the possibility to record on a file any operation executed (log file). This is especially useful for search results and queries.

A new report was produced [6], putting together evaluators’ and designers’ replies, that served as "seed" to drive the design process for the World Wide Web version of BRAQUE. This new version was called EINS Web following the decision of the international EINS consortium that was going to exploit the interface.

The report provided a form containing the evaluation criteria and rules to be followed in the development of the new EINS-Web GUI. The EINS-Web heuristic evaluation form contains a suitable subset of the 101 usability heuristics as retrieved and analyzed from a database of 249 usability problems by Nielsen in [7]. The heuristics have been selected according to our judgment that they were likely to fully describe problems in the WEB interface and that they could be easily understood by different evaluators (see Table 3).

A second evaluation session was then run. Students of the Department of Computer Science of the University of Bologna, who had taken the one-year course in User Interface Design and Evaluation, carried on the evaluation in the Library of the CNR. Before arranging for the EINS-Web evaluation session, the students were given a training seminar on architecture and languages of information retrieval systems, which most of them had never previously used. The heuristic evaluation form (see Table 3) was then distributed to the evaluators and discussed with them. For each usability problem a separate column was given to be filled with a rating value: we agreed to assign values from 1 (the interface does not keep into consideration the usability problem at all) to 5 (the usability problem has been completely worked out). The evaluators were requested to identify potential usability problems and to tie each problem found to the specific heuristic it violated. As in the previous evaluation, multiple heuristics could be linked to any given violation. Finally, the evaluation session was carried on. The same evaluation, dealing with same information problems, was run by four different groups of students in parallel sessions. One group tested the interface using the Internet Explorer browser, while the others performed the evaluation using the Netscape browser.


Figure 4

The feedback to the designers (see figure 4) allowed to modify the design during the implementation phase.

3. Lessons learned

Here is our assessment of the evaluation methodology:

Interface evaluation:

The second issue related to user background knowledge and user conceptual model. It was in fact recognized that EINS-Web should provide the same or improved functionality already present in EINS PC. The presence of a time-out resulted also very "annoying". Apparently it was not related to any particular user action or network error. Users get disconnected without notice or error message (or, after having waited for the result of a query, and having stopped the execution with the Browser’s STOP button, the system replied with the time-out disconnection message error). This actually happens because there are three different time-outs, not tied or synchronized to each other, that convey to the user the erroneous model of a "single wild time-out".

As preliminary conclusion we feel that the interaction among evaluators and designers proved to improve the success of the spiral design methodology depicted in figure 4, which is needed to cope with the requirements of designing interfaces targeted at the rapidly evolving Internet world.


We are indebted with Prof. Cesare Maioli of the Department of Computer Science of the University of Bologna, and his students of the User Interface Design and Evaluation course, for the time and effort spent in the evaluation of EINS Web interface. CINECA designed the look and feel of the EINS Web interface as available to the EINS users at http://www.eins.org. The technical implementation was done by Vitrociset for the European Space Agency and the EINS consortium in the framework of the BRIDGE and CIME projects co-financed by the European Union.


[1] Nielsen, J., Mack, R. L. Editors; Usability Inspection Methods, Wiley and Sons Inc. , 1994

[2] Belkin N., Marchetti P.G., Cool C. BRAQUE: Design of an Interface to Support User Interaction in Information Retrieval, Information Processing & Management, vol.29 n.3, 1993, pp.325-344

[3] Marchetti, P.G.; BRAQUE: A Hypertext-based Interface for Accessing Large Text Databases, Informatica e Diritto, Special Issue: on Hypertext and Hypermedia, vol. III (1994) - n.2 pp.10-112

[4] Nielsen J. and Molich R. Improving a Human-Computer Dialogue; CACM, 33 (3), March 1990, pp. 338-348

[5] Mangiaracina S. , Merivot. F., Statti F.; BRAQUE evaluation report, Technical Report: BRIDGE - IMPACT Project BIS-4017 10284/0 BRIDGE/WP3/EV/PGM/01/00

[6] Zappaterreno P., Marchetti P.G. BRAQUE-WEB Project. An heuristic evaluation guideline

Technical Report: BRIDGE - IMPACT Project BIS-4017 10284/0 - BRIDGE/WP3//HEG/PZ/01

[7] Nielsen, J. Enhancing the explanatory power of usability Heuristics; Proceedings ACM CHI’94 Conf., April 1994, pp.152-158


Table 1 Nielsen’s heuristic guidelines

Visibility of system status

The system should always keep the user informed about what is going on by providing him or her with appropriate feedback within reasonable time.

Match between system and the real world

The dialogue should be expressed clearly in words, phrases, and concepts familiar to the user rather than in system-oriented terms.

User control and freedom

A system should never capture users in situations that have no visible escape. Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue.

Consistency and standards

Users should not have to wonder whether different words, situations, or actions mean the same thing. A particular system action - when appropriate - should always be achievable by one particular user action. Consistency also means coordination between subsystems and between major independent systems with common user population

Helping users recognize, diagnose, and recover from errors

Good error messages are defensive, precise, and constructive. Defensive error messages blame the problem on system deficiencies and never criticize the user. Precise error messages provide the user with exact information about the cause of the problem. Constructive error messages provide meaningful suggestions to the user about what to do next.

Error prevention

Even better than good messages is a careful design that prevents a problem from occurring in the first place

Recognition rather than recall

The user’s short-term memory is limited. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. Complicated instructions should be simplified.

Flexibility and efficiency of use

The features that make a system easy to learn -such as verbose dialogues and few entry fields on each display - are often cumbersome to the experienced user. Clever shortcuts - unseen by the novice user - may often be included in a system such that the system caters to both inexperienced and experienced users.

Aesthetic and minimalist design

Simple things should be simple complex things should be possible. Things should look good, keep graphic design simple, and follow the graphic language of the interface without introducing arbitrary images to represent concepts.



Table 2 BRAQUE PC evaluation results


BRAQUE PC Evaluation


System status visibility

Normally information on system status is available. Feedback is good, as well as overall direct object manipulation

in some instances, when a task is activated, it is not possible to execute other operations. All that is not signaled. It is preferable, in such cases, to change mouse pointer into the hourglass.


Match of the system with real world

The interaction is driven by the desktop metaphor. It is not necessary to know specific terms to use the system. Users must anyway know English language. In the Italian version most of interaction dialogues is in English

In the Italian version most of the messages are in English. In some message box some of the text is in Italian some in English. We suggest to perform a coherent complete translation or to keep the original version.


Control and user freedom


User control of the GUI is good. It is possible to enable via the "edit" menu the "undo/redo" option for one level

In the "Search Results" environment, it is not possible to cancel sets. If a large number of sets are available the user may be confused. If the cancel utility would be available it would be easier to organize the work and keep the environment under control.

Coherence and standard


In general, the standard coherence rules of GUI have been followed.

  • It is possible to perform the same operation (search) in the "Document Searcher" and in the "Idea Finder". The same search executed in the two environments, leads often to two different results. This is considered the most serious flaw of the interface
  • According to Windows95 specifications "x" and "-" buttons placed in the rightmost angle of the window should close the environment and minimize the window respectively. In EINS windows both buttons minimize the window without closing the application.
  • "Document Searcher" and "Document Pool" icons are ambiguous since their label starts with the same word: Document.
  • Tested version was 1.3, but on-line help referred to 1.2.
  • Password is not masked.

Error prevention

Error prevention is good. It is based on a clear organization of interaction tools.

If a file of type "Term Pool" is opened active window is canceled and replaced with the new archive. In such a way previous archive information is lost. The same is valid for the "Document Pool". In the latter case files are on disk and are not lost. However, is not possible to compare two Document Pools.

Recognition instead than recall


Graphical tools offer options clear and self explanatory, it is not necessary to record the steps needed to perform a search

It is not clear the method to be used to insert search keys in the "Document Searcher". The onscreen explanation is not exhaustive. The user has to remember the default operator inserted among the terms on the same line. The fact that a fixed logical AND links the four entry boxes for the search terms is a constraint for the resulting query

Flexibility and use efficiency

The separation of tasks allows a flexible and efficient use of the system

  • Whilst executing commands in remote mode, it is not possible to insert anything locally.
  • In order to save terms in the "Document Searcher" it is necessary to close at least once the window during the working session and then to reopen it.

Aesthetic characteristics and minimalist design

The structure of the interaction tools, the colors' management, the small number of graphical objects are characteristics of the EINS interface, determining a nice looking interface


User aids in recognize detect and correct mistakes

Error messages are always clear

When leaving EINS, the interface suggests to disconnect using the DISCONNECT command even if the command was already executed, and therefore not active





Table 3 Heuristic evaluation form

Visibility of system status

Score (1-5)

Feedback: keep user informed about what goes on


Provide status information


Feedback: show that input has been received


Features change as user carries out task


Feedback provided for all actions


Feedback timely and accurate


Indicate progress in task performance


Direct manipulation: visible objects, visible results


Identity cues system response vs. user’s goal


Show icons and other visual indicators


WYSIWYG; do not hide features


Match between system and the real world


Speak the user’s language


Contains familiar terms and natural language


Metaphors from the real world


Familiar user’s conceptual model


Use of user’s background knowledge


User control and freedom


Undo e redo should be supported


Obvious way to undo actions


Forgiveness: make actions reversible


Ability to undo prior commands


Clearly marked exits


Ability to re-order or cancel tasks


Modeless interaction


User control: allow user to initiate/control actions


Consistency and standards


Consistency: express same thing same way


Consistency: same things look the same


Uniform command syntax


Conform to platform interface conventions


Show similar inf. at same place on each screen


Error prevention


Prevent errors from occurring in the first place


System designed to prevent errors


What planning mistakes are most likely ?


Recognition rather than recall


See-and-point instead of remember-and-type


Make the repertoire of available actions salient


Seeing and pointing: objects and actions visible


What features often missed and at what cost ?


Provide list of choices and picking from list


Minimise the user’s memory load


Easy or difficult to perform (execute) tasks ?


Allow access to operations from other applications


Show icons and other visual indicators


Flexibility and efficiency of use


Shortcuts: Accelerators should be provided


User tailorability to speed up frequent actions


User interface should be customisable