PROGRAMMING LANGUAGE TECHNOLOGIES
ERCIM News No.36 - January 1999

Eurosearch: a Federation of European Search Engines

by Martin Braschler, Mounia Lalmas, Luigi Madella and Carol Peters


The objective of the EuroSearch project is to build a federation of European search engines. The main aims of the federation are to join forces in order to be able to better compete with global search engines, to enhance the visibility of European web sites, and to help to preserve European language and cultural diversity. The technologies under development will provide linguistic support for querying over different search services and enable the automatic generation of catalogues, best reflecting local cultures within the federation.

The decision of the EuroSearch consortium of industrial (Italia On-Line, Pisa; CINET, Barcelona; EuroSpider Information Technology, Zurich) and academic partners (CNR, Pisa, and Dortmund University) to build a federation of European search engines originated from the consideration that the World Wide Web is still dominated by US culture and so far little effort has been put into promoting European web sites. A study of the incoming traffic of the services provided by the EuroSearch partners determined that about 70% comes from the same country or from countries using the same language; of the outgoing traffic more than 50% is directed to the US, while almost all the rest remains in the country of origin. This situation has been analysed as depending mainly on language barriers between European countries, the poor multilingual support in traditional search engines, and on the US cultural domination of most popular web catalogues.

The EuroSearch project thus aims at helping to restore linguistic and cultural equilibrium on the Web by building a pan-European federation of national search and categorization services. The main objectives of the federation are to:

The Cross-Language Approach

The aim of the EuroSearch distributed, multilingual service is to permit users to enter queries in their own, or their preferred language, and to carry out search and information retrieval over some or all of the federation’s national sites.

Differences in the partners’ document collections and indexing mechanisms have led to the implementation of different search strategies, depending on the collection to be queried. The cross-language search component of EuroSearch thus activates two distinct types of searching:

The languages covered are currently German, Italian and Spanish, plus English. The two approaches are integrated through the development of common translation server interfaces and data exchange formats; this will facilitate future extensions of the Eurosearch components.

A preliminary simplified prototype of a Translation Server has been developed and integrated in the Arianna search engine, allowing queries in Italian to be formulated and directed to Alta Vista. This server will be extended with the addition of a corpus-based query expansion mechanism. In 1999 the integration of the linguistic resources on all the federated services will be completed.

The Automatic Categorization Technology

Another important goal of the project is to facilitate the creation of Web catalogues by developing techniques for the automatic categorization of documents. In this way, even small corporations will be able to develop their own catalogues.

The categorization approach is grounded on an automatic textual analysis of web documents associating weighted terms with documents. The determination of the weighted terms is based on the description-oriented indexing approach developed at the University of Dortmund. It takes into account features:

The weights are probabilistically determined using the Least Square Polynomial (LSP) approach and a test-bed of pre-categorized documents taken from the ‘Computers and Internet’ part of the Yahoo! catalogue.

This approach produces two main results:

The approach is fully automatic, and is portable to the various languages involved in the federation. We are currently applying our techniques to German web documents from the DINO-online catalogue.

A preliminary on-line prototype is now running, and an engineered version is available in the Arianna catalog. This is one of the first examples in the world of automatically generated catalogues available on the Web.

For further information and demos, see the EuroSearch Web site at: http://eurosearch.iol.it/

Please contact:

Luigi Madella - Italia Online
Tel: +39 050 944258
E-mail: l.madella@pisa.iol.it


return to the ERCIM News 36 contents page