Search is one of the most important user interfaceelements in any complex Website, including the Public Administration ones.However, there is no standard methodology to evaluate a site's search qualityand there is a dramatic shortage of evaluation studies. This paper representsa first step in this direction. We present an evaluation methodology thatcovers the three main dimensions of the quality of the search process,namely, efficiency, effectiveness, and user interface. The proposed methodologyis applied to a sample of the Italian Public Administration Websites consistingof 155 sites. The most striking results of our evaluation study are thatmore than half of the sites did not have an internal search engine andthat the search quality of the remaining sites was, in general, low, withsome notable variations. In particular, search efficiency was fairly good,especially to the extent that response times were concerned, while searcheffectiveness and search interface were unsatisfying. One serious shortcomingof almost all sites examined was the lack of features for supporting interactionwith users.
 
 

1. Introduction

Search is the key to accessing the information publishedon the Web. It consists of providing some description of the informationbeing sought on the part of the user and having the system retrieve databaseitems that match the description. According to the biannual WWW surveysconducted at the Graphics, Visualization and Usability Center (GVUC) ofthe Georgia Institute of Technology - one of the more reliable sourceson the user data - about 85% of the Web users surveyed claimed to be usingsearch engines or some kind of search tools to find specific informationof interest.

Search is the predominant paradigm for accessingnot only the information published in the whole Web but also in any complexWebsite. Website usability studies suggests that more than half of allusers are search-dominant, about a fifth of the users are link-dominant,and the rest switch between search and link-following, without an inherentpreference (Nielsen 1997). One of the main reasons for preferring searchis that it leaves full control to the user; search is also seen as a wayout when one gets stuck in navigation (Nielsen 2000).

Over the last few years, the research on more effectiveand usable search techniques has produced a number of advances, some ofwhich have been incorporated into commercially available systems. Multilingualand cross-lingual information retrieval (Oard et al. 1999), link-basedranking (Henzinger 2000), automatic query expansion (Carpineto et al. 2001),graphical user interfaces (Hearst 1999, Berenci et al. 2000), XML searchengines (Luk et al. 2000), and multimedia information retrieval (Hauptmannand Witbrok 1997) are just a few examples of the renewed interest for informationsearch and retrieval.

Search is especially important for institutionalWebsites. Institutional websites contain a large number of pages, theymay offer multiple services in a single site, and their users present alarge variation in terms of skills, Web experience, and knowledge of thedomain being searched. Thus, the possession of search tools of good qualitywould be highly desirable for mastering the inherent complexity of suchsites.

However, the problem of evaluating the quality ofinstitutional Websites' internal search engines has so far received verylittle attention. Most of the studies that have recently attempted to evaluatethe quality of the Italian Public Administration Websites focused on contentsand administrative services (http://www.atenea.it/html/risorse/form_ARPA-C.html)or on different elements of the site's user interface (http://www.aipa.it/attivita[2/gruppi[18/accessibilita[3/bozza[1/index.asp),without addressing the search issue. This paper wants to help fill thisgap by offering a two-fold contribution.

The first contribution is a methodology for evaluatingthe quality of internal search engines. The methodology deals with allmain aspects involved in the search process, including efficiency, effectiveness,and user interface. It is comprehensive, yet simple to use. The secondmain contribution is an evaluation study of the search quality of ItalianPublic Administration Websites performed using the introduced methodology.The results of our study suggest that site's search is somewhat overlooked- internal search is, in general, either lacking or of low quality - andprovide some indications as to how to improve its quality.

The rest of the paper has the following structure.We first discuss the main dimensions that affect the quality of search.Then we describe the evaluation methodolology. At this point the evaluationstudy is presented. After describing the choice of the Website sample,the results are illustrated and their implications are discussed. Finally,some conclusions are drawn.

2. Search quality dimensions

Although statistics and demographics on search engineusage may vary between different sources, they seem to consistently rateamong the most commonly experienced problems: (a) slow access, (b) inabilityto find relevant information, and (c) inadequacy of user interface.

The majority of web users surveyed at GVUC over aperiod of a year-and-a-half left a Website while searching for informationbecause of slow access, and the problem appeared to be growing worse. Otherstudies - see (Huberman et al. 1998) and Kobayashi and Takeda (2000) fora more extensive review - cite the number one and number two reason fordissatisfaction as "slow access" and "the inability to find relevant information".

The latter problem - mainly due to the limited rankingcapabilities of available systems - may be exacerbated by a scarce amountof interaction between the user and the system, as observed in many Websearch sessions. Users typically do not peruse long lists of documents(in this paper we will often use "document" and "page" interchangeably)obtained in response to a query and they are only moderately able or willingto refine their queries. Furthermore, users may make very quick judgementsabout a Website's value based on the quality of one or two sets of searchresults, after which they may decide to abandon the site altogether orto forgo the site's search in favor of external search engines like Google(Nielsen 2000).

The quality of retrieved information is cited asone of the main reasons for dissatifaction also by Lawrence and Giles (1998).In addition, they note that users are not necessarily satisfied with severaluser interface elements such as formats for inputting queries and presentationformats of the retrieval results. Nielsen et al. (2001) analytically studythe impact of the search interface details on systems's retrieval effectivenessand user satisfaction, pointing out a number of useful guidelines.

These observations and findings suggest that thequality of search could be conveniently analyzed along three main dimensions,which we term, for the sake of simplicity, search efficiency, search effectiveness,and search interface. Although these dimensions may present some overlaps,they can be evaluated independently.

Search efficiency is mainly concerned with searchspeed and maintenance. Search speed is given by search engine search andretrieval time plus communication delays. While the latter may be influencedby a number of factors such as the speed of connectivity to the networkby users, the performance of the host Web server, and the uneven concentrationof information packets on the network, the former is an isolated technicalproblem depending on the efficiency of the access and retrieval algorithmsused by the search engines. Speed is not the only factor relevant to searchefficiency. As many sites continue to grow and change fast, their internalsearch engines must be maintained and updated. The newly-created pagesmust be incorporated into the search engine's database, and possible deadlinks and duplicates should be eliminated.

Retrieval effectiveness is affected by the techniquesused by the system for indexing the pages to be retrieved and for processingthe user queries. It is also related to the possibility for the user toperform more sophisticated searches or to receive support from the systemfor entering queries and analyzing results. A number of specific detailsof search capabilities can be used to judge retrieval effectiveness, similarto feature charts used for rating external search engines (see, for instance,Web Search Engine Watch, http://searchenginewatch.com/reports/index.html).

The retrieval effectiveness of search systems canalso be evaluated more quantitatively in terms of precision and recall,which measure, respectively, the ability of the system to retrieve onlyrelevant documents and the ability of the system to retrieve all relevantdocuments. Most "quantitative" evaluation methods are based on a combinationof these two criteria and require the possession of a set of documents,a set of topics, and relevance judgements for each topic (i.e., which documentsare relevant to each topic). Several test collections of this kind areavailable, the best known of which are, perhaps, those developed by theTREC (Text Retrieval Conference) initiative (Voorhees and Harman 2000).Although this evaluation method too has shortcomings, it represents themost used and reliable way to measure the retrieval effectiveness of rankedoutput search systems for single-query searches. One practical difficultyof this method is that the system being evaluated must be run on the testcollection, which may be unfeasible in many circumstances.

The user interface is the third main dimension ofsearch quality that we consider. To build a Website that enables user tosearch successfully, it is necessary to look at the process from the user'spoint of view. Following Nielsen (2001), four main aspects should be dealtwith: (a) the user notices the Website supports search, (b) the user expressesa question as a search query string, (c) the user enters the search stringand submits the query, (d) the user receives and interprets the searchresults.

In the next section we describe a search qualityevaluation methodology - based on a number of specific observable features- that accounts for all three main dimensions described above.

3. Evaluation methodology

We were interested in defining a methodology forevaluating the quality of existing internal search engines, without assumingthat their code is made accessible or that they may be used for indexinguser-specified collections. Thus, we concentrated on those quality featureswhose measurement did not require experimenting with test collections.

For each quality dimension - search efficiency, searcheffectiveness, and search interface - we defined a number of features tobe evaluated. For each feature, we defined an evaluation method and anevaluation measure. The search interface features are borrowed from Nielsenet al. 2001, with some small changes due to the different nature of thedomain of interest (Nielsen et al.'s study focused on commercial Websites).

In the following, the three quality dimensions areexamined in turn. For each quality dimension, it is shown a table illustratingits associated features, including name, evaluation methodology, and evaluationmeasure. A complete list of the features that contains a more detaileddiscussion of their rationale as well as of thier evaluation method isalso provided.

3.1 Search efficiency
 
 

Table 1. Evaluation features for search efficiency
 

Name	EvaluationMethodology	EvaluationMeasure
Fast responsetime	Objective estimateby query sample	Yes/No
High coverage	Objective estimateby page sample	Yes/No
No broken links	Objective estimateby query sample	Yes/No
Freshness	Objective estimateby query sample	Yes/No

 

Fast response time. Speed of response timeis assessed by performing a number of sample searches and averaging theirresponse times. Communication delay is estimated by performing differentoperations that require no additional processing time (e.g., link navigation)on the same site at the same time. If the time strictly necessary for searchengine search and retrieval exceeds two seconds, the response time is notdeemed to be fast.

High coverage. The search engine should be able to retrieve any pagecontained in the site. The coverage of the search engine is consideredto be high if all pages of a randomly-chosen page sample may be retrievedby the search engine.

Freshness. The search engine should be incrementally updated as newpages are added to the site. Freshness is assessed by checking whetherall pages of a sample of recently-created pages may be retrieved by thesearch engine.

No broken links. The search engine should not return results containingbroken links. The absence of broken links is assessed by checking if allqueries of a query sample do not yield broken links.
 
 

3.2 Search effectiveness
 
 

Table 2. Evaluation features for search effectiveness
 

Name	EvaluationMethodology	EvaluationMeasure
Case insensitivity	Objective estimateby page sample	Yes/No
Inexact matching	Objective estimateby page sample	Yes/No
Stemming	Objective estimateby page sample	Yes/No
Support forscoped search	Objective inspection	Yes/No
Support forAdvanced Search	Objective inspection	Yes/No
Support forphrase searching	Objective inspection	Yes/No
Support forBoolean search	Objective inspection	Yes/No
Sort by relevance	Objective inspection	Yes/No
Multilingualsupport	Objective inspection	Yes/No
Selectablenumber of displayed results	Objective inspection	Yes/No
Support forrefinement of search results	Objective inspection	Yes/No
Support forquery reformulation	Objective inspection	Yes/No
Use of metadata	Objective estimateby page sample	Yes/No/Doesn'tapply

 

Case insensitivity. Upper- and lower-casecharacters should not be distinguished when indexing pages and queries.Case insensitivity is assessed by checking whether a randomly-chosen pageis retrieved by the search engine using a same keyword with different cases.

Inexact matching. Users may easily get confusedby their search results when the search engine looks for exact matchingonly. The system should accomodate for multiple-word queries by allowinginexact matching between the words in the query and the words in each document.Inexact matching is assessed by checking whether a two-word query retrievesthe documents that contain either word.

Stemming. The search engine should reduceeach word to word-stem form, at least for the standard inflections of nounsand adjectives. Stemming is assessed by checking whether a randomly-chosenpage is retrieved by the search engine using a same word with differentvariant forms (i.e., singular/plural, male/female).

Support for scoped search. Scoped search letsuser limit the search to results from specific areas of the site. Thisfunctionality may be especially useful when a single site offers multipleservices.

Support for Advanced Search. Advanced searchmay be useful for experienced users in some circumstances, notably as analternative option to default search after a poor search results page.Two specific forms of Advanced Search are considered below.

Support for phrase searching. On request,the system should be able to do an exact search for the whole string enteredby the user.

Support for Boolean search. Boolean searchshould be supported, as this may be more effective than best matching searchfor experienced users.

Sort by relevance. Sorting the results byrelevance is essential when the number of hits retrieved in response toa query is large, as it is often the case.

Multilingual support. Multilingual queryingand document presentation could be useful for many potential users whocannot write or read Italian.

Selectable number of displayed results. Thepossibility for users to choose the number of results returned on the firstpage makes the interpretation or peruse of results more flexible.

Support for refinement of search results.The system should provide the user with mechanisms to refine or enlargethe search results, which are notably useful when the search engine returnstoo many results or no results at all.

Support for query reformulation. Search interfacesshould show users easy ways to refine queries, mainly based on the numberand characteristics of search results and on the knowledge of the contentof the documents being searched.

Use of metadata. Metadata and annotation canfacilitate fast and accurate search and retrieval. This aspect may becomeeven more important as the semantic Web makes progress. Use of metadatais assessed by checking whether a randomly-chosen page is retrieved bythe search engine using the metadata encoded in the page itself, providedthat that page does use metatags.

3.3 Search interface
 
 

Table 3. Evaluation features for search interface
 

Name	EvaluationMethodology	EvaluationMeasure
Accessibilityfor disabled persons	Objective inspection	Yes/No
Search boxon home page	Objective inspection	Yes/No
Large searchbox	Objective inspection	Yes/No
Search button	Objective inspection	Yes/No
Search functionon every page	Objective estimateby page sample	Yes/No
Search boxon every page	Objective estimateby page sample	Yes/No
Scoped searchexplained	Subjectiveinspection	Yes/No/Doesn'tapply
Discreet AdvancedSearch	Subjectiveinspection	Yes/No/Doesn'tapply
Context ofsearch results	Objective inspection	Yes/No
Relevance scoreshidden	Objective inspection	Yes/No
Matching keywordshighlighted	Objective inspection	Yes/No
Visible "NoResults" message	Objective inspection	Yes/No
Search functionon a search results page	Objective inspection	Yes/No
Search boxon a search results page	Objective inspection	Yes/No
Constructiveadvice for "No results" searches	Subjectiveinspection	Yes/No

 

Accessibility for disabled persons. The searchengine should be made accessible to people with handicaps. Platform- andbrowser-independent accessibility for disabled persons requires compliancewith the W3C accessibility guidelines (http://www.w3.org/TR/2000/NOTE-WCAG10-HTML-TECHS-20000920/).This is ascertained by inspecting the source code of the Web page containingthe search engine.

Search box on the home page. The search functionshould be clearly visible when entering a Website. For many users, tabsand links to a separate search page just don't work. It is preferable touse a text box.

Large search box. The search box should beable to handle long queries. We set the lower bound of the maximum numberof characters accepted by a good search box to 30.

Search button. After entrying a query, usersshould start a search by clicking on a button. The button label itselfis less important.

Search function on every page. Easy accessto search should be garanteed from any page in the site, except for thecases when users would risk losing their work (e.g., form filling, checkoutprocess).

Search box on every page. Even on pages otherthan the home page, it is preferable to use a text box.

Scoped search explained. The scope of thesearch - when applicable - should be explicitly stated at the top of theresults page, and suggestions for limiting or enlarging the current scopeshould be provided.

Discreet link to Advanced Search - and back.Advanced Search - when applicabe - should be offered with caution, as itmay easily lead users into trouble. It is better not to make Advanced Searchaccessible from the home page and to make its link less prominent thanthe Search button. When the users get a No Results message in AdvancedSearch, a visible link back to simple Search should be provided.

Context of Search Results. After a user hasentered one or more search keywords and activated the search, the SearchResults page should always display the search criteria - possibly includingthe scope of search - and the total number of search results.

Relevance scores hidden.Relevance scores are just noise for most users, who aren't aware of orinterested in the factors that influence ranking. Scores should be usedonly for ranking the list from highest to lowest.

Matching keywords highlighted. Highlightingthe document keywords which matched the query keywords and showing thesentences surrounding the matching keywords is a simple and effective wayto show why results were included and avoid displaying seemingly irrelevanthits.

Visible "No Results" message. If the searchfails, one must make it sure that the user notices the "No Results" message,without mistaking the No Results page for a Search Results page.

Search function on a search results page.Users shoul not be forced to navigate backward from the Search Resultsor No Results page to try another query. Search functionality should beplaced directly in the content area of such pages.

Search box on a search results page. Searchbox is recommended also for Search Results or No Results pages.

Constructive advice for "No results" searches.After making clear why the search failed,the system should suggest possibleways to avoid the No Results search (e.g., checking the spelling, broadeningthe search) or alternative ways of locating information (e.g., navigation,table of contents, site map, site's FAQ).

4. A case study: the Italian PublicAdministration Websites

We applied the evaluation methodology described aboveto the Websites of the Italian Public Administration. In the next two sections,we describe the Website sample chosen for evaluation and the results ofour study.

4.1 Website sample

In the study, conducted in June 2001, the followingsites were considered.
 
 

- The Chief Towns of Italy. Of the 103 administrations,4 did not have a Website, 8 had an unavailable Website, 65 did not havea search engine or had a search engine which did not work, and only 26had a running system.

- The Regions of Italy. All 20 regions had a Website,12 with and 8 without an internal search engine.

- The Presidency of the Republic and the two branchesof Parliament. The Presidency of the Republic and the Chamber of Deputieshad a search engine, the Senate had a Website without search engine.

- The Presidency of the Council of Ministers andthe Ministries. The former and 12 ministries had a search engine, while7 ministries had a Website without search engine.

- Myscellaneous of main institutional bodies. Weselected 9 administrations (see Table 5 for a detailed list), all of whichhad a site's search engine.

In all, of the 155 administrations considered forevaluation, 4 did not have a Website, 8 had a Website which, however, didnot happen to work , 81 (more than half of the sample) had a Website butthey did not have a search engine or had a search engine which did notwork. Thus, we actually evaluated the search quality of the remaining 62sites.

The evaluation was performed using Internet Exploreron a Personal Computer with a fast connection to the Internet. In the nextsection we present the results.

4.2 Results
 
 

For the sake of illustration, the evaluation featuresare numbered. The numerical lables assigned to the features are shown inTable 4. In Table 5 we report, for each site (denoted by its URL withoutthe prefix "www."), the complete list of search quality features possessedby that site (i.e., "x" means that the evaluation measure for the correspondingsite-feature pair is "yes"). These raw results are the starting point ofour analysis.

Table 6 shows the number and percentage of siteswhich comply with each feature. The most unusual features (with less than25% of complying sites) are underscored, the most usual features (withmore than 75% of complying sites) are bold. The majority of the most unusualfeatures belongs to the search effectiveness dimension (i.e., 9, 13, 14,15, 16, 17) - in particular, to the set of features which should supportthe interaction with the user. One feature (i.e., stemming) refers to searcheffectiveness and four features (18, 24, 25, and 32) to search interface.By contrast, the most usual features are concentrated in the search efficiencygroup (i.e., speed and no broken links, with all search efficiency featureshaving more than half of complying sites) and in the search interface group(21, 29, 30), with only one feature (i.e., case insensitivity) being relatedto search effectiveness.

Table 4. Labeled search quality features
 

Highspeed	1
Highcoverage	2
Nobroken links	3
Freshness	4
Caseinsensitivity	5
Inexactmatching	6
Stemming	7
Supportfor scoped search	8
Supportfor advanced search	9
Supportfor phrase searching	10
Supportfor boolean search	11
Sortby relevance	12
Multilingualsupport	13
Selectablenumber of displayed results	14
Supportfor refinement of search results	15
Supportfor query reformulation	16
Useof metadata	17
Accessibilityfor disabled persons	18
Searchbox on home page	19
Largesearch box	20
Searchbutton	21
Searchfunction on every page	22
Searchbox on every page	23
Scopedsearch explained	24
DiscreetAdvanced Search	25
Contextof search results	26
Relevancescores hidden	27
Matchingkeywords highlighted	28
Visible"no results" message	29
Searchfunction on a search results page	30
Searchbox on a search results page	31
Costructiveadvice for "no results" searches	32

Table 5. Site-feature relation

Table 6. Number and percentage of sites per feature.
 

Features	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32
# Sites	48	33	54	33	58	28	3	18	12	17	33	33	1	7	3	11	9	0	40	31	58	40	20	9	12	47	34	19	56	57	40	7
% Sites	77	53	87	53	94	45	5	29	19	27	53	53	2	11	5	18	15	0	65	50	94	65	32	15	19	76	55	31	90	92	65	11

 

The results shown in Table 6 are useful to considerthe behavior of single features, but they do not show the distributionof features in the sites under examination.

Figure 1 depicts how the number of sites which possessat least a given number of search quality features decreases as the numberof features is increased. The figure clearly shows that the overall searchquality of the sites examined was modest. In particular, while almost allsites had at least 10 features, the number of complying sites decreasedsharply when passing from 11 to 20 features. For

17 features, which is just more than 50% of the featuresconsidered for evaluation, the number of sites reduced to 11 (16 % of thesites). Only two sites had twenty or more features and no site had at least25 features.

As we are interested in evaluating the behavior ofthe examined sites with respect to the main search quality dimensions,in the following we show the analogous of Figure 1 for each quality dimension.

The quality of search efficiency, as illustratedin Figure 2, is more favourable than the general picture. About 50 siteswere found to have at least two features and nearly 40 sites possessedthree of the four features considered for evaluation. Search speed, asalready pointed out, was pretty good while search maintenance was slightlyless satisfying, with both high coverage and freshness scoring about 50%of sites.

Figure 3 shows the search effectiveness quality results.The number of complying sites falls sharply after a few features, reducingto only 10% of sites for less than 50% of features. Clearly, the low qualityof search effectiveness was heavily influenced by the lack of featuressupporting the interaction with users. In fact, this is one of the mostvisible and consistent findings of our study.

Finally, in Figure 4 we report the search interfaceresults. The quality was better than search effectiveness but it was stillunsatisfying. Less than 50% of sites had more than 50% of features, andsome features, as shown in Table 6, were very poorly represented.
 
 

Figure 1. Number of complying sites versus numberof search quality features

Figure 2. Number of complying sites versus numberof search efficiency features.
 
 
 
 

Figure 3. Number of complying sites versus numberof search effectiveness features.
 
 

Figure 4. Number of complying sites versus numberof search interface features.
 
 
 
 

5. Conclusions

We presented a comprehensive methodology for evaluatingthe quality of a site's search and applied it to a sample of the ItalianPublic Administration Websites.

The main conclusions of our evaluation study arethefollowing.

- Half of the sites under consideration were notequipped with an internal search engine.

- The overall search quality of the sites with aninternal search engine was low.

- The quality of search efficiency was fairly acceptable,with good response times and slightly less satisfying search maintenance.

- The quality of search effectiveness and searchinterface was unsatisfying, with a serious shortage of features supportingthe interaction between users and system.

Finally, we would like to emphasize that due to thedifficulty of accessing the source code of the deployed systems, theirranking effectiveness for single-query searches was not taken into accountin our evaluation. This aspect complements the others very well and cannow be reliably assessed using electronically available test collections,provided that the search systems are made available for testing.

Acknowledgments

This research was done as a part of the project "InfoWeb",which aims at assessing and improving the quality of the Italian PublicAdministration Websites. Infoweb is conducted by Fondazione Ugo Bordoniand ISCTI.
 
 

References

Berenci, E., Carpineto, C., Giannini, V, and MizzaroS. (2000). Effectiveness of keyword-based display and selection of retrievalresults for interactive searches. International Journal on Digital Libraries,3, 3, pp. 249-260.

Carpineto, C., De Mori, R., Romano, G., Bigi, B (2001).An information theoretic approach to automatic query expansion, ACM Transactionson Information Systems, 19, 1, pp. 1-27.

Hauptmann, A., and Witbrock, M. (1997). Informedia:News-on-demand multimedia information acquisition and retrieval. In IntelligentMultimedia Information Retrieval, M. Maybury (Ed.), AAAI Press, Menlo Park,CA.

Hearst, M. (1999). User interfaces and visualization.In R. Baeza-Yates and B. Ribeiro-Neto (Eds.), Modern information retrieval,(pp. 257-322), ACM Press, New York.

Huberman, B., Pirolli, P., Pitkow, J., and Lukose,R. (1998). Strong regularities in world wide web surfing, Science, 280,pp. 95-97.

Kobayashi, M, and Takeda, K. (2000). Informationretrieval on the Web. ACM Computing Surveys, 32, 2 , pp. 144-173.

Lawrence, S. and Giles, C. (1998). Searching theworld wide web. Science, 280, pp. 98-100.

Luk, R., Chan, A., Dillon, T., and Leong, H. (2000).A survey of search engines for XML documents. In Proceedings of the ACMSIGIR Workshop on XML and Information Retrieval, Athens, Greece, July 2000.

Nielsen, J. (1997). Search and you may find.Jakob Nielsen's Alertbox, July 15, 1997, http://www.useit.com/alertbox/9707b.html.

Nielsen, J. (2001) Search: Visible and Simple. JakobNielsen's Alertbox, May 13, 2001, http://www.useit.com/alertbox/20010513.html.

Nielsen, J., Molich, R., Snyder, C., and Farrel,S. (2000). Search: 29 Design Guidelines for Usable Search, http://www.nngroup.com/reports/ecommerce/search.html

Oard, D., Peters, C., Ruiz, M., Frederking, R., Klavans,J., and Sheridan, P. (1999). Multilingual information discovery and access(MIDAS): A joint ACM DL '99 / ACM SIGIR '99 workshop. D-Lib Magazine, October1999, http://www.dlib.org

Voorhees, E., and Harman, D. 2000. Overview of theEighth Text Retrieval Conference (TRC-8). In Proceedings of the 8thText REtrieval Conference (TREC-8), Gaithersburg, MD, NIST SpecialPublication 500-246, pp. 1-24.