SEARCH SMARTS Search engines
are stupid. Fast, cheap, simple ... and stupid.
S E T H A. M A I S L I N Boston Chapter
a search engine is like asking someone on the telephone how to fix your
car. Itís possible, but you need to be prepared to do a lot more than
say "My car isnít working right."
basic search engines, the demands of searching are placed on users. They
have to know precisely what theyíre looking for, including spellings,
meanings, and languages. They need to understand higher-level search syntax,
if available. They have to know the scope of the search. Finally, they
have to be prepared to sort through innumerable results with no clear
guide to why certain results are valuable.
On the other
hand, programmers work furiously to overcome the failings of the simple
search to make search engines "smarter." The best algorithms
use multiple sources of additional input: context, dictionaries, thesauri,
grammar, search history, personal preferences, and third-party ratings.
Left alone, however, this pile of information is just that: a pile of
search engines must emulate human judgment.
"human-like" a search engine becomes, the more "intelligent"
it seems, and the better it performs. This article gives some basic approaches
toward improving your search engines. All of them involve the application
of human intelligence to the search process.
Understanding the search
engine is comprised of two parts: the algorithm and the data. The algorithm
evaluates the data by performing a series of valuation steps, from which
the overall rating is determined.
often (unfortunately) called an index, are generated by separating
the total documentation into smaller chunks, each with a label
(unique identifier). For example, the World Wide Web can be subdivided
into individual web pages, each defined by a specific URL; the elements
of a telephone directory are the unique name entries; a user manual is
divided into sections or paragraphs. The size of these chunks determines
the granularity of the search. These chunks are then stored as records
in a database, along with other useful data like date-time information.
algorithm scans the data, collecting the labels of any hits (search
matches). If the algorithm doesnít incorporate a rating system, the labels
remain in database order. Sometimes alphabetical order is treated as a
valuation algorithm, such that labels starting with "C" appear
before labels starting with "D." Finally, the collated labels
are displayed accessibly to the user.
two approaches for improving each of the search engineís two parts (algorithm
and data): making overall improvements and making case-by-case improvements.
The next several sections address these approaches individually.
Improving the data by
In everyday life,
human beings respond to new situations by making immediate categorical
assumptions -- "judging a book by its cover" -- before delving into
specifics. Although never true in all circumstances, these assumptions
provide an excellent framework for later judgments, such as the generalizations
below. Although there are clear exceptions, these assumptions are correct
in more than 50% of ordinary circumstances.
in titles, headings, and meta tags are important.
that appear frequently within a document are important.
words are important.
in bulleted lists are important.
words are important.
words are not particularly important.
prepositions, articles, and auxiliary verbs are particularly unimportant.
assigns auxiliary values to the records by preemptively applying assumptions
such as these. Then, when the search algorithm reads the database, these
values are used to help rate the hits. Consequently, a search of "books"
might determine that a bookstore web site is a better match than an online
no limits regarding the quantity or complexity of these assumptions:
information identifies newer or frequently updated documentation.
written in Spanish can be emphasized for Latin American audiences.
documents with images can be devalued to discourage long downloads.
Touching up the data:
and editors want to classify a database record as more relevant or irrelevant
than usual. Advertisers want their products to appear more relevant than
competitorsí, and authors might think explanations are more desirable
than definitions. These opinions usually reveal themselves only after
reviewing results to a few inadequate searches.
Writers can add editorial
values to the records for the algorithm to consider. For example, note
that text-based search engines cannot effectively rate graphics: important
diagrams and maps often are ignored inadvertently.
By individually "boosting" the ratings of graphics, authors
can guarantee graphics better placement in the results list.
This approach also
can be taken to an effective extreme, often called keywording or
(unfortunately again) indexing.
text, called keywords, is added into each database record. Then,
the search algorithm considers only these keywords, and not the documentation
itself. Further, the keywords can be collected into a standalone document
and scanned by users, as a replacement for searching. This is how indexes
are built for online help documentation. (When applied to entire web sites,
this is known as meta tagging.) With careful keyword selection,
a more accurate search is almost guaranteed. One interesting technique
is adding commonly misspelled words as keywords so that errors and ignorance
donít interfere with a successful search.
Any case-by-case editorial
interpretation of data is resource-intensive. In fact, the entire process
is much more like book indexing than search programming.
Improving the algorithm
overall: Query processing
Basic search algorithms
attempt to find simple equalities: When does the search term match a term
in a database record? Unfortunately, the human definition of "equality"
is not limited to identical words. Users might be interested in any different
word that has the same meaning, has a similar spelling, or shares the
same linguistic root. Several algorithms can use rules of language and
grammar to serve these needs.
For example, a user
searching for "computer" might also want records that contain
the word "computers." This can accomplished using a simple algorithm
that looks for singulars as well as plurals. A more complicated grammatical
algorithm known as stemming compares only linguistic roots: "Computer"
is equivalent to "compute," "computation," and "computational."
A thesaurus identifies
words that share meanings. For example, a search for "computer"
finds examples of "desktops" and "laptops." Thesauri
have to be built by human beings, and some can be purchased. Understanding
the context and scope of the documentation is important. To many applications,
equating "acetaminophen" with its many brand names is not relevant.
The algorithm can
also interpret the userís query before searching the database, such as
testing for common misspellings. (This is the opposite approach to using
misspellings as keywords, mentioned above.) The algorithm can ignore particularly
unimportant words, called stop words (usually prepositions, conjunctions,
and articles) that are too common to be limiting. If the user is allowed
to use search-related syntax such as wildcards, quotation marks, and Boolean
expressions, the algorithm must further interpret the query.
Search engines that
accept natural language queries are an extreme case. These engines,
which attempt to accept queries like "How can I print graphics upside-down?"
and "How far is the Boston Aquarium from Faneuil Hall?" utilize
years of grammatical and syntactical research and continue to improve.
(In contrast, most humans perform this kind of full-sentence interpretation
easily, automatically, and unconsciously!) To use natural language algorithms
in documentation or on web sites, the only sensible option these days
is to license the technology.
Algorithms also can
process hits before displaying them to the user. Parental controls use
this technique to repress content considered vulgar or otherwise inappropriate.
Query processing of
any kind relieves the user from the burden of knowing how to search and
what to search for.
Tweaking the algorithm:
Handling specific searches
Users are often interested
in a particular word only when defined in a specific way or used in a
specific context. For example, there are two meanings for the words "glasses"
and "speakers." Because of this language-based subjectivity
and ambiguity, computers are unable to guess which interpretation the
Instead of ignoring
the ambiguity, the search engine can respond by asking for more information.
For example, if a user searches for "glasses," the search algorithm
gives the user a choice between "drinking glasses" and "eyeglasses."
The response to this choice influences the final database search.
is a natural human process:
a new query based on the results.
An interruptive model
of searching accommodates this process. Unfortunately, most users are
not interested in secondary choices and might think the engine is "showing
off." Thus, itís important to provide results in addition to providing
these options. For example, users searching for "glasses" should
be presented with results for this ambiguous search and the option
to refine the search using either "drinking glasses" or "eyeglasses."
(Also remember that there might be additional interpretations of the query
words that cannot be predicted.)
There is also the
opportunity to override the natural behavior of a search engine for specific
queries. Sometimes there is an advantage to not performing the
search, particularly when the user considers the search input box as the
only viable input opportunity. If a user searches for the word "home,"
it might make sense to send the user to the home page of the site instead
of displaying search results. Likewise, a user who types "lycos.com"
in the search box is probably interested in navigating to the Lycos site.
Clarifying the search
After applying all
this "intelligence" to your search engine, there is still one
major element missing: context. Users often believe they are searching
something different than what is really being searched:
After one search,
users might think they can search the search results.
into a section of documentation, they might expect the search to be
limited to that section.
Users might suspect
that a search engine on a web site actually searches the entire Web.
It is important to
communicate the scope of the search. Instead of labeling the engine with
only the word "Search," be more explicit: "Search our site"
or "Search the Web." Consider allowing users to choose the scope
with radio buttons or drop-down menus: "Search: This Site (or) This
Section (or) This Page." In addition, if there are sections of a
site that are password-protected or otherwise restricted, identify whether
or not users can get search results for those sections without a password.
The more flexibility you allow with search scope, the more search databases
engines cannot handle language without human help. They perform no interpretation
without editorial assistance. Remember that the burden of effective searching
is often on the user, and that the user is rarely as familiar with the site
structure as the writers, editors, and programmers. Use your human language
skills to interpret the data in advance and to build a search algorithm
that incorporates language rules and considers exceptions.
Volume 3, # 1
Maislin is an indexer, information architect, and consultant. Formerly
the senior indexer at O'Reilly & Associates, Seth is a regular conference
presenter, a council member for the Boston STC Chapter, webmaster for
the Indexing SIG, and a member of the American Society of Indexers national