5 1980 – 1989 Rapid evolution

In the early 1980s there was a great deal of interest in the UK around the use of text retrieval software running on mini-computers. The STATUS User Group in particular was very active. These vendors were not especially interested in the commercial success of their products as the development had been justified on the need to meet internal information searching requirements within the organisation.

In the UK the Institute of Information Scientists played a very important role in stimulating interest in the capabilities of these applications through a series of Text Retrieval conferences between 1980 and 1990. The proceedings of these conferences make fascinating reading though sadly none are available in a digital format and only the 1998 and 1999 conference proceedings were published. However, most of the conference documents are held by the British Library.

As far as the technical development of enterprise search was concerned probably the most important advance was the release of the Snowball English language stemmer developed by Dr. Martin Porter. To be pedantic it was first released in 1979 but was not widely promoted until 1980. Martin Porter tells the story from a 2001 perspective on his website where his original stemming code and many more algorithms for various languages are available as open source. According to the Wikipedia entry the name Snowball was chosen as a tribute to the SNOBOL programming language, with which it shares the concept of string patterns delivering signals that are used to control the flow of the program.

Martin Porter, together with John Snyder, also developed the Muscat (MUSeum CATalogue) search application while at Cambridge University. Released in 1984 it sought to take advantage of the work of Stephen Robertson and others on a probabilistic approach to information retrieval. Muscat Ltd. became a successful company with clients that included Fujitsu, the Japanese IT company.  Muscat was eventually rewritten and released as the open source Xapian library which survived the eventual acquisition of Muscat Ltd. by a short-lived dotcom era company and is still available. There is a good summary of Muscat on the Flax website.

By the mid-1980s the IBM STAIRS full-text search application was setting the standard for enterprise search. In 1985 a wide-ranging research study was carried out by Blair and Maron of the retrieval performance of STAIRS, which at that time was being promoted as a litigation support tool. The results were far from impressive (Blair and Maron 1985). This study remains the most comprehensive of its type, with nothing approaching it having been published in the last thirty years. It had commercial implications for the legal sector as this was the time when there started to be a number of major anti-trust cases brought by the US Department of Justice where reliable access to millions of corporate documents was of great importance. It should also be borne in mind that the IBM PC had been launched in 1981 and it was during the 1980s that documents started to be created on personal computers rather than being transcribed onto word processors.

I would suggest that the first commercial enterprise search application other than STAIRS was developed by Fulcrum Technologies, established in Ottawa in 1983. This was a client-server application, rather than mainframe and offered the first API for writing information retrieval applications. It was most visible for the rest of the decade as a provider of search software for CD-ROM applications. From 1983 to 1988 Fulcrum pretty much had the search market to itself but failed to make much headway. The arrival of Verity (see below) born in the entrepreneurial climate of California, marked a gradual decline of Fulcrum as a business. A succession of owners over the 1990s led eventually to Fulcrum being purchased by Hummingbird in 1997, which itself was then acquired by OpenText in 2006.

In 1985 Advanced Decision Systems was set up in San Jose, California with the objective of developing expert system and artificial intelligence applications. In 1986 David Glazer and Philip Nelson developed an innovative search application called Topic which was beta tested with success by the US Strategic Air Command. Topic made use of a probabilistic search ranking engine which offered significantly better management of ranking than the Boolean operators that had been used prior to the release of Topic, though STAIRS also used this model. This early success led to the spin-out of what was to become Verity from ADS, led by Michael Pliner with a technical team led by David Glazer and Philip Nelson. There can be no doubt that Verity was the proto-typical enterprise search application as unlike IBM STAIRS it was platform agnostic. At launch a multi-user licence cost $39,500, quite a substantial licence fee in the late 1990s.

Two other search software companies started out towards the end of the 1980s. David Thede set up dtSearch in 1988, initially offering a desktop search application. dtSearch remains one of the very few search software vendors to have been in the same ownership from start-up to the present day. Also in 1988 but across the other side of the world in Australia Ian Davies was developing the Isys software suite.  This ended up being acquired by Lexmark in 2012. Several others were on the drawing board but did not emerge until the early 1990s.

The decade also marked the birth of a project at CERN in Switzerland to create what would become the World Wide Web. Tim Berners-Lee submitted his report Information Management – A Proposal in March 1989. It is important to appreciate that the initial purpose of the project was to be able to search through CERN documentation and thereby an enterprise search project was the start of the global web search business. W3C has compiled a very useful chronology of the subsequent development of the World Wide Web over the period from 1989 to 1995.

Reference