Full text pipeline workflow

The following diagram shows how data is converted from a document to a stream of terms to index within SQL Anywhere. The workflow for creating a text index, updating it, and querying it, is referred to as the pipeline. The mandatory parts of the pipeline are depicted in light gray. Arrows show the flow of data through the pipeline. Function calls are propagated in the opposite direction.

Graphic showing pipeline for full text indexing and querying. Pipeline for indexing goes from document, through prefilter library, through term breaker library, to database server, with possible character set conversion between each step. Pipeline for querying shows the query going through possible character set conversion to the term breaker library, another possible character set conversion, and to the database server for query processing.

High level view of how the pipeline works

The processing of each document is initiated by the database server calling the begin_document method on the end of the pipeline, which is either the term breaker or the character set converter. Each component in the pipeline calls begin_document on its own producer before returning from its begin_document method invocation.
The database server calls get_words on the end of the pipeline after the begin_document completes successfully.
- While executing get_words, the term breaker calls get_next_piece on its producer to get data to process. If a prefilter exists in the pipeline, the data is filtered by it during the get_next_piece call.
- The term breaker breaks the data it receives from its producer into terms according to its term breaking rules.
The database server applies the minimum and maximum term length settings, as well as the stoplist restrictions to the terms returned from get_words call.
The database server continues to call get_words until no more terms are returned. At that point, the database server calls end_document. This call is propagated through the pipeline in the same manner as the begin_document call.

Note

Character set converters are transparently added to the pipeline by the database server where necessary.

Full text pipeline workflow

High level view of how the pipeline works

Note

Prefilter and term breaker code samples

See also