How to design an external term breaker library

The external term breaker library must be implemented in C/C++, and must:

include of the term breaker interface definition header file, exttbapiv1.h.
implement the a_word_source interface.
provide an entry point function that initializes and returns an instance of a_word_source (term breaker) and the label of the character set supported by the term breaker.

Calling sequence for the term breaker

The following calling sequence is executed by the consumer of the term breaker for each document being processed:

begin_document(a_word_source*, asql_uint32);
get_words(a_word_source*, a_term**, uint32 *num_words)
get_words(a_word_source*, a_term**, uint32 *num_words)
...
end_document(a_word_source*)

The get_words function must call get_next_piece on its producer to get data to break into terms until the array of a_term structures is filled, or there is no more data to process.

Note

end_document can be called multiple times without an intervening begin_document call. For example, if one of the documents to be indexed is empty, the database server may call end_document for that document without calling begin_document.

How to design an external term breaker library

Calling sequence for the term breaker

Note

External term breaker library workflow

See also