Document Store Index

Indexing is collecting data about the documents in a document store. Generically, the proprietary data structures of the document store are called indexes.

Only indexed documents in a document store can be searched. You can immediately index a document store when you create it, or, you can perform an incremental or partial index.

Use an incremental index to index all new and updated documents, and to remove index entries of deleted documents. Use a partial index to identify specific documents from a document store for which you want to create index entries.

The file system partial index process does not check the directory trees for new, modified, or deleted documents, which can save a significant amount of time for large document stores.

All data collected during an indexing session is stored in a data buffer. The data buffer is a RAM-oriented data structure, where data is aggregated, ready to be written to an index stripe. This buffer is flushed when the maximum memory threshold (specified in the system property omniq.index.buffer.maxMemory), is exceeded. The buffer shares this memory allocation with the document store’s active index stripe.

Index data is transferred from the data buffer and written to active or static stripes. The current active stripe stores all the data collected during the indexing session if it can accommodate it; otherwise, the active index stripe is emptied into a new static stripe, and all data collected during the indexing session is stored in the new static index stripe.

Active index stripes - each document store’s collection of index stripes contain exactly zero or one active index stripe. An active index stripe is a collection of RAM-oriented data structures—all of its data is stored in RAM while it keeps a copy on disk for persistence. An active index stripe can always be written to, and thus may contain data collected over numerous indexing sessions.

Static index stripes - each document store’s collection of index stripes contain zero or more static index stripes. A static index stripe is a collection of read-only, disk-oriented data structures.