Tokenization can be applied to columns with more than 64K distinct values.
When you create a permanent table in a Sybase IQ database, IQ stores all column values in a default index. This default index, called an FP (fast projection) index, optimizes projections and enables certain kinds of search conditions to be evaluated.
Each column has one FP index, and each FP is an array of n fixed-length entries where n is the number of rows in the table. Each column value is stored sequentially in ascending RecordID order.
With a small number of distinct or unique values, such as a state, date, or month field, an optimized form of the FP can be created that will reduce the number of disk pages required, dramatically reducing both the storage required for a column and I/O costs for projection.
These optimized FP indexes have two pieces: 1. a lookup table where each distinct value in the column appears exactly once and 2. the logical array of values where each element of the logical array is a key pointing to the location where the cell value is stored in the lookup table.
The sp_iqindexmetadata stored procedure generates a report describing a specified index or indexes belonging to a specified owner or table. The output allows easy checking of whether a given index is a 1-byte, 2-byte, 3-byte, or flat style FP index. For details, see “sp_iqindexmetadata procedure” in Reference: Building Blocks, Tables, and Procedures.
