SecondaryIndexManager (apache-cassandra API)
Date: 2018-02-14

org.apache.cassandra.index

## Class SecondaryIndexManager

- java.lang.Object
- - org.apache.cassandra.index.SecondaryIndexManager

- All Implemented Interfaces:
    IndexRegistry

      public class SecondaryIndexManager
      extends java.lang.Object
      implements IndexRegistry

  Handles the core maintenance functionality associated with indexes: adding/removing them to or from a table, (re)building during bootstrap or other streaming operations, flushing, reloading metadata and so on. The Index interface defines a number of methods which return `Callable<?>`. These are primarily the management tasks for an index implementation. Most of them are currently executed in a blocking fashion via submission to SIM's blockingExecutor. This provides the desired behaviour in pretty much all cases, as tasks like flushing an index needs to be executed synchronously to avoid potentially deadlocking on the FlushWriter or PostFlusher. Several of these `Callable<?>` returning methods on Index could then be defined with as void and called directly from SIM (rather than being run via the executor service). Separating the task defintion from execution gives us greater flexibility though, so that in future, for example, if the flush process allows it we leave open the possibility of executing more of these tasks asynchronously. The primary exception to the above is the Callable returned from Index#addIndexedColumn. This may involve a significant effort, building a new index over any existing data. We perform this task asynchronously; as it is called as part of a schema update, which we do not want to block for a long period. Building non-custom indexes is performed on the CompactionManager. This class also provides instances of processors which listen to updates to the base table and forward to registered Indexes the info required to keep those indexes up to date. There are two variants of these processors, each with a factory method provided by SIM: IndexTransaction: deals with updates generated on the regular write path. CleanupTransaction: used when partitions are modified during compaction or cleanup operations. Further details on their usage and lifecycles can be found in the interface definitions below. Finally, the bestIndexFor method is used at query time to identify the most selective index of those able to satisfy any search predicates defined by a ReadCommand's RowFilter. It returns a thin IndexAccessor object which enables the ReadCommand to access the appropriate functions of the Index at various stages in its lifecycle. e.g. the getEstimatedResultRows is required when StorageProxy calculates the initial concurrency factor for distributing requests to replicas, whereas a Searcher instance is needed when the ReadCommand is executed locally on a target replica.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `ColumnFamilyStore` | `baseCfs` The underlying column family containing the source data for these indexes
    `static int` | `DEFAULT_PAGE_SIZE`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `SecondaryIndexManager(ColumnFamilyStore baseCfs)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `java.util.concurrent.Future<?>` | `addIndex(IndexMetadata indexDef)` Adds and builds a index
    `void` | `buildAllIndexesBlocking(java.util.Collection<SSTableReader> sstables)`
    `void` | `buildIndexBlocking(Index index)`
    `int` | `calculateIndexingPageSize()` Return the page size used when indexing an entire partition
    `void` | `deletePartition(UnfilteredRowIterator partition, int nowInSec)` Delete all data from all indexes for this partition.
    `void` | `executePreJoinTasksBlocking(boolean hadBootstrap)` Performs a blocking execution of pre-join tasks of all indexes
    `void` | `flushAllIndexesBlocking()` Perform a blocking flush all indexes
    `void` | `flushAllNonCFSBackedIndexesBlocking()` Performs a blocking flush of all custom indexes
    `void` | `flushIndexesBlocking(java.util.Set<Index> indexes)` Perform a blocking flush of selected indexes
    `java.util.Set<ColumnFamilyStore>` | `getAllIndexColumnFamilyStores()`
    `java.util.Optional<Index>` | `getBestIndexFor(RowFilter.Expression expression)`
    `Index` | `getBestIndexFor(RowFilter rowFilter)` Called at query time to choose which (if any) of the registered index implementations to use for a given query.
    `java.util.List<java.lang.String>` | `getBuiltIndexNames()`
    `java.util.Set<IndexMetadata>` | `getDependentIndexes(ColumnDefinition column)`
    `Index` | `getIndex(IndexMetadata metadata)`
    `Index` | `getIndexByName(java.lang.String indexName)`
    `static java.lang.String` | `getIndexName(ColumnFamilyStore cfs)` Returns the index name
    `static java.lang.String` | `getIndexName(java.lang.String cfName)` Returns the index name
    `static ColumnFamilyStore` | `getParentCfs(ColumnFamilyStore cfs)` Returns the parent of the specified `ColumnFamilyStore`.
    `static java.lang.String` | `getParentCfsName(java.lang.String cfName)` Returns the parent name of the specified `ColumnFamilyStore`.
    `boolean` | `hasIndexes()`
    `void` | `indexPartition(DecoratedKey key, java.util.Set<Index> indexes, int pageSize)` When building an index against existing data in sstables, add the given partition to the index
    `void` | `invalidateAllIndexesBlocking()` Remove all indexes
    `static boolean` | `isIndexColumnFamily(java.lang.String cfName)` Checks if the specified `ColumnFamilyStore` is the one secondary index.
    `static boolean` | `isIndexColumnFamilyStore(ColumnFamilyStore cfs)` Checks if the specified `ColumnFamilyStore` is a secondary index.
    `boolean` | `isIndexQueryable(Index index)` Checks if the specified index is queryable.
    `java.util.Collection<Index>` | `listIndexes()`
    `void` | `markAllIndexesRemoved()` Called when dropping a Table
    `void` | `markIndexBuilt(java.lang.String indexName)` Marks the specified index as build.
    `void` | `markIndexRemoved(java.lang.String indexName)` Marks the specified index as removed.
    `CleanupTransaction` | `newCleanupTransaction(DecoratedKey key, PartitionColumns partitionColumns, int nowInSec)` Transaction for use when removing partitions during cleanup
    `CompactionTransaction` | `newCompactionTransaction(DecoratedKey key, PartitionColumns partitionColumns, int versions, int nowInSec)` Transaction for use when merging rows during compaction
    `UpdateTransaction` | `newUpdateTransaction(PartitionUpdate update, OpOrder.Group opGroup, int nowInSec)` Transaction for updates on the write path.
    `void` | `rebuildIndexesBlocking(java.util.Collection<SSTableReader> sstables, java.util.Set<java.lang.String> indexNames)` Does a full, blocking rebuild of the indexes specified by columns from the sstables.
    `void` | `registerIndex(Index index)` IndexRegistry methods
    `void` | `reload()` Drops and adds new indexes associated with the underlying CF
    `void` | `removeIndex(java.lang.String indexName)`
    `void` | `truncateAllIndexesBlocking(long truncatedAt)` Truncate all indexes
    `void` | `unregisterIndex(Index index)`
    `void` | `validate(PartitionUpdate update)` Called at write time to ensure that values present in the update are valid according to the rules of all registered indexes which will process it.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_PAGE_SIZE

          public static final int DEFAULT_PAGE_SIZE

      See Also:
        Constant Field Values

    - #### baseCfs

          public final ColumnFamilyStore baseCfs

      The underlying column family containing the source data for these indexes

  - ### Constructor Detail

    - #### SecondaryIndexManager

          public SecondaryIndexManager(ColumnFamilyStore baseCfs)

  - ### Method Detail

    - #### reload

          public void reload()

      Drops and adds new indexes associated with the underlying CF

    - #### addIndex

          public java.util.concurrent.Future<?> addIndex(IndexMetadata indexDef)

      Adds and builds a index
      Parameters:
        `indexDef` - the IndexMetadata describing the index

    - #### isIndexQueryable

          public boolean isIndexQueryable(Index index)

      Checks if the specified index is queryable.
      Parameters:
        `index` - the index
      Returns:
        `true` if the specified index is queryable, `false` otherwise

    - #### removeIndex

          public void removeIndex(java.lang.String indexName)

    - #### getDependentIndexes

          public java.util.Set<IndexMetadata> getDependentIndexes(ColumnDefinition column)

    - #### markAllIndexesRemoved

          public void markAllIndexesRemoved()

      Called when dropping a Table

    - #### rebuildIndexesBlocking

          public void rebuildIndexesBlocking(java.util.Collection<SSTableReader> sstables,
                                             java.util.Set<java.lang.String> indexNames)

      Does a full, blocking rebuild of the indexes specified by columns from the sstables. Caller must acquire and release references to the sstables used here. Note also that only this method of (re)building indexes: a) takes a set of index *names* rather than Indexers b) marks exsiting indexes removed prior to rebuilding
      Parameters:
        `sstables` - the data to build from
        `indexNames` - the list of indexes to be rebuilt

    - #### buildAllIndexesBlocking

          public void buildAllIndexesBlocking(java.util.Collection<SSTableReader> sstables)

    - #### buildIndexBlocking

          public void buildIndexBlocking(Index index)

    - #### isIndexColumnFamilyStore

          public static boolean isIndexColumnFamilyStore(ColumnFamilyStore cfs)

      Checks if the specified `ColumnFamilyStore` is a secondary index.
      Parameters:
        `cfs` - the `ColumnFamilyStore` to check.
      Returns:
        `true` if the specified `ColumnFamilyStore` is a secondary index, `false` otherwise.

    - #### isIndexColumnFamily

          public static boolean isIndexColumnFamily(java.lang.String cfName)

      Checks if the specified `ColumnFamilyStore` is the one secondary index.
      Parameters:
        `cfName` - the name of the `ColumnFamilyStore` to check.
      Returns:
        `true` if the specified `ColumnFamilyStore` is a secondary index, `false` otherwise.

    - #### getParentCfs

          public static ColumnFamilyStore getParentCfs(ColumnFamilyStore cfs)

      Returns the parent of the specified `ColumnFamilyStore`.
      Parameters:
        `cfs` - the `ColumnFamilyStore`
      Returns:
        the parent of the specified `ColumnFamilyStore`

    - #### getParentCfsName

          public static java.lang.String getParentCfsName(java.lang.String cfName)

      Returns the parent name of the specified `ColumnFamilyStore`.
      Parameters:
        `cfName` - the `ColumnFamilyStore` name
      Returns:
        the parent name of the specified `ColumnFamilyStore`

    - #### getIndexName

          public static java.lang.String getIndexName(ColumnFamilyStore cfs)

      Returns the index name
      Parameters:
        `cfs` - the `ColumnFamilyStore`
      Returns:
        the index name

    - #### getIndexName

          public static java.lang.String getIndexName(java.lang.String cfName)

      Returns the index name
      Parameters:
        `cfName` - the `ColumnFamilyStore` name
      Returns:
        the index name

    - #### markIndexBuilt

          public void markIndexBuilt(java.lang.String indexName)

      Marks the specified index as build.

      This method is public as it need to be accessible from the `Index` implementations

      Parameters:
        `indexName` - the index name

    - #### markIndexRemoved

          public void markIndexRemoved(java.lang.String indexName)

      Marks the specified index as removed.

      This method is public as it need to be accessible from the `Index` implementations

      Parameters:
        `indexName` - the index name

    - #### getIndexByName

          public Index getIndexByName(java.lang.String indexName)

    - #### truncateAllIndexesBlocking

          public void truncateAllIndexesBlocking(long truncatedAt)

      Truncate all indexes

    - #### invalidateAllIndexesBlocking

          public void invalidateAllIndexesBlocking()

      Remove all indexes

    - #### flushAllIndexesBlocking

          public void flushAllIndexesBlocking()

      Perform a blocking flush all indexes

    - #### flushIndexesBlocking

          public void flushIndexesBlocking(java.util.Set<Index> indexes)

      Perform a blocking flush of selected indexes

    - #### flushAllNonCFSBackedIndexesBlocking

          public void flushAllNonCFSBackedIndexesBlocking()

      Performs a blocking flush of all custom indexes

    - #### executePreJoinTasksBlocking

          public void executePreJoinTasksBlocking(boolean hadBootstrap)

      Performs a blocking execution of pre-join tasks of all indexes

    - #### getBuiltIndexNames

          public java.util.List<java.lang.String> getBuiltIndexNames()

      Returns:
        all indexes which are marked as built and ready to use

    - #### getAllIndexColumnFamilyStores

          public java.util.Set<ColumnFamilyStore> getAllIndexColumnFamilyStores()

      Returns:
        all backing Tables used by registered indexes

    - #### hasIndexes

          public boolean hasIndexes()

      Returns:
        if there are ANY indexes registered for this table

    - #### indexPartition

          public void indexPartition(DecoratedKey key,
                                     java.util.Set<Index> indexes,
                                     int pageSize)

      When building an index against existing data in sstables, add the given partition to the index

    - #### calculateIndexingPageSize

          public int calculateIndexingPageSize()

      Return the page size used when indexing an entire partition

    - #### deletePartition

          public void deletePartition(UnfilteredRowIterator partition,
                                      int nowInSec)

      Delete all data from all indexes for this partition. For when cleanup rips a partition out entirely. TODO : improve cleanup transaction to batch updates and perform them async

    - #### getBestIndexFor

          public Index getBestIndexFor(RowFilter rowFilter)

      Called at query time to choose which (if any) of the registered index implementations to use for a given query. This is a two step processes, firstly compiling the set of searchable indexes then choosing the one which reduces the search space the most. In the first phase, if the command's RowFilter contains any custom index expressions, the indexes that they specify are automatically included. Following that, the registered indexes are filtered to include only those which support the standard expressions in the RowFilter. The filtered set then sorted by selectivity, as reported by the Index implementations' getEstimatedResultRows method. Implementation specific validation of the target expression, either custom or standard, by the selected index should be performed in the searcherFor method to ensure that we pick the right index regardless of the validity of the expression. This method is only called once during the lifecycle of a ReadCommand and the result is cached for future use when obtaining a Searcher, getting the index's underlying CFS for ReadOrderGroup, or an estimate of the result size from an average index query.
      Parameters:
        `rowFilter` - RowFilter of the command to be executed
      Returns:
        an Index instance, ready to use during execution of the command, or null if none of the registered indexes can support the command.

    - #### getBestIndexFor

          public java.util.Optional<Index> getBestIndexFor(RowFilter.Expression expression)

    - #### validate

          public void validate(PartitionUpdate update)
                        throws InvalidRequestException

      Called at write time to ensure that values present in the update are valid according to the rules of all registered indexes which will process it. The partition key as well as the clustering and cell values for each row in the update may be checked by index implementations
      Parameters:
        `update` - PartitionUpdate containing the values to be validated by registered Index implementations
      Throws:
        `InvalidRequestException`

    - #### registerIndex

          public void registerIndex(Index index)

      IndexRegistry methods
      Specified by:
        `registerIndex` in interface `IndexRegistry`

    - #### unregisterIndex

          public void unregisterIndex(Index index)

      Specified by:
        `unregisterIndex` in interface `IndexRegistry`

    - #### getIndex

          public Index getIndex(IndexMetadata metadata)

      Specified by:
        `getIndex` in interface `IndexRegistry`

    - #### listIndexes

          public java.util.Collection<Index> listIndexes()

      Specified by:
        `listIndexes` in interface `IndexRegistry`

    - #### newUpdateTransaction

          public UpdateTransaction newUpdateTransaction(PartitionUpdate update,
                                                        OpOrder.Group opGroup,
                                                        int nowInSec)

      Transaction for updates on the write path.

    - #### newCompactionTransaction

          public CompactionTransaction newCompactionTransaction(DecoratedKey key,
                                                                PartitionColumns partitionColumns,
                                                                int versions,
                                                                int nowInSec)

      Transaction for use when merging rows during compaction

    - #### newCleanupTransaction

          public CleanupTransaction newCleanupTransaction(DecoratedKey key,
                                                          PartitionColumns partitionColumns,
                                                          int nowInSec)

      Transaction for use when removing partitions during cleanup

Copyright © 2018 The Apache Software Foundation