StandardTokenizerInterface (apache-cassandra API)
Date: 2018-02-14

org.apache.cassandra.index.sasi.analyzer

## Interface StandardTokenizerInterface

-     public interface StandardTokenizerInterface

  Internal interface for supporting versioned grammars.

- - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method and Description
    --- | ---
    `char[]` | `getArray()`
    `byte[]` | `getBytes()`
    `int` | `getNextToken()` Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
    `java.lang.String` | `getText()`
    `int` | `yychar()` Returns the current position.
    `int` | `yylength()` Returns the length of the matched text region.
    `void` | `yyreset(java.io.Reader reader)` Resets the scanner to read from a new input stream.

- - ### Method Detail

    - #### getText

          java.lang.String getText()

    - #### getArray

          char[] getArray()

    - #### getBytes

          byte[] getBytes()

    - #### yychar

          int yychar()

      Returns the current position.

    - #### yylength

          int yylength()

      Returns the length of the matched text region.

    - #### getNextToken

          int getNextToken()
                    throws java.io.IOException

      Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
      Returns:
        the next token, `#YYEOF` on end of stream
      Throws:
        `java.io.IOException` - if any I/O-Error occurs

    - #### yyreset

          void yyreset(java.io.Reader reader)

      Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to `ZZ_INITIAL`.
      Parameters:
        `reader` - the new input stream

Copyright © 2018 The Apache Software Foundation