Class LimitTokenCountAnalyzer

java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.AnalyzerWrapper
org.apache.lucene.analysis.miscellaneous.LimitTokenCountAnalyzer
All Implemented Interfaces:
Closeable, AutoCloseable

public final class LimitTokenCountAnalyzer extends AnalyzerWrapper
This Analyzer limits the number of tokens while indexing. It is a replacement for the maximum field length setting inside IndexWriter.
Since:
3.1
See Also:
  • Field Details

    • delegate

      private final Analyzer delegate
    • maxTokenCount

      private final int maxTokenCount
    • consumeAllTokens

      private final boolean consumeAllTokens
  • Constructor Details

    • LimitTokenCountAnalyzer

      public LimitTokenCountAnalyzer(Analyzer delegate, int maxTokenCount)
      Build an analyzer that limits the maximum number of tokens per field. This analyzer will not consume any tokens beyond the maxTokenCount limit
      See Also:
    • LimitTokenCountAnalyzer

      public LimitTokenCountAnalyzer(Analyzer delegate, int maxTokenCount, boolean consumeAllTokens)
      Build an analyzer that limits the maximum number of tokens per field.
      Parameters:
      delegate - the analyzer to wrap
      maxTokenCount - max number of tokens to produce
      consumeAllTokens - whether all tokens from the delegate should be consumed even if maxTokenCount is reached.
  • Method Details

    • getWrappedAnalyzer

      protected Analyzer getWrappedAnalyzer(String fieldName)
      Description copied from class: AnalyzerWrapper
      Retrieves the wrapped Analyzer appropriate for analyzing the field with the given name
      Specified by:
      getWrappedAnalyzer in class AnalyzerWrapper
      Parameters:
      fieldName - Name of the field which is to be analyzed
      Returns:
      Analyzer for the field with the given name. Assumed to be non-null
    • wrapComponents

      protected Analyzer.TokenStreamComponents wrapComponents(String fieldName, Analyzer.TokenStreamComponents components)
      Description copied from class: AnalyzerWrapper
      Wraps / alters the given TokenStreamComponents, taken from the wrapped Analyzer, to form new components. It is through this method that new TokenFilters can be added by AnalyzerWrappers. By default, the given components are returned.
      Overrides:
      wrapComponents in class AnalyzerWrapper
      Parameters:
      fieldName - Name of the field which is to be analyzed
      components - TokenStreamComponents taken from the wrapped Analyzer
      Returns:
      Wrapped / altered TokenStreamComponents.
    • toString

      public String toString()
      Overrides:
      toString in class Object