it.unipd.dei.ims.circo.lucene.demo
Class BasicAnalyzer

java.lang.Object
  org.apache.lucene.analysis.Analyzer
      it.unipd.dei.ims.circo.lucene.demo.BasicAnalyzer

public class BasicAnalyzer
extends org.apache.lucene.analysis.Analyzer

Provides a basic analyzer which tokenizes documents, removes stop words, and stems them. It is based on the code of StandardAnalyzer.

Since:

0.10

Version:

0.10

Author:

Nicola Ferro

Field Summary

static int	DEFAULT_MAX_TOKEN_LENGTH Default maximum allowed token length
static java.lang.String[]	STOP_WORDS An array containing some common English words that are usually not useful for searching.

Constructor Summary

BasicAnalyzer()
Builds an analyzer with the default stop words (STOP_WORDS).

BasicAnalyzer(java.io.File stopwords)
Builds an analyzer with the stop words from the given file.

BasicAnalyzer(java.io.Reader stopwords)
Builds an analyzer with the stop words from the given reader.

BasicAnalyzer(java.util.Set stopWords)
Builds an analyzer with the given stop words.

BasicAnalyzer(java.lang.String[] stopWords)
Builds an analyzer with the given stop words.

Method Summary

int	getMaxTokenLength()
org.apache.lucene.analysis.TokenStream	reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)
void	setMaxTokenLength(int length) Set maximum allowed token length.
org.apache.lucene.analysis.TokenStream	tokenStream(java.lang.String fieldName, java.io.Reader reader) Constructs a StandardTokenizer filtered by a a LowerCaseFilter, a StopFilter, and a PorterStemFilter

Methods inherited from class org.apache.lucene.analysis.Analyzer

close, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STOP_WORDS

public static final java.lang.String[] STOP_WORDS

An array containing some common English words that are usually not useful for searching.

DEFAULT_MAX_TOKEN_LENGTH

public static final int DEFAULT_MAX_TOKEN_LENGTH

Default maximum allowed token length

See Also:

Constant Field Values

Constructor Detail

BasicAnalyzer

public BasicAnalyzer()

Builds an analyzer with the default stop words (STOP_WORDS).

BasicAnalyzer

public BasicAnalyzer(java.util.Set stopWords)

Builds an analyzer with the given stop words.

BasicAnalyzer

public BasicAnalyzer(java.lang.String[] stopWords)

Builds an analyzer with the given stop words.

BasicAnalyzer

public BasicAnalyzer(java.io.File stopwords)
              throws java.io.IOException

Builds an analyzer with the stop words from the given file.

Throws:

java.io.IOException

See Also:

WordlistLoader.getWordSet(File)

BasicAnalyzer

public BasicAnalyzer(java.io.Reader stopwords)
              throws java.io.IOException

Builds an analyzer with the stop words from the given reader.

Throws:

java.io.IOException

See Also:

WordlistLoader.getWordSet(Reader)

Method Detail

tokenStream

public org.apache.lucene.analysis.TokenStream tokenStream(java.lang.String fieldName,
                                                          java.io.Reader reader)

Constructs a StandardTokenizer filtered by a a LowerCaseFilter, a StopFilter, and a PorterStemFilter

Specified by:

tokenStream in class org.apache.lucene.analysis.Analyzer

setMaxTokenLength

public void setMaxTokenLength(int length)

Set maximum allowed token length. If a token is seen that exceeds this length then it is discarded. This setting only takes effect the next time tokenStream or reusableTokenStream is called.

getMaxTokenLength

public int getMaxTokenLength()

See Also:

setMaxTokenLength(int)

reusableTokenStream

public org.apache.lucene.analysis.TokenStream reusableTokenStream(java.lang.String fieldName,
                                                                  java.io.Reader reader)
                                                           throws java.io.IOException

Overrides:

reusableTokenStream in class org.apache.lucene.analysis.Analyzer

Throws:

java.io.IOException