Closeable
, AutoCloseable
ArabicLetterTokenizer
, LowerCaseTokenizer
public class LetterTokenizer extends CharTokenizer
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required Version
compatibility when creating
LetterTokenizer
:
CharTokenizer
uses an int based API to normalize and
detect token characters. See CharTokenizer.isTokenChar(int)
and
CharTokenizer.normalize(int)
for details.AttributeSource.AttributeFactory, AttributeSource.State
Constructor | Description |
---|---|
LetterTokenizer(Reader in) |
Deprecated.
use
LetterTokenizer(Version, Reader) instead. |
LetterTokenizer(AttributeSource.AttributeFactory factory,
Reader in) |
Deprecated.
|
LetterTokenizer(AttributeSource source,
Reader in) |
Deprecated.
use
LetterTokenizer(Version, AttributeSource, Reader) instead. |
LetterTokenizer(Version matchVersion,
Reader in) |
Construct a new LetterTokenizer.
|
LetterTokenizer(Version matchVersion,
AttributeSource.AttributeFactory factory,
Reader in) |
Construct a new LetterTokenizer using a given
AttributeSource.AttributeFactory . |
LetterTokenizer(Version matchVersion,
AttributeSource source,
Reader in) |
Construct a new LetterTokenizer using a given
AttributeSource . |
Modifier and Type | Method | Description |
---|---|---|
protected boolean |
isTokenChar(int c) |
Collects only characters which satisfy
Character.isLetter(int) . |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
end, incrementToken, isTokenChar, normalize, normalize, reset
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
close, correctOffset
reset
public LetterTokenizer(Version matchVersion, Reader in)
matchVersion
- Lucene version to match See {@link above}in
- the input to split up into tokenspublic LetterTokenizer(Version matchVersion, AttributeSource source, Reader in)
AttributeSource
.public LetterTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
AttributeSource.AttributeFactory
.@Deprecated public LetterTokenizer(Reader in)
LetterTokenizer(Version, Reader)
instead. This
will be removed in Lucene 4.0.@Deprecated public LetterTokenizer(AttributeSource source, Reader in)
LetterTokenizer(Version, AttributeSource, Reader)
instead.
This will be removed in Lucene 4.0.AttributeSource
.@Deprecated public LetterTokenizer(AttributeSource.AttributeFactory factory, Reader in)
LetterTokenizer(Version, AttributeSource.AttributeFactory, Reader)
instead. This will be removed in Lucene 4.0.AttributeSource.AttributeFactory
.protected boolean isTokenChar(int c)
Character.isLetter(int)
.isTokenChar
in class CharTokenizer
Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.