StandardTokenizerImpl
, StandardTokenizerImpl31
, UAX29URLEmailTokenizerImpl
, UAX29URLEmailTokenizerImpl31
, UAX29URLEmailTokenizerImpl34
public interface StandardTokenizerInterface
Modifier and Type | Field | Description |
---|---|---|
static int |
YYEOF |
This character denotes the end of file
|
Modifier and Type | Method | Description |
---|---|---|
int |
getNextToken() |
Resumes scanning until the next regular expression is matched,
the end of input is encountered or an I/O-Error occurs.
|
void |
getText(CharTermAttribute t) |
Copies the matched text into the CharTermAttribute
|
int |
yychar() |
Returns the current position.
|
int |
yylength() |
Returns the length of the matched text region.
|
void |
yyreset(Reader reader) |
Resets the scanner to read from a new input stream.
|
static final int YYEOF
void getText(CharTermAttribute t)
int yychar()
void yyreset(Reader reader)
reader
- the new input streamint yylength()
int getNextToken() throws IOException
YYEOF
on end of streamIOException
- if any I/O-Error occursCopyright © 2000-2018 Apache Software Foundation. All Rights Reserved.