public class DefaultICUTokenizerConfig extends ICUTokenizerConfig
ICUTokenizerConfig
that is generally applicable
to many languages.
Generally tokenizes Unicode text according to UAX#29
(BreakIterator.getWordInstance(ULocale.ROOT)
),
but with the following tailorings:
DictionaryBasedBreakIterator
Modifier and Type | Field | Description |
---|---|---|
static String |
WORD_HANGUL |
Token type for words containing Korean hangul
|
static String |
WORD_HIRAGANA |
Token type for words containing Japanese hiragana
|
static String |
WORD_IDEO |
Token type for words containing ideographic characters
|
static String |
WORD_KATAKANA |
Token type for words containing Japanese katakana
|
static String |
WORD_LETTER |
Token type for words that contain letters
|
static String |
WORD_NUMBER |
Token type for words that appear to be numbers
|
Constructor | Description |
---|---|
DefaultICUTokenizerConfig() |
Modifier and Type | Method | Description |
---|---|---|
com.ibm.icu.text.BreakIterator |
getBreakIterator(int script) |
Return a breakiterator capable of processing a given script.
|
String |
getType(int script,
int ruleStatus) |
Return a token type value for a given script and BreakIterator
rule status.
|
public static final String WORD_IDEO
public static final String WORD_HIRAGANA
public static final String WORD_KATAKANA
public static final String WORD_HANGUL
public static final String WORD_LETTER
public static final String WORD_NUMBER
public com.ibm.icu.text.BreakIterator getBreakIterator(int script)
ICUTokenizerConfig
getBreakIterator
in class ICUTokenizerConfig
public String getType(int script, int ruleStatus)
ICUTokenizerConfig
getType
in class ICUTokenizerConfig
Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.