public final class Tokenizer extends Object
Tokenizer
class is used to split a given text source
into lexicographical tokens which are regognized by the parser.ParserImpl
Modifier and Type | Field and Description |
---|---|
static int |
TT_DOUBLE
Token type which indicates a numeric constant.
|
static int |
TT_EOS
Token type which indicates the end-of-stream.
|
static int |
TT_ESCAPED_NAME
Token type which indicates a name (e.g. variable or function identifier).
|
static int |
TT_HEX_INT
Token type which indicates an hexadecimal integer constant.
|
static int |
TT_INT
Token type which indicates a numeric constant.
|
static int |
TT_KEYWORD
Token type which indicates a name (e.g. variable or function identifier).
|
static int |
TT_NAME
Token type which indicates a name (e.g. variable or function identifier).
|
static int |
TT_OCT_INT
Token type which indicates an hexadecimal integer constant.
|
static int |
TT_SPECIAL
Token type which indicates a character (-sequence) with a special meaning.
|
static int |
TT_STRING
Token type which indicates a string constant.
|
static int |
TT_UNKNOWN
Token type which indicates an unknown token type.
|
Constructor and Description |
---|
Tokenizer(String source)
Constructs a new tokenizer for the given text source.
|
Tokenizer(String source,
boolean caseSensitive)
Constructs a new tokenizer for the given text source.
|
Modifier and Type | Method and Description |
---|---|
static String |
createExternalName(String name)
Create an external name from the given name.
|
int |
getColumn()
Gets the current source column number.
|
int |
getLine()
Gets the current source line number.
|
String |
getToken()
Gets the current token wich was read in due to the last
next() call. |
int |
getType()
Returns the current token type.
|
static boolean |
isExternalName(String name)
Tests whether or not the given name is a valid external name.
|
int |
next()
Reads the next token from the text source passed to the constructor of
this tokenizer.
|
void |
pushBack()
Pushes back the token read by the last
next() call, so
that it is returned again by a following next() call. |
public static final int TT_EOS
public static final int TT_UNKNOWN
public static final int TT_INT
public static final int TT_HEX_INT
public static final int TT_OCT_INT
public static final int TT_DOUBLE
public static final int TT_STRING
public static final int TT_NAME
public static final int TT_KEYWORD
public static final int TT_ESCAPED_NAME
public static final int TT_SPECIAL
public Tokenizer(String source, boolean caseSensitive)
source
- the text source to split into tokenscaseSensitive
- determines if tokenizing is performed case sensitive or notpublic Tokenizer(String source)
source
- the text source to split into tokenspublic int getType()
public int getLine()
public int getColumn()
public String getToken()
next()
call.public void pushBack()
next()
call, so
that it is returned again by a following next()
call.public int next()
public static boolean isExternalName(String name)
name
- the name to testtrue
if the name is a valid external name, false
otherwiseCopyright © 2002–2014 Brockmann Consult GmbH. All rights reserved.