com.bc.jexp.impl
Class Tokenizer

java.lang.Object
  com.bc.jexp.impl.Tokenizer

public final class Tokenizer
extends Object

The Tokenizer class is used to split a given text source into lexicographical tokens which are regognized by the parser.

Version:

$Revision$ $Date$

Author:

Norman Fomferra (norman.fomferra@brockmann-consult.de)

See Also:

ParserImpl

Field Summary

static int	TT_DOUBLE Token type which indicates a numeric constant.
static int	TT_EOS Token type which indicates the end-of-stream.
static int	TT_ESCAPED_NAME Token type which indicates a name (e.g.
static int	TT_HEX_INT Token type which indicates an hexadecimal integer constant.
static int	TT_INT Token type which indicates a numeric constant.
static int	TT_KEYWORD Token type which indicates a name (e.g.
static int	TT_NAME Token type which indicates a name (e.g.
static int	TT_OCT_INT Token type which indicates an hexadecimal integer constant.
static int	TT_SPECIAL Token type which indicates a character (-sequence) with a special meaning.
static int	TT_STRING Token type which indicates a string constant.
static int	TT_UNKNOWN Token type which indicates an unknown token type.

Constructor Summary

Tokenizer(String source)
Constructs a new tokenizer for the given text source.

Tokenizer(String source, boolean caseSensitive)
Constructs a new tokenizer for the given text source.

Method Summary

static String	createExternalName(String name) Create an external name from the given name.
int	getColumn() Gets the current source column number.
int	getLine() Gets the current source line number.
String	getToken() Gets the current token wich was read in due to the last next() call.
int	getType() Returns the current token type.
static boolean	isExternalName(String name) Tests whether or not the given name is a valid external name.
int	next() Reads the next token from the text source passed to the constructor of this tokenizer.
void	pushBack() Pushes back the token read by the last next() call, so that it is returned again by a following next() call.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TT_EOS

public static final int TT_EOS

Token type which indicates the end-of-stream.

See Also:

Constant Field Values

TT_UNKNOWN

public static final int TT_UNKNOWN

Token type which indicates an unknown token type.

See Also:

Constant Field Values

TT_INT

public static final int TT_INT

Token type which indicates a numeric constant.

See Also:

Constant Field Values

TT_HEX_INT

public static final int TT_HEX_INT

Token type which indicates an hexadecimal integer constant.

See Also:

Constant Field Values

TT_OCT_INT

public static final int TT_OCT_INT

Token type which indicates an hexadecimal integer constant.

See Also:

Constant Field Values

TT_DOUBLE

public static final int TT_DOUBLE

Token type which indicates a numeric constant.

See Also:

Constant Field Values

TT_STRING

public static final int TT_STRING

Token type which indicates a string constant.

See Also:

Constant Field Values

TT_NAME

public static final int TT_NAME

Token type which indicates a name (e.g. variable or function identifier).

See Also:

Constant Field Values

TT_KEYWORD

public static final int TT_KEYWORD

Token type which indicates a name (e.g. variable or function identifier).

See Also:

Constant Field Values

TT_ESCAPED_NAME

public static final int TT_ESCAPED_NAME

Token type which indicates a name (e.g. variable or function identifier).

See Also:

Constant Field Values

TT_SPECIAL

public static final int TT_SPECIAL

Token type which indicates a character (-sequence) with a special meaning.

See Also:

Constant Field Values

Constructor Detail

Tokenizer

public Tokenizer(String source,
                 boolean caseSensitive)

Constructs a new tokenizer for the given text source.

Parameters:

source - the text source to split into tokens

caseSensitive - determines if tokenizing is performed case sensitive or not

Tokenizer

public Tokenizer(String source)

Constructs a new tokenizer for the given text source. Tokenizing is performed case insensitive.

Parameters:

source - the text source to split into tokens

Method Detail

getType

public int getType()

Returns the current token type.

getLine

public int getLine()

Gets the current source line number.

getColumn

public int getColumn()

Gets the current source column number.

pushBack

public void pushBack()

Pushes back the token read by the last next() call, so that it is returned again by a following next() call.

getToken

public String getToken()

Gets the current token wich was read in due to the last next() call.

Returns:

the current token string

next

public int next()

Reads the next token from the text source passed to the constructor of this tokenizer. The value returned is one of the regognized token types or for ordinary characters, the character code itself.

Returns:

the type of the current token or for ordinary characters, the character code itself

isExternalName

public static boolean isExternalName(String name)

Tests whether or not the given name is a valid external name. Valid names have a length greater than zero, start with a letter or underscore followed by letters, digits or underscores. The keywords are not allowed as external names.

Parameters:

name - the name to test

Returns:

true if the name is a valid external name, false otherwise

createExternalName

public static String createExternalName(String name)

Create an external name from the given name. If the given name contains character which are not valid in an external name the name is escaped with single quotes.

Parameters:

name - the name

Returns:

a valid external name