Abstraction of a language token. More...

#include <Puma/Token.h>

Inheritance diagram for Puma::Token:

Public Types
enum	{ ID_END_OF_FILE = -1, ID_UNKNOWN = -2, ID_ERROR = -3, ID_WARNING = -4 }
	Special token types. More...

Public Member Functions
	Token (int type=0, LanguageID lang=LanguageID(0), const char *text="")
	Constructor. More...

	Token (const Token &copy)
	Copy-constructor. More...

virtual	~Token ()
	Destructor. More...

void	print (ostream &os) const
	Print the token text on the given stream. More...

virtual ListElement *	duplicate ()
	Duplicate this token. More...

Unit *	unit () const
	Get the unit this token belongs to. More...

void	reset (int type=0, const char *text=0, LanguageID lang=LanguageID(0))
	Reset the token object. More...

void	location (const Location &loc)
	Set the location of the token (usually the line and column in a file). More...

void	cont_lines (Array< int > *cl)
	Set the continues lines appeared in this token. More...

int	line_breaks () const
	Count the number of line breaks in the text of this token. More...

void	macro_call (bool is_call=true)
	Set the token as being a macro call. More...

const char *	text () const
	Get the token text. More...

DString &	dtext () const
	Get the token text. More...

char *	get_static_text () const
	Get the static token text. More...

int	type () const
	Get the type of the token (see Puma::CTokens). More...

const Location &	location () const
	Get the location of the token (usually the line and column in a file). More...

bool	is_macro_generated () const
	Check if the token is macro generated. More...

bool	is_macro_call () const
	Check if the token is a macro call. More...

bool	is_core () const
	Check if this is a core language token. More...

bool	is_identifier () const
	Check if the token is an identifier. More...

bool	is_keyword () const
	Check if the token is a keyword. More...

bool	is_wildcard () const
	Check if this is a wildcard token. More...

bool	is_preprocessor () const
	Check if this is a preprocessor token. More...

bool	is_directive () const
	Check if this is a compiler directive. More...

bool	is_whitespace () const
	Check if this is a white-space token. More...

bool	is_comment () const
	Check if this is a comment. More...

bool	is_macro_op () const
	Check if this is function-like macro operand. More...

bool	is_open () const
	Check if this is a left parenthesis. More...

bool	is_comma () const
	Check if this is a comma. More...

bool	is_close () const
	Check if this is a right parenthesis. More...

void *	operator new (size_t)
	Own operator new reusing memory. More...

void	operator delete (void *)
	Own delete operator. More...

Public Member Functions inherited from Puma::ListElement
	ListElement ()
	Constructor. More...

virtual	~ListElement ()
	Destructor. More...

List *	belonging_to () const
	Get the list this element belongs to. More...

Static Public Member Functions
static char *	get_static_text (int token_type)
	Get the static token text for the given token type. More...

Static Public Attributes
static LanguageID	comment_id
	Comment tokens. More...

static LanguageID	pre_id
	Preprocessor tokens. More...

static LanguageID	dir_id
	Compiler directives. More...

static LanguageID	cpp_id
	C/C++ core language tokens. More...

static LanguageID	white_id
	White-spaces. More...

static LanguageID	wildcard_id
	Wildcards. More...

static LanguageID	keyword_id
	Keywords. More...

static LanguageID	open_id
	Left parenthesis. More...

static LanguageID	macro_op_id
	Function-like macro operands. More...

static LanguageID	identifier_id
	Identifiers. More...

static LanguageID	comma_id
	Comma. More...

static LanguageID	close_id
	Right parenthesis. More...

Detailed Description

Abstraction of a language token.

A token is created by a scanner (see Puma::CScanner) as part a token chain (Puma::Unit). It encapsulates a lexical unit of the scanned text.

The information provided about a lexical unit is

the token type
the location in the scanned text
the portion of the scanned text represented by this token
the language of the token

Member Enumeration Documentation

anonymous enum

Special token types.

Enumerator
ID_END_OF_FILE	End-of-file token.
ID_UNKNOWN	Unknown token type.
ID_ERROR	Error token type.
ID_WARNING	Warning token type.

Constructor & Destructor Documentation

Puma::Token::Token	(	int	type = `0`,
		LanguageID	lang = `LanguageID(0)`,
		const char *	text = `""`
	)

Constructor.

Parameters

type	The token type.
lang	The language of the token.
text	The optional token text (defaults to the empty string).

Puma::Token::Token ( const Token & copy )

Copy-constructor.

Parameters

copy	The token to copy.

virtual Puma::Token::~Token ( )

virtual

Destructor.

Destroys the token text.

Member Function Documentation

void Puma::Token::cont_lines ( Array< int > * cl )

inline

Set the continues lines appeared in this token.

Parameters

cl	The continues line array.

DString& Puma::Token::dtext ( ) const

Get the token text.

virtual ListElement* Puma::Token::duplicate ( )

inlinevirtual

Duplicate this token.

Returns: The copy of this token. Has to be destroyed by the caller.

Implements Puma::ListElement.

char* Puma::Token::get_static_text ( ) const

Get the static token text.

Only for keywords, operators, and so on with a constant token text.

Returns: The static text or NULL for tokens with dynamic text (like identifiers).

static char* Puma::Token::get_static_text ( int token_type )

static

Get the static token text for the given token type.

Only for keywords, operators, and so on with a constant token text.

Returns: The static text or NULL for tokens with dynamic text (like identifiers).

bool Puma::Token::is_close ( ) const

inline

Check if this is a right parenthesis.

bool Puma::Token::is_comma ( ) const

inline

Check if this is a comma.

bool Puma::Token::is_comment ( ) const

inline

Check if this is a comment.

bool Puma::Token::is_core ( ) const

inline

Check if this is a core language token.

bool Puma::Token::is_directive ( ) const

inline

Check if this is a compiler directive.

bool Puma::Token::is_identifier ( ) const

inline

Check if the token is an identifier.

bool Puma::Token::is_keyword ( ) const

inline

Check if the token is a keyword.

bool Puma::Token::is_macro_call ( ) const

inline

Check if the token is a macro call.

bool Puma::Token::is_macro_generated ( ) const

Check if the token is macro generated.

bool Puma::Token::is_macro_op ( ) const

inline

Check if this is function-like macro operand.

bool Puma::Token::is_open ( ) const

inline

Check if this is a left parenthesis.

bool Puma::Token::is_preprocessor ( ) const

inline

Check if this is a preprocessor token.

bool Puma::Token::is_whitespace ( ) const

inline

Check if this is a white-space token.

bool Puma::Token::is_wildcard ( ) const

inline

Check if this is a wildcard token.

int Puma::Token::line_breaks ( ) const

Count the number of line breaks in the text of this token.

void Puma::Token::location ( const Location & loc )

inline

Set the location of the token (usually the line and column in a file).

Parameters

loc	The location.

const Location& Puma::Token::location ( ) const

inline

Get the location of the token (usually the line and column in a file).

void Puma::Token::macro_call ( bool is_call = true )

inline

Set the token as being a macro call.

void Puma::Token::operator delete ( void * )

Own delete operator.

void* Puma::Token::operator new ( size_t )

Own operator new reusing memory.

void Puma::Token::print ( ostream & os ) const

Print the token text on the given stream.

Parameters

os	The output stream.

void Puma::Token::reset	(	int	type = `0`,
		const char *	text = `0`,
		LanguageID	lang = `LanguageID(0)`
	)

Reset the token object.

Parameters

type	The new token type.
text	The new token text.
lang	The new token language.

const char* Puma::Token::text ( ) const

Get the token text.

int Puma::Token::type ( ) const

inline

Get the type of the token (see Puma::CTokens).

Unit* Puma::Token::unit ( ) const

Get the unit this token belongs to.

Member Data Documentation

LanguageID Puma::Token::close_id

static

Right parenthesis.

LanguageID Puma::Token::comma_id

static

Comma.

LanguageID Puma::Token::comment_id

static

Comment tokens.

LanguageID Puma::Token::cpp_id

static

C/C++ core language tokens.

LanguageID Puma::Token::dir_id

static

Compiler directives.

LanguageID Puma::Token::identifier_id

static

Identifiers.

LanguageID Puma::Token::keyword_id

static

Keywords.

LanguageID Puma::Token::macro_op_id

static

Function-like macro operands.

LanguageID Puma::Token::open_id

static

Left parenthesis.

LanguageID Puma::Token::pre_id

static

Preprocessor tokens.

LanguageID Puma::Token::white_id

static

White-spaces.

LanguageID Puma::Token::wildcard_id

static

Wildcards.

Public Types

Public Member Functions

Static Public Member Functions

Static Public Attributes

Detailed Description

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation