Falcon::Tokenizer Class Reference

Base tokenizer base class. More...

#include <tokenizer.h>

Inheritance diagram for Falcon::Tokenizer:

[legend]

List of all members.

Public Member Functions
virtual void	append (const Item &itm)
	Append an item at the end of the sequence.
virtual const Item &	back () const
	Calling this causes a CodeError (unimplemented) to be raised.
virtual void	clear ()
	Calling this causes a CodeError (unimplemented) to be raised.
virtual FalconData *	clone () const
virtual bool	empty () const
	Returns true if the tokenizer knows it can't return any other element.
virtual const Item &	front () const
	Calling this causes a CodeError (unimplemented) to be raised.
virtual void	gcMark (uint32 mark)
const String &	getToken () const
	Returns the currently active range after a succesful next.
bool	hasCurrent () const
bool	isReady () const
	Returns true if the tokenizer has been readied with a stream.
virtual bool	next ()
	Advance to a further token.
void	parse (Stream *in, bool bOwn=false)
	Resets the tokenizer providing new data to be tokenized.
void	parse (const String &data)
	Resets the tokenizer providing new data to be tokenized.
virtual void	prepend (const Item &itm)
	Prepend an item at the beginning of the sequence.
virtual void	rewind ()
	This may fail if the underlying stream doesn't support seek.
	Tokenizer (const Tokenizer &other)
	Tokenizer (TokenizerParams &params, const String &seps, Stream *inp=0, bool bOwn=false)
	Tokenizer (TokenizerParams &params, const String &seps, const String &source)
	Creates a ROStringStream and uses that to read the source.
virtual	~Tokenizer ()
Protected Member Functions
virtual void	copyIterator (Iterator &tgt, const Iterator &source) const
	Copy an iterator so that the two points to the same item.
virtual bool	equalIterator (const Iterator &first, const Iterator &second) const
virtual void	erase (Iterator &iter)
	Deletes the element at position indicated by the iterator.
virtual Item &	getCurrent (const Iterator &iter)
virtual Item &	getCurrentKey (const Iterator &iter)
virtual void	getIterator (Iterator &tgt, bool tail=false) const
	Gets an Iterator valid for this sequence.
virtual bool	hasCurrent (const Iterator &iter) const
virtual bool	hasNext (const Iterator &iter) const
virtual bool	hasPrev (const Iterator &iter) const
virtual void	insert (Iterator &iter, const Item &data)
	Inserts an element in a position indicated by the iterator.
virtual bool	next (Iterator &iter) const
virtual bool	prev (Iterator &iter) const

---

Detailed Description

Base tokenizer base class.

Although this class is declared as a sequence, it only supports empty() and getIterator() operations, needed for the TRAV loop in the vm.

In future, some subclasses may support some specific operations when they are locally buffered.

The tokenizer is designed to operate with subclasses tokenizing a string and or a stream. The StringStream class is not used for string for performance reasons; as a very limited subset of operations are needed (namely, get), and as the visibility of the underlying buffer is useful (i.e. to avoid storing a local copy of the forming buffer), the StringTokenizer has a different, optimized implementation.

The iterator generated by a Tokenizer is one-way only. Every next() operation on an iterator invalidates the others; this means that only one iterator at a time can be used on a tokenizer.

Note:

As this class is used only internally, there is no need to mark the owner on GC as usual. All GC marking must be external (i.e. placing the item as an hidden property of the tokenizer object).

---

Constructor & Destructor Documentation

Falcon::Tokenizer::Tokenizer	(	TokenizerParams &	params,
		const String &	seps,
		const String &	source
	)

Creates a ROStringStream and uses that to read the source.

WARNING: the source must be granted to stay alive for the whole duration of the tokenization, as nothing is going to create a local safe copy of the given string.

Falcon::Tokenizer::Tokenizer	(	TokenizerParams &	params,
		const String &	seps,
		Stream *	inp = 0,
		bool	bOwn = false
	)

Falcon::Tokenizer::Tokenizer

(

const Tokenizer &

other

)

virtual Falcon::Tokenizer::~Tokenizer

(

)

[virtual]

---

Member Function Documentation

virtual void Falcon::Tokenizer::append

(

const Item &

data

)

[virtual]

Append an item at the end of the sequence.

Implements Falcon::Sequence.

virtual const Item& Falcon::Tokenizer::back

(

)

const [virtual]

Calling this causes a CodeError (unimplemented) to be raised.

Subclasses may provide a consistent behavior if they wish.

Implements Falcon::Sequence.

virtual void Falcon::Tokenizer::clear

(

)

[virtual]

Calling this causes a CodeError (unimplemented) to be raised.

Subclasses may provide a consistent behavior if they wish.

Implements Falcon::Sequence.

virtual FalconData* Falcon::Tokenizer::clone

(

)

const [virtual]

Implements Falcon::FalconData.

virtual void Falcon::Tokenizer::copyIterator	(	Iterator &	tgt,
		const Iterator &	source
	)			const [protected, virtual]

Copy an iterator so that the two points to the same item.

The source iterator may point to the past-end element, but must not be invalid.

Note:

The base version of this function just adds the iterator to the iterator list; it MUST be called by all the implementations.

Reimplemented from Falcon::Sequence.

virtual bool Falcon::Tokenizer::empty

(

)

const [virtual]

Returns true if the tokenizer knows it can't return any other element.

Implements Falcon::Sequence.

virtual bool Falcon::Tokenizer::equalIterator	(	const Iterator &	first,
		const Iterator &	second
	)			const [protected, virtual]

Implements Falcon::Sequence.

virtual void Falcon::Tokenizer::erase

(

Iterator &

iter

)

[protected, virtual]

Deletes the element at position indicated by the iterator.

The implementation must check that the iterator is a valid iterator created by this object and pointing to a valid position.

Deletion happens at given position, shifting all the remaining elements backward; after a successful erase, the iterator must point to the element that was previously next in the series, or must be invalidated if the removed element was the last.

If the sequence is empty or the iterator is invalid, an AccessError must be thrown. If the iterator is referencing another sequence, a CodeError must be thrown.

Parameters:

iter

an iterator (possibly invalid or not generated by this class).

Returns:

true if the iterator was valid for this object.

Implements Falcon::Sequence.

virtual const Item& Falcon::Tokenizer::front

(

)

const [virtual]

Calling this causes a CodeError (unimplemented) to be raised.

Subclasses may provide a consistent behavior if they wish.

Implements Falcon::Sequence.

virtual void Falcon::Tokenizer::gcMark

(

uint32

mark

)

[inline, virtual]

Reimplemented from Falcon::Sequence.

References Falcon::Sequence::gcMark().

virtual Item& Falcon::Tokenizer::getCurrent

(

const Iterator &

iter

)

[protected, virtual]

Implements Falcon::Sequence.

virtual Item& Falcon::Tokenizer::getCurrentKey

(

const Iterator &

iter

)

[protected, virtual]

Implements Falcon::Sequence.

virtual void Falcon::Tokenizer::getIterator	(	Iterator &	tgt,
		bool	tail = false
	)			const [protected, virtual]

Gets an Iterator valid for this sequence.

If you need an iterator as a pointer or in the target stack, use Iterator( Sequence*, bool) instead.

The iterator constructor calls back this method to be configured.

It is possible to call this method thereafter to reset the iterator, even if it's gone invalid.

However, it is not legal to call this method with an iterator coming from another sequence; this will cause the program to throw a CodeError.

Note:

The base version of this function just adds the iterator to the iterator list; it MUST be called by all the implementations.

Parameters:

	An	Iterator to be set.
	tail	if false, get an iterator to the first element, else get an iterator to the last element.

Reimplemented from Falcon::Sequence.

const String& Falcon::Tokenizer::getToken

(

)

const [inline]

Returns the currently active range after a succesful next.

virtual bool Falcon::Tokenizer::hasCurrent

(

const Iterator &

iter

)

const [protected, virtual]

Implements Falcon::Sequence.

bool Falcon::Tokenizer::hasCurrent

(

)

const [inline]

virtual bool Falcon::Tokenizer::hasNext

(

const Iterator &

iter

)

const [protected, virtual]

Implements Falcon::Sequence.

virtual bool Falcon::Tokenizer::hasPrev

(

const Iterator &

iter

)

const [protected, virtual]

Implements Falcon::Sequence.

virtual void Falcon::Tokenizer::insert	(	Iterator &	iter,
		const Item &	data
	)			[protected, virtual]

Inserts an element in a position indicated by the iterator.

The implementation must check that the iterator is a valid iterator created by this object and pointing to a valid position.

Insertion happens at given position, shifting all the remaining elements forward; after a successful insert, the iterator must point to the newly inserted element, and the previously current element is found safely in the next() position of the iterator.

Valid iterators (generated by this owner) pointing to invalid positions must be treated as pointing to last-past-one element; insertion causes append on tail, and at return they must be valid and point to the last valid element (the one just inserted).

If the iterator cannot be used, for example because their owner is not this item, this method will raise a CodeError.

Parameters:

	iterator	an iterator.
	data	the item to be inserted

Returns:

true if the iterator was valid for this object.

Implements Falcon::Sequence.

bool Falcon::Tokenizer::isReady

(

)

const [inline]

Returns true if the tokenizer has been readied with a stream.

virtual bool Falcon::Tokenizer::next

(

Iterator &

iter

)

const [protected, virtual]

Implements Falcon::Sequence.

virtual bool Falcon::Tokenizer::next

(

)

[virtual]

Advance to a further token.

Returns:

false if no more token can be found

void Falcon::Tokenizer::parse	(	Stream *	in,
		bool	bOwn = false
	)

Resets the tokenizer providing new data to be tokenized.

If the parameter bOwn is true, ti

void Falcon::Tokenizer::parse

(

const String &

data

)

Resets the tokenizer providing new data to be tokenized.

The string data must stay alive as long as this parser is used, as it is not internally copied anywhere.

virtual void Falcon::Tokenizer::prepend

(

const Item &

data

)

[virtual]

Prepend an item at the beginning of the sequence.

Implements Falcon::Sequence.

virtual bool Falcon::Tokenizer::prev

(

Iterator &

iter

)

const [protected, virtual]

Implements Falcon::Sequence.

virtual void Falcon::Tokenizer::rewind

(

)

[virtual]

This may fail if the underlying stream doesn't support seek.

---

The documentation for this class was generated from the following file:

• /home/gian/Progetti/falcon/core/include/falcon/tokenizer.h