org.netbeans.modules.lexer/2 1.39.1 1

org.netbeans.api.lexer
Class Token<T extends TokenId>

java.lang.Object
  extended by org.netbeans.api.lexer.Token<T>

public abstract class Token<T extends TokenId>
extends Object

Token describes a lexical element of input text.
It mainly provides an identification by id() and a textual body (aka token's image) by text().
Only lexers should produce token instances and they should do it solely by using methods of TokenFactory.

Note: Do not create custom extensions of this class - lexers may only return implementations produced by TokenFactory. Creation of any other token implementations will be refused.

Token guarantees stability of the id() and length() methods. The hashCode() and equals(Object) methods use the default implementations from java.lang.Object.
The two tokens are only equal if they are the same object.


Constructor Summary
protected Token()
          Create token instance.
 
Method Summary
 boolean equals(Object o)
          Make sure the default implementation of equals() is used and the token can safely be used in maps.
abstract  Object getProperty(Object key)
          Get extra property of this token.
 int hashCode()
          Make sure the default implementation of hashCode() is used and the token can safely be used in maps.
abstract  boolean hasProperties()
          Quickly determine whether this token has any extra properties.
abstract  T id()
          Get identification of this token.
abstract  boolean isCustomText()
          Check whether text() returns a custom value that may differ from the original content of the text input.
abstract  boolean isFlyweight()
          Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.
 boolean isRemoved()
          Check whether this token is no longer part of the token hierarchy.
 List<? extends Token<T>> joinedParts()
          Get all token parts comprising this token ordered from lowest to highest part's offset.
 Token<T> joinToken()
          Get a complete token that is joined from multiple parts (this token is one of those parts).
abstract  int length()
          Get number of characters in the original text input that the token spans.
abstract  int offset(TokenHierarchy<?> tokenHierarchy)
          Get the offset at which this token is present in the input or -1 if this token is flyweight (and therefore does not store offset).
abstract  PartType partType()
          Check whether this token represents a complete token or whether it's a particular part of a complete token.
abstract  CharSequence text()
          Get text of this token (aka token's image) as a character sequence.
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Token

protected Token()
Create token instance.

Throws:
IllegalStateException - if a non-lexer-module-implementation token is attempted to be created.
Method Detail

id

public abstract T id()
Get identification of this token.

Returns:
non-null identification of this token.

text

public abstract CharSequence text()
Get text of this token (aka token's image) as a character sequence.
This text usually corresponds to the characters present in the lexed text input unless isCustomText() returns true.

Note for mutable input sources:
This method should only be called within a readonly (or read-write) transaction over the underlying input source (such as javax.swing.text.Document.render() for Swing documents).
The result returned by this method is only valid within a readonly (or read-write) transaction over the input source (method must be re-called during the next readonly transaction).

Returns:
non-null, non-empty text of this token. It may be null in case the token was used for a mutable input and it was removed from the token list for the given input (but even in such case the text can be retained in certain cases).

The behavior of equals() and hashCode() of the returned character sequence is generally undefined.
The returned character sequence can NOT be compared to another character sequence by using its equals() method.
TokenUtilities contains utility methods related to token text comparing.

The returned text is just a pointer to the primary source of the data e.g. a swing document. The character data are not duplicated in the tokens.


isCustomText

public abstract boolean isCustomText()
Check whether text() returns a custom value that may differ from the original content of the text input.
Using custom text may be useful in case when only certain part of the token is useful for the particular use and the token's text can be shrinked and possibly a flyweight text can be used.
Also this is useful when using lexers generated by various lexer generators that generally allow to use a custom text in the produced tokens.

Returns:
true if the text of the token does not correspond to the original characters present in the text input being lexed.

length

public abstract int length()
Get number of characters in the original text input that the token spans.
Usually this is the same value like text().length() unless isCustomText() returns true.
Also this method will return valid length in all cases even when the text of the token could become null.

Returns:
>=0 length of the token.

offset

public abstract int offset(TokenHierarchy<?> tokenHierarchy)
Get the offset at which this token is present in the input or -1 if this token is flyweight (and therefore does not store offset).
Note: Use of TokenSequence.offset() is usually preferred over this method because it returns actual offset even for the flyweight tokens.
If necessary the flyweight token may be replaced by regular token by using TokenSequence.offsetToken().

The complexity of the method should generally be constant regardless of the level of the language embedding.

Parameters:
tokenHierarchy - null should be passed (the parameter is reserved for future use when token hierarchy snapshots will be implemented).
Returns:
>=0 offset of the token in the input or -1 if this token is flyweight.

isFlyweight

public abstract boolean isFlyweight()
Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.
For example keywords or operators are typically flyweight tokens while e.g. identifiers are not flyweight as their text generally varies.
Flyweight tokens may decrease the memory consumption for the tokens considerably for frequently used tokens. For example a single space ' ' may be a useful flyweight token as it's used very often throughout a source. The decision of what tokens are made flyweight is upon the implementor of the particular language.

If the token is flyweight its offset(TokenHierarchy) returns -1.

Returns:
true if the token is flyweight or false otherwise.

isRemoved

public boolean isRemoved()
Check whether this token is no longer part of the token hierarchy.

Returns:
true if the token was removed from the token hierarchy or false if it's still present in the hierarchy.

partType

public abstract PartType partType()
Check whether this token represents a complete token or whether it's a particular part of a complete token.
Some lexers may also use this information to express an incomplete token. For example an unclosed block comment at the end of java source is represented as a BLOCK_COMMENT token id and PartType.START.

Returns:
PartType.COMPLETE for regular token or other part types for particular token parts.

joinToken

public Token<T> joinToken()
Get a complete token that is joined from multiple parts (this token is one of those parts).

Returns:
complete token or null if this token is not a part of any token.

joinedParts

public List<? extends Token<T>> joinedParts()
Get all token parts comprising this token ordered from lowest to highest part's offset.
It's guaranteed that each token part is continuous in the input text (there are no gaps inside the token part's text).
On the other hand there may be textual gaps between two adajcent token parts.

Returns:
list of token parts or null if the token is continuous.

hasProperties

public abstract boolean hasProperties()
Quickly determine whether this token has any extra properties.


getProperty

public abstract Object getProperty(Object key)
Get extra property of this token.
The token properties are defined by the lexer upon token creation. The clients of the API cannot set any property of the token.

Parameters:
key - non-null key of the property to get.
Returns:
non-null value of the property or null if the property does not have any value.
See Also:
hasProperties()

hashCode

public final int hashCode()
Make sure the default implementation of hashCode() is used and the token can safely be used in maps.

Overrides:
hashCode in class Object

equals

public final boolean equals(Object o)
Make sure the default implementation of equals() is used and the token can safely be used in maps.

Overrides:
equals in class Object

org.netbeans.modules.lexer/2 1.39.1 1

Built on December 5 2011.  |  Portions Copyright 1997-2011 Sun Microsystems, Inc. All rights reserved.