org.fife.ui.rsyntaxtextarea
Class Token

java.lang.Object
  extended by org.fife.ui.rsyntaxtextarea.Token
Direct Known Subclasses:
DefaultToken

public abstract class Token
extends Object

A generic token that functions as a node in a linked list of syntax highlighted tokens for some language.

A Token is a piece of text representing some logical token in source code for a programming language. For example, the line of C code:

 int i = 0;
 
would be broken into 8 Tokens: the first representing int, the second whitespace, the third i, the fourth whitespace, the fifth =, etc.

Version:
0.3
Author:
Robert Futrell

Field Summary
static int COMMENT_DOCUMENTATION
           
static int COMMENT_EOL
           
static int COMMENT_MULTILINE
           
static int DATA_TYPE
           
static int ERROR_CHAR
           
static int ERROR_IDENTIFIER
           
static int ERROR_NUMBER_FORMAT
           
static int ERROR_STRING_DOUBLE
           
static int FUNCTION
           
static int IDENTIFIER
           
static int LITERAL_BACKQUOTE
           
static int LITERAL_BOOLEAN
           
static int LITERAL_CHAR
           
static int LITERAL_NUMBER_DECIMAL_INT
           
static int LITERAL_NUMBER_FLOAT
           
static int LITERAL_NUMBER_HEXADECIMAL
           
static int LITERAL_STRING_DOUBLE_QUOTE
           
static int MARKUP_TAG_ATTRIBUTE
           
static int MARKUP_TAG_DELIMITER
           
static int MARKUP_TAG_NAME
           
static int NULL
           
static int NUM_TOKEN_TYPES
           
 int offset
          The offset into the document at which this token resides.
static int OPERATOR
           
static int PREPROCESSOR
           
static int RESERVED_WORD
           
static int SEPARATOR
           
 char[] text
          The text this token represents.
 int textCount
           
 int textOffset
           
 int type
          The type of token this is; for example, FUNCTION.
static int VARIABLE
           
static int WHITESPACE
           
 
Constructor Summary
Token()
          Creates a "null token."
Token(char[] line, int beg, int end, int startOffset, int type)
          Constructor.
Token(Token t2)
          Creates this token as a deep copy of the passed-in token.
 
Method Summary
 StringBuffer appendHTMLRepresentation(StringBuffer sb, RSyntaxTextArea textArea, boolean fontFamily)
          Appends HTML code for painting this token, using the given text area's color scheme.
 boolean containsPosition(int pos)
          Returns whether the token straddles the specified position in the document.
 void copyFrom(Token t2)
          Makes one token point to the same text segment, and have the same value as another token.
 int documentToToken(int pos)
          Returns the position in the token's internal char array corresponding to the specified document position.
 String getHTMLRepresentation(RSyntaxTextArea textArea)
          Returns a String containing HTML code for painting this token, using the given text area's color scheme.
 Token getLastNonCommentNonWhitespaceToken()
          Returns the last token in this list that is not whitespace or a comment.
 Token getLastPaintableToken()
          Returns the last paintable token in this token list, or null if there is no paintable token.
 String getLexeme()
          Returns the text of this token, as a string.
abstract  int getListOffset(RSyntaxTextArea textArea, TabExpander e, float x0, float x)
          Determines the offset into this token list (i.e., into the document) that covers pixel location x if the token list starts at pixel location x0.
 Token getNextToken()
          Returns the token after this one in the linked list.
 int getOffsetBeforeX(RSyntaxTextArea textArea, TabExpander e, float startX, float endBeforeX)
          Returns the position in the document that represents the last character in the token that will fit into endBeforeX-startX pixels.
 float getWidth(RSyntaxTextArea textArea, TabExpander e, float x0)
          Returns the width of this token given the specified parameters.
abstract  float getWidthUpTo(int numChars, RSyntaxTextArea textArea, TabExpander e, float x0)
          Returns the width of a specified number of characters in this token.
 boolean is(int type, String lexeme)
          Returns whether this token is of the specified type, with the specified lexeme.
 boolean isComment()
          Returns whether this token is a comment.
 boolean isHyperlink()
          Returns whether this token is a hyperlink.
 boolean isLeftCurly()
          Returns whether this token is a SEPARATOR representing a single left curly brace.
 boolean isPaintable()
          Returns whether or not this token is "paintable;" i.e., whether or not the type of this token is one such that it has an associated syntax style.
 boolean isRightCurly()
          Returns whether this token is a SEPARATOR representing a single right curly brace.
 boolean isSingleChar(char ch)
          Returns whether this token is the specified single character.
 boolean isWhitespace()
          Returns whether or not this token is whitespace.
abstract  Rectangle listOffsetToView(RSyntaxTextArea textArea, TabExpander e, int pos, int x0, Rectangle rect)
          Returns the bounding box for the specified document location.
 void makeStartAt(int pos)
          Makes this token start at the specified offset into the document.
 void moveOffset(int amt)
          Moves the starting offset of this token.
 float paint(Graphics2D g, float x, float y, RSyntaxTextArea host, TabExpander e)
          Paints this token.
abstract  float paint(Graphics2D g, float x, float y, RSyntaxTextArea host, TabExpander e, float clipStart)
          Paints this token.
protected  void paintBackground(float x, float y, float width, float height, Graphics2D g, int fontAscent, RSyntaxTextArea host, Color color)
          Paints the background of a token.
 void set(char[] line, int beg, int end, int offset, int type)
          Sets the value of this token to a particular segment of a document.
 void setHyperlink(boolean hyperlink)
          Sets whether this token is a hyperlink.
 void setNextToken(Token nextToken)
          Sets the "next token" pointer of this token to point to the specified token.
 int tokenToDocument(int pos)
          Returns the position in the document corresponding to the specified position in this token's internal char array (textOffset - textOffset+textCount-1).
 String toString()
          Returns this token as a String, which is useful for debugging.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

text

public char[] text
The text this token represents. This is implemented as a segment so we can point directly to the text in the document without having to make a copy of it.


textOffset

public int textOffset

textCount

public int textCount

offset

public int offset
The offset into the document at which this token resides.


type

public int type
The type of token this is; for example, FUNCTION.


NULL

public static final int NULL
See Also:
Constant Field Values

COMMENT_EOL

public static final int COMMENT_EOL
See Also:
Constant Field Values

COMMENT_MULTILINE

public static final int COMMENT_MULTILINE
See Also:
Constant Field Values

COMMENT_DOCUMENTATION

public static final int COMMENT_DOCUMENTATION
See Also:
Constant Field Values

RESERVED_WORD

public static final int RESERVED_WORD
See Also:
Constant Field Values

FUNCTION

public static final int FUNCTION
See Also:
Constant Field Values

LITERAL_BOOLEAN

public static final int LITERAL_BOOLEAN
See Also:
Constant Field Values

LITERAL_NUMBER_DECIMAL_INT

public static final int LITERAL_NUMBER_DECIMAL_INT
See Also:
Constant Field Values

LITERAL_NUMBER_FLOAT

public static final int LITERAL_NUMBER_FLOAT
See Also:
Constant Field Values

LITERAL_NUMBER_HEXADECIMAL

public static final int LITERAL_NUMBER_HEXADECIMAL
See Also:
Constant Field Values

LITERAL_STRING_DOUBLE_QUOTE

public static final int LITERAL_STRING_DOUBLE_QUOTE
See Also:
Constant Field Values

LITERAL_CHAR

public static final int LITERAL_CHAR
See Also:
Constant Field Values

LITERAL_BACKQUOTE

public static final int LITERAL_BACKQUOTE
See Also:
Constant Field Values

DATA_TYPE

public static final int DATA_TYPE
See Also:
Constant Field Values

VARIABLE

public static final int VARIABLE
See Also:
Constant Field Values

IDENTIFIER

public static final int IDENTIFIER
See Also:
Constant Field Values

WHITESPACE

public static final int WHITESPACE
See Also:
Constant Field Values

SEPARATOR

public static final int SEPARATOR
See Also:
Constant Field Values

OPERATOR

public static final int OPERATOR
See Also:
Constant Field Values

PREPROCESSOR

public static final int PREPROCESSOR
See Also:
Constant Field Values

MARKUP_TAG_DELIMITER

public static final int MARKUP_TAG_DELIMITER
See Also:
Constant Field Values

MARKUP_TAG_NAME

public static final int MARKUP_TAG_NAME
See Also:
Constant Field Values

MARKUP_TAG_ATTRIBUTE

public static final int MARKUP_TAG_ATTRIBUTE
See Also:
Constant Field Values

ERROR_IDENTIFIER

public static final int ERROR_IDENTIFIER
See Also:
Constant Field Values

ERROR_NUMBER_FORMAT

public static final int ERROR_NUMBER_FORMAT
See Also:
Constant Field Values

ERROR_STRING_DOUBLE

public static final int ERROR_STRING_DOUBLE
See Also:
Constant Field Values

ERROR_CHAR

public static final int ERROR_CHAR
See Also:
Constant Field Values

NUM_TOKEN_TYPES

public static final int NUM_TOKEN_TYPES
See Also:
Constant Field Values
Constructor Detail

Token

public Token()
Creates a "null token." The token itself is not null; rather, it signifies that it is the last token in a linked list of tokens and that it is not part of a "multi-line token."


Token

public Token(char[] line,
             int beg,
             int end,
             int startOffset,
             int type)
Constructor.

Parameters:
line - The segment from which to get the token.
beg - The first character's position in line.
end - The last character's position in line.
startOffset - The offset into the document at which this token begins.
type - A token type listed as "generic" above.

Token

public Token(Token t2)
Creates this token as a deep copy of the passed-in token.

Parameters:
t2 - The token from which to make a copy.
Method Detail

appendHTMLRepresentation

public StringBuffer appendHTMLRepresentation(StringBuffer sb,
                                             RSyntaxTextArea textArea,
                                             boolean fontFamily)
Appends HTML code for painting this token, using the given text area's color scheme.

Parameters:
sb - The buffer to append to.
textArea - The text area whose color scheme to use.
fontFamily - Whether to include the font family in the HTML for this token. You can pass false for this parameter if, for example, you are making all your HTML be monospaced, and don't want any crazy fonts being used in the editor to be reflected in your HTML.
Returns:
The buffer appended to.
See Also:
getHTMLRepresentation(RSyntaxTextArea)

containsPosition

public boolean containsPosition(int pos)
Returns whether the token straddles the specified position in the document.

Parameters:
pos - The position in the document to check.
Returns:
Whether the specified position is straddled by this token.

copyFrom

public void copyFrom(Token t2)
Makes one token point to the same text segment, and have the same value as another token.

Parameters:
t2 - The token from which to copy.

documentToToken

public int documentToToken(int pos)
Returns the position in the token's internal char array corresponding to the specified document position.

Note that this method does NOT do any bounds checking; you can pass in a document position that does not correspond to a position in this token, and you will not receive an Exception or any other notification; it is up to the caller to ensure valid input.

Parameters:
pos - A position in the document that is represented by this token.
Returns:
The corresponding token position >= textOffset and < textOffset+textCount.
See Also:
tokenToDocument(int)

getHTMLRepresentation

public String getHTMLRepresentation(RSyntaxTextArea textArea)
Returns a String containing HTML code for painting this token, using the given text area's color scheme.

Parameters:
textArea - The text area whose color scheme to use.
Returns:
The HTML representation of the token.
See Also:
appendHTMLRepresentation(StringBuffer, RSyntaxTextArea, boolean)

getLastNonCommentNonWhitespaceToken

public Token getLastNonCommentNonWhitespaceToken()
Returns the last token in this list that is not whitespace or a comment.

Returns:
The last non-comment, non-whitespace token, or null if there isn't one.

getLastPaintableToken

public Token getLastPaintableToken()
Returns the last paintable token in this token list, or null if there is no paintable token.

Returns:
The last paintable token in this token list.

getLexeme

public String getLexeme()
Returns the text of this token, as a string.

Note that this method isn't used much by the rsyntaxtextarea package internally, as it tries to limit memory allocation.

Returns:
The text of this token.

getListOffset

public abstract int getListOffset(RSyntaxTextArea textArea,
                                  TabExpander e,
                                  float x0,
                                  float x)
Determines the offset into this token list (i.e., into the document) that covers pixel location x if the token list starts at pixel location x0

. This method will return the document position "closest" to the x-coordinate (i.e., if they click on the "right-half" of the w in awe, the caret will be placed in between the w and e; similarly, clicking on the left-half places the caret between the a and w). This makes it useful for methods such as viewToModel found in javax.swing.text.View subclasses.

This method is abstract so subclasses who paint themselves differently (i.e., VisibleWhitespaceToken is painted a tad differently than DefaultToken when rendering hints are enabled) can still return accurate results.

Parameters:
textArea - The text area from which the token list was derived.
e - How to expand tabs.
x0 - The pixel x-location that is the beginning of tokenList.
x - The pixel-position for which you want to get the corresponding offset.
Returns:
The position (in the document, NOT into the token list!) that covers the pixel location. If tokenList is null or has type Token.NULL, then -1

getNextToken

public Token getNextToken()
Returns the token after this one in the linked list.

Returns:
The next token.
See Also:
setNextToken(org.fife.ui.rsyntaxtextarea.Token)

getOffsetBeforeX

public int getOffsetBeforeX(RSyntaxTextArea textArea,
                            TabExpander e,
                            float startX,
                            float endBeforeX)
Returns the position in the document that represents the last character in the token that will fit into endBeforeX-startX pixels. For example, if you're using a monospaced 8-pixel-per-character font, have the token "while" and startX is 0 and endBeforeX is 30, this method will return the document position of the "i" in "while", because the "i" ends at pixel 24, while the "l" ends at 32. If not even the first character fits in endBeforeX-startX, the first character's position is still returned so calling methods don't go into infinite loops.

Parameters:
textArea - The text area in which this token is being painted.
e - How to expand tabs.
startX - The x-coordinate at which the token will be painted. This is needed because of tabs.
endBeforeX - The x-coordinate for which you want to find the last character of t which comes before it.
Returns:
The last document position that will fit in the specified amount of pixels.

getWidth

public float getWidth(RSyntaxTextArea textArea,
                      TabExpander e,
                      float x0)
Returns the width of this token given the specified parameters.

Parameters:
textArea - The text area in which the token is being painted.
e - Describes how to expand tabs. This parameter cannot be null.
x0 - The pixel-location at which the token begins. This is needed because of tabs.
Returns:
The width of the token, in pixels.
See Also:
getWidthUpTo(int, org.fife.ui.rsyntaxtextarea.RSyntaxTextArea, javax.swing.text.TabExpander, float)

getWidthUpTo

public abstract float getWidthUpTo(int numChars,
                                   RSyntaxTextArea textArea,
                                   TabExpander e,
                                   float x0)
Returns the width of a specified number of characters in this token. For example, for the token "while", specifying a value of 3 here returns the width of the "whi" portion of the token.

This method is abstract so subclasses who paint themselves differently (i.e., VisibleWhitespaceToken is painted a tad differently than DefaultToken when rendering hints are enabled) can still return accurate results.

Parameters:
numChars - The number of characters for which to get the width.
textArea - The text area in which the token is being painted.
e - How to expand tabs. This value cannot be null.
x0 - The pixel-location at which this token begins. This is needed because of tabs.
Returns:
The width of the specified number of characters in this token.
See Also:
getWidth(org.fife.ui.rsyntaxtextarea.RSyntaxTextArea, javax.swing.text.TabExpander, float)

is

public boolean is(int type,
                  String lexeme)
Returns whether this token is of the specified type, with the specified lexeme.

Parameters:
type - The type to check for.
lexeme - The lexeme to check for.
Returns:
Whether this token has that type and lexeme.

isComment

public boolean isComment()
Returns whether this token is a comment.

Returns:
Whether this token is a comment.
See Also:
isWhitespace()

isHyperlink

public boolean isHyperlink()
Returns whether this token is a hyperlink.

Returns:
Whether this token is a hyperlink.
See Also:
setHyperlink(boolean)

isLeftCurly

public boolean isLeftCurly()
Returns whether this token is a SEPARATOR representing a single left curly brace.

Returns:
Whether this token is a left curly brace.
See Also:
isRightCurly()

isRightCurly

public boolean isRightCurly()
Returns whether this token is a SEPARATOR representing a single right curly brace.

Returns:
Whether this token is a right curly brace.
See Also:
isLeftCurly()

isPaintable

public boolean isPaintable()
Returns whether or not this token is "paintable;" i.e., whether or not the type of this token is one such that it has an associated syntax style. What this boils down to is whether the token type is greater than Token.NULL.

Returns:
Whether or not this token is paintable.

isSingleChar

public boolean isSingleChar(char ch)
Returns whether this token is the specified single character.

Parameters:
ch - The character to check for.
Returns:
Whether this token's lexeme is the single character specified.

isWhitespace

public boolean isWhitespace()
Returns whether or not this token is whitespace.

Returns:
true iff this token is whitespace.
See Also:
isComment()

listOffsetToView

public abstract Rectangle listOffsetToView(RSyntaxTextArea textArea,
                                           TabExpander e,
                                           int pos,
                                           int x0,
                                           Rectangle rect)
Returns the bounding box for the specified document location. The location must be in the specified token list; if it isn't, null is returned.

Parameters:
textArea - The text area from which the token list was derived.
e - How to expand tabs.
pos - The position in the document for which to get the bounding box in the view.
x0 - The pixel x-location that is the beginning of tokenList.
rect - The rectangle in which we'll be returning the results. This object is reused to keep from frequent memory allocations.
Returns:
The bounding box for the specified position in the model.

makeStartAt

public void makeStartAt(int pos)
Makes this token start at the specified offset into the document.

Parameters:
pos - The offset into the document this token should start at. Note that this token must already contain this position; if it doesn't, an exception is thrown.
Throws:
IllegalArgumentException - If pos is not already contained by this token.
See Also:
moveOffset(int)

moveOffset

public void moveOffset(int amt)
Moves the starting offset of this token.

Parameters:
amt - The amount to move the starting offset. This should be between 0 and textCount, inclusive.
Throws:
IllegalArgumentException - If amt is an invalid value.
See Also:
makeStartAt(int)

paint

public final float paint(Graphics2D g,
                         float x,
                         float y,
                         RSyntaxTextArea host,
                         TabExpander e)
Paints this token.

Parameters:
g - The graphics context in which to paint.
x - The x-coordinate at which to paint.
y - The y-coordinate at which to paint.
host - The text area this token is in.
e - How to expand tabs.
Returns:
The x-coordinate representing the end of the painted text.

paint

public abstract float paint(Graphics2D g,
                            float x,
                            float y,
                            RSyntaxTextArea host,
                            TabExpander e,
                            float clipStart)
Paints this token.

Parameters:
g - The graphics context in which to paint.
x - The x-coordinate at which to paint.
y - The y-coordinate at which to paint.
host - The text area this token is in.
e - How to expand tabs.
clipStart - The left boundary of the clip rectangle in which we're painting. This optimizes painting by allowing us to not paint paint when this token is "to the left" of the clip rectangle.
Returns:
The x-coordinate representing the end of the painted text.

paintBackground

protected void paintBackground(float x,
                               float y,
                               float width,
                               float height,
                               Graphics2D g,
                               int fontAscent,
                               RSyntaxTextArea host,
                               Color color)
Paints the background of a token.

Parameters:
x - The x-coordinate of the token.
y - The y-coordinate of the token.
width - The width of the token (actually, the width of the part of the token to paint).
height - The height of the token.
g - The graphics context with which to paint.
fontAscent - The ascent of the token's font.
host - The text area.
color - The color with which to paint.

set

public void set(char[] line,
                int beg,
                int end,
                int offset,
                int type)
Sets the value of this token to a particular segment of a document. The "next token" value is cleared.

Parameters:
line - The segment from which to get the token.
beg - The first character's position in line.
end - The last character's position in line.
offset - The offset into the document at which this token begins.
type - A token type listed as "generic" above.

setHyperlink

public void setHyperlink(boolean hyperlink)
Sets whether this token is a hyperlink.

Parameters:
hyperlink - Whether this token is a hyperlink.
See Also:
isHyperlink()

setNextToken

public void setNextToken(Token nextToken)
Sets the "next token" pointer of this token to point to the specified token.

Parameters:
nextToken - The new next token.
See Also:
getNextToken()

tokenToDocument

public int tokenToDocument(int pos)
Returns the position in the document corresponding to the specified position in this token's internal char array (textOffset - textOffset+textCount-1).

Note that this method does NOT do any bounds checking; you can pass in an invalid token position, and you will not receive an Exception or any other indication that the returned document position is invalid. It is up to the user to ensure valid input.

Parameters:
pos - A position in the token's internal char array (textOffset - textOffset+textCount).
Returns:
The corresponding position in the document.
See Also:
documentToToken(int)

toString

public String toString()
Returns this token as a String, which is useful for debugging.

Overrides:
toString in class Object
Returns:
A string describing this token.