Tokens

Tokens are word-like units recognized by a language. SpecMan recognizes five classes of tokens.

Here is the formal definition of a token:

• keyword

• identifier
• constant
• operator
• separators

As the source code is scanned, tokens are extracted in such a way that the longest possible token from the character sequence is selected.