51

I have problem to catch the real meaning of the term 'Token.'

In terms of software development, can you define it generically? (Does it have different meanings in terms of different contexts and languages?)

Thanks!

pencilCake
  • 51,323
  • 85
  • 226
  • 363

7 Answers7

58

If you google define:token, you get (amongst others) the following two definitions that seem applicable:

  • an individual instance of a type of symbol; "the word 'error' contains three tokens of `r'"
  • something serving as a sign of something else

If you combine these two, you will land somewhere near what is commonly meant when talking about tokens in programming; a symbol representing something. Pretty vague, yes, but then it's used in many different contexts.

One example: you have an authentication system where a user logs on. When the system has authenticated the user, instead of repeating this process for every request, a token is created that represents the fact that the user is authenticated. This token is then used in subsequent requests. In this case the something is the fact the the user is authenticated, and the token represents this fact.

Fredrik Mörk
  • 155,851
  • 29
  • 291
  • 343
  • Then if I use the word 'something' instead of Token it won't be wrong. :) I am afraid, I really don't get the point :( – pencilCake Dec 15 '10 at 10:17
  • @Frederik: Thanks Frederik, good example. So, basically it represents a "specific instance" in the OOP context maybe, right? – pencilCake Dec 15 '10 at 10:33
  • @burak: again; depends on context, but yes in a way you could say that. A "specific instance" is "something" :) – Fredrik Mörk Dec 15 '10 at 10:34
  • Very nice definition of token. Just a question : Why should digits and numbers be considered as token? Isn't number enough? –  Sep 12 '16 at 05:22
  • @Ehsan I would say that a digits (such as 1 or 7) could be considered tokens with which you can produce numbers (such as 17). Depending on context, 1 or 7 could of course be considered to be numbers as well, consisting of single tokens. – Fredrik Mörk Sep 12 '16 at 13:09
12

Tokens are: identifiers, keywords, literals, operators, and punctuators. But we can't consider White spaces and comments as tokens, though they act as separators for tokens.

nadun
  • 393
  • 1
  • 4
  • 14
6

In the compiler Lexical analyzer (or scanner ) : Reads the input stream and fuses characters of the source text into tokens of the language. Token : sequence of characters having a collective meaning. The character sequence forming a token is called the Lexeme.

this example might help
Consider the following assignment statement
newvalue = oldvalue + rate * 60
The lexical analyzer will generate the following tokens.

Token       Lexeme
Identifier  newvalue
assignop    =
Identifier  oldvalue
addop       +
identifier  rate
mulop       *
number      6
ChamaraL
  • 351
  • 4
  • 7
3

One of the meaning in terms of Software Development is the authentication signature which is issued by a server for a defined time interval

TalentTuner
  • 17,262
  • 5
  • 38
  • 63
2

A token is a single element of a programming language. There are 5 token categories :

  • Reserved words
  • Operators
  • Identifiers
  • Constants
  • Separators
Max
  • 2,710
  • 1
  • 23
  • 34
1

Token can be seen as a seal, as when in the middle age a courrier representig a king or a duke or a bishop or a pope or a anything went riding from realm to realm, and needed to be authenticated as the true representative of what he claim to be from when passing the gates of each kingdom.

He had to show a letter with that specific seal or showing a medal or a special coin engraved like few others, token is like one of these items.

Antoine Meltzheim
  • 9,579
  • 6
  • 35
  • 41
0

A token is an object of value

http://en.wikipedia.org/wiki/Token

GvS
  • 52,015
  • 16
  • 101
  • 139