Programming Language & Its Applications — Introduction to C Programming, NEC licence examination syllabus (Nepal Engineering Council).
C Tokens: the smallest pieces the compiler recognises
Before a compiler can understand your program, it chops the text into tokens.
Compilation begins with lexical analysis: the compiler scans your source file and breaks it into indivisible units — a keyword here, an identifier there, an operator, a constant. These units are tokens, and every syntax error you have ever seen originates at this level or just above it. Knowing the six categories makes the compiler's complaints far less mysterious.
🌍 Where this lives: the tokeniser you are about to study is the first stage of every compiler, and also of every syntax highlighter. The colours you see in VS Code come from a program doing exactly this — splitting your text into identifiers, keywords, operators and literals. When you mistype a variable name and get "undeclared identifier", that is the tokeniser having handed a token to the next stage which could not find it in the symbol table. If this interests you, search for "lexical analysis with flex" and you can build a working tokeniser in an afternoon.
The six token types
1. KEYWORDS — 32 reserved words in C (auto, break, case,
char, const, continue, default, do, double,
else, enum, extern, float, for, goto, if,
int, long, register, return, short, signed,
sizeof, static, struct, switch, typedef,
union, unsigned, void, volatile, while)
2. IDENTIFIERS — names you invent: variables, functions,
arrays, structs
3. CONSTANTS — fixed values: 42, 3.14, 'A', "hello"
4. STRINGS — a sequence of characters in double quotes
5. OPERATORS — + - * / % ++ -- && || etc.
6. SPECIAL SYMBOLS — [ ] { } ( ) , ; # etc.
Identifier rules
Valid:
· starts with a letter or underscore
· contains only letters, digits, underscores
· case-sensitive: total, Total and TOTAL are DIFFERENT
· cannot be a keyword
int count; ✔
int _temp; ✔
int x2y; ✔
Invalid:
int 2count; ✗ starts with a digit
int my-var; ✗ hyphen is an operator
int float; ✗ keyword
int total sum; ✗ space not allowed
Data types and their sizes
Type Typical size Range (signed)
─────────────────────────────────────────────────────────
char 1 byte −128 to 127
unsigned char 1 byte 0 to 255
short 2 bytes −32 768 to 32 767
int 4 bytes −2 147 483 648 to 2 147 483 647
unsigned int 4 bytes 0 to 4 294 967 295
long 4 or 8 bytes platform dependent
long long 8 bytes ±9.2 × 10¹⁸
float 4 bytes ~6 decimal digits precision
double 8 bytes ~15 decimal digits precision
IMPORTANT: sizes are NOT fixed by the C standard, only
minimums are. Always use sizeof() rather than assuming.
tokens_demo.c
#include <stdio.h>
int main(void) {
int count = 10; /* int=keyword, count=identifier, 10=constant */
float rate = 2.5f;
char grade = 'A'; /* character constant: single quotes */
char name[] = "Manish"; /* string literal: double quotes */
printf("%d items at %.2f each\n", count, rate);
printf("Grade %c for %s\n", grade, name);
printf("int=%zu float=%zu char=%zu\n",
sizeof(int), sizeof(float), sizeof(char));
return 0;
}
Output:
10 items at 2.50 each
Grade A for Manish
int=4 float=4 char=1
A detail that catches everyone: 'A' and "A" are completely different things. 'A' is a single char occupying 1 byte with value 65. "A" is a string — an array of 2 bytes holding 'A' and the terminating '\0'. Passing "A" where a char is expected is a type error, and passing 'A' to %s causes a crash.
Worked example 1 — identifying tokens
Count and classify the tokens in: sum = a + b * 2;
Breaking it down:
sum → identifier
= → operator (assignment)
a → identifier
+ → operator
b → identifier
* → operator
2 → constant (integer)
; → special symbol (statement terminator)
Total: 8 tokens
3 identifiers, 3 operators, 1 constant, 1 special symbol
Note how whitespace is NOT a token — it only separates them.
"sum=a+b*2;" produces exactly the same 8 tokens.
Worked example 2 — storage classes
Storage class Scope Lifetime Default value
──────────────────────────────────────────────────────────
auto block block garbage
register block block garbage
static block/file whole program 0
extern global whole program 0
static has two distinct meanings:
· inside a function → the variable keeps its value
between calls
· at file level → the variable/function is private to
that file (internal linkage)
static_demo.c
#include <stdio.h>
void counter(void) {
int auto_var = 0; /* recreated every call */static int static_var = 0; /* created ONCE, persists */
auto_var++;
static_var++;
printf("auto=%d static=%d\n", auto_var, static_var);
}
int main(void) {
counter();
counter();
counter();
return 0;
}
Output:
auto=1 static=1
auto=1 static=2
auto=1 static=3
The auto variable resets to 0 on every call, so it always prints 1. The static variable is initialised only once, at program start, and remembers its value — which is exactly the behaviour you want for a call counter or a cache.
💡 Exam angle: listing the six token types with examples is a reliable 4-mark question. Identifier rules and the 32 keywords are direct recall. The highest-value details are the 'A' vs "A" distinction and the two meanings of static — both are favourite trap questions. Always write sizeof() rather than assuming an int is 4 bytes.