Python lexical structure
last modified November 16, 2026 Python 3.15
In this article we cover the lexical structure of Python.
Computer languages, like human languages, have a lexical structure. The lexical structure of a programming language is the set of basic rules that define how the source code is split into tokens. A token is an atomic element of the code: an identifier, a keyword, a literal, an operator, or a delimiter. In Python, the line structure and the indentation also produce tokens. Comments and whitespace between tokens are discarded. Python source files are read as UTF-8 by default.
Tokens
The tokenize module splits Python source code into tokens the
same way the interpreter does.
import io
import tokenize
src = '''\
if price > 100: # discount
total = price * 0.9
'''
for tok in tokenize.generate_tokens(io.StringIO(src).readline):
print(f'{tokenize.tok_name[tok.type]:10} {tok.string!r}')
We tokenize a short if statement. Keywords and identifiers are
both NAME tokens, operators and delimiters are OP
tokens. The NEWLINE token ends a logical line; the
INDENT and DEDENT tokens mark the start and the end of
an indented block. The same output can be produced on the command line with
python -m tokenize file.py.
$ python tokens.py NAME 'if' NAME 'price' OP '>' NUMBER '100' OP ':' COMMENT '# discount' NEWLINE '\n' INDENT ' ' NAME 'total' OP '=' NAME 'price' OP '*' NUMBER '0.9' NEWLINE '\n' DEDENT '' ENDMARKER ''
Comments
A comment starts with the # character and extends to
the end of the physical line. Comments are ignored by the interpreter; they
explain the code to human readers.
# Author: Jan Bodnar
# Prints the area of a circle
import math
radius = 3 # in centimetres
print('Area:', math.pi * radius ** 2)
print('# is not a comment inside a string')
def area(r):
"""Return the area of a circle with radius r."""
return math.pi * r ** 2
print(area.__doc__)
The program has full-line comments and an inline comment. A #
inside a string literal does not start a comment.
def area(r):
"""Return the area of a circle with radius r."""
A string literal at the start of a function, class, or module is a
docstring. Unlike a comment, it is kept at run time in the
__doc__ attribute.
$ python comments.py Area: 28.274333882308138 # is not a comment inside a string Return the area of a circle with radius r.
Logical and physical lines
A physical line is a line of the source file. A logical line is a complete statement, which may span several physical lines. Expressions in parentheses, square brackets, or curly braces can be split over lines freely; elsewhere, a backslash at the end of a line joins it with the next one. A semicolon separates several statements on one line.
total = (1 + 2 + 3 +
4 + 5 + 6)
colors = [
'red',
'green',
'blue',
]
message = 'The total is ' \
+ str(total)
x = 1; y = 2; print(x + y)
text = ('implicit '
'concatenation')
print(total, colors, message, text, sep='\n')
Implicit line joining inside brackets is the preferred way of splitting long
lines; a trailing comma after the last item is allowed. Adjacent string
literals are concatenated at compile time. Putting several statements on one
line with semicolons is legal but discouraged; the first line of the output
comes from print(x + y).
$ python lines.py 3 21 ['red', 'green', 'blue'] The total is 21 implicit concatenation
Indentation
Python uses indentation to delimit blocks of code, where other languages use
curly brackets or keywords such as begin and end.
A statement that ends with a colon, such as if,
for, or def, must be followed by an indented block;
returning to the previous indentation level ends the block. The PEP 8 style
guide recommends four spaces per level.
sources = {
'ok': 'if True:\n x = 1\n y = 2\n',
'unexpected': 'x = 1\n y = 2\n',
'expected': 'if True:\nx = 1\n',
'unindent': 'if True:\n x = 1\n y = 2\n',
'tabs': 'if True:\n\tx = 1\n y = 2\n',
}
for name, src in sources.items():
try:
compile(src, name, 'exec')
print(f'{name:10} ok')
except SyntaxError as e:
print(f'{name:10} {type(e).__name__}: {e.msg}')
We compile several code snippets and print the errors. An indented line
outside of a block, a missing block, and a dedent to a level that was never
used raise IndentationError. Mixing tabs and spaces in a way that
makes the meaning depend on the tab width raises TabError.
$ python indentation.py ok ok unexpected IndentationError: unexpected indent expected IndentationError: expected an indented block after 'if' statement on line 1 unindent IndentationError: unindent does not match any outer indentation level tabs TabError: inconsistent use of tabs and spaces in indentation
Identifiers
Identifiers are the names of variables, functions, classes, modules, and other objects. An identifier consists of letters, digits, and underscores, and must not start with a digit. Keywords cannot be used as identifiers. Identifiers are case sensitive.
import keyword
names = ['value', 'value2', 'company_name', '_private', '__dunder__',
'12val', 'exx$', 'first-name', 'class', 'cena', 'größe', 'π']
for name in names:
ok = name.isidentifier() and not keyword.iskeyword(name)
print(f'{name:14} {ok}')
number = 10
Number = 11
NUMBER = 12
print(number, Number, NUMBER)
The str.isidentifier method checks the lexical rules; we also
reject keywords with keyword.iskeyword. The $ and
- characters are not allowed, while letters from other alphabets
are.
number = 10 Number = 11 NUMBER = 12
The three names are different identifiers. While this is possible, it is not considered good practice.
$ python identifiers.py value True value2 True company_name True _private True __dunder__ True 12val False exx$ False first-name False class False cena True größe True π True 10 11 12
Unicode identifiers
Python identifiers can contain Unicode letters. The parser normalizes them to the NFKC form, so identifiers that look different can be the same name.
import unicodedata
π = 3.14159
größe = 180
print(π, größe)
file = 'ligature'
print(file)
print(unicodedata.normalize('NFKC', 'file'))
print('ℌ'.isidentifier(), unicodedata.normalize('NFKC', 'ℌ'))
The file identifier is written with the single ligature character
U+FB01. After NFKC normalization, it becomes file, so printing
file prints the value assigned to file. In the same
way, the Fraktur letter ℌ becomes a plain H.
$ python unicode_names.py 3.14159 180 ligature file True H
Keywords
A keyword is a reserved word with a special meaning in the
language. Keywords cannot be used as identifiers. The keyword
module lists them.
import keyword
print(len(keyword.kwlist))
for i in range(0, len(keyword.kwlist), 7):
print(' '.join(f'{kw:9}' for kw in keyword.kwlist[i:i + 7]).rstrip())
print(keyword.softkwlist)
try:
compile('class = 1', '<test>', 'exec')
except SyntaxError as e:
print(e.msg)
Python has 35 keywords. The softkwlist attribute lists the soft
keywords, described in the next section. Using a keyword as a variable name
is a syntax error.
$ python keywords.py 35 False None True and as assert async await break class continue def del elif else except finally for from global if import in is lambda nonlocal not or pass raise return try while with yield ['_', 'case', 'lazy', 'match', 'type'] invalid syntax
Soft keywords
Soft keywords are reserved only in specific contexts and remain
valid identifiers elsewhere. The match, case, and
_ soft keywords are used in pattern matching (Python 3.10),
type in type alias statements (Python 3.12), and
lazy in lazy imports (Python 3.15, PEP 810).
import sys
match = 'match is a variable'
case = 1
type = 'type is shadowed here'
lazy = True
print(match, case, type, lazy)
command = 'go north'
match command.split():
case ['go', direction]:
print('moving', direction)
case _:
print('unknown command')
lazy import json
print('json' in sys.modules)
print(json.dumps({'lazy': lazy}))
print('json' in sys.modules)
We use the soft keywords as ordinary variable names, and then as keywords in
a match statement and a lazy import.
lazy import json
The lazy import statement binds the name to a proxy without
loading the module. The json module is loaded on its first use,
which is why it is missing from sys.modules before the
dumps call. Python 3.14 and older report a syntax error for this
line.
$ python soft_keywords.py
match is a variable 1 type is shadowed here True
moving north
False
{"lazy": true}
True
Literals
A literal is a notation for a constant value in the source code.
Python has numeric literals (integers, floating point numbers, and imaginary
numbers), string and bytes literals, and the True,
False, None, and ... constants, which
are keywords or tokens of their own.
print(1_000_000, 0b1010, 0o17, 0xff)
print(3.14, 1e-3, 2.5E6, .5, 7.)
print(3 + 4j, 2j * 2j)
print('single', "double", '''triple''')
print(r'raw \n string', 'escape \t tab')
print(b'bytes', rb'\x00')
print(f'{6 * 7}', type(t'{6 * 7}').__name__)
print(True, False, None, ...)
print([1, 2], (1,), {1, 2}, {'a': 1})
Integer literals can be written in decimal, binary, octal, and hexadecimal
notation; underscores group the digits. A j suffix creates an
imaginary number. String literals use single, double, or triple quotes, and
prefixes change their meaning: r for raw strings,
b for bytes, f for f-strings, and t for
template strings (Python 3.14).
print([1, 2], (1,), {1, 2}, {'a': 1})
Lists, tuples, sets, and dictionaries are not literals in the strict sense; they are displays, expressions that build a new object each time they are evaluated.
$ python literals.py
1000000 10 15 255
3.14 0.001 2500000.0 0.5 7.0
(3+4j) (-4+0j)
single double triple
raw \n string escape tab
b'bytes' b'\\x00'
42 Template
True False None Ellipsis
[1, 2] (1,) {1, 2} {'a': 1}
Operators and delimiters
Operators are symbols that perform an operation on values, such as
+ or **. Delimiters separate parts of the
code: brackets, commas, colons, the assignment operator, and the augmented
assignment operators. The token module contains the complete
list of these symbols.
import token
ops = sorted(token.EXACT_TOKEN_TYPES, key=lambda s: (len(s), s))
for n in (1, 2, 3):
print(' '.join(op for op in ops if len(op) == n))
print(len(ops))
We print the 48 symbols grouped by length. The @ operator is
used for matrix multiplication and decorators, -> introduces a
return annotation, and := is the assignment expression (walrus)
operator. The ! symbol appears only in f-string conversions such
as {x!r}. The word operators and, or,
not, in, and is are keywords, so they
are not in the list.
$ python operators.py
! % & ( ) * + , - . / : ; < = > @ [ ] ^ { | } ~
!= %= &= ** *= += -= -> // /= := << <= == >= >> @= ^= |=
**= ... //= <<= >>=
48
Reserved classes of identifiers
Some identifier patterns have a special meaning by convention or by the language rules. A leading underscore marks a name as internal, names with two leading and two trailing underscores are reserved for Python, and names with two leading underscores in a class body are mangled.
class Account:
def __init__(self):
self._balance = 100
self.__pin = 1234
acc = Account()
print([a for a in vars(acc)])
print(acc._balance)
print(acc._Account__pin)
try:
acc.__pin
except AttributeError as e:
print(e)
for _ in range(2):
print('loop variable not needed')
print(__name__, __file__.rsplit('/', 1)[-1])
The __pin attribute is stored as _Account__pin, so
it cannot be accessed as acc.__pin from outside the class. The
single underscore is used for values that are not needed, such as an unused
loop variable. The __name__ and __file__ names are
set by Python for every module.
$ python reserved_names.py ['_balance', '_Account__pin'] 100 1234 'Account' object has no attribute '__pin' loop variable not needed loop variable not needed __main__ reserved_names.py
Source
Lexical analysis – Python language reference
tokenize – Tokenizer for Python source
PEP 810 – Explicit lazy imports
In this article we have covered the lexical structure of Python. See also Python pattern matching and Python operator module.
Author
List all Python tutorials.