ZetCode

Python lexical structure

last modified November 16, 2026 Python 3.15

In this article we cover the lexical structure of Python.

Computer languages, like human languages, have a lexical structure. The lexical structure of a programming language is the set of basic rules that define how the source code is split into tokens. A token is an atomic element of the code: an identifier, a keyword, a literal, an operator, or a delimiter. In Python, the line structure and the indentation also produce tokens. Comments and whitespace between tokens are discarded. Python source files are read as UTF-8 by default.

Tokens

The tokenize module splits Python source code into tokens the same way the interpreter does.

tokens.py
import io
import tokenize

src = '''\
if price > 100:  # discount
    total = price * 0.9
'''

for tok in tokenize.generate_tokens(io.StringIO(src).readline):
    print(f'{tokenize.tok_name[tok.type]:10} {tok.string!r}')

We tokenize a short if statement. Keywords and identifiers are both NAME tokens, operators and delimiters are OP tokens. The NEWLINE token ends a logical line; the INDENT and DEDENT tokens mark the start and the end of an indented block. The same output can be produced on the command line with python -m tokenize file.py.

$ python tokens.py
NAME       'if'
NAME       'price'
OP         '>'
NUMBER     '100'
OP         ':'
COMMENT    '# discount'
NEWLINE    '\n'
INDENT     '    '
NAME       'total'
OP         '='
NAME       'price'
OP         '*'
NUMBER     '0.9'
NEWLINE    '\n'
DEDENT     ''
ENDMARKER  ''

Comments

A comment starts with the # character and extends to the end of the physical line. Comments are ignored by the interpreter; they explain the code to human readers.

comments.py
# Author: Jan Bodnar
# Prints the area of a circle

import math

radius = 3  # in centimetres
print('Area:', math.pi * radius ** 2)
print('# is not a comment inside a string')


def area(r):
    """Return the area of a circle with radius r."""
    return math.pi * r ** 2


print(area.__doc__)

The program has full-line comments and an inline comment. A # inside a string literal does not start a comment.

def area(r):
    """Return the area of a circle with radius r."""

A string literal at the start of a function, class, or module is a docstring. Unlike a comment, it is kept at run time in the __doc__ attribute.

$ python comments.py
Area: 28.274333882308138
# is not a comment inside a string
Return the area of a circle with radius r.

Logical and physical lines

A physical line is a line of the source file. A logical line is a complete statement, which may span several physical lines. Expressions in parentheses, square brackets, or curly braces can be split over lines freely; elsewhere, a backslash at the end of a line joins it with the next one. A semicolon separates several statements on one line.

lines.py
total = (1 + 2 + 3 +
         4 + 5 + 6)

colors = [
    'red',
    'green',
    'blue',
]

message = 'The total is ' \
          + str(total)

x = 1; y = 2; print(x + y)

text = ('implicit '
        'concatenation')

print(total, colors, message, text, sep='\n')

Implicit line joining inside brackets is the preferred way of splitting long lines; a trailing comma after the last item is allowed. Adjacent string literals are concatenated at compile time. Putting several statements on one line with semicolons is legal but discouraged; the first line of the output comes from print(x + y).

$ python lines.py
3
21
['red', 'green', 'blue']
The total is 21
implicit concatenation

Indentation

Python uses indentation to delimit blocks of code, where other languages use curly brackets or keywords such as begin and end. A statement that ends with a colon, such as if, for, or def, must be followed by an indented block; returning to the previous indentation level ends the block. The PEP 8 style guide recommends four spaces per level.

indentation.py
sources = {
    'ok': 'if True:\n    x = 1\n    y = 2\n',
    'unexpected': 'x = 1\n    y = 2\n',
    'expected': 'if True:\nx = 1\n',
    'unindent': 'if True:\n        x = 1\n    y = 2\n',
    'tabs': 'if True:\n\tx = 1\n        y = 2\n',
}

for name, src in sources.items():
    try:
        compile(src, name, 'exec')
        print(f'{name:10} ok')
    except SyntaxError as e:
        print(f'{name:10} {type(e).__name__}: {e.msg}')

We compile several code snippets and print the errors. An indented line outside of a block, a missing block, and a dedent to a level that was never used raise IndentationError. Mixing tabs and spaces in a way that makes the meaning depend on the tab width raises TabError.

$ python indentation.py
ok         ok
unexpected IndentationError: unexpected indent
expected   IndentationError: expected an indented block after 'if' statement on line 1
unindent   IndentationError: unindent does not match any outer indentation level
tabs       TabError: inconsistent use of tabs and spaces in indentation

Identifiers

Identifiers are the names of variables, functions, classes, modules, and other objects. An identifier consists of letters, digits, and underscores, and must not start with a digit. Keywords cannot be used as identifiers. Identifiers are case sensitive.

identifiers.py
import keyword

names = ['value', 'value2', 'company_name', '_private', '__dunder__',
         '12val', 'exx$', 'first-name', 'class', 'cena', 'größe', 'π']

for name in names:
    ok = name.isidentifier() and not keyword.iskeyword(name)
    print(f'{name:14} {ok}')

number = 10
Number = 11
NUMBER = 12
print(number, Number, NUMBER)

The str.isidentifier method checks the lexical rules; we also reject keywords with keyword.iskeyword. The $ and - characters are not allowed, while letters from other alphabets are.

number = 10
Number = 11
NUMBER = 12

The three names are different identifiers. While this is possible, it is not considered good practice.

$ python identifiers.py
value          True
value2         True
company_name   True
_private       True
__dunder__     True
12val          False
exx$           False
first-name     False
class          False
cena           True
größe          True
π              True
10 11 12

Unicode identifiers

Python identifiers can contain Unicode letters. The parser normalizes them to the NFKC form, so identifiers that look different can be the same name.

unicode_names.py
import unicodedata

π = 3.14159
größe = 180

print(π, größe)

file = 'ligature'
print(file)

print(unicodedata.normalize('NFKC', 'file'))
print('ℌ'.isidentifier(), unicodedata.normalize('NFKC', 'ℌ'))

The file identifier is written with the single ligature character U+FB01. After NFKC normalization, it becomes file, so printing file prints the value assigned to file. In the same way, the Fraktur letter ℌ becomes a plain H.

$ python unicode_names.py
3.14159 180
ligature
file
True H

Keywords

A keyword is a reserved word with a special meaning in the language. Keywords cannot be used as identifiers. The keyword module lists them.

keywords.py
import keyword

print(len(keyword.kwlist))

for i in range(0, len(keyword.kwlist), 7):
    print(' '.join(f'{kw:9}' for kw in keyword.kwlist[i:i + 7]).rstrip())

print(keyword.softkwlist)

try:
    compile('class = 1', '<test>', 'exec')
except SyntaxError as e:
    print(e.msg)

Python has 35 keywords. The softkwlist attribute lists the soft keywords, described in the next section. Using a keyword as a variable name is a syntax error.

$ python keywords.py
35
False     None      True      and       as        assert    async
await     break     class     continue  def       del       elif
else      except    finally   for       from      global    if
import    in        is        lambda    nonlocal  not       or
pass      raise     return    try       while     with      yield
['_', 'case', 'lazy', 'match', 'type']
invalid syntax

Soft keywords

Soft keywords are reserved only in specific contexts and remain valid identifiers elsewhere. The match, case, and _ soft keywords are used in pattern matching (Python 3.10), type in type alias statements (Python 3.12), and lazy in lazy imports (Python 3.15, PEP 810).

soft_keywords.py
import sys

match = 'match is a variable'
case = 1
type = 'type is shadowed here'
lazy = True

print(match, case, type, lazy)

command = 'go north'

match command.split():
    case ['go', direction]:
        print('moving', direction)
    case _:
        print('unknown command')

lazy import json

print('json' in sys.modules)
print(json.dumps({'lazy': lazy}))
print('json' in sys.modules)

We use the soft keywords as ordinary variable names, and then as keywords in a match statement and a lazy import.

lazy import json

The lazy import statement binds the name to a proxy without loading the module. The json module is loaded on its first use, which is why it is missing from sys.modules before the dumps call. Python 3.14 and older report a syntax error for this line.

$ python soft_keywords.py
match is a variable 1 type is shadowed here True
moving north
False
{"lazy": true}
True

Literals

A literal is a notation for a constant value in the source code. Python has numeric literals (integers, floating point numbers, and imaginary numbers), string and bytes literals, and the True, False, None, and ... constants, which are keywords or tokens of their own.

literals.py
print(1_000_000, 0b1010, 0o17, 0xff)
print(3.14, 1e-3, 2.5E6, .5, 7.)
print(3 + 4j, 2j * 2j)
print('single', "double", '''triple''')
print(r'raw \n string', 'escape \t tab')
print(b'bytes', rb'\x00')
print(f'{6 * 7}', type(t'{6 * 7}').__name__)
print(True, False, None, ...)
print([1, 2], (1,), {1, 2}, {'a': 1})

Integer literals can be written in decimal, binary, octal, and hexadecimal notation; underscores group the digits. A j suffix creates an imaginary number. String literals use single, double, or triple quotes, and prefixes change their meaning: r for raw strings, b for bytes, f for f-strings, and t for template strings (Python 3.14).

print([1, 2], (1,), {1, 2}, {'a': 1})

Lists, tuples, sets, and dictionaries are not literals in the strict sense; they are displays, expressions that build a new object each time they are evaluated.

$ python literals.py
1000000 10 15 255
3.14 0.001 2500000.0 0.5 7.0
(3+4j) (-4+0j)
single double triple
raw \n string escape 	 tab
b'bytes' b'\\x00'
42 Template
True False None Ellipsis
[1, 2] (1,) {1, 2} {'a': 1}

Operators and delimiters

Operators are symbols that perform an operation on values, such as + or **. Delimiters separate parts of the code: brackets, commas, colons, the assignment operator, and the augmented assignment operators. The token module contains the complete list of these symbols.

operators.py
import token

ops = sorted(token.EXACT_TOKEN_TYPES, key=lambda s: (len(s), s))

for n in (1, 2, 3):
    print(' '.join(op for op in ops if len(op) == n))

print(len(ops))

We print the 48 symbols grouped by length. The @ operator is used for matrix multiplication and decorators, -> introduces a return annotation, and := is the assignment expression (walrus) operator. The ! symbol appears only in f-string conversions such as {x!r}. The word operators and, or, not, in, and is are keywords, so they are not in the list.

$ python operators.py
! % & ( ) * + , - . / : ; < = > @ [ ] ^ { | } ~
!= %= &= ** *= += -= -> // /= := << <= == >= >> @= ^= |=
**= ... //= <<= >>=
48

Reserved classes of identifiers

Some identifier patterns have a special meaning by convention or by the language rules. A leading underscore marks a name as internal, names with two leading and two trailing underscores are reserved for Python, and names with two leading underscores in a class body are mangled.

reserved_names.py
class Account:

    def __init__(self):
        self._balance = 100
        self.__pin = 1234


acc = Account()

print([a for a in vars(acc)])
print(acc._balance)
print(acc._Account__pin)

try:
    acc.__pin
except AttributeError as e:
    print(e)

for _ in range(2):
    print('loop variable not needed')

print(__name__, __file__.rsplit('/', 1)[-1])

The __pin attribute is stored as _Account__pin, so it cannot be accessed as acc.__pin from outside the class. The single underscore is used for values that are not needed, such as an unused loop variable. The __name__ and __file__ names are set by Python for every module.

$ python reserved_names.py
['_balance', '_Account__pin']
100
1234
'Account' object has no attribute '__pin'
loop variable not needed
loop variable not needed
__main__ reserved_names.py

Source

Lexical analysis – Python language reference

tokenize – Tokenizer for Python source

PEP 810 – Explicit lazy imports

In this article we have covered the lexical structure of Python. See also Python pattern matching and Python operator module.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Python tutorials.