Python generators
last modified November 17, 2026 Python 3.15
In this article we show how to work with generators in Python.
A generator is a function that produces a sequence of values
lazily, one value at a time. Any function that contains the
yield keyword is a generator function. Calling it
does not run its body; instead, it returns a generator object,
which is an iterator. Each time the next value is requested, the function
runs until it reaches a yield, hands the value to the caller,
and is suspended with all its local variables intact. The next request
resumes it right after that yield.
Because values are computed only when they are needed, generators use very little memory, can represent infinite sequences, and can be chained into data-processing pipelines. They are the simplest way to write an iterator in Python; the Python iterator article covers the underlying iterator protocol.
The examples were tested with Python 3.15.
Simple generator
The following example defines a generator that counts down from a given number.
def countdown(n):
while n > 0:
yield n
n -= 1
for i in countdown(5):
print(i)
The countdown function contains a yield
statement, which makes it a generator function.
while n > 0:
yield n
n -= 1
In each iteration of the while loop, the current value of
n is yielded to the caller. The function is then paused; when
the next value is requested, it continues with n -= 1. When
the loop ends and the function returns, the iteration is over.
for i in countdown(5):
print(i)
The for loop requests the values one by one and stops
automatically when the generator is exhausted.
$ python simple_gen.py 5 4 3 2 1
Generator objects
Calling a generator function returns a generator object. We can drive it
manually with the built-in next function.
import inspect
def colours():
yield 'red'
yield 'green'
yield 'blue'
gen = colours()
print(type(colours))
print(type(gen))
print(inspect.isgeneratorfunction(colours))
print(inspect.getgeneratorstate(gen))
print(next(gen))
print(inspect.getgeneratorstate(gen))
print(next(gen))
print(next(gen))
try:
next(gen)
except StopIteration:
print('StopIteration raised')
print(inspect.getgeneratorstate(gen))
print(next(gen, 'no more colours'))
The colours function is an ordinary function object, but the
value it returns is a generator. The
inspect.getgeneratorstate function shows in which state the
generator is.
print(next(gen))
The next function runs the generator until the next
yield and returns the yielded value. Before the first call
the generator is in the GEN_CREATED state; between calls it
is GEN_SUSPENDED.
try:
next(gen)
except StopIteration:
print('StopIteration raised')
When there are no more values, next raises the
StopIteration exception and the generator becomes
GEN_CLOSED. This is the signal a for loop uses
to end the iteration.
print(next(gen, 'no more colours'))
With a second argument, next returns that default value
instead of raising StopIteration.
$ python gen_object.py <class 'function'> <class 'generator'> True GEN_CREATED red GEN_SUSPENDED green blue StopIteration raised GEN_CLOSED no more colours
Lazy execution
The body of a generator does not run when the generator is created. It runs piece by piece, interleaved with the code that consumes it.
def numbers():
print(' start')
yield 1
print(' after first yield')
yield 2
print(' after second yield')
yield 3
print(' end')
gen = numbers()
print('generator created')
for n in gen:
print(f'got {n}')
The generator prints a message before and after each yield.
The output shows that nothing happens when numbers() is
called; the first message appears only when the for loop asks
for the first value. Control then alternates between the loop body and the
generator. The final end message is printed when the loop
requests a fourth value and the function runs to its end.
$ python lazy_exec.py generator created start got 1 after first yield got 2 after second yield got 3 end
Generator expressions
A generator expression is a compact way to create a generator. It has the same syntax as a list comprehension but uses parentheses instead of square brackets.
words = ['sky', 'forest', 'cup', 'rock', 'falcon', 'owl']
lengths = (len(word) for word in words)
print(lengths)
print(list(lengths))
total = sum(len(word) for word in words)
print(total)
longest = max((w for w in words if len(w) > 3), key=len)
print(longest)
print(any(w.startswith('f') for w in words))
print(', '.join(w.upper() for w in words if 'o' in w))
The lengths variable holds a generator object; no length has
been computed yet. Calling list on it consumes all the
values.
total = sum(len(word) for word in words)
When a generator expression is the only argument of a function call, the
extra parentheses can be omitted. Functions that consume iterables, such as
sum, max, any, all, or
str.join, work naturally with generator expressions. The
any function even stops consuming the generator as soon as it
finds the first true value. When there are other arguments, such as
key in max, the generator expression must have
its own parentheses.
$ python gen_expressions.py <generator object <genexpr> at 0x7b233502c580> [3, 6, 3, 4, 6, 3] 25 forest True FOREST, ROCK, FALCON, OWL
Memory usage
A list stores all of its elements, while a generator stores only its current state. The difference becomes significant for large sequences.
import sys
squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))
print(f'list: {sys.getsizeof(squares_list):>9,} bytes')
print(f'generator: {sys.getsizeof(squares_gen):>9,} bytes')
print(sum(squares_list))
print(sum(squares_gen))
We create one million squares, once as a list and once as a generator
expression. The sys.getsizeof function returns the size of the
object itself. The list holds a million references (and the integer objects
they point to come on top of that), while the generator has a small fixed
size regardless of how many values it will produce. Both give the same sum;
the generator, however, never holds more than one square in memory.
$ python memory_usage.py list: 8,448,728 bytes generator: 216 bytes 333332833333500000 333332833333500000
Infinite sequences
Since values are produced on demand, a generator can represent a sequence that never ends. The consumer decides how many values to take.
from itertools import islice
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
print(list(islice(fibonacci(), 15)))
for n in fibonacci():
if n > 1000:
print(f'first Fibonacci number above 1000: {n}')
break
The fibonacci generator yields Fibonacci numbers in an endless
while True loop. Calling list on it directly
would never finish; the itertools.islice function takes only
the first fifteen values. In the for loop, we use
break to stop the iteration once we find the value we are
looking for.
$ python infinite.py [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377] first Fibonacci number above 1000: 1597
Generators are single-use
A generator object can be iterated only once. After it is exhausted, it yields nothing more.
gen = (x * 10 for x in range(4))
print(list(gen))
print(list(gen))
gen = (x * 10 for x in range(4))
print(20 in gen)
print(list(gen))
class Squares:
def __init__(self, n):
self.n = n
def __iter__(self):
for i in range(1, self.n + 1):
yield i * i
squares = Squares(4)
print(list(squares))
print(list(squares))
print(sum(squares))
The second list(gen) call returns an empty list because the
generator has already been consumed. The in operator consumes
a generator too: it reads values until it finds 20, so only
30 is left afterwards.
def __iter__(self):
for i in range(1, self.n + 1):
yield i * i
If we need to iterate the same data several times, we create a class whose
__iter__ method is a generator. Each iteration calls
__iter__ again and gets a fresh generator object, so the
Squares object can be iterated any number of times.
$ python exhaustion.py [0, 10, 20, 30] [] True [30] [1, 4, 9, 16] [1, 4, 9, 16] 30
Returning a value
A generator may contain a return statement. It ends the
generator, and the returned value is attached to the
StopIteration exception.
def read_values(data):
count = 0
for value in data:
if value is None:
return count
count += 1
yield value
return count
gen = read_values([4, 8, None, 15])
while True:
try:
print(next(gen))
except StopIteration as e:
print(f'generator finished, returned {e.value}')
break
The read_values generator yields values until it encounters
None, then returns the number of values it produced. The
caller reads the return value from the value attribute of
the exception. A for loop silently discards this value; it
is mostly useful with yield from, shown in the next section.
$ python return_value.py 4 8 generator finished, returned 2
Delegating with yield from
The yield from expression delegates to another iterable or
generator. It yields all of its values and evaluates to the return value
of the delegated generator.
def flatten(items):
for item in items:
if isinstance(item, list):
yield from flatten(item)
else:
yield item
nested = [1, [2, 3, [4, 5]], [], [6, [7, [8]]], 9]
print(list(flatten(nested)))
def header():
yield '<table>'
def rows(data):
for name, age in data:
yield f' <tr><td>{name}</td><td>{age}</td></tr>'
return len(data)
def table(data):
yield from header()
n = yield from rows(data)
yield '</table>'
yield f'<!-- {n} rows -->'
users = [('Peter', 34), ('Lucy', 28), ('Robert', 41)]
for line in table(users):
print(line)
The recursive flatten generator turns arbitrarily nested
lists into a flat sequence. For each nested list, it delegates to a new
flatten generator with yield from.
def table(data):
yield from header()
n = yield from rows(data)
yield '</table>'
yield f'<!-- {n} rows -->'
The table generator is composed of smaller generators. The
rows generator returns the number of rows, and
yield from passes that value to the n variable.
Besides values, yield from also forwards values sent with
send and exceptions thrown with throw to the
inner generator.
$ python yield_from.py [1, 2, 3, 4, 5, 6, 7, 8, 9] <table> <tr><td>Peter</td><td>34</td></tr> <tr><td>Lucy</td><td>28</td></tr> <tr><td>Robert</td><td>41</td></tr> </table> <!-- 3 rows -->
Sending values into a generator
The yield keyword is also an expression. The
send method resumes the generator and makes the paused
yield expression evaluate to the sent value. This turns a
generator into a two-way channel.
def running_average():
total = 0
count = 0
average = None
while True:
value = yield average
total += value
count += 1
average = total / count
avg = running_average()
print(next(avg))
for value in [10, 20, 60, 30]:
print(f'sent {value}, average {avg.send(value)}')
The running_average generator keeps a running total and a
count in its local variables, which survive between calls.
value = yield average
This line first yields the current average to the caller. When the caller
calls send, the generator resumes and the sent value is
assigned to value. The new average is then yielded by the
next pass of the loop and becomes the return value of send.
avg = running_average() print(next(avg))
A freshly created generator has not reached any yield yet,
so it cannot receive a value. We must first prime it with
next (or the equivalent send(None)), which runs
it to the first yield. That yield produces
None because no value has been averaged yet.
$ python send_values.py None sent 10, average 10.0 sent 20, average 15.0 sent 60, average 30.0 sent 30, average 30.0
The throw and close methods
The throw method raises an exception inside the generator at
the point where it is paused. The close method raises
GeneratorExit there, which ends the generator.
def worker():
print('worker started')
try:
while True:
try:
task = yield
print(f'processing {task}')
except ValueError as e:
print(f'bad task: {e}')
finally:
print('worker cleaned up')
w = worker()
next(w)
w.send('task 1')
w.throw(ValueError('empty task'))
w.send('task 2')
w.close()
print('---')
w = worker()
next(w)
w.send('task 3')
del w
print('end of script')
The worker handles ValueError inside its loop, so a thrown
ValueError is reported and the worker keeps running. The
close call ends the loop, and the finally block
performs the cleanup.
del w
print('end of script')
A suspended generator is closed automatically when it is garbage collected.
In CPython, deleting the last reference closes it immediately, so the
cleanup message appears before end of script. Other Python
implementations may run the cleanup later, so code that depends on it
should call close explicitly or use a with
statement.
$ python close_throw.py worker started processing task 1 bad task: empty task processing task 2 worker cleaned up --- worker started processing task 3 worker cleaned up end of script
Generator pipelines
Generators can be connected so that the output of one is the input of the next. Each stage processes one item at a time, so even a huge input file is never loaded into memory as a whole. In this example, we analyse a web server access log.
192.168.0.12 GET /index.html 200 5120 192.168.0.15 GET /about.html 200 3277 192.168.0.12 GET /missing.png 404 0 10.0.0.7 POST /login 302 0 192.168.0.21 GET /index.html 200 5120 10.0.0.7 GET /dashboard 200 18450 192.168.0.15 GET /old-page 404 0 192.168.0.33 GET /images/logo.png 200 24576 10.0.0.9 GET /admin 403 512 192.168.0.12 GET /index.html 200 5120
Each line of the log contains the client address, the HTTP method, the URL, the status code, and the response size in bytes.
from collections import Counter
def read_lines(path):
with open(path) as f:
for line in f:
yield line.rstrip('\n')
def parse(lines):
for line in lines:
ip, method, url, status, size = line.split()
yield {'ip': ip, 'url': url, 'status': int(status), 'size': int(size)}
def successful(records):
return (r for r in records if r['status'] == 200)
lines = read_lines('logs/access.log')
records = parse(lines)
ok = successful(records)
hits = Counter()
total = 0
for r in ok:
hits[r['url']] += 1
total += r['size']
print(hits.most_common(3))
print(f'{total:,} bytes served')
The pipeline has three stages: read_lines reads the file
line by line, parse turns each line into a dictionary, and
successful keeps only the records with status 200. The last
stage is written as a generator expression returned from a function.
lines = read_lines('logs/access.log')
records = parse(lines)
ok = successful(records)
Building the pipeline does not read anything. The file is opened and read
only when the final for loop starts pulling records, and each
line travels through all the stages before the next one is read. The loop
counts the hits per URL with a Counter and sums the bytes
served.
$ python pipeline.py
[('/index.html', 3), ('/about.html', 1), ('/dashboard', 1)]
61,663 bytes served
Reading files in chunks
Text files can be iterated line by line, but binary files have no lines. A generator can split them into fixed-size chunks.
import hashlib
def read_chunks(path, size=4096):
with open(path, 'rb') as f:
while chunk := f.read(size):
yield chunk
sha = hashlib.sha256()
nchunks = 0
for chunk in read_chunks('logs/access.log', size=64):
sha.update(chunk)
nchunks += 1
print(f'{nchunks} chunks')
print(sha.hexdigest())
The read_chunks generator reads the file in blocks of
size bytes. The walrus operator assigns the block and tests
it in one step; read returns an empty bytes object at the end
of the file, which ends the loop. We feed the chunks into a SHA-256 hash,
so the file is hashed without ever being loaded completely. A tiny chunk
size is used here to show that the 354-byte log is split into several
pieces.
$ python chunks.py 6 chunks 7eee419bb13bc09a84fee9a68acead5496f6bad1397c6d24e7bc6b9ea30c014e
Generators and itertools
The itertools module provides building blocks that consume
and produce iterators lazily. They combine well with generators.
from itertools import batched, count, islice, takewhile, chain, accumulate
def primes():
found = []
for n in count(2):
if all(n % p for p in takewhile(lambda p: p * p <= n, found)):
found.append(n)
yield n
print(list(islice(primes(), 12)))
print(list(takewhile(lambda p: p < 30, primes())))
print(list(islice(primes(), 100, 105)))
for batch in batched(islice(primes(), 10), 4):
print(batch)
print(list(chain('ab', range(3), (x.upper() for x in 'cd'))))
print(list(accumulate(islice(primes(), 6))))
The primes generator produces an infinite sequence of prime
numbers. It iterates over count(2), an endless counter, and
checks each number against the primes found so far;
takewhile limits the check to primes not greater than the
square root of the number.
The islice function takes a slice of an iterator, here the
first twelve primes and then the primes at positions 100 to 104;
takewhile takes values while a condition holds. The
batched function, added in Python 3.12, groups values into
tuples of a given size. The chain function joins several
iterables into one and accumulate yields running totals.
$ python itertools_gen.py [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37] [2, 3, 5, 7, 11, 13, 17, 19, 23, 29] [547, 557, 563, 569, 571] (2, 3, 5, 7) (11, 13, 17, 19) (23, 29) ['a', 'b', 0, 1, 2, 'C', 'D'] [2, 5, 10, 17, 28, 41]
Generator-based context managers
The contextlib.contextmanager decorator turns a generator
with a single yield into a context manager for the
with statement.
import time
from contextlib import contextmanager
@contextmanager
def timer(label):
start = time.perf_counter()
try:
yield
finally:
elapsed = time.perf_counter() - start
print(f'{label}: {elapsed:.2f} s')
@contextmanager
def opened(path, mode='r'):
print(f'opening {path}')
f = open(path, mode)
try:
yield f
finally:
f.close()
print(f'closed {path}')
with timer('sleep'):
time.sleep(0.25)
with opened('logs/access.log') as f:
print(f.readline().rstrip())
The code before yield runs when the with block
is entered, and the code after it runs when the block is left. The
yielded value is bound to the variable after as, as in the
opened function. Placing the yield in a
try/finally statement guarantees that the
cleanup runs even if the with block raises an exception.
$ python timer_ctx.py sleep: 0.25 s opening logs/access.log 192.168.0.12 GET /index.html 200 5120 closed logs/access.log
StopIteration inside a generator
Since Python 3.7 (PEP 479), a StopIteration exception that
escapes from a generator body is converted into a
RuntimeError. This prevents bugs where a generator would
silently stop early.
def first_words(lines):
it = iter(lines)
while True:
line = next(it)
yield line.split()[0]
try:
print(list(first_words(['old falcon', 'blue sky'])))
except RuntimeError as e:
print(f'RuntimeError: {e}')
print(f'caused by {type(e.__cause__).__name__}')
def first_words_fixed(lines):
for line in lines:
yield line.split()[0]
print(list(first_words_fixed(['old falcon', 'blue sky'])))
The first_words generator calls next on an
iterator in an endless loop. When the iterator is exhausted,
next raises StopIteration, which is turned into
a RuntimeError; the original exception is available as
__cause__. The fixed version uses a for loop,
which handles the end of the iteration itself. If we have to call
next manually, we catch StopIteration and
return.
$ python stop_iteration.py RuntimeError: generator raised StopIteration caused by StopIteration ['old', 'blue']
When generator expressions are evaluated
A generator expression evaluates its first (outermost) iterable immediately, when the expression is created. Everything else is evaluated lazily, when the values are consumed.
numbers = [1, 2, 3]
factor = 10
gen = (n * factor for n in numbers)
numbers = [7, 8, 9]
factor = 100
print(list(gen))
try:
gen = (n for n in 42)
except TypeError as e:
print(f'TypeError at creation: {e}')
After creating the generator, we rebind both numbers and
factor. The generator still iterates over the original list
[1, 2, 3], because that list was evaluated at creation, but it
multiplies by the new value of factor, which is looked up
only when each value is computed. For the same reason, a non-iterable
object raises TypeError right when the generator expression
is created, not later when it is consumed.
$ python late_binding.py [100, 200, 300] TypeError at creation: 'int' object is not iterable
Type hints for generators
Generators are annotated with the Iterator or
Generator types from collections.abc.
from collections.abc import Generator, Iterator
def evens(limit: int) -> Iterator[int]:
yield from range(0, limit, 2)
def echo() -> Generator[str, str, int]:
count = 0
received = yield 'ready'
while received != 'quit':
count += 1
received = yield received.upper()
return count
def names() -> Generator[str]:
yield 'Jan'
yield 'Mia'
print(list(evens(10)))
gen = echo()
print(next(gen))
print(gen.send('hello'))
print(gen.send('world'))
try:
gen.send('quit')
except StopIteration as e:
print(f'echoed {e.value} messages')
print(list(names()))
When a generator only yields values, Iterator[int] is
enough. The full form Generator[YieldType, SendType,
ReturnType] also describes the values accepted by
send and the return value. The echo generator
yields strings, receives strings, and returns an integer. Since Python
3.13, the send and return types default to None, so
Generator[str] can be used for a generator that only yields
strings.
$ python type_hints.py [0, 2, 4, 6, 8] ready HELLO WORLD echoed 2 messages ['Jan', 'Mia']
Async generators
An async def function that contains yield is an
asynchronous generator. It can await between
yields and is consumed with async for.
import asyncio
import time
async def ticker(n, delay):
for i in range(n):
await asyncio.sleep(delay)
yield i
async def collect(name, n, delay):
async for i in ticker(n, delay):
print(f'{name} {i}')
return name
async def main():
start = time.perf_counter()
done = await asyncio.gather(collect('fast', 3, 0.1), collect('slow', 2, 0.25))
print(done)
print(f'{time.perf_counter() - start:.1f} s')
squares = [i * i async for i in ticker(5, 0)]
print(squares)
asyncio.run(main())
The ticker async generator waits for the given delay before
each value. Two consumers run concurrently with
asyncio.gather, so their output interleaves and the whole
run takes about half a second, the time of the slower one. Async
generators also work in asynchronous comprehensions, as in the
squares list. The
Python async/await article covers
asynchronous programming in more detail.
$ python async_gen.py fast 0 fast 1 slow 0 fast 2 slow 1 ['fast', 'slow'] 0.5 s [0, 1, 4, 9, 16]
Source
Python tutorial - Generators
Python reference - Yield expressions
Functional Programming HOWTO - Generators
PEP 479 - Change StopIteration handling inside generators
In this article we have worked with generators in Python. We have covered
yield, generator objects, generator expressions,
yield from, the send, throw, and
close methods, pipelines, context managers, and async
generators. See also the Python iterator,
Python itertools,
Python list comprehensions,
and Python closures articles.
Author
List all Python tutorials.