ZetCode

Python generators

last modified November 17, 2026 Python 3.15

In this article we show how to work with generators in Python.

A generator is a function that produces a sequence of values lazily, one value at a time. Any function that contains the yield keyword is a generator function. Calling it does not run its body; instead, it returns a generator object, which is an iterator. Each time the next value is requested, the function runs until it reaches a yield, hands the value to the caller, and is suspended with all its local variables intact. The next request resumes it right after that yield.

Because values are computed only when they are needed, generators use very little memory, can represent infinite sequences, and can be chained into data-processing pipelines. They are the simplest way to write an iterator in Python; the Python iterator article covers the underlying iterator protocol.

The examples were tested with Python 3.15.

Simple generator

The following example defines a generator that counts down from a given number.

simple_gen.py
def countdown(n):
    while n > 0:
        yield n
        n -= 1


for i in countdown(5):
    print(i)

The countdown function contains a yield statement, which makes it a generator function.

while n > 0:
    yield n
    n -= 1

In each iteration of the while loop, the current value of n is yielded to the caller. The function is then paused; when the next value is requested, it continues with n -= 1. When the loop ends and the function returns, the iteration is over.

for i in countdown(5):
    print(i)

The for loop requests the values one by one and stops automatically when the generator is exhausted.

$ python simple_gen.py
5
4
3
2
1

Generator objects

Calling a generator function returns a generator object. We can drive it manually with the built-in next function.

gen_object.py
import inspect


def colours():
    yield 'red'
    yield 'green'
    yield 'blue'


gen = colours()

print(type(colours))
print(type(gen))
print(inspect.isgeneratorfunction(colours))
print(inspect.getgeneratorstate(gen))

print(next(gen))
print(inspect.getgeneratorstate(gen))
print(next(gen))
print(next(gen))

try:
    next(gen)
except StopIteration:
    print('StopIteration raised')

print(inspect.getgeneratorstate(gen))
print(next(gen, 'no more colours'))

The colours function is an ordinary function object, but the value it returns is a generator. The inspect.getgeneratorstate function shows in which state the generator is.

print(next(gen))

The next function runs the generator until the next yield and returns the yielded value. Before the first call the generator is in the GEN_CREATED state; between calls it is GEN_SUSPENDED.

try:
    next(gen)
except StopIteration:
    print('StopIteration raised')

When there are no more values, next raises the StopIteration exception and the generator becomes GEN_CLOSED. This is the signal a for loop uses to end the iteration.

print(next(gen, 'no more colours'))

With a second argument, next returns that default value instead of raising StopIteration.

$ python gen_object.py
<class 'function'>
<class 'generator'>
True
GEN_CREATED
red
GEN_SUSPENDED
green
blue
StopIteration raised
GEN_CLOSED
no more colours

Lazy execution

The body of a generator does not run when the generator is created. It runs piece by piece, interleaved with the code that consumes it.

lazy_exec.py
def numbers():
    print('  start')
    yield 1
    print('  after first yield')
    yield 2
    print('  after second yield')
    yield 3
    print('  end')


gen = numbers()
print('generator created')

for n in gen:
    print(f'got {n}')

The generator prints a message before and after each yield. The output shows that nothing happens when numbers() is called; the first message appears only when the for loop asks for the first value. Control then alternates between the loop body and the generator. The final end message is printed when the loop requests a fourth value and the function runs to its end.

$ python lazy_exec.py
generator created
  start
got 1
  after first yield
got 2
  after second yield
got 3
  end

Generator expressions

A generator expression is a compact way to create a generator. It has the same syntax as a list comprehension but uses parentheses instead of square brackets.

gen_expressions.py
words = ['sky', 'forest', 'cup', 'rock', 'falcon', 'owl']

lengths = (len(word) for word in words)
print(lengths)
print(list(lengths))

total = sum(len(word) for word in words)
print(total)

longest = max((w for w in words if len(w) > 3), key=len)
print(longest)

print(any(w.startswith('f') for w in words))
print(', '.join(w.upper() for w in words if 'o' in w))

The lengths variable holds a generator object; no length has been computed yet. Calling list on it consumes all the values.

total = sum(len(word) for word in words)

When a generator expression is the only argument of a function call, the extra parentheses can be omitted. Functions that consume iterables, such as sum, max, any, all, or str.join, work naturally with generator expressions. The any function even stops consuming the generator as soon as it finds the first true value. When there are other arguments, such as key in max, the generator expression must have its own parentheses.

$ python gen_expressions.py
<generator object <genexpr> at 0x7b233502c580>
[3, 6, 3, 4, 6, 3]
25
forest
True
FOREST, ROCK, FALCON, OWL

Memory usage

A list stores all of its elements, while a generator stores only its current state. The difference becomes significant for large sequences.

memory_usage.py
import sys

squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))

print(f'list:      {sys.getsizeof(squares_list):>9,} bytes')
print(f'generator: {sys.getsizeof(squares_gen):>9,} bytes')

print(sum(squares_list))
print(sum(squares_gen))

We create one million squares, once as a list and once as a generator expression. The sys.getsizeof function returns the size of the object itself. The list holds a million references (and the integer objects they point to come on top of that), while the generator has a small fixed size regardless of how many values it will produce. Both give the same sum; the generator, however, never holds more than one square in memory.

$ python memory_usage.py
list:      8,448,728 bytes
generator:       216 bytes
333332833333500000
333332833333500000

Infinite sequences

Since values are produced on demand, a generator can represent a sequence that never ends. The consumer decides how many values to take.

infinite.py
from itertools import islice


def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


print(list(islice(fibonacci(), 15)))

for n in fibonacci():
    if n > 1000:
        print(f'first Fibonacci number above 1000: {n}')
        break

The fibonacci generator yields Fibonacci numbers in an endless while True loop. Calling list on it directly would never finish; the itertools.islice function takes only the first fifteen values. In the for loop, we use break to stop the iteration once we find the value we are looking for.

$ python infinite.py
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]
first Fibonacci number above 1000: 1597

Generators are single-use

A generator object can be iterated only once. After it is exhausted, it yields nothing more.

exhaustion.py
gen = (x * 10 for x in range(4))

print(list(gen))
print(list(gen))

gen = (x * 10 for x in range(4))
print(20 in gen)
print(list(gen))


class Squares:

    def __init__(self, n):
        self.n = n

    def __iter__(self):
        for i in range(1, self.n + 1):
            yield i * i


squares = Squares(4)
print(list(squares))
print(list(squares))
print(sum(squares))

The second list(gen) call returns an empty list because the generator has already been consumed. The in operator consumes a generator too: it reads values until it finds 20, so only 30 is left afterwards.

def __iter__(self):
    for i in range(1, self.n + 1):
        yield i * i

If we need to iterate the same data several times, we create a class whose __iter__ method is a generator. Each iteration calls __iter__ again and gets a fresh generator object, so the Squares object can be iterated any number of times.

$ python exhaustion.py
[0, 10, 20, 30]
[]
True
[30]
[1, 4, 9, 16]
[1, 4, 9, 16]
30

Returning a value

A generator may contain a return statement. It ends the generator, and the returned value is attached to the StopIteration exception.

return_value.py
def read_values(data):
    count = 0
    for value in data:
        if value is None:
            return count
        count += 1
        yield value
    return count


gen = read_values([4, 8, None, 15])

while True:
    try:
        print(next(gen))
    except StopIteration as e:
        print(f'generator finished, returned {e.value}')
        break

The read_values generator yields values until it encounters None, then returns the number of values it produced. The caller reads the return value from the value attribute of the exception. A for loop silently discards this value; it is mostly useful with yield from, shown in the next section.

$ python return_value.py
4
8
generator finished, returned 2

Delegating with yield from

The yield from expression delegates to another iterable or generator. It yields all of its values and evaluates to the return value of the delegated generator.

yield_from.py
def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item


nested = [1, [2, 3, [4, 5]], [], [6, [7, [8]]], 9]
print(list(flatten(nested)))


def header():
    yield '<table>'


def rows(data):
    for name, age in data:
        yield f'  <tr><td>{name}</td><td>{age}</td></tr>'
    return len(data)


def table(data):
    yield from header()
    n = yield from rows(data)
    yield '</table>'
    yield f'<!-- {n} rows -->'


users = [('Peter', 34), ('Lucy', 28), ('Robert', 41)]
for line in table(users):
    print(line)

The recursive flatten generator turns arbitrarily nested lists into a flat sequence. For each nested list, it delegates to a new flatten generator with yield from.

def table(data):
    yield from header()
    n = yield from rows(data)
    yield '</table>'
    yield f'<!-- {n} rows -->'

The table generator is composed of smaller generators. The rows generator returns the number of rows, and yield from passes that value to the n variable. Besides values, yield from also forwards values sent with send and exceptions thrown with throw to the inner generator.

$ python yield_from.py
[1, 2, 3, 4, 5, 6, 7, 8, 9]
<table>
  <tr><td>Peter</td><td>34</td></tr>
  <tr><td>Lucy</td><td>28</td></tr>
  <tr><td>Robert</td><td>41</td></tr>
</table>
<!-- 3 rows -->

Sending values into a generator

The yield keyword is also an expression. The send method resumes the generator and makes the paused yield expression evaluate to the sent value. This turns a generator into a two-way channel.

send_values.py
def running_average():
    total = 0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count


avg = running_average()
print(next(avg))

for value in [10, 20, 60, 30]:
    print(f'sent {value}, average {avg.send(value)}')

The running_average generator keeps a running total and a count in its local variables, which survive between calls.

value = yield average

This line first yields the current average to the caller. When the caller calls send, the generator resumes and the sent value is assigned to value. The new average is then yielded by the next pass of the loop and becomes the return value of send.

avg = running_average()
print(next(avg))

A freshly created generator has not reached any yield yet, so it cannot receive a value. We must first prime it with next (or the equivalent send(None)), which runs it to the first yield. That yield produces None because no value has been averaged yet.

$ python send_values.py
None
sent 10, average 10.0
sent 20, average 15.0
sent 60, average 30.0
sent 30, average 30.0

The throw and close methods

The throw method raises an exception inside the generator at the point where it is paused. The close method raises GeneratorExit there, which ends the generator.

close_throw.py
def worker():
    print('worker started')
    try:
        while True:
            try:
                task = yield
                print(f'processing {task}')
            except ValueError as e:
                print(f'bad task: {e}')
    finally:
        print('worker cleaned up')


w = worker()
next(w)
w.send('task 1')
w.throw(ValueError('empty task'))
w.send('task 2')
w.close()

print('---')

w = worker()
next(w)
w.send('task 3')
del w
print('end of script')

The worker handles ValueError inside its loop, so a thrown ValueError is reported and the worker keeps running. The close call ends the loop, and the finally block performs the cleanup.

del w
print('end of script')

A suspended generator is closed automatically when it is garbage collected. In CPython, deleting the last reference closes it immediately, so the cleanup message appears before end of script. Other Python implementations may run the cleanup later, so code that depends on it should call close explicitly or use a with statement.

$ python close_throw.py
worker started
processing task 1
bad task: empty task
processing task 2
worker cleaned up
---
worker started
processing task 3
worker cleaned up
end of script

Generator pipelines

Generators can be connected so that the output of one is the input of the next. Each stage processes one item at a time, so even a huge input file is never loaded into memory as a whole. In this example, we analyse a web server access log.

logs/access.log
192.168.0.12 GET /index.html 200 5120
192.168.0.15 GET /about.html 200 3277
192.168.0.12 GET /missing.png 404 0
10.0.0.7 POST /login 302 0
192.168.0.21 GET /index.html 200 5120
10.0.0.7 GET /dashboard 200 18450
192.168.0.15 GET /old-page 404 0
192.168.0.33 GET /images/logo.png 200 24576
10.0.0.9 GET /admin 403 512
192.168.0.12 GET /index.html 200 5120

Each line of the log contains the client address, the HTTP method, the URL, the status code, and the response size in bytes.

pipeline.py
from collections import Counter


def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip('\n')


def parse(lines):
    for line in lines:
        ip, method, url, status, size = line.split()
        yield {'ip': ip, 'url': url, 'status': int(status), 'size': int(size)}


def successful(records):
    return (r for r in records if r['status'] == 200)


lines = read_lines('logs/access.log')
records = parse(lines)
ok = successful(records)

hits = Counter()
total = 0
for r in ok:
    hits[r['url']] += 1
    total += r['size']

print(hits.most_common(3))
print(f'{total:,} bytes served')

The pipeline has three stages: read_lines reads the file line by line, parse turns each line into a dictionary, and successful keeps only the records with status 200. The last stage is written as a generator expression returned from a function.

lines = read_lines('logs/access.log')
records = parse(lines)
ok = successful(records)

Building the pipeline does not read anything. The file is opened and read only when the final for loop starts pulling records, and each line travels through all the stages before the next one is read. The loop counts the hits per URL with a Counter and sums the bytes served.

$ python pipeline.py
[('/index.html', 3), ('/about.html', 1), ('/dashboard', 1)]
61,663 bytes served

Reading files in chunks

Text files can be iterated line by line, but binary files have no lines. A generator can split them into fixed-size chunks.

chunks.py
import hashlib


def read_chunks(path, size=4096):
    with open(path, 'rb') as f:
        while chunk := f.read(size):
            yield chunk


sha = hashlib.sha256()
nchunks = 0

for chunk in read_chunks('logs/access.log', size=64):
    sha.update(chunk)
    nchunks += 1

print(f'{nchunks} chunks')
print(sha.hexdigest())

The read_chunks generator reads the file in blocks of size bytes. The walrus operator assigns the block and tests it in one step; read returns an empty bytes object at the end of the file, which ends the loop. We feed the chunks into a SHA-256 hash, so the file is hashed without ever being loaded completely. A tiny chunk size is used here to show that the 354-byte log is split into several pieces.

$ python chunks.py
6 chunks
7eee419bb13bc09a84fee9a68acead5496f6bad1397c6d24e7bc6b9ea30c014e

Generators and itertools

The itertools module provides building blocks that consume and produce iterators lazily. They combine well with generators.

itertools_gen.py
from itertools import batched, count, islice, takewhile, chain, accumulate


def primes():
    found = []
    for n in count(2):
        if all(n % p for p in takewhile(lambda p: p * p <= n, found)):
            found.append(n)
            yield n


print(list(islice(primes(), 12)))
print(list(takewhile(lambda p: p < 30, primes())))

print(list(islice(primes(), 100, 105)))

for batch in batched(islice(primes(), 10), 4):
    print(batch)

print(list(chain('ab', range(3), (x.upper() for x in 'cd'))))
print(list(accumulate(islice(primes(), 6))))

The primes generator produces an infinite sequence of prime numbers. It iterates over count(2), an endless counter, and checks each number against the primes found so far; takewhile limits the check to primes not greater than the square root of the number.

The islice function takes a slice of an iterator, here the first twelve primes and then the primes at positions 100 to 104; takewhile takes values while a condition holds. The batched function, added in Python 3.12, groups values into tuples of a given size. The chain function joins several iterables into one and accumulate yields running totals.

$ python itertools_gen.py
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37]
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
[547, 557, 563, 569, 571]
(2, 3, 5, 7)
(11, 13, 17, 19)
(23, 29)
['a', 'b', 0, 1, 2, 'C', 'D']
[2, 5, 10, 17, 28, 41]

Generator-based context managers

The contextlib.contextmanager decorator turns a generator with a single yield into a context manager for the with statement.

timer_ctx.py
import time
from contextlib import contextmanager


@contextmanager
def timer(label):
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = time.perf_counter() - start
        print(f'{label}: {elapsed:.2f} s')


@contextmanager
def opened(path, mode='r'):
    print(f'opening {path}')
    f = open(path, mode)
    try:
        yield f
    finally:
        f.close()
        print(f'closed {path}')


with timer('sleep'):
    time.sleep(0.25)

with opened('logs/access.log') as f:
    print(f.readline().rstrip())

The code before yield runs when the with block is entered, and the code after it runs when the block is left. The yielded value is bound to the variable after as, as in the opened function. Placing the yield in a try/finally statement guarantees that the cleanup runs even if the with block raises an exception.

$ python timer_ctx.py
sleep: 0.25 s
opening logs/access.log
192.168.0.12 GET /index.html 200 5120
closed logs/access.log

StopIteration inside a generator

Since Python 3.7 (PEP 479), a StopIteration exception that escapes from a generator body is converted into a RuntimeError. This prevents bugs where a generator would silently stop early.

stop_iteration.py
def first_words(lines):
    it = iter(lines)
    while True:
        line = next(it)
        yield line.split()[0]


try:
    print(list(first_words(['old falcon', 'blue sky'])))
except RuntimeError as e:
    print(f'RuntimeError: {e}')
    print(f'caused by {type(e.__cause__).__name__}')


def first_words_fixed(lines):
    for line in lines:
        yield line.split()[0]


print(list(first_words_fixed(['old falcon', 'blue sky'])))

The first_words generator calls next on an iterator in an endless loop. When the iterator is exhausted, next raises StopIteration, which is turned into a RuntimeError; the original exception is available as __cause__. The fixed version uses a for loop, which handles the end of the iteration itself. If we have to call next manually, we catch StopIteration and return.

$ python stop_iteration.py
RuntimeError: generator raised StopIteration
caused by StopIteration
['old', 'blue']

When generator expressions are evaluated

A generator expression evaluates its first (outermost) iterable immediately, when the expression is created. Everything else is evaluated lazily, when the values are consumed.

late_binding.py
numbers = [1, 2, 3]
factor = 10

gen = (n * factor for n in numbers)

numbers = [7, 8, 9]
factor = 100

print(list(gen))


try:
    gen = (n for n in 42)
except TypeError as e:
    print(f'TypeError at creation: {e}')

After creating the generator, we rebind both numbers and factor. The generator still iterates over the original list [1, 2, 3], because that list was evaluated at creation, but it multiplies by the new value of factor, which is looked up only when each value is computed. For the same reason, a non-iterable object raises TypeError right when the generator expression is created, not later when it is consumed.

$ python late_binding.py
[100, 200, 300]
TypeError at creation: 'int' object is not iterable

Type hints for generators

Generators are annotated with the Iterator or Generator types from collections.abc.

type_hints.py
from collections.abc import Generator, Iterator


def evens(limit: int) -> Iterator[int]:
    yield from range(0, limit, 2)


def echo() -> Generator[str, str, int]:
    count = 0
    received = yield 'ready'
    while received != 'quit':
        count += 1
        received = yield received.upper()
    return count


def names() -> Generator[str]:
    yield 'Jan'
    yield 'Mia'


print(list(evens(10)))

gen = echo()
print(next(gen))
print(gen.send('hello'))
print(gen.send('world'))
try:
    gen.send('quit')
except StopIteration as e:
    print(f'echoed {e.value} messages')

print(list(names()))

When a generator only yields values, Iterator[int] is enough. The full form Generator[YieldType, SendType, ReturnType] also describes the values accepted by send and the return value. The echo generator yields strings, receives strings, and returns an integer. Since Python 3.13, the send and return types default to None, so Generator[str] can be used for a generator that only yields strings.

$ python type_hints.py
[0, 2, 4, 6, 8]
ready
HELLO
WORLD
echoed 2 messages
['Jan', 'Mia']

Async generators

An async def function that contains yield is an asynchronous generator. It can await between yields and is consumed with async for.

async_gen.py
import asyncio
import time


async def ticker(n, delay):
    for i in range(n):
        await asyncio.sleep(delay)
        yield i


async def collect(name, n, delay):
    async for i in ticker(n, delay):
        print(f'{name} {i}')
    return name


async def main():
    start = time.perf_counter()
    done = await asyncio.gather(collect('fast', 3, 0.1), collect('slow', 2, 0.25))
    print(done)
    print(f'{time.perf_counter() - start:.1f} s')

    squares = [i * i async for i in ticker(5, 0)]
    print(squares)


asyncio.run(main())

The ticker async generator waits for the given delay before each value. Two consumers run concurrently with asyncio.gather, so their output interleaves and the whole run takes about half a second, the time of the slower one. Async generators also work in asynchronous comprehensions, as in the squares list. The Python async/await article covers asynchronous programming in more detail.

$ python async_gen.py
fast 0
fast 1
slow 0
fast 2
slow 1
['fast', 'slow']
0.5 s
[0, 1, 4, 9, 16]

Source

Python tutorial - Generators
Python reference - Yield expressions
Functional Programming HOWTO - Generators
PEP 479 - Change StopIteration handling inside generators

In this article we have worked with generators in Python. We have covered yield, generator objects, generator expressions, yield from, the send, throw, and close methods, pipelines, context managers, and async generators. See also the Python iterator, Python itertools, Python list comprehensions, and Python closures articles.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Python tutorials.