Pandas Memory Usage
last modified November 17, 2026 Python 3.15
Memory usage is the number of bytes a DataFrame occupies in RAM. Knowing it matters when a dataset approaches the available memory: a column stored as 64-bit integers can take eight times more space than the same values stored as 8-bit integers, and repeated text can take far more space than a categorical code. Pandas provides several ways to measure and reduce it.
The byte counts below come from real runs of pandas 3.1.1 with pyarrow 26.0.1 installed. Exact numbers depend on the platform, the Pandas version and the actual values, so treat them as illustrations rather than constants.
Shallow and deep memory_usage
The memory_usage method returns bytes used per column, including
the index. By default it is shallow: for text columns it counts the
references to strings, not the strings themselves. Passing
deep=True counts the real string data as well, which is the
number to trust for object-like and string columns.
import pandas as pd
df = pd.DataFrame({'name': ['apple', 'banana', 'cherry'] * 100})
print('shallow (per column):')
print(df.memory_usage())
print()
print('deep total:', df.memory_usage(deep=True).sum(), 'bytes')
The shallow view reports 4100 bytes for the 300 names plus 132 bytes for the
index. The deep total, 4232 bytes, adds the actual characters. Calling
sum() on the result turns the per-column Series into one total.
$ python mem_usage.py shallow (per column): Index 132 name 4100 dtype: int64 deep total: 4232 bytes
A quick overview with info
The info method prints a compact summary of a DataFrame: its
class, the number of rows and columns, each column's non-null count and dtype,
and a memory total. With memory_usage='deep' that total includes
the real string bytes, equivalent to memory_usage(deep=True).sum().
import pandas as pd
df = pd.DataFrame({'name': ['apple', 'banana', 'cherry'] * 100})
df.info(memory_usage='deep')
The output lists one column, name, with 300 non-null entries and
dtype str, and ends with a memory usage of 4.1 KB. This is a fast
way to inspect both dtypes and memory before deciding what to change.
$ python mem_info.py <class 'pandas.DataFrame'> RangeIndex: 300 entries, 0 to 299 Data columns (total 1 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 name 300 non-null str dtypes: str(1) memory usage: 4.1 KB
Downcasting numeric dtypes
A dtype describes how values are stored. Pandas often chooses a
wide default, such as 64-bit integer (int64), even when the values
would fit in something smaller. astype converts a column to a
narrower dtype; here int8 stores each value in a single byte.
import pandas as pd
df = pd.DataFrame({'x': [1, 2, 3] * 100})
print('int64:', df.memory_usage(deep=True)['x'], 'bytes')
small = df.astype({'x': 'int8'})
print('int8: ', small.memory_usage(deep=True)['x'], 'bytes')
The same 300 values need 2400 bytes as int64 and only 300 bytes
as int8. Only downcast when every value fits: int8
holds numbers from -128 to 127. For floating-point columns,
float32 is the corresponding smaller choice.
$ python mem_downcast.py int64: 2400 bytes int8: 300 bytes
Strings and StringDtype
In pandas 3 the default text dtype is str, shown as
str in info output. An alternative is
pd.StringDtype with storage='pyarrow', which keeps
the strings in Arrow-backed arrays; pyarrow is installed in the environment
used here. The dtype attribute tells you which representation a
column uses.
import pandas as pd
s = pd.Series(['apple', 'banana', 'cherry'] * 100)
print('default dtype:', s.dtype)
print('default deep: ', s.memory_usage(deep=True), 'bytes')
a = s.astype(pd.StringDtype(storage='pyarrow'))
print('arrow dtype: ', a.dtype)
print('arrow deep: ', a.memory_usage(deep=True), 'bytes')
The default dtype is str and the Arrow-backed one is
string. On these short, repeated strings both report 4232 bytes,
so there is no saving here: the optional dtype is chosen for its nullable
semantics and Arrow interoperability, not as a guaranteed memory win.
$ python mem_string.py default dtype: str default deep: 4232 bytes arrow dtype: string arrow deep: 4232 bytes
Categoricals for repeated values
When a column contains only a handful of distinct strings, converting it to a
category dtype stores each distinct value once and uses small
integer codes for the rows. This is often the biggest single saving available
for text data, and it is described in more detail in the
categorical data tutorial.
import pandas as pd
s = pd.Series(['apple', 'banana', 'cherry'] * 100)
print('object deep: ', s.memory_usage(deep=True), 'bytes')
c = s.astype('category')
print('category deep:', c.memory_usage(deep=True), 'bytes')
The deep figure drops from 4232 bytes to 474 bytes because the three words are stored once instead of three hundred times. The saving grows with the number of rows and shrinks as the number of distinct values grows.
$ python mem_cat.py object deep: 4232 bytes category deep: 474 bytes
Reading large files in chunks
A chunk is one piece of a file read into memory at a time. Passing
chunksize to read_csv makes it return an iterator of
DataFrames instead of one huge frame. Each chunk is processed and then
released, so peak memory depends on the chunk size rather than the file size.
import pandas as pd
path = '/tmp/opencode/agentG_big.csv'
pd.DataFrame({'a': range(10000), 'b': ['x'] * 10000}).to_csv(path, index=False)
total_rows = 0
chunks = 0
for chunk in pd.read_csv(path, chunksize=2000):
total_rows += len(chunk)
chunks += 1
print('chunks:', chunks)
print('rows read:', total_rows)
The example writes a temporary ten-thousand-row CSV, then reads it five chunks of 2000 rows at a time and adds up their lengths. In a real program you would aggregate or filter each chunk before discarding it, keeping only the small result in memory.
$ python mem_chunks.py chunks: 5 rows read: 10000
Source
Pandas DataFrame.memory_usage Documentation, Pandas DataFrame.info Documentation, Pandas Categorical Data Documentation, and Pandas read_csv Documentation
In this article we have measured shallow and deep memory, used
info for an overview, downcast numeric columns, compared the
default str dtype with a pyarrow-backed
StringDtype, saved space with categoricals, and read a large file
in chunks. For the categorical representation see
Pandas categorical data, and for how
sharing affects memory see Pandas
copy-on-write.
Author
List all Pandas tutorials.