ZetCode

Pandas Duplicate Rows

last modified November 17, 2026 Python 3.15

A duplicate row is a row whose values match those of another row in every column. Real data sets often contain repeated records, and leaving them in can skew a calculation. Pandas gives us two closely related tools: duplicated, which reports where the repeats are, and drop_duplicates, which removes them.

Detecting and removing are separate jobs. Detection leaves the data untouched and answers the question “which rows repeat?”. Removal builds a new table with the extra copies stripped out. The examples below were verified with pandas 3.1.1.

Detecting duplicates with duplicated

A boolean mask is a Series of True and False values that lines up row by row with the data. Calling duplicated on a DataFrame returns exactly such a mask: True marks a row that has appeared earlier in the table, so only later copies are flagged.

duplicated.py
import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob'],
    'score': [85, 90, 85, 78, 90]
})
print(df.duplicated())

The mask has one entry per row, indexed just like the DataFrame. Rows 2 and 4 are True because the pairs (Alice, 85) and (Bob, 90) were already seen at rows 0 and 1. Notice that the first occurrence of each pair is False.

$ python duplicated.py
0    False
1    False
2     True
3    False
4     True
dtype: bool

Removing duplicates with drop_duplicates

drop_duplicates returns a new DataFrame with the flagged rows removed. By default it keeps the first occurrence of each repeated row — the same rows that duplicated marked False — and leaves the original index values in place.

drop_duplicates.py
import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob'],
    'score': [85, 90, 85, 78, 90]
})
print(df.drop_duplicates())

Only the three unique rows survive. The original labels 0, 1 and 3 are kept rather than renumbered, which tells you where each surviving row came from. We change that with ignore_index later.

$ python drop_duplicates.py
    name  score
0  Alice     85
1    Bob     90
3  Carol     78

Choosing which copy to keep

The keep parameter decides which member of a group of equal rows survives. It accepts 'first' (the default), 'last', or False. The last choice discards the entire group, keeping only rows that have no duplicate at all.

keep.py
import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Alice', 'Bob'],
    'score': [85, 90, 85, 90]
})
print("keep='first':")
print(df.drop_duplicates(keep='first'))
print()
print("keep='last':")
print(df.drop_duplicates(keep='last'))
print()
print("keep=False:")
print(df.drop_duplicates(keep=False))

With 'first' rows 0 and 1 remain; with 'last' rows 2 and 3 remain, carrying the later index labels. With keep=False every row belongs to a duplicated pair, so nothing is left and an empty DataFrame is printed.

$ python keep.py
keep='first':
    name  score
0  Alice     85
1    Bob     90

keep='last':
    name  score
2  Alice     85
3    Bob     90

keep=False:
Empty DataFrame
Columns: [name, score]
Index: []

Limiting the check to selected columns

A subset is the list of columns that Pandas compares when deciding whether two rows match. Passing subset= to either method ignores every other column. This is useful when a table has an identifier column that should be unique, plus other columns that legitimately differ.

subset.py
import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Alice', 'Bob', 'Bob'],
    'score': [85, 90, 78, 78]
})
print("duplicated(subset=['name']):")
print(df.duplicated(subset=['name']))
print()
print("drop_duplicates(subset=['name'], keep='first'):")
print(df.drop_duplicates(subset=['name'], keep='first'))

Here the scores differ, so on the full table no row is a duplicate. Comparing only the name column, however, flags the second Alice and the second Bob. Dropping with that subset leaves one row per name: Alice with her first score, 85, and Bob with his first score, 78.

$ python subset.py
duplicated(subset=['name']):
0    False
1     True
2    False
3     True
dtype: bool

drop_duplicates(subset=['name'], keep='first'):
    name  score
0  Alice     85
2    Bob     78

Resetting the index

The index is the row-label Series that identifies each row. Because drop_duplicates preserves the original labels, the result can have gaps such as 0, 1, 3. Setting ignore_index=True replaces them with a fresh continuous range starting at zero.

ignore_index.py
import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Alice', 'Carol'],
    'score': [85, 90, 85, 78]
})
print(df.drop_duplicates())
print()
print(df.drop_duplicates(ignore_index=True))

The first output keeps the gap at label 3. The second output renumbers the surviving rows 0, 1, 2. The data is identical; only the labels changed.

$ python ignore_index.py
    name  score
0  Alice     85
1    Bob     90
3  Carol     78

    name  score
0  Alice     85
1    Bob     90
2  Carol     78

Counting duplicates

Because duplicated returns a boolean mask, counting is simply a matter of adding up its True values — Python treats True as 1 and False as 0, and a Series provides a sum method. The number of unique rows is the length of the table minus that count, or the length of the result of drop_duplicates.

counting.py
import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'],
    'score': [85, 90, 85, 78, 90, 85]
})
mask = df.duplicated()
print("duplicate mask:", mask.to_list())
print("duplicate count:", int(mask.sum()))
print("unique count:", int(df.drop_duplicates().shape[0]))

The mask shows three True entries, so there are three extra copies: rows 2, 4 and 5. Subtracting them from the six original rows leaves three unique rows.

$ python counting.py
duplicate mask: [False, False, True, False, True, True]
duplicate count: 3
unique count: 3

Detecting versus removing on a Series

Both methods work on a Series as well as a DataFrame. On a one-dimensional Series there are no columns to choose from, so each value is compared directly with the values before it. This makes the difference between the two methods especially clear: one returns labels and booleans, the other returns the values that survive.

series_duplicates.py
import pandas as pd

s = pd.Series(['x', 'y', 'x', 'z', 'y', 'x'])
print("detect (boolean mask):")
print(s.duplicated())
print()
print("remove (values kept):")
print(s.drop_duplicates())

The first call is a diagnostic: a boolean Series telling us that positions 2, 4 and 5 repeat. The second call is the cleanup: a shorter Series holding x, y and z, with their original labels 0, 1 and 3. Use duplicated to investigate, then drop_duplicates when you want the cleaned data.

$ python series_duplicates.py
detect (boolean mask):
0    False
1    False
2     True
3    False
4     True
5     True
dtype: bool

remove (values kept):
0    x
1    y
3    z
dtype: str

Source

Pandas DataFrame.duplicated Documentation, Pandas DataFrame.drop_duplicates Documentation, and Pandas Series.duplicated Documentation

In this article we have detected duplicate rows with duplicated, removed them with drop_duplicates, chosen which copy to keep, restricted the comparison to a subset of columns, reset the index, counted the repeats, and seen the difference between detecting and removing. For related clean-up steps see dropping missing values and filling missing values; for other ways to narrow a table down see boolean indexing.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Pandas tutorials.