Linux uniq Command
last modified October 23, 2026
The uniq command in Linux filters out repeated lines. It
collapses runs of identical adjacent lines into a single line and can also
count the repetitions or print only duplicated or only unique lines.
The key point about uniq is that it compares only adjacent
lines. To find duplicates anywhere in a file, the input must first be sorted
with the sort command. The pair
sort | uniq is one of the most common idioms in shell pipelines.
The examples use GNU uniq from coreutils.
Removing adjacent duplicates
Without options, uniq prints one copy of each run of identical
lines.
uniq words.txt
The two adjacent sky lines and the three falcon
lines are merged. However, the word sky still appears twice in
the output, because its third occurrence is not adjacent to the others.
$ cat words.txt sky sky forest falcon falcon falcon sky owl $ uniq words.txt sky forest falcon sky owl
Removing all duplicates
Sorting the input first brings equal lines together.
sort words.txt | uniq
Each word now appears exactly once. The same result is produced by
sort -u; uniq is needed when we want to use its
other options, such as counting.
$ sort words.txt | uniq falcon forest owl sky
Counting occurrences
The -c option prefixes each line with the number of times it
occurred.
uniq -c words.txt sort words.txt | uniq -c | sort -rn
On unsorted input, the counts refer to adjacent runs only. The second pipeline is the standard way to build a frequency table: sort, count, and then sort the counts numerically in descending order.
$ uniq -c words.txt
2 sky
1 forest
3 falcon
1 sky
1 owl
$ sort words.txt | uniq -c | sort -rn
3 sky
3 falcon
1 owl
1 forest
Duplicated and unique lines
The -d option prints only lines that are repeated,
-D prints all copies of the repeated lines, and -u
prints only lines that are not repeated.
sort words.txt | uniq -d sort words.txt | uniq -D sort words.txt | uniq -u
-d is useful for finding duplicate entries, for example
duplicate user names or IDs in a list. -u finds the items that
occur only once.
$ sort words.txt | uniq -d falcon sky $ sort words.txt | uniq -D falcon falcon falcon sky sky sky $ sort words.txt | uniq -u forest owl
Ignoring case
The -i option compares lines case-insensitively.
uniq -i fruits.txt uniq -ic fruits.txt
Without -i, the different spellings of apple are
distinct lines. With it, they form one group, represented by its first line.
When combining with sort, use sort -f so that the
lines are grouped case-insensitively as well.
$ uniq fruits.txt
Apple
apple
APPLE
banana
Banana
cherry
$ uniq -i fruits.txt
Apple
banana
cherry
$ uniq -ic fruits.txt
3 Apple
2 banana
1 cherry
Skipping fields
The -f option skips the given number of fields at the beginning
of each line before comparing. Fields are separated by blanks.
uniq -f 1 events.txt uniq -f 1 -c events.txt
The events start with a timestamp, which differs on every line. By skipping the first field, we compare only the event and the user, so consecutive repeated events are collapsed and the first timestamp of each run is kept.
$ uniq -f 1 events.txt
09:00 login jano
09:07 logout jano
09:10 login alice
09:30 logout alice
$ uniq -f 1 -c events.txt
2 09:00 login jano
1 09:07 logout jano
2 09:10 login alice
1 09:30 logout alice
Comparing parts of lines
The -w option compares only the first N characters, and
-s skips the first N characters.
printf "%s\n" error:disk error:network warn:cpu | uniq -w 5 -c printf "%s\n" A1-x A2-x B1-x | uniq -s 3 -c
With -w 5, the lines are grouped by their first five characters,
so the two error lines form one group. With -s 3,
only the text after the third character is compared, which is the same for
all three lines. When -f and -s are combined, the
fields are skipped first.
$ printf "%s\n" error:disk error:network warn:cpu | uniq -w 5 -c
2 error:disk
1 warn:cpu
$ printf "%s\n" A1-x A2-x B1-x | uniq -s 3 -c
3 A1-x
Analyzing a log file
A typical use of uniq is summarizing log files.
192.0.2.10 - - [23/Oct/2026:09:00:01] "GET / HTTP/1.1" 200 198.51.100.7 - - [23/Oct/2026:09:00:03] "GET /about HTTP/1.1" 200 192.0.2.10 - - [23/Oct/2026:09:00:04] "GET /contact HTTP/1.1" 200 203.0.113.5 - - [23/Oct/2026:09:00:09] "GET /admin HTTP/1.1" 403 192.0.2.10 - - [23/Oct/2026:09:00:12] "GET / HTTP/1.1" 200 198.51.100.7 - - [23/Oct/2026:09:00:15] "GET /blog HTTP/1.1" 404 203.0.113.5 - - [23/Oct/2026:09:00:16] "GET /admin HTTP/1.1" 403 192.0.2.10 - - [23/Oct/2026:09:00:20] "GET /blog HTTP/1.1" 200
The access log contains the client address, the time, the request, and the status code of each request.
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn
awk '$NF == 403 {print $1}' access.log | sort | uniq -c
cut -d'"' -f2 access.log | sort | uniq -c | sort -rn | head -n 3
The first pipeline ranks the clients by the number of requests. The second counts the forbidden requests per client, which can reveal someone probing the admin page. The third extracts the request between the quotes and shows the most requested pages; among equal counts, the order is determined by the reverse sort of the whole line.
$ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn
4 192.0.2.10
2 203.0.113.5
2 198.51.100.7
$ awk '$NF == 403 {print $1}' access.log | sort | uniq -c
2 203.0.113.5
$ cut -d'"' -f2 access.log | sort | uniq -c | sort -rn | head -n 3
2 GET / HTTP/1.1
2 GET /blog HTTP/1.1
2 GET /admin HTTP/1.1
Writing to a file
uniq accepts an optional output file after the input file.
uniq words.txt unique-words.txt
The result is written to unique-words.txt instead of standard
output. Unlike with sort -o, the output file must not be the
same as the input file.
$ uniq words.txt unique-words.txt $ cat unique-words.txt sky forest falcon sky owl
Best practices for uniq
- Sort first:
uniqonly detects adjacent duplicates. - Use the same rules: Combine
sort -fwithuniq -i, and sort by the same fields you compare. - Rank with sort -rn: Use
sort | uniq -c | sort -rnfor frequency tables. - Prefer sort -u: When only removing duplicates, a single
sort -uis enough.
Source
In this article, we have used the uniq command to remove
duplicate lines, count occurrences, find repeated and unique lines, compare
parts of lines, and analyze a log file.
Author
List all Linux tutorials.