Blog

03/06/2019

Unix / Linux: Remove duplicate lines from a text file using awk or perl



I have a text file with exact duplicates of lines. I need to remove all those duplicates lines and preserves the order too on Linux or Unix-like system. How do I delete duplicate lines from a text file?

You can use Perl or awk or Python to delete all duplicate lines from a text file on Linux, OS X, and Unix-like system.

Sample data file

$ cat data.txt
this is a test
Hi, User!
this is a test
this is a line
this is another line
call 911
this vs that
that vs this
How to Call 911
that and that
Hi, User!
this vs that
call 911

How to remove duplicate lines inside a text file using awk

The syntax is as follows to preserves the order of the text file:
awk '!seen[$0]++' input > output
awk '!seen[$0]++' data.txt > output.txt
more output.txt

Sample outputs:

this is a test
Hi, User!
this is a line
this is another line
call 911
this vs that
that vs this
How to Call 911
that and that

How to remove duplicates line from multiple text file in Perl?

The syntax is:

perl -lne '$seen{$_}++ and next or print;' input > output
perl -lne '$seen{$_}++ and next or print;' data.txt > output.txt
more output.txt

Sample outputs:

this is a test
Hi, User!
this is a line
this is another line
call 911
this vs that
that vs this
How to Call 911
that and that

Posted by: SXI ADMIN

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.

20/08/2019

Start your Bitcoin Exchange with our Software

Start your Bitcoin Exchange with our Software Setup your white label bitcoin exchange right away. Or create your own cryptocurrency using...
14/08/2019

How to KVM, QEMU start or stop virtual machine from command line (CLI)

KVM or Kernel Based Virtual Machine is a popular virtualization technology. It allows you to run virtual guest machines over a host machine. To start...
14/08/2019

How to Docker backup Saving and restoring your volumes

Running a Docker volume backup First, we spin up a temporary container, and we mount the backup folder and the target Docker volume to this container....