WriteBash - In this article, we will talk about using the
uniq command to delete duplicate lines. This is one of the commands that is also used quite a lot while working on Linux systems.
Uniq command and sort command
In the article about sort command, we know that the sort command will help you output an sorted list. This
uniq command is often used in conjunction with the
In sorted list data, duplicate data may exist. This
uniq command will work to delete duplicate lines, leaving only one line.
Use the uniq command to delete duplicate lines
uniq command can use sorted list data from the standard input or from a single file argument.
In the example below, we create a text file with unsorted data. Next, we will use both
uniq commands to see how the results are.
$ cat demo-uniq.txt | sort | uniq
Look at the image above, we have 6 data lines. After using the
sort command, we can have 2 duplicate lines
1 - line number 1.
Recommended Reading: What is pipeline and how to use it
Then, we use the
uniq command in the pipeline, the
uniq command is right next the
sort command. You can see in the result after using the uniq command, there is only one line
1 - line number 1.
If you don’t want to delete the duplicate line from the standard output, you just want to display the duplicate line that tells you. Use the
-d option as the example below.
$ cat demo-uniq-txt | sort | uniq -d
If you want to count the number of duplicate lines, use the
-c option (count).
$ cat demo-uniq.txt | sort | uniq -c
Through this article, you have learned another command to process the pipeline data. It is very useful in filtering data.If you liked this article, then please subscribe to our YouTube Channel for more video tutorials. You can also find us on Twitter and Facebook.
If you appreciate what we share in this blog, you can support us by:
- Stay connected to: Facebook | Twitter | Google Plus | YouTube
- Subscribe email to recieve new posts from us: Sign up now.
- Start your own blog with SSD VPS - Free Let's Encrypt SSL ($2.5/month).
- Become a Supporter - Make a contribution via PayPal.
- Support us by purchasing Ribbon Lite Child theme being using on this website.
We are thankful for your support.