Use uniq command to delete duplicate lines

Share this post to your friends !

This entry is part 5 of 10 in the series Linux commands: Part 6 - Redirection
Use uniq command to delete duplicate lines
5 (100%) 1 vote[s]

WriteBash - In this article, we will talk about using the uniq command to delete duplicate lines. This is one of the commands that is also used quite a lot while working on Linux systems.

Uniq command and sort command

In the article about sort command, we know that the sort command will help you output an sorted list. This uniq command is often used in conjunction with the sort command.

In sorted list data, duplicate data may exist. This uniq command will work to delete duplicate lines, leaving only one line.

Use the uniq command to delete duplicate lines

The uniq command can use sorted list data from the standard input or from a single file argument.

In the example below, we create a text file with unsorted data. Next, we will use both sort and uniq commands to see how the results are.

$ cat demo-uniq.txt | sort | uniq
use-uniq-command-to-delete-duplicate-lines-01 Use uniq command to delete duplicate lines
Use uniq command to delete duplicate lines.

Look at the image above, we have 6 data lines. After using the sort command, we can have 2 duplicate lines 1 - line number 1.

Then, we use the uniq command in the pipeline, the uniq command is right next the sort command. You can see in the result after using the uniq command, there is only one line 1 - line number 1.

If you don’t want to delete the duplicate line from the standard output, you just want to display the duplicate line that tells you. Use the -d option as the example below.

$ cat demo-uniq-txt | sort | uniq -d
use-uniq-command-to-delete-duplicate-lines-02 Use uniq command to delete duplicate lines
Use uniq command to view duplicate lines or count number duplicate lines.

If you want to count the number of duplicate lines, use the -c option (count).

$ cat demo-uniq.txt | sort | uniq -c

Conclusion

Through this article, you have learned another command to process the pipeline data. It is very useful in filtering data.

Continue reading the series«« Previous part: Use sort command to sort pipeline dataNext part: Use wc command to count number lines »»
If you liked this article, then please subscribe to our YouTube Channel for more video tutorials. You can also find us on Twitter and Facebook.


Share this post to your friends !
If you appreciate what we share in this blog, you can support us by:
  1. Stay connected to: Facebook | Twitter | Google Plus | YouTube
  2. Subscribe email to recieve new posts from us: Sign up now.
  3. Start your own blog with SSD VPS - Free Let's Encrypt SSL ($2.5/month).
  4. Become a Supporter - Make a contribution via PayPal.
  5. Support us by purchasing Ribbon Lite Child theme being using on this website.

We are thankful for your support.

single post bottom banner
«« »»

Got something to say? Join the discussion

Please keep in mind that all comments are subject to our Comment Policy. Your email address will not be published.
This site uses Akismet to reduce spam. Learn how your comment data is processed.