When working with file processing in Linux system, may needed to remove duplicate lines from a file. It can be useful with log files which often contains repeated lines. This tutorial shows how to remove duplicate lines from file in Linux.
Create a new file for testing:
printf "Line1\nLine2\nLine2\nLine1\nLine3\n" > test.txt
awk command can be used process text files. Execute the following command to remove duplicate lines from a file:
awk -i inplace '!seen[$0]++' test.txt
-i option with value
inplace will edit file in-place.