Delete duplicate lines within a text file in two steps:

• sort file : sort command
• extract lines and ignore duplicate ones : awk ‘!x[$0]++’

> sort input.txt | awk '!x[$0]++' >> output.txt