Remove Duplicate Lines
Remove repeated lines from a list and see exactly how many were taken out.
How to use Remove Duplicate Lines
- 1Paste your list, one item per line.
- 2Decide whether case and surrounding whitespace should matter.
- 3Choose to keep unique lines, or invert it to keep only the duplicates.
- 4Copy the cleaned list.
How it works
Lines are compared using a hash set, which keeps the work proportional to the number of lines rather than to their square — so a hundred thousand lines deduplicate as fast as a hundred.
The comparison options matter more than they look. With case sensitivity off, 'Apple' and 'apple' are treated as the same line and the first occurrence is kept. With whitespace trimming on, lines that differ only by trailing spaces — extremely common in data pasted from spreadsheets — collapse together.
Order is always preserved: the first occurrence of each line stays where it was. Inverting the mode shows only the lines that appeared more than once, which is a fast way to find accidental repeats in a list you expected to be unique.
Your text is processed by JavaScript running in this page and is never sent over the network. That matters when you are pasting a draft, a client brief, meeting notes or anything else you would rather not hand to a server.