Remove Duplicate Lines

1. 1Paste your list, one item per line.
2. 2Decide whether case and surrounding whitespace should matter.
3. 3Choose to keep unique lines, or invert it to keep only the duplicates.
4. 4Copy the cleaned list.

Lines are compared using a hash set, which keeps the work proportional to the number of lines rather than to their square — so a hundred thousand lines deduplicate as fast as a hundred.

The comparison options matter more than they look. With case sensitivity off, 'Apple' and 'apple' are treated as the same line and the first occurrence is kept. With whitespace trimming on, lines that differ only by trailing spaces — extremely common in data pasted from spreadsheets — collapse together.

Order is always preserved: the first occurrence of each line stays where it was. Inverting the mode shows only the lines that appeared more than once, which is a fast way to find accidental repeats in a list you expected to be unique.

Your text is processed by JavaScript running in this page and is never sent over the network. That matters when you are pasting a draft, a client brief, meeting notes or anything else you would rather not hand to a server.

• Turn on whitespace trimming for lists pasted from spreadsheets, where trailing spaces are common.
• Turn off case sensitivity for email addresses and names, so different capitalisation still counts as a duplicate.
• Invert the mode to see only the repeated lines when you are checking a list that should be unique.
• Sort the result afterwards with Sort Lines if you want it in alphabetical order.

• How to remove duplicate lines from a list
• How to sort a list properly — and why item10 comes before item2
• How to pick a random winner fairly — and prove it
• How to check an email address is valid before you send
• How to make barcodes for inventory and stock labels