How to remove duplicate lines from a list

Merge two contact lists, paste a column from a spreadsheet, or combine exports from several systems, and duplicates appear. Removing them by eye is slow and unreliable in a list of a few hundred lines. A deduplication tool does it in a moment — if you set it up for the kind of duplicates your list actually contains.

Remove duplicates step by step

1. Paste the list into Remove Duplicate Lines, one item per line.
2. Decide whether capital letters and surrounding spaces should matter.
3. Keep unique lines, or invert the mode to see only the lines that were repeated.
4. Copy the cleaned list. The tool tells you how many lines were removed.

The first occurrence of each line stays where it was, so the list keeps its original order. Even very long lists are handled quickly, and the text never leaves your browser.

A worked example

Take a list of six lines pasted from two sign-up sheets:

• Ann Lee
• Ravi Shah
• ann lee
• Tom Burke
• Tom Burke
• Ravi Shah

With default settings only the exact repeat of “Ravi Shah” is removed: five lines remain. Turn off case sensitivity and “ann lee” goes too. Turn on whitespace trimming and the second “Tom Burke”, which had a trailing space, goes as well. Three lines remain — Ann Lee, Ravi Shah and Tom Burke — in their original order.

Near-duplicates: the real problem

Lines that look identical to a person often differ to a computer:

• Capital letters. “Apple” and “apple”. Turn off case sensitivity for names, email addresses and most everyday lists.
• Trailing spaces. Extremely common in data pasted from spreadsheets, and invisible. Turn on whitespace trimming.
• Invisible characters such as non-breaking or zero-width spaces, which arrive from web pages and documents. Run the list through Text Cleaner first if duplicates survive that should not.

Find the duplicates instead

Sometimes the useful question is not “give me a clean list” but “which items were entered twice?” — a double booking, a product listed under two codes, a person who signed up twice. Inverting the mode shows only the lines that appeared more than once, with their counts.

Email lists

For mailing lists, turn off case sensitivity and turn on trimming, so “Ann@Example.com ” and “ann@example.com” count as the same address. Note that addresses which differ in other ways — a plus tag such as ann+news@example.com — are different text, and a deduplication tool rightly keeps both.

Sorting afterwards

Deduplication keeps the original order. If you want the result alphabetical or numeric, run it through Sort Lines afterwards. Doing it in that order keeps the first occurrence of each line — useful when the order means something, such as the date people signed up.

Spreadsheets and databases

Spreadsheets have their own remove-duplicates command, which works on whole rows or chosen columns. Use that when the data has several columns and a row only counts as a duplicate if several fields match. A line-based tool is quicker for a single column you have just copied, and shows you exactly what it removed.