How to find and remove invisible characters in text

Two words look identical, yet a spreadsheet says they are different. A coupon code copied from an email is rejected. A config file fails with an error on line 1, column 1. Often the culprit is a character you cannot see: a zero-width space, a byte-order mark or a direction mark that came along with copied text.

The usual suspects

• Zero-width space (U+200B): marks a possible line break without showing a space. Common in text copied from web pages and chat apps.
• Byte-order mark (U+FEFF): added to the start of files by some Windows editors. Breaks scripts, CSV headers and JSON parsers.
• Zero-width joiner and non-joiner (U+200D, U+200C): control how characters combine.
• Direction marks: left-to-right and right-to-left marks used in mixed-language text.
• Soft hyphen (U+00AD): shows only when a word breaks at a line end.
• Non-breaking space (U+00A0): visible as a space but a different character, so searches and comparisons fail.

Detect them

Paste the text into the Text Cleaner. The panel shows how many invisible characters it found. Another clue: the Character Counter shows a higher character count than the letters you can see, and its visible-character figure is lower than its total.

Remove them step by step

1. Paste the text into the Text Cleaner.
2. Turn on “Remove invisible characters”. It removes zero-width spaces, joiners, byte-order marks and direction marks.
3. Turn on “Straighten smart quotes and dashes” as well, which turns non-breaking spaces into ordinary ones.
4. Copy the cleaned text and try again.

Cleaning happens in your browser, so private text is not uploaded.

When not to remove them

Some invisible characters are doing a job. Removing all of them can damage text in these cases:

• Emoji sequences: family, profession and some flag emoji are built from several emoji glued together with zero-width joiners. Removing the joiners splits 👨‍👩‍👧 into separate people.
• Some scripts: Persian, Arabic and several Indian scripts use joiners and non-joiners to control how letters connect. Removing them can change how words display.
• Right-to-left text mixed with English may rely on direction marks to display in the right order.

For plain English data, codes, addresses, IDs and code, removing them is safe and usually what you want.

Prevent them

• Save files as UTF-8 without a byte-order mark; most code editors have this option.
• Type short codes and passwords instead of copying them from formatted emails.
• Paste into a plain-text editor first when moving text from the web into data files.

For curly quotes and dashes, see converting curly quotes to straight. For broken JSON, see how to fix invalid JSON.