Unicode lets the same visual character be encoded in more than one way. This can create invisible mismatches that break search, file systems, and security checks. The characters involved are valid and well known; the difficulty is that they are not interchangeable, even when they look identical on screen.
One letter, two encodings
Take the letter é. It can be encoded as a single code point U+00E9 (composed), or as the letter e followed by a combining acute accent U+0301 (decomposed). They look the same. Many systems treat them as different strings.
This is not a rendering quirk. Both encodings are legal, and the code points really are different, so byte comparison, hashing, and sorting all see two distinct strings even though a reader sees one character.
What normalization does
Unicode defines four standard forms: NFC, NFD, NFKC, and NFKD. They describe how to convert equivalent sequences into a canonical form. NFC is the most common choice on the web. Normalization is the process of rewriting a string into one of those forms so that two equivalent sequences become byte-identical. It does not change meaning, case, or spacing; it only picks a single representation among the ones the standard treats as equivalent.
The four forms, and when each is used
NFC and NFD keep compatibility distinctions and differ mainly in how combining sequences are stored. NFC prefers a single composed character where one exists, so é stays U+00E9. NFD prefers the decomposed sequence, so the same letter becomes e followed by U+0301. For text that only needs to be read, the two forms display the same way and are usually interchangeable.
NFKC and NFKD add a further step called compatibility folding, which collapses characters that are visually or historically related but technically distinct. A full-width digit, a typographic ligature, or a superscript number can be folded to its plain equivalent. Because folding can change characters a reader might expect to keep, NFKC and NFKD are normally reserved for identifiers, search keys, and matching logic rather than for text that must be preserved exactly.
Why it matters
Normalization problems often show up alongside other Unicode tricks. See Trojan Source: When Text Order Lies for another way hidden characters can change how text is interpreted.
- Two URLs that look identical may route to different servers because one uses a composed character and the other uses decomposed characters.
- A username check may fail if one side normalizes and the other does not.
- Search indexes may miss content encoded in a different normalization form.
Failures like these are silent. A lookup simply does not match, and unless you suspect an encoding difference, the real cause is easy to overlook.
How a mismatch shows up in practice
- A pasted name or address does not match the stored record even though both look the same on screen.
- A password or token comparison fails for one user because their input arrived in a different form.
- A filter that checks for a forbidden substring misses it when the text uses a compatibility variant the filter never normalizes.
- A deduplication step keeps two copies of the same entry because the two strings hash differently.
- A file cannot be found by one tool because another tool stored or queried the name in a different form.
What the scanner does today
This tool reports when the text contains characters that would change under NFC normalization. It does not rewrite the text automatically; it only points out where a mismatch could happen. You decide whether to normalize.
How to handle normalization in practice
- Pick one form per purpose. NFC is the usual default for text shown on the web; NFKC is common for identifiers and search keys.
- Normalize on input and on storage. Converting only at comparison time still lets two different forms reach the database.
- Normalize both sides of every comparison. If a stored value and a user input are compared, run both through the same form first.
- Use the same form for the query and the index. A search that normalizes only one of the two will miss equivalent characters.
- Leave display text alone unless there is a reason. Rewriting the bytes of text you intend to show verbatim can introduce changes you did not want.
Common misunderstandings about normalization
- "One letter always has one code point." Many characters have both a composed and a decomposed encoding.
- "Normalization is the same as lowercasing or trimming." It only rewrites characters into an equivalent form; it does not change case or remove whitespace.
- "NFC and NFKC are interchangeable." Compatibility folding in NFKC can change characters that NFC would leave untouched.
- "If two strings look the same, they are the same." Rendering and storage are different layers, and only the bytes decide equality.
Tip: If you are building a system that compares text, normalize both sides to the same form before comparing. Do not assume two visually identical strings have the same bytes.