UTF-8 Decoder and Encoder

How it works

• Decode mode accepts hex ("43 61 66 C3 A9" or packed "436166C3A9") or decimal bytes, and walks the sequence the way a real UTF-8 decoder does — reporting the exact byte index and reason when a sequence is invalid, rather than silently producing question marks.
• Mojibake repair reverses a specific accident: UTF-8 bytes that were read as Latin-1 or Windows-1252. Each character is mapped back to the byte it was mis-read from, and that byte string is decoded as UTF-8 again. Text that went through the mistake twice is repaired in two passes.
• Normalization converts between composed and decomposed forms. NFC stores é as one code point, NFD as e plus a combining accent — identical on screen, different in a database comparison.
• Because the base is ambiguous, the decoder chooses hex or decimal once for the whole input rather than per value, and tells you which it used.

Use cases

Why é appears instead of é

This is the most common encoding failure there is, and it has one cause. In UTF-8 the character é is stored as two bytes, C3 and A9. If a program reads those bytes one at a time as Latin-1 or Windows-1252 — one byte per character — it displays them as the two separate characters à and ©. Nothing has been lost. The bytes are still exactly what they were; only the interpretation is wrong, which is why the repair is deterministic rather than guesswork.

The same mechanism explains the other familiar artefacts. A right single quotation mark becomes ’ because its three UTF-8 bytes map to those three Windows-1252 characters. An em dash becomes —. A non-breaking space becomes  followed by a space, which is why mangled text often has stray  characters sprinkled through it. Recognising the pattern is enough to know it is recoverable.

Repair works by reversing the steps: map each character back to the single byte it was mis-read from, then decode that byte string as UTF-8 properly. Text that passed through the mistake twice needs two passes, which happens more often than you would expect when data moves between several systems. The tool reports how many passes it applied.

When mojibake cannot be repaired

The repair depends on the original bytes surviving. They usually do, because the mis-decoding is a display and storage problem rather than a lossy conversion. But there are cases where the information is genuinely gone, and it is worth recognising them rather than hunting for a better tool.

If the text passed through a step that replaced unmappable characters with question marks or with the replacement character U+FFFD, the original bytes were discarded at that point. A database column with a restrictive collation, an export that forced ASCII, or a transcoding step configured to substitute rather than fail will all do this. Once a character has become "?" there is no way to know what it was.

The practical implication is to fix the pipeline before re-exporting. If you can regenerate the file from its source with the encoding declared correctly, do that instead of repairing the output — and if you cannot, repair it once and store the result, because running a repair pass over already-correct text can corrupt legitimate characters that happen to look like artefacts.

NFC and NFD: two spellings of the same word

Unicode allows some characters to be written more than one way. The letter é can be the single code point U+00E9, or it can be a plain e followed by a combining acute accent at U+0301. Both render identically. Neither is wrong. But they are different strings, so they do not compare as equal, do not match the same search, and do not produce the same hash.

Normalization resolves this by picking a canonical spelling. NFC composes: it uses the single code point wherever one exists, and this is what you want for storing text, comparing strings, and sending data between systems, because it is what the web and most software expect. NFD decomposes into base characters plus marks, which is useful when stripping accents — decompose, drop the combining marks, and you have the unaccented text — and is what macOS has historically used for filenames.

The K forms, NFKC and NFKD, go further and apply compatibility mappings: a fullwidth Ａ becomes A, a ligature fi becomes fi, and the superscript ² becomes 2. That is destructive in the sense that it loses formatting distinctions, which makes it right for search indexing and wrong for storing what the user typed.

FAQ