Unicode vs UTF-8: What They Are and Why Text Gets Garbled

A support ticket once landed on my desk with a screenshot showing a customer's name rendered as 'José' instead of 'José', and the reporting agent's proposed fix was to manually retype every affected name in the database by hand. That would have worked for the fifty visible cases and completely missed the several thousand more sitting quietly in fields nobody had looked at yet, because the actual cause wasn't a typo — it was a single character encoding mismatch applied consistently across an entire import, and understanding what encoding even is turns that fifty-row manual fix into a one-line database correction.

Numbers are all a computer actually stores

Every letter, symbol, and emoji you see on a screen is, underneath, just a number, and a character encoding is the agreed table that maps those numbers back to visible characters. The letter 'A' being represented by the number 65 isn't a fundamental law — it's a convention, specifically part of the ASCII standard drawn up in the early 1960s. ASCII only ever accounted for 128 characters, which was fine for plain English text but left no room for accented letters, currency symbols, or the thousands of other scripts and symbols people actually need to write.

Unicode is the master list; UTF-8 is one way of writing it down

This distinction confuses almost everyone at first. Unicode is an enormous, continually expanding catalog assigning a unique reference number — a code point — to every character anyone has requested support for, from Latin letters to Cyrillic to emoji. UTF-8 is a specific method for translating those reference numbers into actual bytes that get stored or transmitted. Other methods exist too, like UTF-16 or the older Latin-1, but UTF-8 has become the dominant default across the web mostly because it stays backward-compatible with plain ASCII and keeps common English text compact.

The exact mechanism behind garbled text

Mojibake happens when bytes written under one encoding scheme get interpreted using a different scheme's rulebook. The letter 'é' is stored in UTF-8 as two specific bytes; a program that mistakenly reads those same two bytes as Latin-1 instead sees two separate, unrelated characters rather than one combined letter, and the result is exactly the doubled, garbled text pattern that shows up in poorly configured emails, web pages missing an explicit charset declaration, or spreadsheet exports opened with the wrong assumed encoding.

Why 'one character' isn't always as simple as it looks

A single visible glyph on screen isn't guaranteed to be a single underlying code point. Some accented letters can be stored either as one combined code point or as a plain base letter followed by a separate combining accent mark — two genuinely different byte sequences that render identically to a human but compare as unequal in code performing a naive string match. This is a real, recurring source of bugs in search boxes and deduplication logic, which is why serious text-processing tools include explicit normalization functions to force text into one consistent representation before comparing it.

The everyday situations where this actually bites

You don't have to write any parsing code to run into this. Curly quotes pasted from a word processor into a plain-text field, an old accounting system's export, a URL containing non-English characters, or a webpage missing its charset meta tag can all produce visible corruption. When it does, checking a suspicious string's actual byte content — running it through a base64 encoder, for instance, or viewing raw bytes directly — tells you quickly whether the underlying data is intact and only the display is wrong, versus a genuine loss where the original characters no longer exist anywhere.

A related but separate problem: percent-encoding in URLs

URLs can't safely contain certain characters directly — spaces, ampersands, non-ASCII letters — so those get percent-encoded instead, with a space becoming %20. This is a different mechanism from the UTF-8-versus-Latin-1 confusion described above, but the two problems frequently get tangled together in the same bug report, since a non-English character typed into a form field needs correct text encoding first and correct URL encoding on top of that. Running a suspicious link through a URL encoder / decoder is the fastest way to tell whether a broken-looking link is a percent-encoding issue or something deeper.

Habits that prevent most of this from ever happening

Default every new file, database column, and API response to UTF-8 explicitly rather than relying on whatever the operating system happens to assume. Declare the encoding directly in a webpage's head section and in HTTP response headers instead of leaving it to inference. When text misbehaves, check the raw bytes before assuming the underlying data itself is corrupted — often only the interpretation is wrong, and re-reading the exact same bytes with the correct encoding restores everything perfectly. And normalize text before any programmatic comparison or deduplication, so visually identical characters are guaranteed to be treated as equal.

Answers to the questions people actually type into search

Why do some emoji show up as boxes or question marks instead of the actual symbol? That's usually a font or rendering environment that lacks a glyph for that specific code point — the underlying data is often perfectly fine, the display simply can't draw it.

Is UTF-8 the right default for basically any new project? For nearly all modern web and application work, yes — it's the closest thing to a universal standard, supports every written language, and stays compact for English-heavy content.

Can mojibake permanently destroy the original data? Sometimes — if software re-saves misread text using the wrong encoding rather than just displaying it wrong, the original bytes can genuinely be lost, which is why diagnosing the encoding before any bulk re-save is worth the extra few minutes.

What's the fastest way to check a file's actual encoding? Viewing raw bytes near the top of the file in a plain text or hex editor often reveals an obvious byte-order mark or clearly wrong sequence immediately; dedicated encoding-detection tools handle the more ambiguous cases visual inspection can't resolve confidently.