Text, Unicode, UTF-8, and locale
Separate storage units, code points, displayed characters, UTF-8 validation, and locale-dependent conversion.
In this chapter8 sections
A learner’s name may contain accented letters, Hindi text, or emoji. Counting storage is only the first question. We also need to know the encoding and which kind of text boundary the application cares about. One visible character does not always occupy one char element.
1. Separate the layers of text
A storage unit is one element used to hold encoded text; for a narrow C string, it is a char-sized C byte. An encoding gives stored sequences their text meaning. Unicode gives characters and other text elements numbered code points. A Unicode scalar value is a code point outside the reserved surrogate range. UTF-8 encodes each scalar value in one to four eight-bit code units. Remember that a C byte itself can have more than eight bits.
An extended grapheme cluster groups text that is often treated as one user-perceived character. It can contain several scalar values. Display width adds another layer: fonts, shaping, and terminal conventions affect the result. Neither strlen nor a simple code-point count measures all of these ideas.
2. The same-looking text can have different counts
#include <assert.h>
#include <string.h>
int main(void)
{
const char composed[] = u8"\u00E9";
const char decomposed[] = u8"e\u0301";
assert(sizeof composed == 3);
assert(strlen(composed) == 2);
assert(sizeof decomposed == 4);
assert(strlen(decomposed) == 3);
return 0;
}
| Source spelling | UTF-8 units / Unicode scalar values | Extended grapheme clusters in this example |
|---|---|---|
u8"\u00E9" | 2 / 1 | 1 |
u8"e\u0301" | 3 / 2 | 1 |
The first literal contains U+00E9. The second contains U+0065 followed by U+0301, a combining acute accent. These forms commonly display as the same accented letter, but their stored sequences differ. sizeof includes the terminating null element; strlen counts units before it. The assertions concern storage, not what a particular terminal displays.
3. Choose the boundary before truncating
A byte limit is useful for storage budgets, but cutting UTF-8 at an arbitrary byte can split a sequence. Cutting only at code-point boundaries can still split a grapheme cluster. Decide whether the requirement limits encoded units, scalar values, grapheme clusters, or display columns. State that unit in the interface and error message.
UTF-8 validation rejects malformed sequences, overlong encodings, surrogate values, and values beyond U+10FFFF. Validation does not provide normalization, sorting by a language’s conventions, or grapheme segmentation. For substantial text work, select a suitable Unicode library and follow its versioned contracts rather than inventing a partial decoder.
4. Locale-dependent conversion is a different contract
C starts with the "C" locale. setlocale(LC_CTYPE, "") requests the environment’s native character locale and can fail; it does not promise UTF-8. The current locale affects multibyte conversion functions. Changing global locale while other code depends on it needs coordination. An explicitly UTF-8 format should be described independently of an assumed user locale.
#include <assert.h>
#include <stddef.h>
#include <wchar.h>
int main(void)
{
/* The unchanged startup locale is the C locale. */
mbstate_t state = {0};
wchar_t result = 0;
size_t consumed = mbrtowc(&result, "A", 1, &state);
assert(consumed == 1 && result == L'A');
return 0;
}
mbrtowc uses an mbstate_t conversion state. A result from one through the supplied input length reports the number of C bytes consumed. Zero reports a null character. The special values (size_t)-2 and (size_t)-1 report incomplete input and an encoding error, respectively. Keep state across the appropriate input pieces. After an encoding error the state is unspecified; recovery needs a deliberate policy. The basic-character example avoids assuming the machine has a particular installed locale.
5. A wide character is not a universal Unicode character
wchar_t is implementation-defined in its encoding and range. Do not assume it is always UTF-16, UTF-32, or one complete displayed character. wcslen counts wide-character units before a wide null. It is not a universal human-perceived character counter. Wide and narrow stream orientation also have rules; converting storage is different from mixing output APIs.
Does successfully validating UTF-8 prove that taking its first ten bytes preserves ten visible characters?
Read the explanation
No. The limit may split a multi-unit scalar value or a multi-scalar grapheme cluster. Choose and enforce the intended text boundary separately.
Primary references and further reading
- WG14 N1570 — public C11 draft: 5.2.1, 6.4.5, 7.11, 7.29.6.3.2: C17 baseline text and locale contracts.
- WG14 N3096 — public C23 draft: 6.4.5 and 7.30: character types and Unicode conversions.
- RFC 3629 — UTF-8: Sections 3–4: valid encoded sequences.
- Unicode UAX #29: Extended grapheme cluster boundaries; Unicode segmentation is separate from C storage counting.
Before you move on
- Storage units, Unicode scalar values, grapheme clusters, and display width measure different things.
- UTF-8 validation does not automatically provide normalization or grapheme boundaries.
- Locale-dependent C conversions require their stated state and error-handling rules.
Keep these ideas separate
- Byte limit / Text boundary
- A byte cut can split an encoding sequence; a code-point cut can still split a grapheme cluster.
Why might two strings showing the same accented letter have different strlen results?
Optional: say it aloud or keep a note. You can continue whenever you are ready.Check your understanding
Optional. Answers are not saved or sent, and reading stays unrestricted.
Read the explanation for every choice
- No. It counts char elements before the first null, not grapheme clusters.
✓ Correct reasoning. UTF-8 code points can use several bytes, and one visible unit can involve several code points.
- Yes, every visible letter is exactly one C char.
That is not a general Unicode or UTF-8 guarantee.
- It automatically decodes and normalizes all Unicode text.
strlen searches for the terminating null; it is not a Unicode normalization operation.
C17
Ready for the next idea?
Mark this chapter as read whenever you feel ready.