YOUR C LEARNING COMPANION

Find your next idea.

Learning path
Memory & dataChapter 27 of 73

Text, Unicode, UTF-8, and locale

Separate storage units, code points, displayed characters, UTF-8 validation, and locale-dependent conversion.

5 min readDevelopingC17 baseline
Offline reading
In this chapter8 sections

A learner’s name may contain accented letters, Hindi text, or emoji. Counting storage is only the first question. We also need to know the encoding and which kind of text boundary the application cares about. One visible character does not always occupy one char element.

1. Separate the layers of text

A storage unit is one element used to hold encoded text; for a narrow C string, it is a char-sized C byte. An encoding gives stored sequences their text meaning. Unicode gives characters and other text elements numbered code points. A Unicode scalar value is a code point outside the reserved surrogate range. UTF-8 encodes each scalar value in one to four eight-bit code units. Remember that a C byte itself can have more than eight bits.

An extended grapheme cluster groups text that is often treated as one user-perceived character. It can contain several scalar values. Display width adds another layer: fonts, shaping, and terminal conventions affect the result. Neither strlen nor a simple code-point count measures all of these ideas.

2. The same-looking text can have different counts

Complete C17 program
#include <assert.h>
#include <string.h>

int main(void)
{
    const char composed[] = u8"\u00E9";
    const char decomposed[] = u8"e\u0301";
    assert(sizeof composed == 3);
    assert(strlen(composed) == 2);
    assert(sizeof decomposed == 4);
    assert(strlen(decomposed) == 3);
    return 0;
}
Source spellingUTF-8 units / Unicode scalar valuesExtended grapheme clusters in this example
u8"\u00E9"2 / 11
u8"e\u0301"3 / 21

The first literal contains U+00E9. The second contains U+0065 followed by U+0301, a combining acute accent. These forms commonly display as the same accented letter, but their stored sequences differ. sizeof includes the terminating null element; strlen counts units before it. The assertions concern storage, not what a particular terminal displays.

3. Choose the boundary before truncating

A byte limit is useful for storage budgets, but cutting UTF-8 at an arbitrary byte can split a sequence. Cutting only at code-point boundaries can still split a grapheme cluster. Decide whether the requirement limits encoded units, scalar values, grapheme clusters, or display columns. State that unit in the interface and error message.

UTF-8 validation rejects malformed sequences, overlong encodings, surrogate values, and values beyond U+10FFFF. Validation does not provide normalization, sorting by a language’s conventions, or grapheme segmentation. For substantial text work, select a suitable Unicode library and follow its versioned contracts rather than inventing a partial decoder.

4. Locale-dependent conversion is a different contract

C starts with the "C" locale. setlocale(LC_CTYPE, "") requests the environment’s native character locale and can fail; it does not promise UTF-8. The current locale affects multibyte conversion functions. Changing global locale while other code depends on it needs coordination. An explicitly UTF-8 format should be described independently of an assumed user locale.

Complete C17 program
#include <assert.h>
#include <stddef.h>
#include <wchar.h>

int main(void)
{
    /* The unchanged startup locale is the C locale. */
    mbstate_t state = {0};
    wchar_t result = 0;
    size_t consumed = mbrtowc(&result, "A", 1, &state);
    assert(consumed == 1 && result == L'A');
    return 0;
}

mbrtowc uses an mbstate_t conversion state. A result from one through the supplied input length reports the number of C bytes consumed. Zero reports a null character. The special values (size_t)-2 and (size_t)-1 report incomplete input and an encoding error, respectively. Keep state across the appropriate input pieces. After an encoding error the state is unspecified; recovery needs a deliberate policy. The basic-character example avoids assuming the machine has a particular installed locale.

5. A wide character is not a universal Unicode character

wchar_t is implementation-defined in its encoding and range. Do not assume it is always UTF-16, UTF-32, or one complete displayed character. wcslen counts wide-character units before a wide null. It is not a universal human-perceived character counter. Wide and narrow stream orientation also have rules; converting storage is different from mixing output APIs.

Think it through

Does successfully validating UTF-8 prove that taking its first ten bytes preserves ten visible characters?

Read the explanation

No. The limit may split a multi-unit scalar value or a multi-scalar grapheme cluster. Choose and enforce the intended text boundary separately.

Primary references and further reading

Before you move on

  • Storage units, Unicode scalar values, grapheme clusters, and display width measure different things.
  • UTF-8 validation does not automatically provide normalization or grapheme boundaries.
  • Locale-dependent C conversions require their stated state and error-handling rules.
Keep these ideas separate
Byte limit / Text boundary
A byte cut can split an encoding sequence; a code-point cut can still split a grapheme cluster.
Explain it in your own words

Why might two strings showing the same accented letter have different strlen results?

Optional: say it aloud or keep a note. You can continue whenever you are ready.

Check your understanding

Optional. Answers are not saved or sent, and reading stays unrestricted.

For a valid null-terminated UTF-8 string in an 8-bit-byte char buffer, does strlen count visible letters?
Read the explanation for every choice
  1. No. It counts char elements before the first null, not grapheme clusters.

    ✓ Correct reasoning. UTF-8 code points can use several bytes, and one visible unit can involve several code points.

  2. Yes, every visible letter is exactly one C char.

    That is not a general Unicode or UTF-8 guarantee.

  3. It automatically decodes and normalizes all Unicode text.

    strlen searches for the terminating null; it is not a Unicode normalization operation.

C17

Ready for the next idea?

Mark this chapter as read whenever you feel ready.

CHAPTER 27 / 73

In this chapter

Text, Unicode, UTF-8, and locale

Search all 73 chapters in both languages.EN + HG

Try “pointer”, “why int”, or “undefined behavior”.

Concepts connect. Follow your curiosity.↑ ↓ to explore · Enter to open
MAKE YOURSELF COMFORTABLE

Your reading space

Appearance

A clear canvas with balanced contrast.

Text size

100%
AA

Reading tools

Your data stays in this browser

Export a backup before changing browsers or replacing an older version of the reader.

YOUR PERSONAL MARGINS

My notebook

Saved on this device
A LITTLE CLARITY

The glossary

A starting meaning, then a chapter to go deeper.

A QUICK DEFINITION

A little clarity

Read the full explanation →

A thought worth keeping