Text, Unicode, UTF-8 aur locale
Storage units, code points, displayed characters, UTF-8 validation aur locale conversion alag samjhein.
Is chapter mein8 sections
Learner ke naam mein accented letters, Hindi text ya emoji ho sakte hain. Storage count karna first question hai. Encoding aur application ke needed text boundaries bhi jaanni hain. Ek visible character hamesha ek char element occupy nahi karta.
1. Text ki layers alag karein
Storage unit encoded text hold karne ka ek element hai; narrow C string mein yeh char-sized C byte hai. Encoding stored sequence ko text ka meaning deti hai. Unicode characters aur doosre text elements ko numbered code points deta hai. Reserved surrogate range ke bahar ka code point Unicode scalar value hai. UTF-8 har scalar value ko one se four eight-bit code units mein encode karta hai. C byte khud eight se zyada bits ki ho sakti hai.
Extended grapheme cluster aise text ko group karta hai jo often ek user-perceived character maana jata hai. Ismein kai scalar values ho sakti hain. Display width alag layer hai: fonts, shaping aur terminal conventions result affect karte hain. strlen ya simple code-point count in sabko measure nahi karte.
2. Similar dikhne wale text ke counts alag ho sakte hain
#include <assert.h>
#include <string.h>
int main(void)
{
const char composed[] = u8"\u00E9";
const char decomposed[] = u8"e\u0301";
assert(sizeof composed == 3);
assert(strlen(composed) == 2);
assert(sizeof decomposed == 4);
assert(strlen(decomposed) == 3);
return 0;
}
| Source spelling | UTF-8 units / Unicode scalar values | Is example ke extended grapheme clusters |
|---|---|---|
u8"\u00E9" | 2 / 1 | 1 |
u8"e\u0301" | 3 / 2 | 1 |
First literal mein U+00E9 hai. Second mein U+0065 aur combining acute accent U+0301 hai. Yeh forms commonly same accented letter dikhate hain, lekin stored sequences alag hain. sizeof terminating null element include karta hai; strlen usse pehle ke units count karta hai. Assertions storage ke hain, terminal display ke nahi.
3. Truncate karne se pehle boundary choose karein
Storage budget ke liye byte limit useful hai, lekin arbitrary byte par UTF-8 cut karne se sequence toot sakti hai. Code-point boundary par cut karne se bhi grapheme cluster toot sakta hai. Decide karein ki limit encoded units, scalar values, grapheme clusters ya display columns ki hai. Interface aur error mein unit clear rakhein.
UTF-8 validation malformed sequences, overlong encodings, surrogate values aur U+10FFFF se aage ki values reject karti hai. Validation normalization, language-based sorting ya grapheme segmentation nahi deti. Substantial text work ke liye suitable Unicode library aur uske versioned contracts use karein; partial decoder invent na karein.
4. Locale conversion ka contract alag hai
C startup mein "C" locale use karti hai. setlocale(LC_CTYPE, "") environment ki native character locale request karta hai aur fail ho sakta hai; UTF-8 ka promise nahi. Current locale multibyte conversions affect karti hai. Doosra code depend kare toh global locale changes coordinate karein. Explicit UTF-8 format ko assumed user locale se alag describe karein.
#include <assert.h>
#include <stddef.h>
#include <wchar.h>
int main(void)
{
/* The unchanged startup locale is the C locale. */
mbstate_t state = {0};
wchar_t result = 0;
size_t consumed = mbrtowc(&result, "A", 1, &state);
assert(consumed == 1 && result == L'A');
return 0;
}
mbrtowc mbstate_t conversion state use karta hai. One se supplied input length tak ka result consumed C bytes batata hai. Zero null character batata hai. Special values (size_t)-2 incomplete input aur (size_t)-1 encoding error batati hain. Appropriate input pieces ke beech state rakhein. Encoding error ke baad state unspecified hai; deliberate recovery policy chahiye. Basic-character example kisi installed locale ko assume nahi karta.
5. Wide character universal Unicode character nahi hai
wchar_t ki encoding aur range implementation-defined hain. Ise hamesha UTF-16, UTF-32 ya complete displayed character na samjhein. wcslen wide null se pehle wide-character units count karta hai. Yeh universal human-perceived character counter nahi. Wide aur narrow stream orientation ke rules bhi hain; storage conversion aur output APIs mix karna alag hain.
UTF-8 successfully validate hone par kya first ten bytes lena ten visible characters preserve karta hai?
Explanation padhein
Nahi. Limit multi-unit scalar value ya multi-scalar grapheme cluster tod sakti hai. Intended text boundary alag choose aur enforce karein.
Primary references aur aage padhne ke links
- WG14 N1570 — public C11 draft: 5.2.1, 6.4.5, 7.11, 7.29.6.3.2: C17 baseline text and locale contracts.
- WG14 N3096 — public C23 draft: 6.4.5 and 7.30: character types and Unicode conversions.
- RFC 3629 — UTF-8: Sections 3–4: valid encoded sequences.
- Unicode UAX #29: Extended grapheme cluster boundaries; Unicode segmentation is separate from C storage counting.
Aage badhne se pehle
- Storage units, Unicode scalar values, grapheme clusters aur display width alag measurements hain.
- UTF-8 validation automatically normalization ya grapheme boundaries nahi deti.
- Locale-dependent C conversions ko stated state aur error rules follow karne hote hain.
In ideas ko alag rakhein
- Byte limit / Text boundary
- Byte cut encoding sequence tod sakta hai; code-point cut bhi grapheme cluster tod sakta hai.
Same accented letter dikhane wali do strings ka strlen different kyun ho sakta hai?
Optional: bolkar samjhein ya note rakhein. Tayyar hon toh aage badh sakte hain.Apni understanding check karein
Optional hai. Answers save ya send nahi hote; chapter kabhi lock nahi hota.
Har choice ki explanation padhein
- Nahi. Yeh first null se pehle char elements count karta hai, grapheme clusters nahi.
✓ Sahi reasoning. UTF-8 code points kai bytes use kar sakte hain aur ek visible unit mein kai code points ho sakte hain.
- Haan, har visible letter exactly ek C char hai.
Yeh general Unicode ya UTF-8 guarantee nahi.
- Yeh automatically saara Unicode text decode aur normalize karta hai.
strlen terminating null dhoondhta hai; Unicode normalization operation nahi.
C17
Agla idea padhne ke liye tayyar?
Mark this chapter as read whenever you feel ready.