External data formats: an agreement about bytes
Design an explicit record format, validate its boundaries and version, and separate serialization from raw object storage.
In this chapter8 sections
A saved file may be read by another build, another language, or another computer. Those readers need an agreed format. Serialization is the process of encoding logical values into that format; decoding reconstructs and validates the values. The format is an interface across time and systems.
1. Object storage is not automatically a file format
Writing a structure with fwrite copies its object representation. That can include padding and implementation-specific member encodings. Byte order, integer width, floating formats, and layout can differ. Pointer values refer to a program’s runtime context; saving those bytes does not save the pointed-to data or recreate a usable pointer in another run.
A raw dump can be useful under a deliberately narrow, matching environment contract. A portable external format needs its own field definitions and validation. A filename extension such as .bin does not supply those definitions. Binary mode controls stream behavior; it does not standardise object representations.
2. Write the format before the encoder
Our teaching format stores one count in exactly five octets. An octet has exactly eight bits. We explicitly require CHAR_BIT == 8 so one array element matches one format octet. The two value octets store the most significant part first, commonly called big-endian order. The high octet of 256 is one and the low octet is zero.
| Octet positions | Stored values (hexadecimal) | Meaning |
|---|---|---|
| 0–1 | 43 52 | Format identification |
| 2 | 01 | Version 1 |
| 3–4 | high low | Unsigned value 0–65535, most significant octet first |
The identification and version octets help a decoder reject unrelated or unsupported records. They are not authentication, integrity protection, or a guarantee against malicious input. Real formats also need policies for unknown versions, maximum lengths, truncated records, and later schema changes.
3. Validate first, then commit
#include <assert.h>
#include <stdbool.h>
#include <stddef.h>
#include <limits.h>
#include <string.h>
_Static_assert(CHAR_BIT == 8, "This format requires eight-bit C bytes");
/* The record is exactly five octets: 43 52 01 high low. */
static bool decode_count(const unsigned char *source, size_t size, unsigned *result)
{
if (source == NULL || result == NULL || size != 5) return false;
if (source[0] != 0x43 || source[1] != 0x52 || source[2] != 1) return false;
unsigned value = ((unsigned)source[3] << 8) | (unsigned)source[4];
*result = value;
return true;
}
/* destination provides capacity writable bytes. Failure leaves it unchanged. */
static bool encode_count(unsigned value, unsigned char *destination, size_t capacity)
{
if (destination == NULL || capacity < 5 || value > 65535U) return false;
unsigned char record[5] = {
0x43, 0x52, 1, (unsigned char)(value >> 8), (unsigned char)(value & 255U)
};
memcpy(destination, record, sizeof record);
return true;
}
int main(void)
{
const unsigned values[] = {0U, 255U, 256U, 65535U};
for (size_t i = 0; i < sizeof values / sizeof values[0]; ++i) {
unsigned char record[5] = {0};
unsigned result = 17U;
assert(encode_count(values[i], record, sizeof record));
assert(decode_count(record, sizeof record, &result));
assert(result == values[i]);
}
unsigned char encoded[5] = {0};
const unsigned char expected[5] = {0x43, 0x52, 1, 1, 0};
assert(encode_count(256U, encoded, sizeof encoded));
assert(memcmp(encoded, expected, sizeof expected) == 0);
unsigned char record[6] = {0x43, 0x52, 1, 1, 0, 0};
unsigned result = 17U;
for (size_t size = 0; size < 5; ++size) {
assert(!decode_count(record, size, &result));
assert(result == 17U);
}
assert(!decode_count(record, 6, &result));
record[2] = 2;
assert(!decode_count(record, 5, &result) && result == 17U);
record[2] = 1;
record[0] = 0;
assert(!decode_count(record, 5, &result) && result == 17U);
assert(!decode_count(NULL, 5, &result));
assert(!decode_count(record, 5, NULL));
assert(!encode_count(1U, NULL, 5));
unsigned char saved[6];
memcpy(saved, record, sizeof saved);
assert(!encode_count(1U, record, 4));
assert(memcmp(saved, record, sizeof saved) == 0);
#if UINT_MAX > 65535U
assert(!encode_count(65536U, record, sizeof record));
assert(memcmp(saved, record, sizeof saved) == 0);
#endif
return 0;
}
decode_count checks the pointers, exact record length, identification, and version before assigning the output. An invalid record leaves the earlier result unchanged. The encoder checks capacity and range before copying any bytes. These are transactional update policies, not automatic properties of every encoder.
The cast to unsigned occurs before the left shift. It prevents a small unsigned char from being promoted to a signed int whose range could be too small for the shifted value on a 16-bit implementation. Every standard unsigned int can represent 0 through 65535, matching this field.
4. A size argument does not verify accessible storage
The pointer and size checks do not discover how much storage really exists. A non-null source must designate the stated readable region, and a non-null destination must provide the stated writable capacity. The caller must establish those facts. Decode only bytes that a successful input operation actually provided.
For variable-length records, validate lengths before allocation and check size arithmetic before adding headers or multiplying counts. Separate complete, incomplete, and invalid input states. A short network or file read is not permission to interpret uninitialised trailing storage as the rest of a record.
5. Text formats also need agreements
Text can make inspection easier, but it still needs a grammar, encoding, numeric range, escaping rules, and limits. Decide whether decimal separators are fixed or locale-dependent. Decide how embedded delimiters and newlines are represented. A standard format and a suitable parser can be a better choice than inventing a format for each project.
Does opening a file with "wb" make writing a structure a portable serialization scheme?
Read the explanation
No. Binary mode does not define the structure’s layout, padding, member representation, or meaning. Specify and encode the external fields explicitly.
Primary references and further reading
- WG14 N1570 — public C11 draft: 6.2.6.1, 6.3.1.1, 6.5.7, 7.21.2 and 7.21.8: representations, promotions, shifts, streams, and transfers.
- RFC 4506 — External Data Representation: A primary example of an explicitly specified external representation; the small teaching format above is original and is not XDR.
Before you move on
- An external format defines fields and encodings independently of a structure’s raw layout.
- Width, octet order, version, and length rules belong in the reader/writer agreement.
- Validate the complete record before committing results, and handle partial input explicitly.
Keep these ideas separate
- Binary stream / Portable serialization
- Binary mode does not standardise padding, member representations, pointer values, or a schema.
Why is an exact expected-byte test useful in addition to an encoder/decoder round trip?
Optional: say it aloud or keep a note. You can continue whenever you are ready.Check your understanding
Optional. Answers are not saved or sent, and reading stays unrestricted.
Read the explanation for every choice
- No. Layout, padding, byte order, and representations still need an explicit format agreement.
✓ Correct reasoning. Define the fields and encoding, then serialize and validate them deliberately.
- Yes, binary mode standardizes every structure’s layout.
Binary mode does not impose a universal structure representation.
- Yes, sizeof alone describes the complete external protocol.
A local byte size does not specify the meanings or encoding of those bytes.
C17
Ready for the next idea?
Mark this chapter as read whenever you feel ready.