YOUR C LEARNING COMPANION

Find your next idea.

Learning path
Correctness & careChapter 64 of 73

External data formats: an agreement about bytes

Design an explicit record format, validate its boundaries and version, and separate serialization from raw object storage.

6 min readDevelopingC17 baseline
Offline reading
In this chapter8 sections

A saved file may be read by another build, another language, or another computer. Those readers need an agreed format. Serialization is the process of encoding logical values into that format; decoding reconstructs and validates the values. The format is an interface across time and systems.

1. Object storage is not automatically a file format

Writing a structure with fwrite copies its object representation. That can include padding and implementation-specific member encodings. Byte order, integer width, floating formats, and layout can differ. Pointer values refer to a program’s runtime context; saving those bytes does not save the pointed-to data or recreate a usable pointer in another run.

A raw dump can be useful under a deliberately narrow, matching environment contract. A portable external format needs its own field definitions and validation. A filename extension such as .bin does not supply those definitions. Binary mode controls stream behavior; it does not standardise object representations.

2. Write the format before the encoder

Our teaching format stores one count in exactly five octets. An octet has exactly eight bits. We explicitly require CHAR_BIT == 8 so one array element matches one format octet. The two value octets store the most significant part first, commonly called big-endian order. The high octet of 256 is one and the low octet is zero.

Octet positionsStored values (hexadecimal)Meaning
0–143 52Format identification
201Version 1
3–4high lowUnsigned value 0–65535, most significant octet first

The identification and version octets help a decoder reject unrelated or unsupported records. They are not authentication, integrity protection, or a guarantee against malicious input. Real formats also need policies for unknown versions, maximum lengths, truncated records, and later schema changes.

3. Validate first, then commit

Complete C17 program
#include <assert.h>
#include <stdbool.h>
#include <stddef.h>
#include <limits.h>
#include <string.h>

_Static_assert(CHAR_BIT == 8, "This format requires eight-bit C bytes");

/* The record is exactly five octets: 43 52 01 high low. */
static bool decode_count(const unsigned char *source, size_t size, unsigned *result)
{
    if (source == NULL || result == NULL || size != 5) return false;
    if (source[0] != 0x43 || source[1] != 0x52 || source[2] != 1) return false;
    unsigned value = ((unsigned)source[3] << 8) | (unsigned)source[4];
    *result = value;
    return true;
}

/* destination provides capacity writable bytes. Failure leaves it unchanged. */
static bool encode_count(unsigned value, unsigned char *destination, size_t capacity)
{
    if (destination == NULL || capacity < 5 || value > 65535U) return false;
    unsigned char record[5] = {
        0x43, 0x52, 1, (unsigned char)(value >> 8), (unsigned char)(value & 255U)
    };
    memcpy(destination, record, sizeof record);
    return true;
}

int main(void)
{
    const unsigned values[] = {0U, 255U, 256U, 65535U};
    for (size_t i = 0; i < sizeof values / sizeof values[0]; ++i) {
        unsigned char record[5] = {0};
        unsigned result = 17U;
        assert(encode_count(values[i], record, sizeof record));
        assert(decode_count(record, sizeof record, &result));
        assert(result == values[i]);
    }
    unsigned char encoded[5] = {0};
    const unsigned char expected[5] = {0x43, 0x52, 1, 1, 0};
    assert(encode_count(256U, encoded, sizeof encoded));
    assert(memcmp(encoded, expected, sizeof expected) == 0);
    unsigned char record[6] = {0x43, 0x52, 1, 1, 0, 0};
    unsigned result = 17U;
    for (size_t size = 0; size < 5; ++size) {
        assert(!decode_count(record, size, &result));
        assert(result == 17U);
    }
    assert(!decode_count(record, 6, &result));
    record[2] = 2;
    assert(!decode_count(record, 5, &result) && result == 17U);
    record[2] = 1;
    record[0] = 0;
    assert(!decode_count(record, 5, &result) && result == 17U);
    assert(!decode_count(NULL, 5, &result));
    assert(!decode_count(record, 5, NULL));
    assert(!encode_count(1U, NULL, 5));
    unsigned char saved[6];
    memcpy(saved, record, sizeof saved);
    assert(!encode_count(1U, record, 4));
    assert(memcmp(saved, record, sizeof saved) == 0);
#if UINT_MAX > 65535U
    assert(!encode_count(65536U, record, sizeof record));
    assert(memcmp(saved, record, sizeof saved) == 0);
#endif
    return 0;
}

decode_count checks the pointers, exact record length, identification, and version before assigning the output. An invalid record leaves the earlier result unchanged. The encoder checks capacity and range before copying any bytes. These are transactional update policies, not automatic properties of every encoder.

The cast to unsigned occurs before the left shift. It prevents a small unsigned char from being promoted to a signed int whose range could be too small for the shifted value on a 16-bit implementation. Every standard unsigned int can represent 0 through 65535, matching this field.

4. A size argument does not verify accessible storage

The pointer and size checks do not discover how much storage really exists. A non-null source must designate the stated readable region, and a non-null destination must provide the stated writable capacity. The caller must establish those facts. Decode only bytes that a successful input operation actually provided.

For variable-length records, validate lengths before allocation and check size arithmetic before adding headers or multiplying counts. Separate complete, incomplete, and invalid input states. A short network or file read is not permission to interpret uninitialised trailing storage as the rest of a record.

5. Text formats also need agreements

Text can make inspection easier, but it still needs a grammar, encoding, numeric range, escaping rules, and limits. Decide whether decimal separators are fixed or locale-dependent. Decide how embedded delimiters and newlines are represented. A standard format and a suitable parser can be a better choice than inventing a format for each project.

Think it through

Does opening a file with "wb" make writing a structure a portable serialization scheme?

Read the explanation

No. Binary mode does not define the structure’s layout, padding, member representation, or meaning. Specify and encode the external fields explicitly.

Primary references and further reading

Before you move on

  • An external format defines fields and encodings independently of a structure’s raw layout.
  • Width, octet order, version, and length rules belong in the reader/writer agreement.
  • Validate the complete record before committing results, and handle partial input explicitly.
Keep these ideas separate
Binary stream / Portable serialization
Binary mode does not standardise padding, member representations, pointer values, or a schema.
Explain it in your own words

Why is an exact expected-byte test useful in addition to an encoder/decoder round trip?

Optional: say it aloud or keep a note. You can continue whenever you are ready.

Check your understanding

Optional. Answers are not saved or sent, and reading stays unrestricted.

Does binary file mode make a raw structure dump a portable exchange format?
Read the explanation for every choice
  1. No. Layout, padding, byte order, and representations still need an explicit format agreement.

    ✓ Correct reasoning. Define the fields and encoding, then serialize and validate them deliberately.

  2. Yes, binary mode standardizes every structure’s layout.

    Binary mode does not impose a universal structure representation.

  3. Yes, sizeof alone describes the complete external protocol.

    A local byte size does not specify the meanings or encoding of those bytes.

C17

Ready for the next idea?

Mark this chapter as read whenever you feel ready.

CHAPTER 64 / 73

In this chapter

External data formats: an agreement about bytes

Search all 73 chapters in both languages.EN + HG

Try “pointer”, “why int”, or “undefined behavior”.

Concepts connect. Follow your curiosity.↑ ↓ to explore · Enter to open
MAKE YOURSELF COMFORTABLE

Your reading space

Appearance

A clear canvas with balanced contrast.

Text size

100%
AA

Reading tools

Your data stays in this browser

Export a backup before changing browsers or replacing an older version of the reader.

YOUR PERSONAL MARGINS

My notebook

Saved on this device
A LITTLE CLARITY

The glossary

A starting meaning, then a chapter to go deeper.

A QUICK DEFINITION

A little clarity

Read the full explanation →

A thought worth keeping