Catalog My learning Blog
gRPC and Protobuf Tutorial: Fast Microservice APIs · Phase 2 · Lesson 11 · 3 min read

Protobuf Field Numbers and Wire Format Explained

On this page
  1. Decoding real protobuf bytes
  2. Protobuf wire format: tags, wire types and varints
  3. What the wire format means for schema design
  4. Protobuf compactness, measured
  5. Check yourself

Decoding real protobuf bytes

Understanding the encoding explains most protobuf rules: why field numbers matter forever, why some changes are safe and others aren't, and why some types are cheaper. Here's a small message:

Protobuf
message HelloRequest {
  string name = 1;
  int32 age = 2;
  bool vip = 16;
}

Encoded with protoc --encode and dumped with xxd:

YAML
name: "Al"                       0a 02 41 6c
name: "Al" age: 150              0a 02 41 6c 10 96 01
name: "Al" age: 150 vip: true    0a 02 41 6c 10 96 01 80 01 01
name: "" age: 0                  (0 bytes)

Protobuf wire format: tags, wire types and varints

Each field is a tag followed by a value. The tag is a varint: (field_number << 3) | wire_type, which is field number × 8 + wire type.

BytesMeaning
0aField 1, wire type 2 (length-delimited): 1 × 8 + 2 = 10
02Length: 2 bytes follow
41 6c"Al" in UTF-8
10Field 2, wire type 0 (varint): 2 × 8 + 0 = 16
96 01Varint 150: 7 bits per byte, low bits first, high bit means "more bytes"
80 01Field 16, wire type 0: 16 × 8 = 128, which needs two bytes as a varint
01true

Wire types: 0 varint (ints, bools, enums), 1 fixed 64-bit, 2 length-delimited (strings, bytes, nested messages, packed repeated fields), 5 fixed 32-bit.

protoc --decode_raw decodes bytes without the schema, showing only numbers, which is all the wire contains:

YAML
1: "Al"
2: 150
16: 1
Each field is encoded as a tag (field number and wire type) followed by its value; names never appear, defaults are omitted, and unknown fields can be skipped by their wire type.
Each field is encoded as a tag (field number and wire type) followed by its value; names never appear, defaults are omitted, and unknown fields can be skipped by their wire type.

What the wire format means for schema design

  • Field numbers are the identity of a field. Names don't travel. Rename a field freely (for the binary format; JSON uses names, Day 12), but never change or reuse a number.
  • Numbers 1 to 15 take one byte of tag; 16 to 2047 take two. Give the smallest numbers to the most frequent fields.
  • Defaults aren't encoded: an empty string and a zero age produced zero bytes. Cheap, and the reason presence needs optional (Day 3).
  • Unknown fields can be skipped: a reader that doesn't know field 16 reads its wire type, skips the right number of bytes, and continues. That's what makes adding fields safe.
  • Order isn't significant and messages can be concatenated: parsing two encoded messages one after the other merges them, the last value winning for single fields (Day 21 shows this with oneof).
  • Negative int32 values cost 10 bytes in the value (Day 3): use sint32 for signed data.

Protobuf compactness, measured

The same data as JSON is {"name":"Al","age":150,"vip":true}: 34 bytes against 10. Real messages typically shrink by a third to a fifth versus JSON, and parse faster. With compression on top (gRPC supports gzip per call), the gap narrows for large text-heavy messages, but parsing cost stays lower.

Check yourself

It's the tag: field number 1 with wire type 2, length-delimited, since (1 << 3) | 2 = 10, which is 0a in hex. The next byte gives the length of the string or nested message.

The tag is (field_number << 3) | wire_type. For field 16 that's 128, which doesn't fit in one varint byte's 7 bits, so it needs two. Fields 1 to 15 fit in one byte, so give them to frequently set fields.

It reads the tag's wire type, which says how long the value is, skips the value, and keeps the bytes as an unknown field, so re-serializing the message can preserve them.

Numeric repeated fields are written as one length-delimited block of values instead of a tag before each value. Proto3 packs them by default.

Partly: protoc --decode_raw shows field numbers, wire types and values, but not field names or exact types, since an int and a bool look the same on the wire. It's useful for debugging, but you need the schema to interpret the data correctly.