Protobuf Field Numbers and Wire Format Explained
On this page
Decoding real protobuf bytes
Understanding the encoding explains most protobuf rules: why field numbers matter forever, why some changes are safe and others aren't, and why some types are cheaper. Here's a small message:
message HelloRequest {
string name = 1;
int32 age = 2;
bool vip = 16;
}Encoded with protoc --encode and dumped with xxd:
name: "Al" 0a 02 41 6c
name: "Al" age: 150 0a 02 41 6c 10 96 01
name: "Al" age: 150 vip: true 0a 02 41 6c 10 96 01 80 01 01
name: "" age: 0 (0 bytes)Protobuf wire format: tags, wire types and varints
Each field is a tag followed by a value. The tag is a varint: (field_number << 3) | wire_type, which is field number × 8 + wire type.
| Bytes | Meaning |
|---|---|
0a | Field 1, wire type 2 (length-delimited): 1 × 8 + 2 = 10 |
02 | Length: 2 bytes follow |
41 6c | "Al" in UTF-8 |
10 | Field 2, wire type 0 (varint): 2 × 8 + 0 = 16 |
96 01 | Varint 150: 7 bits per byte, low bits first, high bit means "more bytes" |
80 01 | Field 16, wire type 0: 16 × 8 = 128, which needs two bytes as a varint |
01 | true |
Wire types: 0 varint (ints, bools, enums), 1 fixed 64-bit, 2 length-delimited (strings, bytes, nested messages, packed repeated fields), 5 fixed 32-bit.
protoc --decode_raw decodes bytes without the schema, showing only numbers, which is all the wire contains:
1: "Al"
2: 150
16: 1What the wire format means for schema design
- Field numbers are the identity of a field. Names don't travel. Rename a field freely (for the binary format; JSON uses names, Day 12), but never change or reuse a number.
- Numbers 1 to 15 take one byte of tag; 16 to 2047 take two. Give the smallest numbers to the most frequent fields.
- Defaults aren't encoded: an empty string and a zero age produced zero bytes. Cheap, and the reason presence needs
optional(Day 3). - Unknown fields can be skipped: a reader that doesn't know field 16 reads its wire type, skips the right number of bytes, and continues. That's what makes adding fields safe.
- Order isn't significant and messages can be concatenated: parsing two encoded messages one after the other merges them, the last value winning for single fields (Day 21 shows this with oneof).
- Negative int32 values cost 10 bytes in the value (Day 3): use sint32 for signed data.
Protobuf compactness, measured
The same data as JSON is {"name":"Al","age":150,"vip":true}: 34 bytes against 10. Real messages typically shrink by a third to a fifth versus JSON, and parse faster. With compression on top (gRPC supports gzip per call), the gap narrows for large text-heavy messages, but parsing cost stays lower.
Check yourself
It's the tag: field number 1 with wire type 2, length-delimited, since (1 << 3) | 2 = 10, which is 0a in hex. The next byte gives the length of the string or nested message.
The tag is (field_number << 3) | wire_type. For field 16 that's 128, which doesn't fit in one varint byte's 7 bits, so it needs two. Fields 1 to 15 fit in one byte, so give them to frequently set fields.
It reads the tag's wire type, which says how long the value is, skips the value, and keeps the bytes as an unknown field, so re-serializing the message can preserve them.
Numeric repeated fields are written as one length-delimited block of values instead of a tag before each value. Proto3 packs them by default.
Partly: protoc --decode_raw shows field numbers, wire types and values, but not field names or exact types, since an int and a bool look the same on the wire. It's useful for debugging, but you need the schema to interpret the data correctly.