Nanopb: Protocol Buffers with small code size

Nanopb is an ANSI-C library for encoding and decoding messages in Google's Protocol Buffers format with minimal requirements for RAM and code space. It is primarily suitable for 32-bit microcontrollers.

Overall structure

For the runtime program, you always need pb.h for type declarations. Depending on whether you want to encode, decode, or both, you also need pb_encode.h/c or pb_decode.h/c.

The high-level encoding and decoding functions take an array of pb_field_t structures, which describes the fields of a message structure. Usually you want these autogenerated from a .proto file. The tool script nanopb_generator.py accomplishes this.

generator_flow.png

So a typical project might include these files:

  1. Nanopb runtime library:
    • pb.h
    • pb_common.h and pb_common.c (always needed)
    • pb_decode.h and pb_decode.c (needed for decoding messages)
    • pb_encode.h and pb_encode.c (needed for encoding messages)
  2. Protocol description (you can have many):
    • person.proto (just an example)
    • person.pb.c (autogenerated, contains initializers for const arrays)
    • person.pb.h (autogenerated, contains type declarations)

Features and limitations

Features

  1. Pure C runtime
  2. Small code size (2–10 kB depending on processor, plus any message definitions)
  3. Small ram usage (typically ~300 bytes, plus any message structs)
  4. Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
  5. No malloc needed: everything can be allocated statically or on the stack. Optional malloc support available.
  6. You can use either encoder or decoder alone to cut the code size in half.
  7. Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, oneofs, packed arrays, extension fields.
  8. Callback mechanism for handling messages larger than can fit in available RAM.
  9. Extensive set of tests.

Limitations

  1. Some speed has been sacrificed for code size.
  2. Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
  3. The deprecated Protocol Buffers feature called "groups" is not supported.
  4. Fields in the generated structs are ordered by the tag number, instead of the natural ordering in .proto file.
  5. Unknown fields are not preserved when decoding and re-encoding a message.
  6. Reflection (runtime introspection) is not supported. E.g. you can't request a field by giving its name in a string.
  7. Numeric arrays are always encoded as packed, even if not marked as packed in .proto.
  8. Cyclic references between messages are supported only in callback and malloc mode.

Getting started

For starters, consider this simple message:

message Example {
   required int32 value = 1;
}

Save this in message.proto and compile it:

user@host:~$ protoc -omessage.pb message.proto
user@host:~$ python nanopb/generator/nanopb_generator.py message.pb

You should now have in message.pb.h:

typedef struct {
   int32_t value;
} Example;

extern const pb_field_t Example_fields[2];

Now in your main program do this to encode a message:

Example mymessage = {42};
uint8_t buffer[10];
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
pb_encode(&stream, Example_fields, &mymessage);

After that, buffer will contain the encoded message. The number of bytes in the message is stored in stream.bytes_written. You can feed the message to protoc --decode=Example message.proto to verify its validity.

For a complete example of the simple case, see example/simple.c. For a more complex example with network interface, see the example/network_server subdirectory.

Compiler requirements

Nanopb should compile with most ansi-C compatible compilers. It however requires a few header files to be available:

  1. string.h, with these functions: strlen, memcpy, memset
  2. stdint.h, for definitions of int32_t etc.
  3. stddef.h, for definition of size_t
  4. stdbool.h, for definition of bool

If these header files do not come with your compiler, you can use the file extra/pb_syshdr.h instead. It contains an example of how to provide the dependencies. You may have to edit it a bit to suit your custom platform.

To use the pb_syshdr.h, define PB_SYSTEM_HEADER as "pb_syshdr.h" (including the quotes). Similarly, you can provide a custom include file, which should provide all the dependencies listed above.

Running the test cases

Extensive unittests and test cases are included under the tests folder.

To build the tests, you will need the scons build system. The tests should be runnable on most platforms. Windows and Linux builds are regularly tested.

In addition to the build system, you will also need a working Google Protocol Buffers protoc compiler, and the Python bindings for Protocol Buffers. On Debian-based systems, install the following packages: protobuf-compiler, python-protobuf and libprotobuf-dev.