This document details all the breaking changes that have been made to nanopb since its initial release. For each change, the rationale and required modifications of user applications are explained. Also any error indications are included, in order to make it easier to find this document.
Rationale: Python 2 interpreter was deprecated in 2020. For backward compatibility, nanopb has retained support for running the generator with Python 2 for the 0.4.x series. That has required several tricks that complicate the codebase.
Changes: Removed Python 2 support files and code hacks needed to make it work.
Required actions: Upgrade to Python 3 and ensure
python-protobuf is installed.
Rationale: Previously CMake rules primarily looked
for protoc in system path. This was often an outdated
version installed from package manager, and not necessarily compatible
with python-protobuf version installed from
pip.
Changes: CMake rules now default to using
generator/protoc, which in turn uses
grpc_tools Python package if available. If it is not
available, system path is searched for protoc.
Required actions: For most users, no actions are
needed. In case of version incompatibilities,
pip install --user --upgrade grpcio-tools protobuf is
recommended. If needed, PROTOBUF_PROTOC_EXECUTABLE can be
set to override the default.
Error indications:
Failed to import generator/proto/nanopb_pb2.py if versions
of protoc selected by CMake is different than installed
python-protobuf.
Rationale: Previously pb_byte_t was
always defined as uint8_least_t. This could be annoying on
some platforms without this define, or when some compiles might warn on
conversion from uint8_t. However not all platforms support
uint8_t sized access.
Changes: The stdint.h header will
define UINT8_MAX exactly if uint8_t is
available. Use it to select which type to typedef.
Required actions: Usually none. If any compiler
warnings are generated, they can either be fixed or
PB_BYTE_T_OVERRIDE can be defined to
uint_least8_t to restore old behavior.
Error indications: Implicit conversion from
uint_least8_t to uint8_t.
Rationale: Due to the shortcomings of the WORKSPACE system, Bzlmod is going to replace the legacy WORKSPACE system in future Bazel releases. Therefore, nanopb has been migrated to use bzlmod to better support newer bazel versions.
Changes * upgrade bazel deps * bazel_skylib: 1.7.1 * rules_python: 0.34.0 * rules_proto: 6.0.2 * protobuf: 24.4 * rules_proto_grpc: 5.0.0 * Start using bzlmod (MODULE.bazel)
Required actions: bazel build using WORKSPACE has been deprecated. To use bzlmod, adding below content to your MODULE.bazel
bazel_dep(name = "nanopb", version = "0.4.9")
git_override(
module_name = "nanopb",
remote = "https://github.com/nanopb/nanopb.git",
commit = "<commit>",
)noted that the name of the module has been changed to
nanopb, to better fit the convention of bzlmod. If the old
name com_github_nanopb_nanopb is preferred, can add
repo_name parameter to indicate the repo name.
bazel_dep(name = "nanopb", version = "0.4.9", repo_name="com_github_nanopb_nanopb")Rationale: Nanopb-0.4.7 extended
int_size option to affect enums. This is only supported by
C++11 and C23 compilers. The generation used #ifdef to
limit size option to use on C++ compilers. This caused binary
incompatibility when project mixed C and C++ files.
Changes: enum_intsize is now a separate
option, and does not use #ifdef. If compiler does not
support the setting, compilation will fail.
Required actions: If using the recently introduced
int_size option on enums, update to use
enum_intsize instead.
Error indications: Enum integer sizes use defaults as the old setting is ignored.
Rationale: Previously CMakeLists.txt
installed nanopb Python module under name proto and include
file directly as /usr/include/pb.h. These names have
potential to conflict with other libraries.
Changes: Python module is installed as
nanopb and include files under
/usr/include/nanopb.
Required actions: Only affects users who install
nanopb using the cmake build system. Does not affect use of
FindNanopb.cmake. Calling nanopb generator should work as
before. Include path may need adjustment if not using
nanopb-targets.cmake to determine it.
Error indications: Include file pb.h
not found when compiling against a system-wide installation done with
CMake.
This option was separated to enum_intsize in
nanopb-0.4.9. This migration notice has been updated to
match.
Rationale: The packed_enum option does
not work with MSVC due to #pragma pack not supporting enums
with MSVC. To workaround this, enum sizes can be specified with the new
int_size option. Note that this is only supported when
generating C++.
Changes: The int_sizeenum_intsize option can be specified for enums.
Required actions: Any users concerned about the
size of the generated C++ enums and are setting the int_size of enums
via a wildcard (e.g. MyMessage.* int_size=IS_8) will need
to instead set the int_size option for individual
fields.
Error indications: The size of generated C++
enums has changed.
Changes: The include path passed to
protoc by the CMake rules was updated.
Required actions: No changes needed for most users.
In some specific cases it could change the directory hierarchy generated
by protoc. More details in pull request
#822.
Error indications: Generated .pb.c or
.pb.h file not found when building with CMake rules.
Changes: To ease NANOPB_VERSION macro
usage, the value is directly a string.
Required actions: Most nanopb users probably never
used that macro. If so, you certainly use the #
preprocessor to convert it as string. You, now, only have to call it
directly, like this for example:
strcpy(myvar, NANOPB_VERSION);
Changes: The default options passing method now uses
--plugin-opt which is supported by protoc 3.6.0 and newer
(released in 2018).
Required actions: Update protoc if
needed, or alternatively install grpcio-tools package from
pip. If neither is possible, the
NANOPB_PROTOC_OLDER_THAN_3_6_0 cmake option can be used to
restore the old style option passing. Note that it has problems with
special characters such as :.
Error indications:
“protoc: Unknown flag: --nanopb_opt”
Rationale: The nanopb generated headers use static
assertions to catch errors at compile time. There are several mechanisms
to implement this. The most widely supported is C11
_Static_assert keyword. Previously the code used negative
size array definition trick, which is supported already in C99 but does
not work with every compiler and can produce confusing error
messages.
Changes: Now _Static_assert is used by
default.
Required actions: If the keyword is not recognized,
set the compiler to C11 standard mode if available. If it is not
available, define either PB_C99_STATIC_ASSERT or
PB_NO_STATIC_ASSERT in pb.h or on compiler
command line.
Error indications:
Undefined identifier _Static_assert
Changes: Back in 2018, it was considered in pull request #241 to move nanopb generator options to a separate namespace. For this reason, a transitional file was added. It was later abandoned and is now removed to avoid confusion.
Required actions: Most nanopb users probably never
used that transitional file at all. If your .proto files
import it, change to using
generator/proto/nanopb.proto.
Error indications: Errors about missing file
options.proto when running the generator.
Changes: New fields
required_field_count and largest_tag were
added to pb_msgdesc_t and existing fields were
reordered.
Required actions: All .pb.c files must
be recompiled. Regeneration is not needed.
Error indications: Messages may fail to encode or
decode, or the code can crash inside
load_descriptor_values() in pb_common.c.
Rationale: Previously
nanopb-generator.py had hashbang of
#!/usr/bin/env python, which would execute with Python 2 on
most systems. Python 2 is now deprecated and many libraries are dropping
support for it, which makes installing dependencies difficult. While
nanopb_generator.py has worked with Python 3 for years now,
and overriding the python version was possible with virtualenv, that was
an extra complication.
Changes: Hashbang now uses
#!/usr/bin/env python3. New file
nanopb_generator.py2 can be used to run with Python 2, if
necessary.
Required actions: If possible, just verify Python 3
is installed and necessary dependencies are installed for it. For
example pip3 install protobuf grpcio-tools should take care
of it. If this is not possible, call nanopb_generator.py2
from your build scripts instead.
Error indications:
python3: command not found if Python 3 is not installed.
Could not import the Google protobuf Python libraries if
dependencies are only installed for Python 2.
Rationale: Previously information about struct
fields was stored as an array of pb_field_t structures.
This was a straightforward method, but required allocating space for
e.g. submessage type and array size for all fields, even though most
fields are not submessages nor arrays.
Changes: Now field information is encoded more
efficiently in uint32_t array in a variable-length format.
Old pb_field_t structure has been removed and it is now a
typedef for pb_field_iter_t. This retains compatibility
with most old callback definitions. The field definitions in
.pb.h files are now of type pb_msgdesc_t.
Required actions: If your own code accesses the
low-level field information in pb_field_t, it must be
modified to do so only through the functions declared in
pb_common.h.
Error indications:
incompatible pointer type errors relating to
pb_field_t
Rationale: Previously nanopb_generator added a
timestamp header to generated files and used only basename of files in
#include directives. This is different than what the
protoc C++ backend does.
Changes: Now default options are
--no-timestamp and --no-strip-path.
Required actions: If old behaviour is desired, add
--timestamp and --strip-path options to
nanopb_generator.py or on protoc command line
as --nanopb_out=--timestamp,--strip-path:outdir.
Error indications: Compiler error: cannot find
include file mymessage.pb.h when compiling
mymessage.pb.c.
Rationale: Google’s Python protobuf library, which
is used in nanopb generator, has included plugin_pb2 with
it since version 3.1.0. It is not necessary to bundle it with nanopb
anymore.
Required actions: Update
python-protobuf to version 3.1.0 or newer.
Error indications:
ImportError: No module named compiler.plugin_pb2
Rationale: Previously field names in
.options file were case-sensitive on Linux and
case-insensitive on Windows. This was by accident. Because
.proto files are case-sensitive, .options
files should be too.
Changes: Now field names in .options
are always case-sensitive, and matched by fnmatchcase()
instead of fnmatch().
Required actions: If field names in
.options are not capitalized the same as in
.proto, they must be updated.
CHAR_BIT define is
now neededRationale: To check whether the platform has 8-bit
or larger chars, the C standard CHAR_BIT macro is
needed.
Changes: pb.h now includes
limits.h for this macro.
Required actions: If your platform doesn’t have
limits.h available, you can define the macro in
pb_syshdr.h. There is an example in extra
directory.
Error indications:
"Cannot find include file <limits.h>." or
"Undefined identifier: CHAR_BIT."
Rationale: Previously pb_encode() would
accept non-terminated strings and assume that they are the full length
of the defined array. However, pb_decode() would reject
such messages because null terminator wouldn’t fit in the array.
Changes: pb_encode() will now return an
error if null terminator is missing. Maximum encoded message size
calculation is changed accordingly so that at most
max_size-1 strings are assumed. New field option
max_length can be used to define the maximum string length,
instead of the array size.
Required actions: If your strings were previously filling the whole allocated array, increase the size of the field by 1.
Error indications: pb_encode() returns
error unterminated string.
Rationale: Previously nanopb declared a
fieldname_default constant variable for each field with a
default value, and used these internally to initialize messages. This
however used unnecessarily large amount of storage for the values. The
variables were mostly for internal usage, but were available in the
header file.
Changes: Default values are now stored as an encoded protobuf message.
Required actions: If your code previously used
default constants, it will have to be adapted to take the default value
in some other way, such as by defining
static const MyMessage msg_default = MyMessage_init_default;
and accessing msg_default.fieldname.
Error indications: Compiler error about
fieldname_default being undeclared.
Rationale: Previously nanopb has allowed messages to be terminated by a null byte, which is read as zero tag value. Most other protobuf implementations don’t support this, so it is not very useful feature. It has also been noted that this can complicate debugging issues with corrupted messages.
Changes: pb_decode() now gives error
when it encounters zero tag value. A new function
pb_decode_ex() supports flag
PB_DECODE_NULLTERMINATED that supports decoding null
terminated messages.
Required actions: If application uses null
termination for messages, switch it to use pb_decode_ex()
and pb_encode_ex(). If compatibility with 0.3.9.x is
needed, there are also pb_decode_nullterminated() and
pb_encode_nullterminated() macros, which work both in 0.4.0
and 0.3.9.
Error indications: Error message from
pb_decode(): zero_tag.
Rationale: Previously nanopb considered proto3 submessages as present only when their contents was non-zero. Most other protobuf libraries allow explicit null state for submessages.
Changes: Submessages now have separate
has_field in proto3 mode also.
Required actions: When using submessages in proto3
mode, user code must now set mymsg.has_submsg = true for
each submessage that is present. Alternatively, the field option
proto3_singular_msgs can be used to restore the old
behavior.
Error indications: Submessages do not get encoded.
Rationale: Back in 2013, function signature for
callbacks was changed. The PB_OLD_CALLBACK_STYLE option
allowed compatibility with old code, but complicated code and testing
because of the different options.
Changes: PB_OLD_CALLBACK_STYLE option
no-longer has any effect.
Required actions: If
PB_OLD_CALLBACK_STYLE option was in use previously,
function signatures must be updated to use double pointers
(void** and void * const *).
Error indications: Assignment from incompatible pointer type.
Rationale: Protoc allows including comments in form
@@protoc_insertion_point to identify locations for other
plugins to insert their own extra content. Previously these were
included by default, but they clutter the generated files and are rarely
used.
Changes: Insertion points are now included only when
--protoc-insertion-points option is passed to the
generator.
Rationale: Nanopb generator makes #defines for enum minimum and maximum value. Previously these defines incorrectly had the first and last enum value, instead of the actual minimum and maximum. (issue #405)
Changes: Minimum define now always has the smallest value, and maximum define always has the largest value.
Required actions: If these defines are used and enum values in .proto file are not defined in ascending order, user code behaviour may change. Check that user code doesn't expect the old, incorrect first/last behaviour.
Rationale: In C99, bool variables are
not allowed to have other values than true and
false. Compilers use this fact in optimization, and
constructs like int foo = msg.has_field ? 100 : 0; will
give unexpected results otherwise. Previously nanopb didn't enforce that
decoded bool fields had valid values.
Changes: Bool fields are now handled separately as
PB_LTYPE_BOOL. The LTYPE descriptor numbers
for other field types were renumbered.
Required actions: Source code files must be
recompiled, but regenerating .pb.h/.pb.c files
from .proto is not required. If user code directly uses the
nanopb internal field representation (search for
PB_LTYPE_VARINT in source), it may need updating.
Rationale: Previously nanopb didn’t properly decode
special character escapes like \200 emitted by protoc. This
caused these escapes to end up verbatim in the default values in .pb.c
file.
Changes: Escapes are now decoded, and
e.g. \200 or \x80 results in {0x80} for bytes
field and "\x80" for string field.
Required actions: If code has previously relied on
\ in default value being passed through verbatim, it must
now be changed to \\.
Rationale: If the substream functions were called directly and the caller did not completely empty the substring before closing it, the parent stream would be put into an incorrect state.
Changes: pb_close_string_substream can
now error and returns a boolean.
Required actions: Add error checking onto any call
to pb_close_string_substream.
Rationale: Previously two oneofs in a single message would be erroneously handled as part of the same union.
Changes: Oneofs fields now use special
PB_DATAOFFSET_UNION offset type in generated .pb.c files to
distinguish whether they are the first or following field inside an
union.
Required actions: Regenerate
.pb.c/.pb.h files with new nanopb version if oneofs are
used.
Rationale: Some platforms cannot access 8-bit sized
values directly, and do not define uint8_t. Nanopb
previously didn't support these platforms.
Changes: References to uint8_t were
replaced with several alternatives, one of them being a new
pb_byte_t typedef. This in turn uses
uint_least8_t which means the smallest available type.
Required actions: If your platform does not have a
standards-compliant stdint.h, it may lack the definition
for [u]int_least8_t. This must be added manually, example
can be found in extra/pb_syshdr.h.
Error indications: Compiler error:
"unknown type name 'uint_least8_t'".
Rationale: Previously nanopb did not support the
oneof construct in .proto files. Those fields
were generated as regular optional fields.
Changes: OneOfs are now generated as C unions. Callback fields are not supported inside oneof and generator gives an error.
Required actions: The generator option
no_unions can be used to restore old behaviour and to allow
callbacks to be used. To use unions, one change is needed: use
which_xxxx field to detect which field is present, instead
of has_xxxx. Compare the value against
MyStruct_myfield_tag.
Error indications: Generator error:
"Callback fields inside of oneof are not supported".
Compiler error: "Message" has no member named
"has_xxxx".
Rationale: Originally, the field iteration logic was
simple enough to be duplicated in pb_decode.c and
pb_encode.c. New field types have made the logic more
complex, which required the creation of a new file to contain the common
functionality.
Changes: There is a new file,
pb_common.c, which must be included in builds.
Required actions: Add pb_common.c to
build rules. This file is always required. Either
pb_decode.c or pb_encode.c can still be left
out if some functionality is not needed.
Error indications: Linker error: undefined reference
to pb_field_iter_begin, pb_field_iter_next or
similar.
Rationale: Often nanopb is used with small arrays,
such as 255 items or less. Using a full size_t field to
store the array count wastes memory if there are many arrays. There
already exists parameters PB_FIELD_16BIT and
PB_FIELD_32BIT which tell nanopb what is the maximum size
of arrays in use.
Changes: Generator will now use
pb_size_t for the array _count fields. The
size of the type will be controlled by the PB_FIELD_16BIT
and PB_FIELD_32BIT compilation time options.
Required actions: Regenerate all .pb.h
files. In some cases casts to the pb_size_t type may need
to be added in the user code when accessing the _count
fields.
Error indications: Incorrect data at runtime, crashes. But note that other changes in the same version already require regenerating the files and have better indications of errors, so this is only an issue for development versions.
Rationale: Some names in nanopb core were badly chosen and conflicted with ISO C99 reserved names or lacked a prefix. While they haven't caused trouble so far, it is reasonable to switch to non-conflicting names as these are rarely used from user code.
Changes: The following identifier names have changed:
Required actions: Regenerate all .pb.c
files. If you use any of the above identifiers in your application code,
perform search-replace to the new name.
Error indications: Compiler errors on lines with the macro/type names.
Rationale: Some compilers do not accept filenames
with two dots (like in default extension .pb.c). The -e
option to the generator allowed changing the extension, but not skipping
the extra dot.
Changes: The -e option in generator
will no longer add the prepending dot. The default value has been
adjusted accordingly to .pb.c to keep the default behaviour
the same as before.
Required actions: Only if using the generator -e option. Add dot before the parameter value on the command line.
Error indications: File not found when trying to compile generated files.
Rationale: In the initial pointer encoding support
since nanopb-0.2.5, the bytes type used a separate
pb_bytes_ptr_t type to represent bytes fields.
This made it easy to encode data from a separate, user-allocated buffer.
However, it made the internal logic more complex and was inconsistent
with the other types.
Changes: Dynamically allocated bytes fields now have
the pb_bytes_array_t type, just like statically allocated
ones.
Required actions: Only if using pointer-type fields
with the bytes datatype. Change any access to
msg->field.size to msg->field->size.
Change any allocation to reserve space of amount
PB_BYTES_ARRAY_T_ALLOCSIZE(n). If the data pointer was
begin assigned from external source, implement the field using a
callback function instead.
Error indications: Compiler error: unknown type name
pb_bytes_ptr_t.
Rationale: Having the option in the headers required the functions to be non-static, even if the option is not used. This caused errors on some static analysis tools.
Changes: The \#ifdef and associated
functions were removed from the header.
Required actions: Only if the
NANOPB_INTERNALS option was previously used. Actions are as
listed under nanopb-0.1.3 and nanopb-0.1.6.
Error indications: Compiler warning: implicit
declaration of function pb_dec_string,
pb_enc_string, or similar.
Rationale: Previously the auxiliary data to field
callbacks was passed as void*. This allowed passing of any
data, but made it unnecessarily complex to return a pointer from
callback.
Changes: The callback function parameter was changed
to void**.
Required actions: You can continue using the old
callback style by defining PB_OLD_CALLBACK_STYLE.
Recommended action is to:
void** for
decoders and void * const * for encoders.instead ofarg`.Error indications: Compiler warning: assignment from
incompatible pointer type, when initializing funcs.encode
or funcs.decode.
Rationale: Previously the generator made a list of C
pb_field_t initializers in the .pb.c file. This led to a
need to regenerate all .pb.c files after even small changes to the
pb_field_t definition.
Changes: Macros were added to pb.h which allow for cleaner definition of the .pb.c contents. By changing the macro definitions, changes to the field structure are possible without breaking compatibility with old .pb.c files.
Required actions: Regenerate all .pb.c files from the .proto sources.
Error indications: Compiler warning: implicit
declaration of function pb_delta_end.
Rationale: The pb_type_t was previously
an enumeration type. This caused warnings on some compilers when using
bitwise operations to set flags inside the values.
Changes: The pb_type_t was changed to
typedef uint8_t. The values were changed to
#define. Some value names were changed for consistency.
Required actions: Only if you directly access the
pb_field_t contents in your own code, something which is
not usually done. Needed changes:
PB_HTYPE_ARRAY to
PB_HTYPE_REPEATED.PB_HTYPE_CALLBACK to PB_ATYPE() and
PB_ATYPE_CALLBACK.Error indications: Compiler error:
PB_HTYPE_ARRAY or PB_HTYPE_CALLBACK
undeclared.
Rationale: Similarly to field encoders in nanopb-0.1.3.
Changes: New functions with names
pb_decode_* were added.
Required actions: By defining NANOPB_INTERNALS, you
can still keep using the old functions. Recommended action is to replace
any calls with the newer pb_decode_* equivalents.
Error indications: Compiler warning: implicit
declaration of function pb_dec_string,
pb_dec_varint, pb_dec_submessage or
similar.
Rationale: The old pb_enc_* functions
were designed mostly for the internal use by the core. Because they are
internally accessed through function pointers, their signatures had to
be common. This led to a confusing interface for external users.
Changes: New functions with names
pb_encode_* were added. These have easier to use
interfaces. The old functions are now only thin wrappers for the new
interface.
Required actions: By defining NANOPB_INTERNALS, you
can still keep using the old functions. Recommended action is to replace
any calls with the newer pb_encode_* equivalents.
Error indications: Compiler warning: implicit
declaration of function pb_enc_string,
*pb_enc_varint,pb_enc_submessage\ or similar.