Handling of Packed Repeated Fields in libosmium’s PBF Parser

[lukaskabrt]

What version of libosmium are you using?

2.20.0 (packed with the latest available version of osmium-tool 1.16.0 on Ubuntu 24.04) but the problematic code is still in master branch

What operating system and compiler are you using?

Ubuntu 24.04,

Tell us something about your system

Ubuntu on WSL2 (but I think it isn't relevant)

What did you do exactly?

I am trying to read PBF file serialized with Protobuf-net.

osmium cat way-protobufnet.pbf -o way.osm.xml

What did you expect to happen?

I expect one way with highway=residential tag in the output

What did happen instead?

Output doesn't contain any tags.

What did you do to try analyzing the problem?

I traced problem down to the way libosmium handles packed repeated fields. I am going to demonstrate the problem on tags on ways, but I suppose other packed repeated fields would behave the same.

In pbf_decoder.hpp, libosmium’s PBF decoder only treats protozero::pbf_wire_type::length_delimited as valid for packed repeated fields. However, the Protocol Buffers guide states:

Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa.

Protobuf‑net optimizes single-element arrays by emitting them with the varint wire type, which libosmium currently skips. If array contains more items (in this case tags) Protobuf‑net emits length_delimited field and the file deserializes just fine.

I have attached sample files in OSM XML format, PBF format serialized by Protobuf-net and PBF serialized by Osmium.

osmium_debug way-protobufnet.pbf produces

HEADER:
  generator=
WAY: [40]
  id=25863135
  version=8
  uid=0
  user=||
  changeset=0
  timestamp=2025-06-16T20:11:33Z
  visible=yes

osmium_debug way-osmium.pbf produces

HEADER:
  generator=osmium/1.16.0
WAY: [72]
  id=25863135
  version=8
  uid=0
  user=||
  changeset=0
  timestamp=2025-06-16T20:11:33Z
  visible=yes
  TAGS: [28]
    k=|highway| v=|residential|

Additional testing shows, that JOSM is able to handle PBF serialized by Protobuf-net.

Thank you for considering this report! Please let me know if you need additional details.

[joto]

To get the best performance libosmium is taking some "shortcuts" compared to the Protobuf spec and this has never been an issue so far. It needs some careful look to see whether that's something we can change without compromising performance.