Skip to content

Replace "protobuf-dynamic" lib with built-in protobuf methods #18401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Aug 15, 2025

Conversation

amaechler
Copy link
Contributor

@amaechler amaechler commented Aug 14, 2025

Description

The inline and file-based protobuf decoders depended on an external library protobuf-dynamic. Initially, I thought the library (which appears to be no longer actively maintained) doesn't support maps at all, but upon further testing, I was unable to confirm this.

However, instead of using a third-party lib, we can achieve the same functionality (building a Protobuf descriptor from a file or Base64-encoded string) using the standard protobuf library. This PR does so.

Additionally, this PR improves the code generation used in unit tests a lot:

  • Generate the proto wrapper files as part of the build (using https://github.com/ascopes/protobuf-maven-plugin). This allows us to remove previously generated files and also clean up the checkstyle exclusions.
  • Add documentation on how to generate individual descriptor files (which some tests use) and add a Dockerfile providing a container to generate those files.

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever it would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

public abstract class DescriptorBasedProtobufBytesDecoder implements ProtobufBytesDecoder
{
private Descriptors.Descriptor descriptor;

/**
* An optional message Protobuf message type in the descriptor. Both short name and fully qualified name are accepted.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copied from our docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added a few comments to this file (and made the constructor a bit cleaner) while reading through this file to understand what is happening.

@jon-wei jon-wei merged commit d1dd682 into apache:master Aug 15, 2025
71 checks passed
@amaechler amaechler deleted the protobuf-schema-lib branch August 15, 2025 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants