`encode_proto` fails to encode a message if `google.protobuf.Struct` is used

[simplepad]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Problem

Trying to encode an object field using a protobuf .desc file generated from a proto file that uses google.protobuf.Struct results in an empty protobuf map.

Example test.proto file:

syntax = "proto3";

import "google/protobuf/struct.proto";

package test;

message TestMessage {
  google.protobuf.Struct data = 1;
}

Compile it into a .desc file using protoc -o test.desc --include_imports test.proto

Example vector config:

sources:
  in:
    type: demo_logs
    format: json
    count: 1
transforms:
  tr:
    type: remap
    inputs:
      - in
    source: |-
      .data = {"hello": "world"}
      .encoded_proto = encode_base64!(encode_proto!(., "test.desc", "test.TestMessage"))
sinks:
  out:
    type: console
    inputs:
      - tr
    encoding:
      codec: json

Observe the encoded_proto field in the output. Try to decode it using an external protobuf decoder (for example cyberchef). The encoded protobuf only contains an empty map.

It might be that the code needs special handling for google structs.

vrl/src/protobuf/encode.rs

Lines 98 to 121 in bddc786

	(Value::Object(o), Kind::Message(message_descriptor)) => {
	if message_descriptor.is_map_entry() {
	let value_field = message_descriptor
	.get_field_by_name("value")
	.ok_or("Internal error with proto map processing")?;
	let mut map: HashMap<MapKey, prost_reflect::Value> = HashMap::new();
	for (key, val) in o.into_iter() {
	match convert_value(&value_field, val, options) {
	Ok(prost_val) => {
	map.insert(MapKey::String(key.into()), prost_val);
	}
	Err(e) => return Err(e),
	}
	}
	Ok(prost_reflect::Value::Map(map))
	} else {
	// if it's not a map, it's an actual message
	Ok(prost_reflect::Value::Message(encode_message(
	message_descriptor,
	Value::Object(o),
	options,
	)?))
	}
	}

VRL Program

VRL and/or Vector Version

latest commit on the main branch

Debug Output

Example

No response

Additional Context

No response

References

No response