Python: support for optional python_package (or similar) to make import explicit

[brocaar]

What language does this apply to?

Python.

Describe the problem you are trying to solve.

The Python code generator uses the directory structure of the .proto files to generate the generated Python code directory layout and imports (e.g. when Protobuf file A imports Protobuf file B). It doesn't sanitize Python reserved keywords.

This can be problematic in two cases:

1. The directory structure contains a reserved keyword. For example when it contains as. In python import as fails but also import as.foo.bar is invalid.

2. When in case of A depending on B, A will contain an absolute import that might not be on the Python path (e.g. when the generated code is packaged so instead of import b it should do import mypackage.b). This was mentioned as a fix in the discussions: python: use relative imports in generated modules #1491, but I would consider this a workaround.

Describe the solution you'd like

For several languages, there are ways to make the package / module name explicit by defining an option, e.g. option go_package, option java_package, ... (https://developers.google.com/protocol-buffers/docs/proto3#packages). (I believe there is also a ruby_package #4627).

I believe an optional option python_package could solve the above issues as a as/api.proto file then could contain an option python_package = "as_pb.api" (or "my_package.as_pb.api").

The Protobuf code generator will then take the python_package into account when:

• Generating the directory structure for the generated Python code so that it can be imported as import as_pb.api (as_pb/api.py).
• When defining imports (Protobuf file A depends on B). The generated file A then imports B using the (optional) defined python_package.

Describe alternatives you've considered

My initial approach was to re-define the structure of the original Protobuf files. However, when the Protobuf files are used to generate other languages bindings, this has side-effects (e.g. the Go code now ends up in as_pb/api, it should be in as/api.

I believe the supported languages should not be leading in how the Protobuf structure looks like.

Additional context

Explicit is better than implicit.

(https://www.python.org/dev/peps/pep-0020/)

😉

See also brocaar/chirpstack-api#5 .

[tjerkw]

I this this is a duplicate of #2283 I also really need it. For now i think we have todo rewriting of the imports after the code has been generated.

[prateek2408]

I face the same issue where the compiled proto files generated have the wrong import statements.

[djeer]

Very old problem and no fix expected

[gitpushdashf]

I'm also interested in this.

[hungrybirder]

Hi, Any progress on this issue?

[stefanondisponibile]

same problem here. 🤦‍♂️

[vavsab]

same problem

[rmurthy716]

We also have the same problem...

[daicoden]

Is there some automated viable workaround? It seems like a necessity to conform to language guidelines and restrictions.

Instead of renaming hyphen to underscore automagically, I would vote it be explicit. I'm seeing it break down with other tools, i.e. bazel, and am having to fork and rename packages.

[stefanondisponibile]

@daicoden at the moment the only "automated" workaround I'm using is some bash vodoo to edit what I need after compiling the protobufs. Something along this:

sed -i 's/^import \(.\+\) as/from . import \1 as/' some/pkg/protobufs/*.py