[Bug]: ValueError: Can't use 'id' as a column name

[gabfssilva]

Steps to reproduce

Version 0.17.0 introduces — what I believe it is – a bug for columns with name "id".

To reproduce it:

customer_schema = pw.schema_builder(columns={
    'id': pw.column_definition(dtype=str, primary_key=True),
    'first_name': pw.column_definition(dtype=str),
    'last_name': pw.column_definition(dtype=str),
    'email': pw.column_definition(dtype=str)
}, name="customer_schema")

customers = pw.io.kafka.read(
    rdkafka_settings=settings,
    topic="dbserver1.inventory.customers",
    format='json',
    schema=customer_schema,
)

pw.io.kafka.write(
    rdkafka_settings=settings,
    topic_name="customers",
    table=customers,
)

What we get is a ValueError: Can't use 'id' as a column name. It looks like it was added on purpose to avoid something, probably maybe to some default behavior Python adds for id fields, but it introduces a major limitation for any table that contains this field. I also tried to find a way to provide an alias, but I couldn't find anything.

Relevant log output

Traceback (most recent call last):
  File "/Users/user/workspace/debezium-showcase/main.py", line 54, in <module>
    pw.io.kafka.write(
    ~~~~~~~~~~~~~~~~~^
        rdkafka_settings=settings,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
        topic_name="customers",
        ^^^^^^^^^^^^^^^^^^^^^^^
        table=customers,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/io/_utils.py", line 304, in wrapper
    return f(*args, **kwargs)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/runtime_type_check.py", line 19, in with_type_validation
    return beartype.beartype(f)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "<@beartype(pathway.io.kafka.write) at 0x11d679c60>", line 177, in write
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/trace.py", line 134, in _pathway_trace_marker
    _reraise_with_user_frame(e)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/trace.py", line 100, in _reraise_with_user_frame
    raise e
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/io/kafka/__init__.py", line 647, in write
    output_format = MessageQueueOutputFormat.construct(
        table,
    ...<4 lines>...
        headers=headers,
    )
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/io/_utils.py", line 355, in construct
    table = table.select(*columns_to_extract)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/trace.py", line 134, in _pathway_trace_marker
    _reraise_with_user_frame(e)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/trace.py", line 110, in _reraise_with_user_frame
    raise e
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/desugaring.py", line 351, in wrapper
    return func(*args, **kwargs)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/arg_handlers.py", line 19, in inner
    return func(*args, **kwargs)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/decorators.py", line 22, in wrapper
    return G.add_operator(
           ~~~~~~~~~~~~~~^
        lambda id: operator_cls(fn_spec, id),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        lambda operator: operator(*args, **kwargs),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/parse_graph.py", line 152, in add_operator
    result = call_operator(node)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/decorators.py", line 24, in <lambda>
    lambda operator: operator(*args, **kwargs),
                     ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/operator.py", line 226, in __call__
    result = self.func_spec.func(*args, **kwargs)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/table.py", line 414, in select
    all_args = combine_args_kwargs(args, kwargs)
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/desugaring.py", line 302, in combine_args_kwargs
    add(expr.smart_name(expression), expression)
    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/desugaring.py", line 296, in add
    raise ValueError("Can't use 'id' as a column name")
ValueError: Can't use 'id' as a column name
Occurred here:
    Line: pw.io.kafka.write(
    File: /Users/user/workspace/debezium-showcase/main.py:54
/Users/user/workspace/debezium-showcase/.venv/lib/python3.13/site-packages/pathway/internals/parse_graph.py:249: UserWarning: There are operators in the computation graph that haven't been used. Use pathway.run() (or similar) to run the computation involving these nodes.
  warnings.warn(

Process finished with exit code 1

What did you expect to happen?

Columns with name id work the same as the other ones.

Version

0.17.0

Docker Versions (if used)

No response

OS

MacOS

On which CPU architecture did you run Pathway?

ARM64 (AArch64, Apple silicon)

[lewymati]

Hi @gabfssilva

Indeed, Pathway uses column "id" for internal identifiers. We realise this is limiting, especially for input/output connectors.
For temporary workarounds you might want to try:

• for reading from kafka, please use json_field_paths parameter to map id column to something else, e.g. pw.io.kafka.read(…, json_field_paths: {'my_id': '/id'})
• for writing id field to kafka (assuming json), you can use Pathway to generate a string containing json and use pw.io.kafka.write with format="plaintext"

We will strive to come up with a better solution, but for now you should be unblocked.