Skip to content

Support Type Coercion in SchemaAdapter #6381

@tustvold

Description

@tustvold

Is your feature request related to a problem or challenge?

Currently SchemaAdapter::map_projections returns an error if the provided file_schema contains columns of a different type to the expected table_schema.

In order to accomodate schema evolution, it would be advantageous to be able to instead cast the array data from the file schema to the table schema where possible.

Describe the solution you'd like

impl SchemaAdapter {

    fn map_schema(&self, file_schema: &Schema) -> Result<SchemaMapping> {
        ...
    }
}

struct SchemaMapping {
    ...
}

impl SchemaMapping {
    fn map_batch(batch: RecordBatch) -> Result<RecordBatch> {
        ...
    }
}

Where map_schema would use can_cast_types to determine if conversion is possible, and map_batch would use cast to perform this conversion

Describe alternatives you've considered

We could not do this

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions