modify records without altering the entire column

[ozgurkalan]

Hi,
When reading from sources, especially from parquet files, there happened to be some missing records or mini-fixes. I wish we could mutate and modify them with a simpler where, maybe in mutate function.

my sample case:
from 500k records of 50 columns, two have missing iso2c, country, and cluster fields:

(
t
.mutate(iso2c = ibis.where(_.country_name_given=="St.Kitts&Nevis", "KN", _.iso2c))
.mutate(country = ibis.where(_.country_name_given=="St.Kitts&Nevis", "St. Kitts and Nevis", _.country))
.mutate(cluster = ibis.where(_.country_name_given=="St.Kitts&Nevis", "NAM", _.cluster))

)

[cpcloud]

@ozgurkalan Thanks for opening the discussion!

Are you asking for something with more concise syntax than what you're showing here?

[ozgurkalan]

@cpcloud Phillip hi, yes, indeed.

could be very long shot but something similar was in my mind:

t.modify(columns=["iso2c","country","cluster"], values=["KN", "St. Kitts and Nevis", "NAM"], where=[_.iso2c.isnull(), _.country.isnull()])

[cpcloud]

@ozgurkalan How about this version with the across selector?

REPLACEMENTS = {"iso2c": "KN", "country": "St. Kitts and Nevis", "cluster": "NAM"}
t.mutate(
    s.across(
        REPLACEMENTS,
        lambda c: ibis.where(
            t.country_name_given == "St.Kitts&Nevis", REPLACEMENTS[c.get_name()], c
        ),
    )
)

[ozgurkalan]

looks brilliant. Thank you.