Need help to implement pandas drop_duplicates function with keep=first

[abhishekricky]

Hi,

Need help to implement pandas drop_duplicates function with keep=first.

I know currently datatable doesnt support it but any help to achieve with existing functions would be helpful

Thanks,
Abhishek

[samukweku]

you could add an example, with expected output, as that would be helpful for constructing a solution

[samukweku]

One way to simulate drop_duplicates is a combination of by, and sort :

from datatable import dt, by, sort

DT = dt.Frame({
    'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
    'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
    'rating': [4, 4, 3.5, 15, 5]
})

DT

   | brand    style   rating
   | str32    str32  float64
-- + -------  -----  -------
 0 | Yum Yum  cup        4  
 1 | Yum Yum  cup        4  
 2 | Indomie  cup        3.5
 3 | Indomie  pack      15  
 4 | Indomie  pack       5  
[5 rows x 3 columns]



# keep only the first row
DT[0, :, by(DT.names), sort(DT.names)]

   | brand    style   rating
   | str32    str32  float64
-- + -------  -----  -------
 0 | Indomie  cup        3.5
 1 | Indomie  pack       5  
 2 | Indomie  pack      15  
 3 | Yum Yum  cup        4  
[4 rows x 3 columns]

Note that the order is not maintained, since at the moment in datatable, by implicitly generates a sort.

Does this work for all use cases? is it performant? hopefully you can try it out and see where it fails to hold

[oleksiyskononenko]

I'm not sure why you need additional sort here.

>>> DT[0, :, by(DT.names), sort(DT.names)]
   | brand    style   rating
   | str32    str32  float64
-- + -------  -----  -------
 0 | Indomie  cup        3.5
 1 | Indomie  pack       5  
 2 | Indomie  pack      15  
 3 | Yum Yum  cup        4  
[4 rows x 3 columns]

and

>>> DT[0, :, by(DT.names)]
   | brand    style   rating
   | str32    str32  float64
-- + -------  -----  -------
 0 | Indomie  cup        3.5
 1 | Indomie  pack       5  
 2 | Indomie  pack      15  
 3 | Yum Yum  cup        4  
[4 rows x 3 columns]

both produce the same result, because by already does all the sorting job.