First column of csv file cannot be cut

[steinm]

I have a very simple csv file

KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR;BG;CZ;RU;GR;HU;SE;CR;SLO;SK
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz;;;;;;;;;
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah;;;;;;;;;

and try to extract the first column

mlr --csv --fs ; cut -f KEY Colours-import.csv

The result is a list of empty lines. If I add a second column (-f KEY,EN) its cuts out only the second column but still not the first one. It appears like the first column cannot be addressed. Im using 2.3.2.

Is this a bug or am I just to stupid to use the program?

[johnkerl]

If you do file Colours-import.csv do you get ASCII text or ASCII text, with CRLF line terminators?

CSV's default record separator is CRLF (due to RFC-4180) and so for native text files on Unix-like platforms we need mlr --csv --rs lf ....

I'll also expand the FAQ entry a bit.

[johnkerl]

$ cat Colours-import.csv 
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR;BG;CZ;RU;GR;HU;SE;CR;SLO;SK
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz;;;;;;;;;
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah;;;;;;;;;

$ mlr --csv --rs lf --fs semicolon cut -f KEY Colours-import.csv 
KEY
masterdata_colourcode_1
masterdata_colourcode_2

$ mlr --csv --rs lf --fs semicolon cut -f KEY,EN Colours-import.csv 
KEY;EN
masterdata_colourcode_1;White
masterdata_colourcode_2;Black

$ mlr --csv --irs lf --ifs semicolon --oxtab cat Colours-import.csv 
KEY masterdata_colourcode_1
DE  Weiß
EN  White
ES  Blanco
FI  Valkoinen
FR  Blanc
IT  Bianco
NL  Wit
PL  Biały
RO  Alb
TR  Beyaz
BG  
CZ  
RU  
GR  
HU  
SE  
CR  
SLO 
SK  

KEY masterdata_colourcode_2
DE  Schwarz
EN  Black
ES  Negro
FI  Musta
FR  Noir
IT  Nero
NL  Zwart
PL  Czarny
RO  Negru
TR  Siyah
BG  
CZ  
RU  
GR  
HU  
SE  
CR  
SLO 
SK  

[johnkerl]

http://johnkerl.org/miller/doc/faq.html#Diagnosing_delimiter_specifications

[steinm]

Finally found it when I checked the csv file with 'od -x -c'. There were some chars at the beginning of the file were not visible in vi. Even replacing the first line and saving the file didn't remove them. The bottom line is, miller is just working fine. Thanks for your help and thanks for miller.

[jungle-boogie]

@steinm,

You may also use cat -e file.csv to show hidden characters.

[johnkerl]

Aaargh, that happened to me just the other day (unprintable characters the shell wasn't showing me). I put a note in the FAQ about that as well. I think this happens pretty easily when we pluck data out of other places.

http://johnkerl.org/miller/doc/faq.html