-
Notifications
You must be signed in to change notification settings - Fork 2
Make CSV column headers case insensitive #110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! The changes are really minimal given that you had to change so much! I had a few small things inline and then I had a few other thoughts:
- It looks like this line in CSVObservationExtractor.test.js should also be updated if you agree with the other similar comments.
- It might be good to test that the CSV Validator can successfully validate headers that are in various casing that doesn't match the schema. I think that should cover the changes in
csvValidator.jsandCSVModule.js. - I think we'll need a small PR in the E-MEF because the ClinicalTrialExtractor there uses the CSV Module, so when we update the MEF, I think it will be looking for the lower case changes.
|
Okay, I think I fixed the comments mentioned here, so give that another look. I also created a PR on the E-MEF (https://gitlab.mitre.org/mcode/epic-mcode-extraction-framework/-/merge_requests/128) for the Epic Clinical Trial Information extractor there that also uses CSVs |
julianxcarter
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great!
jafeltra
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! Thanks for making those changes!
|
I think this just needs a rebase to get back in line with |
3debca4 to
ca71c5e
Compare
Summary
Column headers of CSV files read as input for the MEF can now have any casing and will still be treated the same by the program
New behavior
The program should still run the same as before, but now CSV headers can have any casing (e.g.
enrollmentStatus,ENROLLMENTSTATUS, andEnRoLlMeNtStAtUsare all valid and refer to the same thing).Code changes
csvModule.jsnow converts all headers to lowercase upon reading the CSV and will convert any keys provided to a get function to lowercase to properly find valuescsvValidoator.jshas been updated to convert the CSV schema and actual CSV headers to lowercase before comparing themTesting guidance