Skip to content

Conversation

@Dtphelan1
Copy link
Contributor

Summary

CSVExtractors now support ingesting a dataDirectory & fileName argument pair, with which the extractor constructs an absolute path to a data file. This technique for specifying the location of a CSV file on disk is now the preferred method, in contrast with our previous filePath argument technique. Documentation, example config files, and schemas have been updated to reflect this.

New behavior

All CSVExtractors extending from the BaseCSVExtractor should now support the ingestion of fileName and dataDirectory arguments. The BaseCSVExtractor now considers a new case in the initial extractor construction to handle the consumption of these arguments.

Code changes

  • Updated the README to reflect this argument pair as our preferred approach;
  • Updated example config to use this approach by default;
  • Updated the BaseCSVExtractor to build that absolute file path when the two arguments are provided;
  • Updated the BaseCSVExtractor to throw when the dataDirectory is not an absolute path;
  • Updated all XCSVExtractors, where X is a ResourceType, to pass the dataDirectory and fileName arguments to their super constructors;
  • Added BaseCSVExtractor to the extractors exported by MEF/src/extractors/index.js;
  • Updated the config schema to reflect new properties, adding descriptions for human readability;
  • Added tests to ensure that the BaseCSVExtractor handles its argument cases appropriately, throwing when expected.

Testing guidance

  • Ensure all tests pass;
  • Ensure local extraction works once you've updated your Config appropriately;
  • Ensure that the schema updates are sufficient;
  • Ensure that the approach is reasonable.
  • Ensure that there aren't any missed documentation updates needed

@Dtphelan1 Dtphelan1 changed the title CSVExtractor's now support dataDirectory & fileName arguments CSVExtractors now support dataDirectory & fileName arguments Dec 14, 2021
@Dtphelan1 Dtphelan1 changed the title CSVExtractors now support dataDirectory & fileName arguments CSVExtractors support dataDirectory & fileName arguments Dec 14, 2021
@julianxcarter julianxcarter self-assigned this Dec 15, 2021
Copy link
Contributor

@julianxcarter julianxcarter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great and I think the approach makes a lot of sense! I just have a few suggestions that I'd like your thoughts on :).

@Dtphelan1 Dtphelan1 merged commit d81800d into develop Dec 16, 2021
@Dtphelan1 Dtphelan1 deleted the data-directory-config branch December 16, 2021 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants