Skip to content

Option skip unparse-able DBF fields #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 23, 2025

Conversation

irees
Copy link
Contributor

@irees irees commented May 23, 2025

hi @twpayne

I use your excellent libraries extensively in my work. Thank you so much for making my projects possible - I would not have been able to use Go otherwise.

I'd like to use go-shapefile to load various shapefiles in my system, and replace a shell script that uses ogr2ogr.

However, there's one significant behavior difference that is preventing me from doing this (without otherwise pre-processing the files). The default behavior of ogr2ogr is to ignore fields that fail to parse (setting the value to null), and continue processing additional records in the shapefile. Currently, go-shapefile stops processing and returns an error in this case.

To match the ogr2ogr behavior, I'd like to propose a new option in ReadDBFOptions called SkipBrokenFields, which when set simply ignores errors returned by ParseRecord.

I've included a test file (Natural Earth 10m Populated Places, which has ** as a value in an integer field and demonstrates the error) and tests that check both the expected error behavior and behavior when SkipBrokenFields is set, for both the normal reader and the Scanner reader.

I'm open to feedback and alternate approaches. Please let me know your thoughts.

@twpayne
Copy link
Owner

twpayne commented May 23, 2025

Thank you for both your kind words and this excellent PR full of tests :)

Yup, this PR makes total sense. I'll merge it now. Do you have more PRs in the pipeline? If not, I'll tag a new version of go-shapefile after merging so you have a clean version for your go.mod.

@twpayne twpayne merged commit 4d4820d into twpayne:master May 23, 2025
3 checks passed
@irees
Copy link
Contributor Author

irees commented May 23, 2025

@twpayne

Thank you for accepting my PR!

I'm incorporating go-shapefile into a couple different projects (some public repos, some private repos) and may have additional contributions when everything is in production, but no other requests at this time, so you are welcome to tag a new version.

@twpayne
Copy link
Owner

twpayne commented May 23, 2025

OK, I tagged v0.0.6 with your contribution. Thank you again!

irees added a commit to interline-io/transitland-lib that referenced this pull request May 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants