Skip to content

Conversation

lauraghiorghisor-tw
Copy link
Contributor

@lauraghiorghisor-tw lauraghiorghisor-tw commented Aug 14, 2025

This is part of creating a finder for MOD armed forces covenant pledges, originally here https://www.gov.uk/government/publications/armed-forces-corporate-covenant-signed-pledges.

We're importing the data from a csv via a rake task, from csv file here: https://docs.google.com/spreadsheets/d/1kUmeMRQPhO8qplC4Lbh3DRXdftYczxW5TedjRvWjolQ/edit?gid=1449320752#gid=1449320752


Testing:

  • ✅ cut down on execution time (done by removing some of the repeated schema loading lines) -> 24 mins run on local
  • ✅ test out idempotency of rake task (subsequent reruns) -> we confirmed that no new records will be created on subsequent executions of the rake task; 422 status will be returned from publishing-api if we have to rerun, but we're ok with this.
  • ✅ test out incorrect rows / missing date (none)
  • ✅ test out duplicated rows -> we're going to remove these duplicates when we run in prod
- Actemium Mechanical & Electrical Ltd
- Baily Garner LLP
- Buckinghamshire Council
- Building Heroes Education Foundation
- Code 9 Security Ltd
- Grosvenor Medical Centre
- Knights Security Ltd
- Learning Curve (NE) Group Ltd
- London Stock Exchange Group
- Marden Medical Practice
- MERE Supplies Ltd
- National Grid PLC
- Sheffield Children's NHS Foundation Trust
- Thales UK Ltd
- Walsall Community Association
- Whitecross Hereford High School

Content - to be confirmed with department. Comms in email thread.

  • Are the extra rows of broken data at the bottom of the .ods file needed? -> NO
  • For the duplicate rows, what date do we need to consider? Or should we keep both? -> Latest data to be kept
  • Bespoke pledge behaviours -> this is the only bit that would require making changes to the code of the rake task. -> Importing as is.

See revised import file.

Import screenshots:
image
image

JIRA: https://gov-uk.atlassian.net/browse/WHIT-1981

@lauraghiorghisor-tw lauraghiorghisor-tw changed the base branch from main to mod-covenant-pledges August 14, 2025 14:05
Base automatically changed from mod-covenant-pledges to main August 14, 2025 16:10
@lauraghiorghisor-tw lauraghiorghisor-tw force-pushed the add-rake-task-for-mod-imports branch 3 times, most recently from 6c05cb1 to ac9726b Compare August 18, 2025 08:49
afc_business = ArmedForcesCovenantBusiness.new(**document_data[:attributes].deep_symbolize_keys)

unless dry_run
afc_business.save
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We did a sleep on the IPO rake task to ensure the order in the csv is preserved, but this one will take forever as it's a much larger csv file, and I think we're ok with the documents being created in a slightly messy order.


unless dry_run
if dry_run
afc_business.valid?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit superficial on the dry run but it can at least show us any date validations etc.

lauraghiorghisor-tw and others added 5 commits October 1, 2025 21:35
- this imports data from csv and creates business documents
- add succcesful logs
- add error logs
- add total error count

The document `save` actually registers `:base` errors on the model for 422 errors from publishing-api, which will be handy to count. We will likely get some console logs from the request itself, but a custom log will be easier to read and search for.
- Some capital case corrections
- Importing the bespoke pledges as they are rather than trying to do anything extra with the "-"
- Add line to summary
- Capital letter corrections to the body template
- Changed date preposition so it reads better in the filters
- Moved "Other" option to be last
@lauraghiorghisor-tw lauraghiorghisor-tw force-pushed the add-rake-task-for-mod-imports branch from ac9726b to 50446f3 Compare October 2, 2025 07:20
@lauraghiorghisor-tw lauraghiorghisor-tw marked this pull request as ready for review October 2, 2025 07:22
- needed to test emails
Copy link
Contributor

@TonyGDS TonyGDS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lauraghiorghisor-tw lauraghiorghisor-tw merged commit f6bc7de into main Oct 2, 2025
11 checks passed
@lauraghiorghisor-tw lauraghiorghisor-tw deleted the add-rake-task-for-mod-imports branch October 2, 2025 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants