Enhance & Refactor SDG #895
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Adds orchestrator module, synthetic data generation class, strategies for generating data - Renames sdg-config to sdg-metadata to be more apt - Removes old tightly-coupled modules - Makes counting bytes more accurate by using pickle for serialization - Ensures that more data is produced than less - Revamps unittests and brings coverage up to 80+% - Gives users options to remove conflicting generated files before generation - Moves Dask instantiation right before its needed (within Synthetic Data Generator class) Signed-off-by: Ian Hoang <[email protected]>
IanHoang
requested review from
gkamat,
beaioun,
rishabh6788,
VijayanB and
OVI3D0
as code owners
Signed-off-by: Ian Hoang <[email protected]>
Signed-off-by: Ian Hoang <[email protected]>
Signed-off-by: Ian Hoang <[email protected]>
Signed-off-by: Ian Hoang <[email protected]>
gkamat
reviewed
Jul 25, 2025
| existing_files = [] | ||
|
|
||
| for file in os.listdir(output_path): | ||
| if (file.startswith(index_name) and file.endswith(".json")) or (file.startswith(index_name) and file.endswith('_record.json')): |
There was a problem hiding this comment.
Are there some expected characters between the prefix and suffix? If so, should they be checked for? If not, the pattern can be made more precise.
osbenchmark/synthetic_data_generator/synthetic_data_generator.py
Outdated
Show resolved
Hide resolved
osbenchmark/synthetic_data_generator/synthetic_data_generator.py
Outdated
Show resolved
Hide resolved
osbenchmark/synthetic_data_generator/synthetic_data_generator.py
Outdated
Show resolved
Hide resolved
gkamat
approved these changes
Jul 25, 2025
osbenchmark/synthetic_data_generator/strategies/mapping_strategy.py
Outdated
Show resolved
Hide resolved
| self.logger = logging.getLogger(__name__) | ||
| # TODO: Set self.mapping_config to automatically point to MappingSyntheticDataGenerator | ||
| self.mapping_config = mapping_config or {} | ||
| self.mapping_config = mapping_generation_values if mapping_generation_values else {} | ||
|
|
||
| self.generic = Generic(locale=Locale.EN) |
There was a problem hiding this comment.
Is this tied to the English locale? Perhaps document text could be in other languages as well.
There was a problem hiding this comment.
I'm planning to enhance mapping SDG in a subsequent PR and will include expanding to different locales.
osbenchmark/synthetic_data_generator/strategies/mapping_strategy.py
Outdated
Show resolved
Hide resolved
Comment on lines
+67
to
+76
| document = { | ||
| "dog_driver_id": f"DD{generic.numeric_string.generate(length=4)}", | ||
| "dog_name": random_mimesis.choice(custom_lists['dog_names']), | ||
| "dog_breed": random_mimesis.choice(custom_lists['dog_breeds']), | ||
| "license_number": f"{random_mimesis.choice(custom_lists['license_plates'])}{generic.numeric_string.generate(length=4)}", | ||
| "favorite_treats": random_mimesis.choice(custom_lists['treats']), | ||
| "preferred_tip": random_mimesis.choice(custom_lists['tips']), | ||
| "vehicle_type": random_mimesis.choice(custom_lists['vehicle_types']), | ||
| "vehicle_make": random_mimesis.choice(custom_lists['vehicle_makes']), | ||
| "vehicle_model": random_mimesis.choice(custom_lists['vehicle_models']), |
There was a problem hiding this comment.
Would be a good example for the documentation.
There was a problem hiding this comment.
Will add this to documentation
…c_document and others Signed-off-by: Ian Hoang <[email protected]>
|
Addressed @gkamat's feedback and have confirmed E2E tests work. Different inputs that have been tested:
Ensured that:
|
gkamat
pushed a commit
that referenced
this pull request
Aug 1, 2025
Signed-off-by: Ian Hoang <[email protected]>
gkamat
pushed a commit
to gkamat/opensearch-benchmark
that referenced
this pull request
Aug 6, 2025
Signed-off-by: Ian Hoang <[email protected]>
gkamat
pushed a commit
that referenced
this pull request
Aug 19, 2025
Signed-off-by: Ian Hoang <[email protected]>
IanHoang
added a commit
that referenced
this pull request
Aug 20, 2025
Signed-off-by: Ian Hoang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
These are a combination of changes that focus on enhancing and refactoring SDG.
Issues Resolved
#836
#833
Testing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.