[HUDI-9620] Refactor HoodieHadoopFsRelationFactory #13527
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
||
| override val fileIndex = new HoodieIncrementalFileIndex( | ||
| sparkSession, metaClient, schemaSpec, options, FileStatusCache.getOrCreate(sparkSession), false, true) | ||
| private val incrementalFileIndex = new HoodieIncrementalFileIndex( |
There was a problem hiding this comment.
Should this be lazy as well?
There was a problem hiding this comment.
It doesn't matter because we always use it right away
|
assignment can only be done to committers and PMCs. assigning this to Ethan. but let @the-other-tim-brown do the first review and hand it off to ethan |
4 tasks
nsivabalan
approved these changes
Jul 21, 2025
jonvex
changed the title
jonvex
changed the title
azure ci all passing
|
|
Typo in the ticket number. HUDI-9620 is correct: https://issues.apache.org/jira/browse/HUDI-9620 |
jonvex
changed the title
rahil-c
pushed a commit
to rahil-c/hudi
that referenced
this pull request
Aug 5, 2025
rahil-c
pushed a commit
to rahil-c/hudi
that referenced
this pull request
Aug 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
refactor hadoopfs relation factory
HoodieCDCFileIndex extends HoodieIncrementalFileIndex but doesn't use any of the methods, the only reason is because the extension in the HoodieHadoopFsRelationFactory getRequiredFilters and fileIndex.isInstanceOf[HoodieCDCFileIndex] are used as params to HoodieFileGroupReaderBasedParquetFileFormat. Additionally, buildFileFormat() has a lot of duplicate code between classes, and it is difficult to read with all the boolean params. Finally, everything is extended from mor snapshot, and a bunch of vals are overridden.
This refactor we get rid of the dependence between HoodieCDCFileIndex and HoodieIncrementalFileIndex. We reduce duplicated code by moving as much as we can to the abstract class and use methods to override instead of vals. The refactored code has good structure and breaks mor and cow up in a more reasonable manner. We need to dependency inject HoodieIncrementalFileIndex with v1 and v2 relations, so that is some of the motivation for this refactor. Additionally, it was very tedious to change anything in HoodieHadoopFsRelationFactory because there was a chain of constructors like 8 classes deep that all needed to be changed.
Before this patch:
After this patch:
Impact
code is more clean
Risk level (write none, low medium or high below)
low
Documentation Update
N/A
Contributor's checklist