NIFI-15062 Add PutIcebergRecord Processor and Services #10400
Conversation
- Added RESTIcebergCatalog implementation of IcebergCatalog - Added S3FileIOProvider implementation of IcebergFileIOProvider - Added ParquetIcebergWriter implementation of IcebergWriter
| final ListIterator<RecordField> recordFields = recordSchema.getFields().listIterator(); | ||
| while (recordFields.hasNext()) { | ||
| final RecordField recordField = recordFields.next(); | ||
| final int id = recordFields.nextIndex(); |
There was a problem hiding this comment.
Isn't it going to be an issue for nested records? When getStructType() calls itself for nested records we will have duplicate IDs in the same schema tree, no? Iceberg requires every ID in a schema tree to be globally unique.
There was a problem hiding this comment.
That's a very good point, most of the initial testing did not use nested types. I will review the implementation options and implement a better strategy for nested field identification.
There was a problem hiding this comment.
I refactored PutIcebergRecord to use the Table Schema and Struct instead of depending on this conversion from Record Schema to Iceberg Struct Type. Although this may be useful for other scenarios such as initial table creation, it is not needed in the current implementation, so I have removed this class.
...sors/src/main/java/org/apache/nifi/processors/iceberg/record/StandardStructTypeProvider.java
Outdated
Show resolved
Hide resolved
...sors/src/main/java/org/apache/nifi/processors/iceberg/record/StandardStructTypeProvider.java
Outdated
Show resolved
Hide resolved
…cord Schema - Removed StructTypeProvider and implementation
Summary
NIFI-15062 Adds a
PutIcebergRecordProcessor and several Controller Services that provide initial integration for storing records in Apache Iceberg tables.The Apache Iceberg ecosystem supports a wide variety of catalogs, storage providers, and file formats. The purpose of this pull request is to provide several Controller Service abstractions that enable extensible integration, with a specific implementation of each Controller Service. With the number of potential integration options, Apache NiFi should not necessarily implement support for every possible solution, but should provide extension points that enable focused types of integration. The
iceberg-apilibrary is the foundation for this approach.The
nifi-iceberg-bundleincludes multiple modules that have the following dependency hierarchy:nifi-iceberg-shared-narnifi-iceberg-services-api-narnifi-iceberg-processors-narnifi-iceberg-rest-catalog-narnifi-iceberg-aws-narnifi-iceberg-parquet-writer-narThe
nifi-iceberg-shared-narcontains theiceberg-apiandiceberg-corelibraries along with transitive dependencies.The
nifi-iceberg-services-api-nardepends oniceberg-apiand incorporates the Apache NiFi Controller Service interfaces that align withiceberg-apiinterfaces.The
nifi-iceberg-processors-narcontains thePutIcebergRecordProcessor, which references properties for the following Controller Services:IcebergCatalogIcebergWriterIcebergFileIOProviderThese three interfaces define the primary extension points for external integration.
The
nifi-iceberg-rest-catalog-narcontains theRESTIcebergCatalogimplementation of theIcebergCatalogController Service. This implementation configures theRESTSessionCatalogfrom theiceberg-corelibrary and supports Catalog Authentication using OAuth2 with Client Credentials or Bearer Tokens. Building on theiceberg-corelibrary provided innifi-iceberg-shared-nar, thenifi-iceberg-rest-catalog-nardoes not have any additional dependencies. TheRESTIcebergCatalogdefines aFileIO Providerproperty that supports configurable Controller Services for IcebergFileIOimplementations.The
nifi-iceberg-aws-narcontains theS3IcebergFileIOProviderwhich configures and returns the IcebergS3FileIOclass. Support for S3 requires a number of AWS SDK 2 libraries, which is one of the primary reasons for separate packaging ofFileIOProviderimplementations. The S3 implementation supports configurable authentication using Basic or Session Credentials, as well as Vended Credentials, where the REST Catalog is expected to provide the required credentials.The
nifi-iceberg-parquet-writer-narcontains theParquetIcebergWriterController Service, supporting Apache Parquet serialization. Apache Parquet has a number of transitive dependencies, including a dependency on thehadoop-commonlibrary. The NAR packaging excludes many unnecessary transitive dependencies and has an explicit list of dependencies required at runtime for Parquet serialization.This implementation structure and Controller Service design strategy should serve as the basis for additional storage provider implementations. With Apache Parquet being the predominant format for Apache Iceberg, direct support for other file formats may not be necessary. The variety of Iceberg REST Catalog implementations may require additional configuration options in the future, but the core
IcebergCatalogController Service abstraction provides a decoupled strategy for future implementation.Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000Pull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation