Skip to content

Releases: dimajix/flowman

0.18.0

13 Oct 17:37
@kupferk kupferk
0.18.0
27de909
Compare
Choose a tag to compare
0.18.0
  • Improve automatic schema migration for Hive and JDBC relations
  • Improve support of CHAR(n) and VARCHAR(n) types. Those types will now be propagates to Hive with newer Spark versions
  • Support writing to dynamic partitions for file relations, Hive tables, JDBC relations and Delta tables
  • Fix the name of some config variables (floman.* => flowman.*)
  • Added new config variables flowman.default.relation.migrationPolicy and flowman.default.relation.migrationStrategy
  • Add plugin for supporting DeltaLake (https://delta.io), which provides deltaTable and deltaFile relation types
  • Fix non-deterministic column order in schema mapping, values mapping and values relation
  • Mark Hive dependencies has 'provided', which reduces the size of dist packages
  • Significantly reduce size of AWS dependencies in AWS plugin
  • Add new build profile for Cloudera CDP-7.1
  • Improve Spark configuration of LocalSparkSession and TestRunner
  • Update Spark 3.0 build profile to Spark 3.0.3
  • Upgrade Impala JDBC driver from 2.6.17.1020 to 2.6.23.1028
  • Upgrade MySQL JDBC driver from 8.0.20 to 8.0.25
  • Upgrade MariaDB JDBC driver from 2.2.4 to 2.7.3
  • Upgrade several Maven plugins to latest versions
  • Add new config option flowman.workaround.analyze_partition to workaround CDP 7.1 issues
  • Fix migrating Hive views to tables and vice-versa
  • Add new option "-j " to allow running multiple job instances in parallel
  • Add new option "-j " to allow running multiple tests in parallel
  • Add new uniqueKey assertion
  • Add new schema assertion
  • Update Swagger libraries for swagger schema
  • Implement new openapi plugin to support OpenAPI 3.0 schemas
  • Add new readHive mapping
  • Add new simpleReport and report hook
  • Implement new templates
Loading

0.17.1

18 Jun 09:20
@kupferk kupferk
0.17.1
d9b6215
Compare
Choose a tag to compare
0.17.1
  • Bump CDH version to 6.3.4
  • Fix scope of some dependencies
  • Update Spark to 3.1.2
  • Add new values relation
Loading

0.17.0

04 Jun 08:02
@kupferk kupferk
0.17.0
1bac111
Compare
Choose a tag to compare
0.17.0
  • New Flowman Kernel and Flowman Studio application prototypes
  • New ParallelExecutor
  • Fix before/after dependencies in count target
  • Default build is now Spark 3.1 + Hadoop 3.2
  • Remove build profiles for Spark 2.3 and CDH 5.15
  • Add MS SQL Server plugin containing JDBC driver
  • Speed up file listing for file relations
  • Use Spark JobGroups
  • Better support running Flowman on Windows with appropriate batch scripts
Loading

0.16.0

26 Apr 05:41
@kupferk kupferk
0.16.0
70a0dba
Compare
Choose a tag to compare
0.16.0
  • Add logo to Flowman Shell
  • Fix name of config option flowman.execution.executor.class
  • Add new groupedAggregate mapping
  • Reimplement target ordering, configurable via flowman.execution.scheduler.class
  • Implement new assertions columns and expression
Loading

0.15.0

23 Mar 13:56
@kupferk kupferk
0.15.0
35b3cd8
Compare
Choose a tag to compare
0.15.0
  • New configuration variable floman.default.target.rebalance
  • New configuration variable floman.default.target.parallelism
  • Changed behaviour: The mergeFile target now does not assume any more that the target is local. If you already
    use mergeFiles with a local file, you need to prefix the target file name with file://.
  • Add new -t argument for selectively building a subset of targets
  • Remove example-plugin
  • Add quickstart guide
  • Add new "flowman-parent" BOM for projects using Flowman
  • Move com.dimajix.flowman.annotations package to com.dimajix.flowman.spec.annotations
  • Add new log redaction
  • Integrate Scala scode coverage analysis
  • assemble will fail when trying to use non-existing columns
  • Move swagger and json schema support into separate plugins
  • Change default build to Spark 3.0 and Hadoop 3.2
  • Update Spark to 3.0.2
  • Rename class Executor to Execution - watch your plugins!
  • Implement new configurable Executor class for executing build targets.
  • Add build profile for Spark 3.1.x
  • Update ScalaTest to 3.2.5 - watch your unittests for changed ScalaTest API!
  • Add new case mapping
  • Add new --dry-run command line option
  • Add new mock and null mapping types
  • Add new mock relation
  • Add new values mapping
  • Add new values dataset
  • Implement new testing capabilities
  • Rename update mapping to upsert mapping, which better describes its functionality
  • Introduce new VALIDATE phase, which is executed even before CREATE phase
  • Implement new validate and verify targets
  • Implement new deptree command in Flowman shell
Loading

0.14.2

12 Oct 10:04
@kupferk kupferk
0.14.2
f0dab8d
Compare
Choose a tag to compare
0.14.2
  • Upgrade to Spark 2.4.7 and Spark 3.0.1
  • Clean up dependencies
  • Disable build of Docker image
  • Update examples
Loading

0.14.1

03 Oct 14:26
@kupferk kupferk
0.14.1
0ee3476
Compare
Choose a tag to compare
0.14.1
  • Fix dropping of partitions which could cause issues on CDH6
Loading

0.14.0

14 Dec 11:25
@kupferk kupferk
0.14.0
174e4b3
Compare
Choose a tag to compare
0.14.0
  • Fix AWS plugin for Hadoop 3.x
  • Improve setup of logging
  • Shade Velocity for better interoperability with Spark 3
  • Add new web hook facility in namespaces and jobs
  • Existing targets will not be overwritten anymore by default. Either use the --force command line option, or set
    the configuration property flowman.execution.target.forceDirty to true for the old behaviour.
  • Add new command line option --keep-going
  • Implement new com.dimajix.spark.io.DeferredFileCommitProtocol which can be used by setting the Spark configuration
    parameter spark.sql.sources.commitProtocolClass
  • Add new flowshell application
Loading

0.13.1

14 Dec 11:25
@kupferk kupferk
0.13.1
6a9ce9d
Compare
Choose a tag to compare
0.13.1
  • Code improvements
  • Do not implicitly set SPARK_MASTER in configuration
  • Add support for CDH6
  • Add support for Spark 3.0
  • Improve support for Hadoop 3.x
Loading

0.13.0

14 Dec 11:26
@kupferk kupferk
0.13.0
adc55d2
Compare
Choose a tag to compare
0.13.0
  • Refactor Maven module structure
  • Implement new Scala DSL for creating projects
  • Fix ordering bug in target execution
  • Merge 'migrate' phase into 'create' phase
  • Rename 'input' field to 'mapping' in most targets
  • Lots of minor code improvements
Loading