Skip to content

Commit 1308c13

Browse files
jrosendphixMedavidjgosswslulciucinanalper
authored
sync upstream 0.50.0 (#5)
* Fixing data quality display. (MarquezProject#2937) Signed-off-by: phixMe <[email protected]> * Dataset Version call simplification (MarquezProject#2938) * Fixing data quality display. Signed-off-by: phixMe <[email protected]> * Fixing dataset version calls. Signed-off-by: phixMe <[email protected]> --------- Signed-off-by: phixMe <[email protected]> * feat: allow db-migrate without version (MarquezProject#2936) Signed-off-by: David Goss <[email protected]> * Display full `runID` and check icon when copied (MarquezProject#2940) Signed-off-by: Willy Lulciuc <[email protected]> * Deferred copy revert. (MarquezProject#2941) Signed-off-by: phixMe <[email protected]> * Long text handling (MarquezProject#2942) * Deferred copy revert. Signed-off-by: phixMe <[email protected]> * Long text handling. Signed-off-by: phixMe <[email protected]> * Adding search back in. Signed-off-by: phixMe <[email protected]> --------- Signed-off-by: phixMe <[email protected]> * Use project root for docker volume prefix (MarquezProject#2943) Signed-off-by: Willy Lulciuc <[email protected]> * fix: Correct SQL query pagination for DatasetVersion findAll method (MarquezProject#2945) Signed-off-by: Alper İnan <[email protected]> Signed-off-by: Alper <[email protected]> * Update changelog for `0.50.0` Signed-off-by: Willy Lulciuc <[email protected]> * Replace `redoc-cli` with `redocly` Signed-off-by: Willy Lulciuc <[email protected]> * Prepare for release 0.50.0 Signed-off-by: Willy Lulciuc <[email protected]> * Prepare next development version 0.51.0-SNAPSHOT Signed-off-by: Willy Lulciuc <[email protected]> * Templatize event time in `metadata.json` (MarquezProject#2946) * Templatize event time in `metadata.json` Signed-off-by: Willy Lulciuc <[email protected]> * Use `metadata.template.json` Signed-off-by: Willy Lulciuc <[email protected]> --------- Signed-off-by: Willy Lulciuc <[email protected]> * Update CHANGELOG.md * Update `web/docs/demo.gif` (MarquezProject#2948) Signed-off-by: Willy Lulciuc <[email protected]> * fix(deps): update dependency io.openlineage:openlineage-java to v1.23.0 (MarquezProject#2907) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix(deps): update dependency org.assertj:assertj-core to v3.26.3 (MarquezProject#2909) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Willy Lulciuc <[email protected]> * fix(deps): update dependency org.postgresql:postgresql to v42.7.4 (MarquezProject#2912) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix(deps): update dependency org.opensearch.client:opensearch-rest-client to v2.17.1 (MarquezProject#2911) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Willy Lulciuc <[email protected]> * fix(deps): update dependency org.apache.commons:commons-lang3 to v3.17.0 (MarquezProject#2908) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Ignore `**/stats/**` (MarquezProject#2952) Signed-off-by: Willy Lulciuc <[email protected]> * Update compatibility for `0.50.0` * fix(deps): update dependency org.opensearch.client:opensearch-java to v2.16.0 (MarquezProject#2910) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * `Dataset.currentVersionUuid` `->` `DatasetVersion.uuid` (MarquezProject#2954) Signed-off-by: Willy Lulciuc <[email protected]> * Update Events Page (MarquezProject#2955) * Tuning the events page for longer events. Signed-off-by: phixMe <[email protected]> * Adding events file. Signed-off-by: phixMe <[email protected]> * Refetch jobs button. Signed-off-by: phixMe <[email protected]> * Refetch jobs button. Signed-off-by: phixMe <[email protected]> * Lint Signed-off-by: phixMe <[email protected]> --------- Signed-off-by: phixMe <[email protected]> Co-authored-by: Willy Lulciuc <[email protected]> * Lineage run attachment issue. (MarquezProject#2953) Signed-off-by: phixMe <[email protected]> Co-authored-by: Willy Lulciuc <[email protected]> * feature: Better handling of missing environment variables in setupProxy.js file. (MarquezProject#2956) Signed-off-by: Artur Owczarek <[email protected]> --------- Signed-off-by: phixMe <[email protected]> Signed-off-by: David Goss <[email protected]> Signed-off-by: Willy Lulciuc <[email protected]> Signed-off-by: Alper İnan <[email protected]> Signed-off-by: Alper <[email protected]> Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Signed-off-by: Artur Owczarek <[email protected]> Co-authored-by: Peter Hicks <[email protected]> Co-authored-by: davidjgoss <[email protected]> Co-authored-by: Willy Lulciuc <[email protected]> Co-authored-by: Alper İnan <[email protected]> Co-authored-by: Willy Lulciuc <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Artur Owczarek <[email protected]>
1 parent bd3cfe7 commit 1308c13

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+1098
-2730
lines changed

‎.circleci/api-load-test.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
set -e
1515

1616
# Build version of Marquez
17-
readonly MARQUEZ_VERSION=0.50.0-SNAPSHOT
17+
readonly MARQUEZ_VERSION=0.51.0-SNAPSHOT
1818
# Fully qualified path to marquez.jar
1919
readonly MARQUEZ_JAR="api/build/libs/marquez-api-${MARQUEZ_VERSION}.jar"
2020

‎.circleci/db-migration.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
# Version of PostgreSQL
1414
readonly POSTGRES_VERSION="14"
1515
# Version of Marquez
16-
readonly MARQUEZ_VERSION=0.49.0
16+
readonly MARQUEZ_VERSION=0.50.0
1717
# Build version of Marquez
1818
readonly MARQUEZ_BUILD_VERSION="$(git log --pretty=format:'%h' -n 1)" # SHA1
1919
readonly POSTGRES_PORT=5432

‎.env.example‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ API_ADMIN_PORT=5001
33
WEB_PORT=3000
44
POSTGRES_PORT=5432
55
SEARCH_PORT=9200
6-
TAG=0.49.0
6+
TAG=0.50.0

‎CHANGELOG.md‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,45 @@
11
# Changelog
22

3-
## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.48.0...HEAD)
3+
## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.50.0...HEAD)
4+
5+
## [0.50.0](https://github.com/MarquezProject/marquez/compare/0.49.0...0.50.0) - 2024-10-23
6+
7+
### Added
8+
9+
* Web: **New** _Data Observability_ dashboard for _stats_ on OpenLineage events (`24hrs`, past `7.days`); views are also available for _sources_, _datasets_, and _jobs_; **new** job list view has also been introduced displaying the latest `N` runs (and duration) for a given job [`#2913`](https://github.com/MarquezProject/marquez/pull/2913) [@phixMe](https://github.com/phixMe)
10+
* Web: `404` page [`#2890`](https://github.com/MarquezProject/marquez/pull/2890) [@phixMe](https://github.com/phixMe)
11+
* Web: Display _parent_ job (if present) in _job_ panel [`#2868`](https://github.com/MarquezProject/marquez/pull/2868) [@phixMe](https://github.com/phixMe)
12+
* Web: Allow override of `web.port` via `WEB_PORT` environment variable [`#2838`](https://github.com/MarquezProject/marquez/pull/2838) [@bidlako](https://github.com/bidlako)
13+
* Web: Allow _nullable_ columns for schema in _dataset_ panel (use `N/A`) [`#2896`](https://github.com/MarquezProject/marquez/pull/2896) [@phixMe](https://github.com/phixMe)
14+
* Web: Better feedback when lineage events are loading [`#2916`](https://github.com/MarquezProject/marquez/pull/2916) [@NisargChokshi45](https://github.com/NisargChokshi45)
15+
* API: `Job` object will now return `Job.latestRuns` (latest `N` runs) and `Job.latestRun` (last run to execute) [`#2901`](https://github.com/MarquezProject/marquez/pull/2901) [@phixMe](https://github.com/phixMe)
16+
* API: Use `io.openlineage.server.*` pkg and **class** [`Metadata`](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/models/Metadata.java) (utility class for `OpenLineage.RunEvent`) [`#2853`](https://github.com/MarquezProject/marquez/pull/2853) [@wslulciuc](https://github.com/wslulciuc)
17+
* API: Use `TIMESTAMPTZ` for _timestamps_ in database; supports _Data Observability_ dashboard with timezone of user [`#2924`](https://github.com/MarquezProject/marquez/pull/2924) [@wslulciuc](https://github.com/wslulciuc)
18+
* API: Set `current_run_uuid` in **table** `jobs` optimizing query for `JobDao.findAll()` [`#2929`](https://github.com/MarquezProject/marquez/pull/2929) [@wslulciuc](https://github.com/wslulciuc)
19+
* API: **New** `GET` `/api/v1/jobs` [`#2930`](https://github.com/MarquezProject/marquez/pull/2930) [@wslulciuc](https://github.com/wslulciuc)
20+
* CLI: **New** cmd args for [`cli.MetadataCommand`](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/cli/MetadataCommand.java) [`#2923`](https://github.com/MarquezProject/marquez/pull/2923) [@wslulciuc](https://github.com/wslulciuc)
21+
* `--jobs`: _limits OL jobs up to N (default: 5)_
22+
* `--runs-per-job`: _limits OL run executions per job up to N (default: 10)_
23+
* `--runs-active`: _limits OL run executions marked as active (='RUNNING') up to N_
24+
* `--max-run-fails-per-job`: _maximum OL run fails per job (default: 2)_
25+
* `--min-run-duration`: _minimum OL run duration (in seconds) per execution (default: 300)_
26+
* `--run-start-time`: _specifies the OL run start time in UTC ISO ('YYYY-MM-DDTHH:MM:SSZ'); used for the initial OL run, with subsequent runs starting relative to the initial start time. (default: 2024-10-15T01:00:11.080828Z)_
27+
* `--run-end-time`: _specifies the OL run end time in UTC ISO ('YYYY-MM-DDTHH:MM:SSZ'); used for the initial OL run, with subsequent runs ending relative to the initial end time. (default: 2024-10-15T01:07:25.080828Z)_
28+
29+
### Fixed
30+
31+
* Web: Better rendering of long text [`#2942`](https://github.com/MarquezProject/marquez/pull/2942) [@phixMe](https://github.com/phixMe)
32+
* Web: Display full `runID` and check icon when copied [`#2940`](https://github.com/MarquezProject/marquez/pull/2940) [`#2941`](https://github.com/MarquezProject/marquez/pull/2941) [@wslulciuc](https://github.com/wslulciuc) [@phixMe](https://github.com/phixMe)
33+
* Web: Use **DatasetVersionAPI** to display latest schema and remove extra job facets API call in _dataset_ panel [`#2938`](https://github.com/MarquezProject/marquez/pull/2938) [@phixMe](https://github.com/phixMe)
34+
* Web: Use **DatasetAPI** for data quality assertions in _dataset_ panel [`#2937`](https://github.com/MarquezProject/marquez/pull/2937) [@phixMe](https://github.com/phixMe)
35+
* Web: Fill-in _job_ node in lineage graph with correct color for `JobEvent`s [`#2934`](https://github.com/MarquezProject/marquez/pull/2934) [@phixMe](https://github.com/phixMe)
36+
* Web: Fill-in _job_ node in lineage graph with correct color for run states `RUNNING`, `COMPLETED`, etc [`#2897`](https://github.com/MarquezProject/marquez/pull/2897) [@phixMe](https://github.com/phixMe)
37+
* API: Pagination for `DatasetVersion.findAll()`; not all dataset versions were returned for `GET` `/api/v1/namespaces/{namespace}/datasets/{dataset}/versions` [`#2944`](https://github.com/MarquezProject/marquez/pull/2945) [@inanalper](https://github.com/inanalper)
38+
* API: `null` namespace and dataset name in **view** `dataset_view` for old versions; use **table** `dataset_versions` instead in column lineage query [#2881](https://github.com/MarquezProject/marquez/pull/2881) [@sophiely](https://github.com/sophiely)
39+
* API: Missing `DELETE CASCADE` on **table** `job_facets` [`#2878`](https://github.com/MarquezProject/marquez/pull/2878) [@mattwparas](https://github.com/mattwparas)
40+
* API: Ensure `Job.latestRun` in `Job` object is set for runs in a `RUNNING` state; before `Job.latestRun` was set only for a run in a _done_ state (`COMPLETED` / `FAILED`) [`#2933`](https://github.com/MarquezProject/marquez/pull/2933) [@phixMe](https://github.com/phixMe)
41+
* CLI: Repurpose cmd `db-migrate` to run all pending database migrations, no longer coupling migrations with HTTP server startup [`#2936`](https://github.com/MarquezProject/marquez/pull/2936) [@davidjgoss](https://github.com/davidjgoss)
42+
* Chart: Missing common `labels` for `deployment.replicas` [`#2877`](https://github.com/MarquezProject/marquez/pull/2877) [@alaturqua](https://github.com/alaturqua)
443

544
## [0.49.0](https://github.com/MarquezProject/marquez/compare/0.48.0...0.49.0) - 2024-08-07
645

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -98,8 +98,8 @@ Versions of Marquez are compatible with OpenLineage unless noted otherwise. We e
9898
| **Marquez** | **OpenLineage** | **Status** |
9999
|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------|---------------|
100100
| [`UNRELEASED`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `CURRENT` |
101-
| [`0.49.0`](https://github.com/MarquezProject/marquez/blob/0.49.0/CHANGELOG.md#0490---2024-08-07) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `RECOMMENDED` |
102-
| [`0.48.0`](https://github.com/MarquezProject/marquez/blob/0.45.0/CHANGELOG.md#0480---2024-08-05) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `MAINTENANCE` |
101+
| [`0.50.0`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#0500---2024-10-23) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `RECOMMENDED` |
102+
| [`0.49.0`](https://github.com/MarquezProject/marquez/blob/0.49.0/CHANGELOG.md#0490---2024-08-07) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `MAINTENANCE` |
103103

104104
> **Note:** The [`openlineage-python`](https://pypi.org/project/openlineage-python) and [`openlineage-java`](https://central.sonatype.com/artifact/io.openlineage/openlineage-java) libraries will a higher version than the OpenLineage [specification](https://github.com/OpenLineage/OpenLineage/tree/main/spec) as they have different version requirements.
105105

‎api/build.gradle‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,8 @@ dependencies {
5151
implementation 'com.graphql-java:graphql-java:20.9'
5252
implementation 'com.graphql-java-kickstart:graphql-java-servlet:12.0.0'
5353

54-
implementation 'org.opensearch.client:opensearch-rest-client:2.15.0'
55-
implementation 'org.opensearch.client:opensearch-java:2.6.0'
54+
implementation 'org.opensearch.client:opensearch-rest-client:2.17.1'
55+
implementation 'org.opensearch.client:opensearch-java:2.16.0'
5656

5757
testImplementation "io.dropwizard:dropwizard-testing:${dropwizardVersion}"
5858
testImplementation "org.jdbi:jdbi3-testing:${jdbi3Version}"

‎api/src/main/java/marquez/MarquezApp.java‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
import marquez.api.filter.JobRedirectFilter;
2929
import marquez.api.filter.exclusions.Exclusions;
3030
import marquez.api.filter.exclusions.ExclusionsConfig;
31-
import marquez.cli.DbMigrationCommand;
31+
import marquez.cli.DbMigrateCommand;
3232
import marquez.cli.DbRetentionCommand;
3333
import marquez.cli.MetadataCommand;
3434
import marquez.cli.SeedCommand;
@@ -90,6 +90,7 @@ public void initialize(@NonNull Bootstrap<MarquezConfig> bootstrap) {
9090
new EnvironmentVariableSubstitutor(ERROR_ON_UNDEFINED)));
9191

9292
// Add CLI commands
93+
bootstrap.addCommand(new DbMigrateCommand());
9394
bootstrap.addCommand(new DbRetentionCommand());
9495
bootstrap.addCommand(new MetadataCommand());
9596
bootstrap.addCommand(new SeedCommand());
@@ -202,12 +203,6 @@ public void registerResources(
202203
}
203204
}
204205

205-
@Override
206-
protected void addDefaultCommands(Bootstrap<MarquezConfig> bootstrap) {
207-
bootstrap.addCommand(new DbMigrationCommand<>(this));
208-
super.addDefaultCommands(bootstrap);
209-
}
210-
211206
private void registerServlets(@NonNull Environment env) {
212207
log.debug("Registering servlets...");
213208

‎api/src/main/java/marquez/MarquezContext.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ private MarquezContext(
155155
this.tagService = new TagService(baseDao);
156156
this.tagService.init(tags);
157157
this.openLineageService = new OpenLineageService(baseDao, runService);
158-
this.lineageService = new LineageService(lineageDao, jobDao);
158+
this.lineageService = new LineageService(lineageDao, jobDao, runDao);
159159
this.columnLineageService = new ColumnLineageService(columnLineageDao, datasetFieldDao);
160160
this.searchService = new SearchService(searchConfig);
161161
this.statsService = new StatsService(statsDao);

‎api/src/main/java/marquez/cli/DbMigrateCommand.java‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/*
2+
* Copyright 2018-2022 contributors to the Marquez project
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
package marquez.cli;
7+
8+
import io.dropwizard.cli.ConfiguredCommand;
9+
import io.dropwizard.db.DataSourceFactory;
10+
import io.dropwizard.db.ManagedDataSource;
11+
import io.dropwizard.setup.Bootstrap;
12+
import lombok.NonNull;
13+
import lombok.extern.slf4j.Slf4j;
14+
import marquez.MarquezConfig;
15+
import marquez.db.DbMigration;
16+
import net.sourceforge.argparse4j.inf.Namespace;
17+
18+
/**
19+
* A command to manually run database migrations. This command to be used to run migrations
20+
* decoupled from application deployment.
21+
*/
22+
@Slf4j
23+
public class DbMigrateCommand extends ConfiguredCommand<MarquezConfig> {
24+
25+
public DbMigrateCommand() {
26+
super("db-migrate", "A command to manually run database migrations.");
27+
}
28+
29+
@Override
30+
protected void run(
31+
@NonNull Bootstrap<MarquezConfig> bootstrap,
32+
@NonNull Namespace namespace,
33+
@NonNull MarquezConfig configuration)
34+
throws Exception {
35+
36+
final DataSourceFactory sourceFactory = configuration.getDataSourceFactory();
37+
final ManagedDataSource source =
38+
sourceFactory.build(bootstrap.getMetricRegistry(), "MarquezApp-source");
39+
40+
DbMigration.migrateDbOrError(configuration.getFlywayFactory(), source, true);
41+
}
42+
}

‎api/src/main/java/marquez/cli/DbMigrationCommand.java‎

Lines changed: 0 additions & 113 deletions
This file was deleted.

0 commit comments

Comments
 (0)