Overview

Core 2.9.1 is a patch release focused on bugfixes and security. We also introduce a number of important features related to cyclic pipelines, usability and cost-effectiveness:

Bug fixes

• Allow Core 2 to work reliably with pod disruption budgets (PDBs) (#6560). Previously, terminating/draining pods remained Ready: True, which meant that they were still considered non-disrupted from a PDB perspective.
• Make model-gateway request timeouts configurable (previously 10 minutes, new default 2 minutes) (#6522). This is a fix for particular cases when inference requests overload the available inference server replicas, leading to significant latency increases. Previously in such cases, a large backlog of unprocessed entries gathered in kafka model input topics, leading to new requests always timing out (enough requests ahead of them in the queue for timeout at LB-level). The new default is likely too high for most usecases, but is set conservatively so as to not prevent inferences for slow models. However, timeouts can be set matching your own workloads at the model-gateway pod level, via the MODELGATEWAY_WORKER_TIMEOUT_MS environment variable.
• Fix dataflow-engine edge cases for communicating to the scheduler (#6506). In the case of multiple dataflow-engine replicas, or a single restarting replica, the status of a Pipeline was set based on the last received message. Sometimes, due to the way locking was done, the status update from a terminating dataflow-engine replica was processed after updates from the replica started by k8s to replace it. This meant that pipelines would transition into PipelineTerminated states in a non-deterministic way.

New features

• Allow pipelines to have cycles, with a bounded number of iterations (#6413, #6480, docs). This feature is enabled on a pipeline-by-pipeline basis via a newly added spec.allowCycles field in the CR.
• Allow the Core 2 operators to be installed within their own namespaces but manage Core 2 CRs in a list of other namespaces (#6434, docs)
• Allow end-users to delete the kafka topics associated with a model or pipeline when deleting the model/pipeline (#6353, #6383, docs). Care needs to be taken in using this feature because it implies loosing observability with respect to historic inference requests/responses, potentially targeted at models within a pipeline.

Docs improvements

• Expanded documentation on Autoscaling and Performance Tuning
• Quickstart guide

Core 2 release images

• Core 2 images published on docker hub now have embedded SBOM attestations.
• At the moment of the release, security scans show zero CVEs within those images.

CRD Updates

All CRD changes in this release maintain backward compatibility, so clusters with existing CRs can be migrated seamlessly. Please see specific feature docs for changes.

Kudos:

We recognise the significant contributions to this release from @RobertSamoilescu

With contributions from @RobertSamoilescu, @lc525, @Rajakavitha1, @paulb-seldon

Changelog

Dates are displayed in UTC. Generated by auto-changelog.

v2.9.1

9 July 2025

• build(deps): fix k6 Dockerfile after version update #6603
• build(deps): update internal package references, k6 #6602
• build(deps): bump grafana/grafana from 11.5.2 to 12.0.2 in /scheduler #6543
• build(deps): bump ubi9/ubi-minimal from 9.5 to 9.6 in /scheduler #6596
• Update README.md #6598
• Update core-autoscaling-servers.md #6599
• fix(ci): kustomize version environment var #6581
• fix(build): scheduler Makefile for building docker images with SBOM #6580
• fix(github): Update image build action to support SBOM attestations #6579
• feat(Agent): add readiness probes #6560
• fix(operator/lint): fix deprecated golangci settings #5887
• fix(o11s): Filter gpu metrics on exported_namespace #6377
• Re-generate license info #6575
• Bump envoyproxy/envoy from v1.33.0 to v1.34.1 in /scheduler #6444
• build(deps): update Core 2 dependencies #6574
• Update README.md - fix link #6572
• Replaced time.sleep with asyncio.sleep #6569
• add quickstart demo #6475
• feat(modelgateway): Added support for custom worker timeout #6522
• Autoscaling Docs #6500
• Update concepts #6437
• Update SUMMARY.md #6539
• Revert "Update autoscaling.md (#6534)" #6536
• Update autoscaling.md #6534
• update readme initial #6523
• Update istio.md #6517
• Update faqs.md #6515
• docs: update FAQs #6511
• docs: Update faqs.md #6510
• fix: dataflow message reconcile #6506
• Updated docs with maxStepRevisits #6507
• feat: detect and stop cycles #6480
• Perf-tuning docs #6491
• docs: added page descriptions #6492
• Comment out broken links #6478
• Update Scaling Docs #6476
• Updated docs for installing the operator with watchNamspaces flag. #6455
• docs(pipelines): Docs for cyclic pipelines (counter) #6440
• docs(Kafka): Included docs for managing Kafka topics #6438
• Update 2.8 to 2.9 upgrade notes #6450
• Link to new docs across our gh repo #6448
• (docs) Update the dependencies table #6446
• docs: updating docs to support kubectl and seldon-cli #6219
• feat: operator watch namespaces #6434
• feat: custom kafka stream join window #6419
• feat: allow removal of kafka pipeline topics #6383
• docs: Update the concepts section #6417
• feat: allow removal of kafka model topics #6353
• feat: cyclic pipelines #6413
• docs added a banner #6390
• docs(pipelines): Fix "Chain" example in the "Pipeline" doc #6370
• docs: add the release notes #6384
• Update SUMMARY.md #6385
• docs fix: added a banner #6376
• feat(o11y): GPU Grafana dashboard #6374
• Generating changelog for v2.9.1 8111172
• Setting version for helm charts 2d91dee
• Setting version for yaml manifests 0cdcb04

Overview

Core 2.9 is a feature-packed release looking to improve usability, simplify operations via autoscaling and scheduling improvements, and support streaming usecases (via model response streaming for both REST and gRPC clients).

Core 2 also has new docs, with revamped content and structure. Documentation will continuously improve to address advanced configurations and use cases.

CRD Updates:

All CRD changes in this release maintain backward compatibility, so clusters with existing CRs can be migrated seamlessly.

• Add status.availableReplicas field to the Model CRD (#5873). Part of the partial scheduling feature. Field not directly set by end-users, but updated by the seldon k8s operator
• Add spec.llm field to the Model CRD (#6234). The field is used by the PromptRuntime (in Seldon's LLM Module) to reference a LLM model. Only one of spec.llm and spec.explainer should be set at a given time. This allows the deployment of multiple "models" acting as prompt generators for the same LLM.

(Main) Features:

• We add inference response streaming support for REST (via SSE) and gRPC for MLServer models that have streaming support (#6293, #6292). This requires MLServer >= 1.6.0.

• We introduce partial scheduling for model replicas (#6221, docs), improving the behaviour of Core 2 during autoscaling. With this new feature, the Core 2 scheduler will try to load as many of the requested model replicas as possible, even when no inference server has sufficient replicas to meet this request.
  Partial scheduling is only active when end-users provide spec.minReplicas in a model manifest (as a user-provided minimum for considering the model "available"), and takes effect when there is a suitable inference server with at least this number of replicas. With partial scheduling, a model can be:

  • Fully scheduled: spec.replicas == status.availableReplicas; The ModelReady condition is True with message ModelAvailable. All requested replicas serve inference requests.
  • Partially scheduled: status.availableReplicas >= spec.minReplicas but status.availableReplicas < spec.replicas; The ModelReady condition is True with message ModelAvailable. Core 2 was not able to find sufficient server replicas to load all requested replicas for this model. This state may be transitory, for example when new server replicas are being created but not yet available. The available model replicas serve inference requests.
  • Not able to schedule: no suitable inference servers that have a number of replicas greater or equal to the model's spec.minReplicas could be found. The ModelReady condition is False with message ScheduleFailed. Some model replicas may still be available for inference requests (for example, if the model was previously loaded on a server that was forced to scale-down below the model's spec.minReplicas)

• We introduce mixed native/HPA autoscaling (#6218, #6222, #6235, with docs for model and server autoscaling) that:

  1. enables end-users to configure a single HPA manifest, controlling model replicas.
  2. works for multi-model serving scenarios (MMS)

  When using this feature, servers are scaled-up/down natively by Core 2 in response to changes in model replicas. If a model scales up and there aren't sufficient server replicas to host it, the number of server replicas is increased; if a model scales down and a server replica remains without any loaded models, the number of server replicas is reduced.

  We also introduce experimental functionality to pack models on fewer inference servers on model scale-down, but this is disabled by default and will be improved in future releases. See the scale-down docs for details.

• Model scheduling now takes into account model memory requirements based on the inference server config and how many in-memory copies of one model it creates (the parallel_workes MLServer setting and instance_group configurations in Triton). For triton, only KIND_CPU instance groups are considered at this point (#6054)

• Log levels for all internal components (#6312) and the envoy accesslog (#6295) can now be controlled in a consistent way.

Features configuration & helm chart updates

• Server spec.minReplicas and spec.maxReplicas can be configured via helm (#6283) via the following values:

  • mlserver.minReplicas
  • mlserver.maxReplicas
  • triton.minReplicas
  • triton.maxReplicas

• Native autoscaling features control (#6301, #6286). All options here have corresponding command-line arguments that can be passed to seldon-scheduler when not using helm as the install method. The following helm values can be set

  • autoscaling.autoscalingModelEnabled, with corresponding cmd line arg: --enable-model-autoscaling (defaults to false): enable or disable native model autoscaling based on lag thresholds. Enabling this assumes that lag (number of inference requests "in-flight") is a representative metric based on which to scale your models in a way that makes efficient use of resources.
  • autoscaling.autoscalingServerEnabled with corresponding cmd line arg: --enable-server-autoscaling(defaults to "true"): enable to use native server autoscaling, where the number of server replicas is set according to the number of replicas required by the models loaded onto that server.
  • autoscaling.serverPackingEnabled with corresponding cmd line arg: --server-packing-enabled(experimental, defaults to "false"): enable server packing to try and reduce the number of server replicas on model scale-down.
  • autoscaling.serverPackingPercentage with corresponding cmd line arg: --server-packing-percentage(experimental, defaults to "0.0"): controls the percentage of model replica removals (due to model scale-down or deletion) that should trigger packing

• The inference server PVC retention policy can now be configured via helm (#6056) via the following values:

  • mlserver.statefulSetPersistentVolumeClaimRetentionPolicy.whenDeleted
  • mlserver.statefulSetPersistentVolumeClaimRetentionPolicy.whenScaled
  • triton.statefulSetPersistentVolumeClaimRetentionPolicy.whenDeleted
  • triton.statefulSetPersistentVolumeClaimRetentionPolicy.whenScaled

• Logging options can be configured either via helm values or by passing arguments to individual Core 2 components. Docs for component log levels and the envoy access log are available

Please consult the helm chart docs for a full list of options

(Main) Bugs Fixed:

• Transient NC 503 error in envoy logs when rolling out a new version of a model
  • fixed in #6082 by configuring envoy to use the Aggregated Discovery Service (ADS) in order to add guarantees regarding the order of routing updates coming from the Core 2 scheduler.
• Mislabelled operational metrics for experiments in Prometheus
  • fixed in #6118 by recording the actual model name in the model label rather than the experiment name;
• Some errors encountered by modelgateway were silently ignored
  • fixed in #6014 by propagating those errors via Kafka, by writing into an error topic (DLQ)
• seldon-scheduler pod spec overrides (alongisde any other stateful-set pod spec overrides) defined within the SeldonRuntime CR were ignored
  • fixed in #6349

Priority bugfixes scheduled for the next patch releases of Core 2.9

• [BUG] When there is a network partition between dataflow-engine and the kafka cluster, and dataflow-engine is restarted, pipelines may sometimes be marked as PipelineTerminated with the message "pipeline removed` after the network partiton is solved. The current workaround is to delete any Pipeline in this state and re-deploy the same manifest into the Core 2 cluster.

Kudos:

We would like to highlight the exceptional contributions that the following team members have brought to this release and to Core 2 so far:

• Sherif Akoush (@sakoush)

With contributions from @sakoush, @lc525, @driev, @RobertSamoilescu, @Rajakavitha1, @paulb-seldon, @tyndria

Changelog

Dates are displayed in UTC. Generated by auto-changelog.

v2.9.0

7 April 2025

• fix(dataflow): Update default kafka log level for dataflow engine #6367
• feat(docs): Server native autoscaling #6356
• Bump ubi9/openjdk-17-runtime from 1.20 to 1.22 in /scheduler #6359
• docs(pipelines): Minor Pipelines doc improvements #6351
• fix: adjust logging level for dataflow #6350
• fix(operator): Apply scheduler runtime podSpec override #6349
• Spelling fix #6340
• Add a missing space [#6338](https:...

Overview

This release introduces stability fixes related to upgrading from previous versions and recovering from network issues gracefully (by using keepalive patterns). It also adds some usability improvements of the system.

We also improve documentations on using HPA autoscaling and Core 2 architecture.

(Main) Features:

• Add keepalive settings (server-side: #6016, client-side: #5978)
• Expose stateful set persistent volume claim retention policy #5946
• Set a time-to-live (TTL) on deleted pipelines and experiments #5948
• Improve control plane synchronisation #6029, #6021 and #6020

(Main) Bugs Fixed:

• Make status.selector field optional #5985
• Closing off envoy admin interface #5936

Changelog

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v2.8.5

7 November 2024

• fix(ci): Changes from v2 for release 2.8.5 #6034
• add keepalive settings to all envoy configs #6033
• fix(ci): Merge changes from v2 for release 2.8.5 (4) #6032
• fix: Use generation id to bootstrap version id #6029
• feat(docs): add more configuration details to the HPA docs #6019
• fix(scheduler): always send model events for deleted models #5992
• feat(scheduler): exposing the deleted resource ttl as a CLI param #5994
• fix(ci): Changes from v2 for release 2.8.5 (3) #6023
• fix(dataflow): make each replica use unique subscription names #6021
• fix(ci): fix flaky test #6022
• fix(agent): allow serial order for servers connection per instance #6020
• fix(ci): Changes from v2 for release 2.8.5 (2) #6018
• fix(modelgateway): Allow modelgateway consumers for transient error #6017
• fix: Add server keep alive enforcement policy #6016
• fix(docs): Update docs for scheduler #6013
• fixed the yaml format #5998
• fix(docs): Migrated the traefik service mesh content #5996
• fix(scheduler): test for deleted resources that are recreated #5993
• ci: Merge changes from v2 branch for 2.8.5 release #5988
• fix: Deal with zero servers control plane synchronisation #5987
• fix: Make status.selector field optional #5985
• feat(scheduler): set a ttl on deleted pipelines and experiments #5948
• feat(envoy): closing off the admin interface #5936
• fix: Use keep alive grpc settings on client side #5978
• feat: stateful set persistent volume claim retention policy #5946
• feat(k6): Add k6 controlplane with dataplane runs #5977
• Update Changelog #5972
• Generating changelog for v2.8.5 545eb41
• checkout changelog from v2 bfa5d64
• Generating changelog for v2.8.5-rc1 74f6872
• Setting version for helm charts 91a90fa
• Setting version for helm charts 337013e

Overview

This release introduces a number of features that help users run core 2 with dynamic loads. We added support for HPA model & server autoscaling based on custom metrics in the single-model-serving case. Documentation showing an example of scaling based on model RPS is described here.

This release also reduces the risk of data plane outage in the case of control plane issues (specifically when seldon-scheduler is restarting). This is done by synchronising the process of the scheduler (re)start with the rest of the system.

A number of features have been added, most notably users can now upgrade a model to a new version that is part of an ongoing experiment.

We are migrating docs to a new site here (experimental).

(Main) Features:

• Add gitbook support (experimental) #5943
• Add metadata object to a seldonconfig component #5918
• seldon-scheduler startup synchronisation #5930
• Envoy Grafana Dashbaord #5894
• Upgrade models in experiment #5874
• Add pipeline and model name validation #5872
• Add retry parameters #5875

(Main) Bugs Fixed:

• Skip out of order control messages in seldon-agent #5969
• Use StatefulSet.Spec.Replicas as reference for Server status #5945
• Cleaning up models that fail to load #5857 and #5830

Changelog

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v2.8.4

11 October 2024

• fix(ci): Changes from v2 for release 2.8.4 (5) #5970
• fix(docs): revert changes for changelog.md #5971
• fix(agent): Skip agent out of order control messages #5969
• Update README.md #5955
• fix(docs): changed the name from core v2 to Core 2 #5963
• fix(ci): Changes from v2 for release 2.8.4 (4) #5968
• increase drain grace period wait #5967
• refactor: error checks and nil pointer check #5964
• fix(ci): changes from v2 for release 1.8.4 (3) #5966
• fix(scheduler): Add grace period for model unload #5965
• Update README.md #5962
• fix(ci): Changes from v2 for release 2.8.4 (2) #5956
• fix(controller): Adjust controller -> scheduler state recreation upon scheduler disconnect. #5944
• feat(docs): improve HPA autoscaling docs #5953
• fix(docs): fix HPA-autoscaling formula in old docs #5950
• feat: bump k8s kind to 1.31 #5954
• Merge GitBook Docs with V2 master #5943
• feat: adding -ldflag and -trimpath to go build #5940
• fix(k8s/yaml): add timing environment variables to triton agent #5949
• fix(operator): Use StatefulSet.Spec.Replicas as reference for Server status #5945
• Bump ubi9/ubi-minimal in /scheduler #5941
• Bump rclone/rclone from 1.68.0 to 1.68.1 in /scheduler #5942
• fix(ci): Release 2.8.4 #5934
• feat(charts): adding metadata object to a seldonconfig component #5918
• feat(operator): add Model selector for scale subresource to enable HPA-based scaling #5932
• fix(scheduler): Scheduler wait on server connect #5930
• Bump envoyproxy/envoy from v1.31.1 to v1.31.2 in /scheduler #5923
• fix headers key for http k6 test #5920
• feat(chart): exposed retry constants as helm chart #5889
• create a CDS cluster per model #5916
• feat(o11y): Envoy Grafana Dashbaord #5894
• Bump ubi9/ubi-minimal from 9.4-1227 to 9.4-1227.1725849298 in /scheduler #5913
• Bump rclone/rclone from 1.67.0 to 1.68.0 in /scheduler #5914
• Bump envoyproxy/envoy from v1.31.0 to v1.31.1 in /scheduler #5915
• fix(scheduler): Controller to update the list of servers to scheduler on reconnect #5893
• fix(bug): time unit fix #5892
• feat(env): parameters exposed as env variables #5875
• Bump ubi9/ubi-micro from 9.4-13 to 9.4-15 in /scheduler #5877
• Bump ubi9/ubi-minimal from 9.4-1194 to 9.4-1227 in /scheduler #5878
• Bump ubi9/ubi-micro from 9.4-13 to 9.4-15 in /hodometer #5879
• Bump github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka #5881
• Bump ubi9/ubi-micro from 9.4-13 to 9.4-15 in /operator #5885
• feat: pipeline and model name validation #5872
• fix(envoy): upgrade models in experiment #5874
• feat(grafana): add performance dashboard #5852
• fix(golangci-lint): version bump and location fix #5853
• Re-generate license info #5871
• Re-generate license info #5869
• Bump github.com/envoyproxy/go-control-plane in /scheduler #5837
• Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 in /operator #5841
• Bump grafana/grafana from 11.1.3 to 11.2.0 in /scheduler #5858
• Bump google.golang.org/grpc from 1.65.0 to 1.66.0 in /hodometer #5861
• fix(model-gateway): Create consumer if model can serve traffic #5865
• fix(agent): cleaning up models that fail to load #5857
• Bump google.golang.org/protobuf from 1.34.1 to 1.34.2 in /operator #5820
• Bump grafana/grafana from 11.1.1 to 11.1.3 in /scheduler #5823
• fix(agent): cleanup rclone directory on errors #5830
• feat(scheduler): Add max elapsed duration for model load/unload #5819
• Bump github.com/onsi/gomega from 1.33.1 to 1.34.0 in /scheduler #5799
• fix: Add experiments pipeline tests for envoy processor #5810
• Re-generate license info #5813
• Re-generate license info #5811
• Bump ubi9/ubi-minimal from 9.4-1134 to 9.4-1194 in /scheduler #5800
• Bump grafana/grafana from 11.1.0 to 11.1.1 in /scheduler #5801
• Bump ubi9/ubi-micro from 9.4-9 to 9.4-13 in /hodometer #5795
• Bump ubi9/ubi-micro from 9.4-9 to 9.4-13 in /operator #5798
• Bump ubi9/ubi-micro from 9.4-9 to 9.4-13 in /scheduler [#5802](https://github.com/SeldonIO/seldo...

Overview

This release is targeted to fix the experiments worflow so that the system can deal with deleted experiments properly on restart of components, specifically seldon-scheduler.

As part of the work done the system can now:

• Handle different versions of the seldon-scheduler local embedded db (BadgerDB), with the ability to migrate to the latest version seamlessly.
• Recover the state of the scheduler from k8s if this local embedded db is corrupted / lost, which includes recovering both experiments and pipelines that exist in k8s.

(Main) Bugs Fixed:

• scheduler/operator: Deal with deleted experiments when restoring from cache #5726
• scheduler: Do not report back draining servers for status #5761

(Main) Features:

• operator: add custom printcolumns for seldon-specific CRs #5736

(Main) Upgrades:

• use mlserver 1.6.0 #5724

Changelog

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v2.8.3

17 July 2024

• ci: Merge change from v2 for release 2.8.3 #5762
• fix(scheduler): Do not report back draining servers for status #5761
• ci: Changes from v2 for release 2.8.3 #5751
• fix: Deal with deleted experiments when restoring from cache #5726
• fix(ansible): add explicit dependency on community.docker collection #5746
• fix(golang-lint): update lint config file to remove deprecated settings #5747
• build(grpc): update Makefile responsible for code generation #5749
• Bump ubi9/openjdk-17-runtime from 1.18 to 1.20 in /scheduler #5666
• Bump github.com/go-playground/validator/v10 in /scheduler #5712
• Bump org.junit.jupiter:junit-jupiter-params in /scheduler/data-flow #5729
• Re-generate license info #5737
• feat(operator): add custom printcolumns for seldon-specific CRs #5736
• Re-generate license info #5735
• Re-generate license info #5734
• Bump envoyproxy/envoy from v1.30.2 to v1.30.4 in /scheduler #5731
• Bump grafana/grafana from 11.0.0 to 11.1.0 in /scheduler #5732
• Re-generate license info #5725
• use mlserver 1.6.0 #5724
• Trigger license check on changes to v2 branch #5721
• Re-generate license info #5723
• 3rd party license #5708
• feat: Add experiments to stress tests #5714
• feat(k6): add automated state consistency checking #5661
• Generating changelog for v2.8.3 a5989f0
• Generating changelog for v2.8.3-rc1 346fd1e
• Setting version for helm charts 7f07554
• Setting version for helm charts c4795ef
• Setting version for helm charts 63da23e

Bugs Fixed:

• Pipeline terminating fix on rebalance #5703
• Do not create new KafkaStreams app for existing pipelines #5550
• Handle unload too quick after load #5504

Features:

• Add pipeline version to Kafka headers #5493

Changelog

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v2.8.2

26 June 2024

• ci: Merge changes from v2 to release 2.8 branch #5705
• fix(scheduler): Pipeline terminating fix on rebalance #5703
• Bump ubi9/ubi-micro from 9.4-6.1716471860 to 9.4-9 in /operator #5682
• Bump ubi9/ubi-micro from 9.4-6.1716471860 to 9.4-9 in /hodometer #5687
• Bump github.com/go-logr/logr from 1.4.1 to 1.4.2 in /operator #5686
• Bump google.golang.org/protobuf from 1.34.1 to 1.34.2 in /scheduler #5691
• Bump ubi9/ubi-micro from 9.4-6.1716471860 to 9.4-9 in /scheduler #5695
• Bump ubi9/ubi-minimal from 9.4-949.1717074713 to 9.4-1134 in /scheduler #5696
• Bump rclone/rclone from 1.66.0 to 1.67.0 in /scheduler #5697
• Bump ubi9/ubi-minimal in /scheduler #5667
• Bump envoyproxy/envoy from v1.30.1 to v1.30.2 in /scheduler #5668
• Bump go.opentelemetry.io/otel/sdk from 1.26.0 to 1.27.0 in /scheduler #5677
• Bump go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux #5679
• Bump ubi9/ubi-micro from 9.4-6 to 9.4-6.1716471860 in /hodometer #5654
• Bump go.opentelemetry.io/otel/trace from 1.26.0 to 1.27.0 in /scheduler #5640
• Bump grafana/grafana from 10.4.3 to 11.0.0 in /scheduler #5644
• Bump ubi9/ubi-micro from 9.4-6 to 9.4-6.1716471860 in /operator #5656
• Bump github.com/hashicorp/go-retryablehttp in /hodometer #5657
• Bump ubi9/ubi-minimal in /scheduler #5659
• Bump ubi9/ubi-micro from 9.4-6 to 9.4-6.1716471860 in /scheduler #5660
• bump(deps): major version updates in /scheduler/data-flow #5639
• fix(ansible): allow multiple custom secrets without namespaces #5638
• feat: Add pipelines to core2 qa control plane tests #5636
• test(k6): Add k8s-based control plane tests #5609
• Bump go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux #5627
• Bump google.golang.org/protobuf from 1.34.0 to 1.34.1 in /scheduler #5628
• Bump grafana/grafana from 10.4.2 to 10.4.3 in /scheduler #5611
• build(ansible): add options to install grafana and dashboards #5589
• fix(deps): Bump org.apache.kafka:kafka-streams-test-utils from 7.6.0-ccs to 7.6.1-ccs in /scheduler/data-flow #5538
• Bump google.golang.org/protobuf from 1.33.0 to 1.34.1 in /operator #5590
• Bump github.com/confluentinc/confluent-kafka-go/v2 in /operator #5591
• Bump github.com/confluentinc/confluent-kafka-go/v2 in /scheduler #5596
• docs: Reference MLServer infer deprecation #5606
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace in /scheduler #5595
• Bump github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka #5597
• Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp #5598
• Bump ubi9/ubi-minimal from 9.4-949 to 9.4-949.1714662671 in /scheduler #5599
• Bump github.com/prometheus/client_golang in /scheduler #5600
• Bump github.com/hashicorp/go-retryablehttp in /hodometer #5601
• docs: replace references to Seldon Deploy with Seldon Enterprise Platform #5551
• fix(k6): allow k8s CR updates during load test #5587
• feat(k6): add support for k8s loading/unloading of Seldon CRs #5563
• Bump com.michael-bull.kotlin-retry:kotlin-retry in /scheduler/data-flow #5537
• Bump google.golang.org/protobuf from 1.33.0 to 1.34.0 in /scheduler #5575
• Bump ubi9/ubi-micro from 9.3-15 to 9.4-6 in /operator #5579
• Bump github.com/go-playground/validator/v10 in /scheduler #5576
• Bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc #5577
• Bump github.com/prometheus/client_golang in /scheduler #5578
• Bump ubi9/ubi-micro from 9.3-15 to 9.4-6 in /scheduler #5581
• Bump ubi9/ubi-micro from 9.3-15 to 9.4-6 in /hodometer #5567
• Bump ubi9/ubi-minimal from 9.3-1612 to 9.4-949 in /scheduler #5580
• fix(deps): Upgrade sigs.k8s.io/controller-runtime to 0.17.4 #5562
• fix(dataflow): do not create new KafkaStreams app for existing pipelines #5550
• Bump go.opentelemetry.io/otel/sdk from 1.24.0 to 1.26.0 in /scheduler #5554
• Bump github.com/onsi/gomega from 1.32.0 to 1.33.1 in /operator #5558
• Bump github.com/onsi/gomega from 1.32.0 to 1.33.1 in /scheduler #5559
• Bump envoyproxy/envoy from v1.29.3 to v1.30.1 in /scheduler #5548
• fix: Upgrade go 1.21 and golint to 1.57 #5556
• Update CONTRIBUTING.md to add notice on licensing changes and links to previous contributions #5555
• Re-generate license info #5528
• fix: Deal with multi model env vars #5527
• build(ansible): use of local dev images in kind cluster #5318
• build(deps): bump io.grpc:grpc* from 1.62.2 to 1.63.0 in scheduler/dataflow #5522
• Bump google.golang.org/grpc from 1.62.1 to 1.63.2 in /hodometer #5516
• Bump envoyproxy/envoy from v1.29.2 to v1.29.3 in /scheduler #5503
• Bump google.golang.org/grpc from 1.62.1 to 1.63.2 in /operator #5514
• Bump google.golang.org/grpc from 1.62.1 to 1.63.2 in /scheduler #5518
• Bump grafana/grafana from 10.4.1 to 10.4.2 in /scheduler #5521
• feat: Add kube api for k6 scaffolding #5494
• fix: Handle unload too quick after load #5504
• feat(dataflow): Add pipeline version to Kafka headers #5493
• build(dataflow): Add ktlint plugin in Gradle, run formatter, and fix lint issues #5491
• Bump github.com/cenkalti/backoff/v4 from 4.2.1 to 4.3.0 in /scheduler #5487
• Bump github.com/cen...

What's Changed

• docs(openapi): Use command with file by @jesse-c in #5349
• fix(logging): adserver is very noisy for feedback requests, possibly affecting performance by @majolo in #5351
• feat(build): use github actions to build docker images by @RafalSkolasinski in #5380
• fix(build): fix pipeline by @RafalSkolasinski in #5389
• Create license.yml dummy for v2 by @sakoush in #5466
• add licensing update to contributing.md by @paulb-seldon in #5510
• Update oss_commits.md by @paulb-seldon in #5529
• Feature: support multi hosts in vs by @ooooona in #5430
• Removed Seldon Deploy and added Seldon Enterprise Platform where necessary by @ramonpzg in #5552
• add generated dependencies licence update by @gsunner in #5637
• fix(helm): helm-docs installation + version bump by @RafalSkolasinski in #5635

New Contributors

• @ooooona made their first contribution in #5430

Full Changelog: v1.18.1...v1.18.2

Changelog

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v2.8.1

21 March 2024

• ci: Merge changes from v2 to release 2.8 branch (for 2.8.1) #5470
• do not mark source in terminate events of old pipelines #5469
• Re-generate license info #5468
• feat(ci): Add license workflow #5465
• ci: Merge changes from v2 to release 2.8 branch (for 2.8.1) #5467
• fix: update 3rd party licenses #5462
• ci: Merge changes from v2 to release 2.8 branch (for 2.8.1) #5460
• fix(dataflow) CVEs related to com.microsoft.azure:adal4j #5458
• Bump ubi9/ubi-micro from 9.3-13 to 9.3-15 in /scheduler #5457
• fix: Add pipelines in stress test #5437
• ci: Merge changes from v2 to release 2.8 branch (for 2.8.1) #5436
• fix(scheduler): Introduce sendMsg timeout for grpc streams (scheduler -> controller) #5434
• build(deps): bump dataflow dependency versions #5429
• Bump ubi9/openjdk-17-runtime from 1.17 to 1.18 in /scheduler #5401
• fix(modelgateway): KafkaAdmin error when fetching OIDC token #5428
• fix(modelgateway): kafka topics with correct number of partitions #5427
• Bump github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka #5420
• Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /scheduler #5421
• Bump github.com/golang/protobuf from 1.5.3 to 1.5.4 in /scheduler #5422
• Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /operator #5424
• Bump rclone/rclone from 1.65.2 to 1.66.0 in /scheduler #5410
• Bump google.golang.org/grpc from 1.62.0 to 1.62.1 in /hodometer #5412
• Bump ubi9/ubi-minimal from 9.3-1552 to 9.3-1612 in /scheduler #5409
• Bump envoyproxy/envoy from v1.29.1 to v1.29.2 in /scheduler #5408
• Bump grafana/grafana from 10.3.3 to 10.4.0 in /scheduler #5407
• Bump ubi9/ubi-micro from 9.3-13 to 9.3-15 in /operator #5417
• Bump ubi9/ubi-micro from 9.3-13 to 9.3-15 in /hodometer #5418
• Bump google.golang.org/grpc from 1.62.0 to 1.62.1 in /scheduler #5423
• Bump google.golang.org/grpc from 1.62.0 to 1.62.1 in /operator #5425
• fix(controller): reload models upon reconnect to the scheduler #5411
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc #5395
• fix(scheduler): Increase event hub buffer inline with other components #5406
• fix(dataflow): catch all exceptions when creating a KafkaStreams object #5405
• docs(SeldonRuntime): Add an example for overriding PodSpec in SeldonRuntime #5404
• fix: Use mlserver 1.5.0 #5403
• Bump google.golang.org/grpc from 1.61.1 to 1.62.0 in /scheduler #5393
• Bump github.com/go-playground/validator/v10 in /scheduler #5392
• Bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc #5394
• build(envoy): Update envoy version to 1.29.1 #5391
• Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 in /scheduler #5396
• Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 in /hodometer #5400
• fix(build): force update dependency of k8s java-client to fix high CVEs #5402
• build(license): Fix license reference and add license files to Dockerfiles #5390
• fix(dataflow) be more strict about error states on pipeline creation #5381
• Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp #5370
• Bump google.golang.org/grpc from 1.61.0 to 1.62.0 in /operator #5368
• Bump go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux #5372
• Bump google.golang.org/grpc from 1.61.1 to 1.62.0 in /hodometer #5361
• Bump go.uber.org/zap from 1.21.0 to 1.27.0 in /operator #5369
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace in /scheduler #5374
• Generating changelog for v2.8.1 942f8c0
• Generating changelog for v2.8.1-rc1 407e693
• Setting version for helm charts 1f2b242
• Setting version for helm charts 0219ee5
• Setting version for helm charts 7ad640c

Changelog

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v2.8.0

28 February 2024

• ci: Merge changes from v2 to release 2.8 branch #5379
• update 3rd party licenses #5377
• fix(gateway): wait for kafka topic creation #5359
• fix(dataflow): wait for kafka topic creation #5375
• build(deps) update dataflow dependencies #5360
• fix(dataflow): handle pipeline errors and clear kafka streams state #5358
• use k8s 1.29.2 in kind #5357
• update Inactive check #5355
• Bump github.com/go-playground/validator/v10 in /scheduler #5327
• fix(dataflow): set default OTEL_EXPORTER_OTLP_PROTOCOL in compose setup #5353
• Bump google.golang.org/grpc from 1.61.0 to 1.61.1 in /scheduler #5326
• Bump grafana/grafana from 10.3.1 to 10.3.3 in /scheduler #5328
• fix(scheduler): Send server statuses on controller reconnect #5350
• Bump google.golang.org/grpc from 1.61.0 to 1.61.1 in /hodometer #5347
• bump(librdkafka): from v1.9.2 to v2.3.0 #5321
• fix(agent): set context deadline for grpc model server control plane #5329
• fix: Experiments and Models state fixes when reconnecting to scheduler #5320
• enable isotime in the logs #5319
• feat(operator): Expose PodSpec in OverrideSpec for SeldonRuntime #5296
• Bump go.opentelemetry.io/otel/sdk from 1.22.0 to 1.23.1 in /scheduler #5311
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace in /scheduler #5312
• Bump go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux #5310
• Bump github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/kafka/splunkkafka #5313
• Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp #5314
• fix: Add kubectl option to pipeline-tests script #5307
• fix(logging): Update otel and tracing config #5291
• fix: Pipeline state during disconnects #5298
• fix: Connection retries when scheduler restarts for dataflow and controller #5292
• fix(pipelinegateway): Use composite key for Kafka when x-request-id header is specified #5275
• build(deps): Update dataflow dependencies #5278
• add sleep to script #5277
• Bump google.golang.org/grpc from 1.56.3 to 1.61.0 in /components/tls #5272
• Bump github.com/otiai10/copy from 1.7.0 to 1.14.0 in /components/tls #5274
• Bump github.com/grpc-ecosystem/go-grpc-middleware in /hodometer #5270
• Bump github.com/stretchr/testify from 1.7.0 to 1.8.4 in /hodometer #5271
• Bump github.com/hashicorp/go-retryablehttp in /hodometer #5269
• Bump google.golang.org/protobuf from 1.31.0 to 1.32.0 in /operator #5268
• Bump github.com/grpc-ecosystem/go-grpc-middleware in /operator #5266
• Bump emperror.dev/errors from 0.8.0 to 0.8.1 in /operator #5265
• Bump github.com/onsi/gomega from 1.18.1 to 1.31.1 in /operator #5267
• Bump github.com/sirupsen/logrus from 1.8.1 to 1.9.3 in /scheduler #5264
• Bump github.com/grpc-ecosystem/go-grpc-middleware in /scheduler #5263
• Bump github.com/rs/xid from 1.3.0 to 1.5.0 in /scheduler #5261
• Bump github.com/dgraph-io/badger/v3 in /scheduler #5262
• Bump github.com/onsi/gomega from 1.18.1 to 1.31.1 in /scheduler #5256
• Bump go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux #5258
• Bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc #5260
• Bump github.com/otiai10/copy from 1.7.0 to 1.14.0 in /scheduler #5259
• Bump github.com/fsnotify/fsnotify from 1.5.1 to 1.7.0 in /scheduler #5253
• Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp #5254
• Bump github.com/google/uuid from 1.4.0 to 1.6.0 in /scheduler #5255
• Bump github.com/envoyproxy/go-control-plane in /scheduler #5247
• Bump github.com/go-playground/validator/v10 in /scheduler #5249
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc #5250
• Bump google.golang.org/grpc from 1.59.0 to 1.61.0 in /operator #5244
• Bump google.golang.org/grpc from 1.59.0 to 1.61.0 in /scheduler #5251
• Bump github.com/google/uuid from 1.3.1 to 1.6.0 in /hodometer #5205
• Bump github.com/dukex/mixpanel in /hodometer #5217
• Bump github.com/jarcoal/httpmock from 1.0.8 to 1.3.1 in /scheduler #5223
• Bump github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/kafka/splunkkafka from 0.8.0 to 1.12.0 in /scheduler #5228
• Bump github.com/go-logr/logr from 1.2.3 to 1.4.1 in /operator #5235
• Bump github.com/sirupsen/logrus from 1.8.1 to 1.9.3 in /hodometer #5240
• Bump ubi9/ubi-minimal from 9.3-1361.1699548032 to 9.3-1552 in /scheduler #5242
• Bump ubi9/ubi-micro from 9.3-6 to 9.3-13 in /operator #5233
• Bump ubi9/ubi-micro from 9.3-9 to 9.3-13 in /hodometer #5239
• Bump ubi9/ubi-micro from 9.3-6 to 9.3-13 in /scheduler #5243
• Bump github.com/confluentinc/confluent-kafka-go from 1.8.2 to 1.9.2 in /operator #5215
• fix(deps): Upgrade to go.1.20.13 and fix high level CVEs #5234
• feat(ci): dependabot worfklow for v2 #5194
• Bump ubi9/ubi-micro from 9.3-6 to 9.3-9 in /hodometer #5208
• Bump google.golang.org/grpc from 1.59.0 to 1.61.0 in /hodometer #5216
• Bump google.golang.org/protobuf from 1.31.0 to 1.32.0 in /scheduler #5229
• Bump github.com/spf13/cobra from 1.4.0 to 1.8.0 in /operator #5220
• Bump rclone/rclone from 1.65.0 to 1.65.1 in /scheduler #5214
• fix(envoy): add readiness probe to Envoy #5158
• Initial commit to remove finalizer - also fixes name typo to fix SeldonRuntime Ready status for dataflow engine #5109
• fix: Start triton server via bash -c tritonserver instead of just tritonserver #5030
• fix(scheduler): Fix deletin...

All notable changes to this project will be documented in this file. Dates are displayed in UTC.

Generated by auto-changelog.

v1.18.1

29 February 2024

• fix pipeline #5389
• feat(build): use github actions to build docker images #5380
• fix(logging): adserver is very noisy for feedback requests, possibly affecting performance #5351
• docs(openapi): Use command with file #5349
• fix(adserver): adserver returns cloudevents compatible response #5348
• fix(docs): Change V2 Inference Protocol to OIP #5297
• fix(docs): Update docs w latest K8s compatibility #5295
• Update README.md #5293
• build(ansible): fix link to minio chart #5276
• feat(ci): dependabot worfklow for v2 #5194
• rewording #5192
• Update index.rst #5191
• Update README.md #5190
• bump versions to 1.19.0-dev #7
• Release 1.18.0 9771f06