Declaring preprocessor functions

[karlkfi]

I want to be able to use a function in a pipeline to perform processing of values AFTER they have been set by a parent package's apply-setters.

The following cluster package works intuitively as a single package, but it doesn't work when it it's inside a parent package that also uses apply-setters, because the parent package mutators are executed after the child package's mutators...

Parent setters:

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: cluster
pipeline:
  mutators:
    - image: gcr.io/kpt-fn/apply-setters:v0.1
      configPath: setters.yaml

Child setters:

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: nodepool
pipeline:
  mutators:
    - image: gcr.io/kpt-fn/apply-setters:v0.1
      configPath: setters.yaml
    - image: gcr.io/kpt-fn/starlark:v0.1
      configPath: truncate-service-accounts.yaml

The workaround I know of is to copy the function to the parent package kptfile, but if I do that, it will affect all resources in the parent package, not just the resources in the child package. And even if I could scope it to the right package, it would be a hassle to have to copy function config from child packages to parent packages.

One way to solve this might be to have early and late mutators.

Order of execution:

• parent "early mutators"
• child "early mutators"
• child "late mutators"
• parent "late mutators"

This would kill two birds with one stone, because it would also allow for parent packages to modify the setter values of child packages. I could put apply-setters as an "early mutator" and the starlark function in a "late mutator" and it would affect the setter values as passed from parent to child without affecting the parent or sibling packages.

For example...

Parent setters:

apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  name: cluster-1
  project-id: example-1234

Child setters:

apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  name: pool-1
  cluster-name: cluster-1 # kpt-set: ${name}
  project-id: example-1234 # kpt-set: ${project-id}

Child resource:

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: gke-cluster-1-pool-1 # kpt-set: gke-${cluster-name}-${name}

Service account names are frequently too long when concatenating other names together. With this configuration the starlark function could truncate it after the setters have been applied. And with this pattern, the name setter means "node pool name" in the child package and "cluster name" in the parent package, allowing for hierarchical setter namespacing.

[frankfarzan]

This is a use case that has been discussed, referred to as "pre-processors" as opposed to current execution order where parent package functions run as "post-processors". This is something that can be added incrementally as it is purely additive (i.e. via a preprocessors field in the pipeline section) so was left out of v1 scope. We should look at this in v1.1 time frame.

[morgante]

Please prioritize this. The lack of this basically makes setter inheritance unworkable as currently implemented. Let's say I want to apply a setter (ex. organization-id) but some functions in lower packages depend on that setter value. The functions in subpackages will always run before the parent apply-setters function, meaning my only option is to manually copy the setter value down into my subpackage.

[phanimarupaka]

@karlkfi Not proposing a solution here, but trying to understand the problem better, does it help if we make the order of hydration configurable ? Currently, the default order if bottom up, where children are hydrated before parents. What if we make it configurable, and hydrate parents first and then children?(top down)

[morgante]

@phanimarupaka That would not solve the problem, as we still have cases where parents need to be applied first (ex. setter inheritance, the one I mentioned). Shifting the burden to users via configuration isn't sufficient.

[phanimarupaka]

So for this case, having early-mutators, early-validators which are applied in top-down fashion(additive change) and mutators, validators which apply in bottom-up fashion(current behavior) should help I guess. Advanced users can pick and choose where to include the functions.

[phanimarupaka]

@morgante Another way is by using target selector for functions. The starlark function can be moved to parent package and specify target functions as mentioned in this #2015.