|
| 1 | +[overlay]: ../docs/glossary.md#overlay |
| 2 | +[target]: ../docs/glossary.md#target |
| 3 | + |
| 4 | +# Demo: combining config data from devops and developers |
| 5 | + |
| 6 | +Scenario: you have a Java-based server storefront in |
| 7 | +production that various internal development teams |
| 8 | +(signups, checkout, search, etc.) contribute to. |
| 9 | + |
| 10 | +The server runs in different environments: |
| 11 | +_development_, _testing_, _staging_ and _production_, |
| 12 | +accepting configuration parameters from java property |
| 13 | +files. |
| 14 | + |
| 15 | +Using one big properties file for each environment is |
| 16 | +difficult to manage. The files change frequently, and |
| 17 | +have to be changed by devops exclusively because |
| 18 | + |
| 19 | + 1. the files must at least partially agree on certain |
| 20 | + values that devops cares about and that developers |
| 21 | + ignore and |
| 22 | + 1. because the production |
| 23 | + properties contain sensitive data like production |
| 24 | + database credentials. |
| 25 | + |
| 26 | +## Property sharding |
| 27 | + |
| 28 | +With some study, we notice that the properties are |
| 29 | +separable into categories. |
| 30 | + |
| 31 | +### Common properties |
| 32 | + |
| 33 | +E.g. internationalization data, static data like |
| 34 | +physical constants, location of external services, etc. |
| 35 | + |
| 36 | +_Things that are the same regardless of environment._ |
| 37 | + |
| 38 | +Only one set of values is needed. |
| 39 | + |
| 40 | +Place them in a file called |
| 41 | + |
| 42 | + * `common.properties` |
| 43 | + |
| 44 | +(relative location defined below). |
| 45 | + |
| 46 | +### Plumbing properties |
| 47 | + |
| 48 | +E.g. serving location of static content (HTML, CSS, |
| 49 | +javascript), location of product and customer database |
| 50 | +tables, ports expected by load balancers, log sinks, |
| 51 | +etc. |
| 52 | + |
| 53 | +_The different values for these properties are |
| 54 | +precisely what sets the environments apart._ |
| 55 | + |
| 56 | +Devops or SRE will want full control over the values |
| 57 | +used in production. Testing will have fixed |
| 58 | +databases supporting testing. Developers will want |
| 59 | +to do whatever they want to try scenarios under |
| 60 | +development. |
| 61 | + |
| 62 | +Places these values in |
| 63 | + |
| 64 | + * `development/plumbing.properties` |
| 65 | + * `staging/plumbing.properties` |
| 66 | + * `production/plumbing.properties` |
| 67 | + |
| 68 | + |
| 69 | +### Secret properties |
| 70 | + |
| 71 | +E.g. location of actual user tables, database |
| 72 | +credentials, decryption keys, etc. |
| 73 | + |
| 74 | +_Things that are a subset of devops controls, that |
| 75 | +nobody else has (or should want) access to._ |
| 76 | + |
| 77 | +Places these values in |
| 78 | + |
| 79 | + * `development/secret.properties` |
| 80 | + * `staging/secret.properties` |
| 81 | + * `production/secret.properties` |
| 82 | + |
| 83 | +[kubernetes secret]: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/ |
| 84 | + |
| 85 | +and control access to them with (for example) unix file |
| 86 | +owner and mode bits, or better yet, put them in |
| 87 | +a server dedicated to storing password protected |
| 88 | +secrets, and use a field called `secretGenerator` |
| 89 | +in your _kustomization_ to create a kubernetes |
| 90 | +secret holding them (not covering that here). |
| 91 | + |
| 92 | +<!-- |
| 93 | +secretGenerator: |
| 94 | +- name: app-tls |
| 95 | + commands: |
| 96 | + tls.crt: "cat tls.cert" |
| 97 | + tls.key: "cat tls.key" |
| 98 | + type: "kubernetes.io/tls" |
| 99 | +EOF |
| 100 | +--> |
| 101 | + |
| 102 | +## A mixin approach to management |
| 103 | + |
| 104 | +The way to create _n_ cluster environments that share |
| 105 | +some common information is to create _n_ overlays of a |
| 106 | +common base. |
| 107 | + |
| 108 | +For the rest of this example, we'll do _n==2_, just |
| 109 | +_development_ and _production_, since adding more |
| 110 | +environments follows the same pattern. |
| 111 | + |
| 112 | +A cluster environment is created by |
| 113 | +running `kustomize build` on a [target] that happens to |
| 114 | +be an [overlay]. |
| 115 | + |
| 116 | +[helloworld]: helloWorld/README.md |
| 117 | + |
| 118 | +The following example will do that, but will focus on |
| 119 | +configMap construction, and not worry about how to |
| 120 | +connect the configMaps to deployments (that is covered |
| 121 | +in the [helloworld] example). |
| 122 | + |
| 123 | + |
| 124 | +All files - including the shared property files |
| 125 | +discussed above - will be created in a directory tree |
| 126 | +that is consistent with the base vs overlay file layout |
| 127 | +defined in the [helloworld] demo. |
| 128 | + |
| 129 | +It will all live in this work directory: |
| 130 | + |
| 131 | +<!-- @makeWorkplace @test --> |
| 132 | +``` |
| 133 | +DEMO_HOME=$(mktemp -d) |
| 134 | +``` |
| 135 | + |
| 136 | +### Create the base |
| 137 | + |
| 138 | +<!-- kubectl create configmap BOB --dry-run -o yaml --from-file db. --> |
| 139 | + |
| 140 | +Make a place to put the base configuration: |
| 141 | + |
| 142 | +<!-- @baseDir @test --> |
| 143 | +``` |
| 144 | +mkdir -p $DEMO_HOME/base |
| 145 | +``` |
| 146 | + |
| 147 | +Make the data for the base. This direction by |
| 148 | +definition should hold resources common to all |
| 149 | +environments. Here we're only defining a java |
| 150 | +properties file, and a `kustomization` file that |
| 151 | +references it. |
| 152 | + |
| 153 | +<!-- @baseKustomization @test --> |
| 154 | +``` |
| 155 | +cat <<EOF >$DEMO_HOME/base/common.properties |
| 156 | +color=blue |
| 157 | +height=10m |
| 158 | +EOF |
| 159 | +
|
| 160 | +cat <<EOF >$DEMO_HOME/base/kustomization.yaml |
| 161 | +configMapGenerator: |
| 162 | +- name: my-configmap |
| 163 | + files: |
| 164 | + - common.properties |
| 165 | +EOF |
| 166 | +``` |
| 167 | + |
| 168 | + |
| 169 | +### Create and use the overlay for _development_ |
| 170 | + |
| 171 | +Make an abbreviation for the parent of the overlay |
| 172 | +directories: |
| 173 | + |
| 174 | +<!-- @overlays @test --> |
| 175 | +``` |
| 176 | +OVERLAYS=$DEMO_HOME/overlays |
| 177 | +``` |
| 178 | + |
| 179 | +Create the files that define the _development_ overlay: |
| 180 | + |
| 181 | +<!-- @developmentFiles @test --> |
| 182 | +``` |
| 183 | +mkdir -p $OVERLAYS/development |
| 184 | +
|
| 185 | +cat <<EOF >$OVERLAYS/development/plumbing.properties |
| 186 | +port=30000 |
| 187 | +EOF |
| 188 | +
|
| 189 | +cat <<EOF >$OVERLAYS/development/secret.properties |
| 190 | +dbpassword=mothersMaidenName |
| 191 | +EOF |
| 192 | +
|
| 193 | +cat <<EOF >$OVERLAYS/development/kustomization.yaml |
| 194 | +bases: |
| 195 | +- ../../base |
| 196 | +namePrefix: dev- |
| 197 | +configMapGenerator: |
| 198 | +- name: my-configmap |
| 199 | + behavior: merge |
| 200 | + files: |
| 201 | + - plumbing.properties |
| 202 | + - secret.properties |
| 203 | +EOF |
| 204 | +``` |
| 205 | + |
| 206 | +One can now generate the configMaps for development: |
| 207 | + |
| 208 | +<!-- @runDev @test --> |
| 209 | +``` |
| 210 | +kustomize build $OVERLAYS/development |
| 211 | +``` |
| 212 | + |
| 213 | +#### Check the ConfigMap name |
| 214 | + |
| 215 | +The name of the generated `ConfigMap` is visible in this |
| 216 | +output. |
| 217 | + |
| 218 | +The name should be something like `dev-my-configmap-b5m75ck895`: |
| 219 | + |
| 220 | + * `"dev-"` comes from the `namePrefix` field, |
| 221 | + * `"my-configmap"` comes from the `configMapGenerator/name` field, |
| 222 | + * `"-b5m75ck895"` comes from a deterministic hash that `kustomize` |
| 223 | + computes from the contents of the configMap. |
| 224 | + |
| 225 | +The hash suffix is critical. If the configMap content |
| 226 | +changes, so does the configMap name, along with all |
| 227 | +references to that name that appear in the YAML output |
| 228 | +from `kustomize`. |
| 229 | + |
| 230 | +The name change means deployments will do a rolling |
| 231 | +restart to get new data if this YAML is applied to the |
| 232 | +cluster using a command like |
| 233 | + |
| 234 | +> ``` |
| 235 | +> kustomize build $OVERLAYS/development | kubectl apply -f - |
| 236 | +> ``` |
| 237 | +
|
| 238 | +A deployment has no means to automatically know when or |
| 239 | +if a configMap in use by the deployment changes. |
| 240 | +
|
| 241 | +If one changes a configMap without changing its name |
| 242 | +and all references to that name, one must imperatively |
| 243 | +restart the cluster to pick up the change. |
| 244 | +
|
| 245 | +The best practice is to treat configMaps as immutable. |
| 246 | +
|
| 247 | +Instead of editing configMaps, modify your declarative |
| 248 | +specification of the cluster's desired state to |
| 249 | +point deployments to _new_ configMaps with _new_ names. |
| 250 | +`kustomize` makes this easy with its |
| 251 | +`configMapGenerator` directive and associated naming |
| 252 | +controls. A GC process in the k8s master eventually |
| 253 | +deletes unused configMaps. |
| 254 | +
|
| 255 | +
|
| 256 | +### Create and use the overlay for _production_ |
| 257 | +
|
| 258 | +Next, create the files for the _production_ overlay: |
| 259 | +
|
| 260 | +
|
| 261 | +<!-- @productionFiles @test --> |
| 262 | +``` |
| 263 | +mkdir -p $OVERLAYS/production |
| 264 | + |
| 265 | +cat <<EOF >$OVERLAYS/production/plumbing.properties |
| 266 | +port=8080 |
| 267 | +EOF |
| 268 | + |
| 269 | +cat <<EOF >$OVERLAYS/production/secret.properties |
| 270 | +dbpassword=thisShouldProbablyBeInASecretInstead |
| 271 | +EOF |
| 272 | + |
| 273 | +cat <<EOF >$OVERLAYS/production/kustomization.yaml |
| 274 | +bases: |
| 275 | +- ../../base |
| 276 | +namePrefix: prod- |
| 277 | +configMapGenerator: |
| 278 | +- name: my-configmap |
| 279 | + behavior: merge |
| 280 | + files: |
| 281 | + - plumbing.properties |
| 282 | + - secret.properties |
| 283 | +EOF |
| 284 | +``` |
| 285 | +
|
| 286 | +One can now generate the configMaps for production: |
| 287 | +
|
| 288 | +<!-- @runProd @test --> |
| 289 | +``` |
| 290 | +kustomize build $OVERLAYS/production |
| 291 | +``` |
| 292 | +
|
| 293 | +A CICD process could apply this directly to |
| 294 | +the cluser using: |
| 295 | +
|
| 296 | +> ``` |
| 297 | +> kustomize build $OVERLAYS/production | kubectl apply -f - |
| 298 | +> ``` |
0 commit comments