-
Notifications
You must be signed in to change notification settings - Fork 444
The CI environment Publishing
This section provides an overview of the principles the publishing is built upon, how it is set up, and how you can interact with it.
The publishing phase concludes each module's pipeline. If all previous tests succeed (i.e., no phase failed) and the pipeline is run in the main or master branch, a new module version is published to all configured target locations. Currently, we support the following target locations:
- Template Specs
- Bicep Registry
-
Azure DevOps Universal Packages.
Note: this is only available if using Azure DevOps pipelines.
Besides the publishing phase's runtime, there is also the possibility to set the switch Publish prerelease module. This switch makes it possible to publish a prerelease version in every workflow run that is not based on main or master. This can be controlled when running the module pipeline leveraging Module pipeline inputs.
Note: The
versionused for publishing any artifact is the same for all three target locations, which reduces the maintenance effort.
Note: The orchestration options described in the solution creation section work differently well with the publishing locations we offer in CARML. To help you select the best location for your use case, we provide further information here section.
The names of published modules differ slightly depending on the location they are published to & settings you may have configured. This is rooted in the different requirements per target location. In the following you can find the rules applied for each:
Template Specs
Actions
The final name is different based on the useApiSpecsAlignedName setting in the settings.yml file. The main difference is, that based on the module's path, the actual resource type identifier will be fetched from the utilities/src/apiSpecsList.json file. Following you can find the general flow and distinct differences in either case:
- (if
useApiSpecsAlignedNameistrue)- Recover the original API Specs reference for the module from the
utilities/src/apiSpecsList.jsonfile - Replace
microsoftwithms
- Recover the original API Specs reference for the module from the
- Remove the root folder name
modulesfrom provided module reference - Make lowercase
- Replace all
\or/with. - Remove all duplications in the path. For example, in an identifier like
virtualNetworks/virtualNetworkPeeringswe'd trim down tovirtualNetworks/peeringsto shorten the path (as there is a maximum character limit for template specs)
Examples
| Module Path | useApiSpecsAlignedName |
Result |
|---|---|---|
/recovery-services/vault |
true |
ms.recoveryservices.vaults |
false |
recovery-services.vault |
|
/recovery-services\vault\replication-fabric\replication-protection-container\replication-protection-container-mapping |
true |
ms.recoveryservices.vaults.replicationfabrics.replicationprotectioncontainers.mappings |
false |
recovery-services.vault.replication-fabric.replication-protection-container.mapping |
Private Bicep Registry (Azure Container Registry)
Actions
The final name is different based on the useApiSpecsAlignedName setting in the settings.yml file. The main difference is, that based on the module's path, the actual resource type identifier will be fetched from the utilities/src/apiSpecsList.json file. Following you can find the general flow and distinct differences in either case:
- (if
useApiSpecsAlignedNameistrue)- Recover the original API Specs reference for the module from the
utilities/src/apiSpecsList.jsonfile
- Recover the original API Specs reference for the module from the
- Remove the root folder name
modulesfrom provided module reference - Make lowercase
- Replace all
\or/with. - Add the
bicep/modulesprefix
Examples
| Module Path | useApiSpecsAlignedName |
Result |
|---|---|---|
/recovery-services/vault |
true |
bicep/modules/microsoft.recoveryservices.vaults |
false |
bicep/modules/recovery-services.vault |
|
/recovery-services\vault\replication-fabric\replication-protection-container\replication-protection-container-mapping |
true |
bicep/modules/microsoft.recoveryservices.vaults.replicationfabrics.replicationprotectioncontainers.replicationprotectioncontainermappings |
false |
bicep/modules/recovery-services.vault.replication-fabric.replication-protection-container.replication-protection-container-mapping |
Azure DevOps Universal Packages
Actions
The final name is different based on the useApiSpecsAlignedName setting in the settings.yml file. The main difference is, that based on the module's path, the actual resource type identifier will be fetched from the utilities/src/apiSpecsList.json file. Following you can find the general flow and distinct differences in either case:
- (if
useApiSpecsAlignedNameistrue)- Recover the original API Specs reference for the module from the
utilities/src/apiSpecsList.jsonfile
- Recover the original API Specs reference for the module from the
- Remove the root folder name
modulesfrom provided module reference - Make lowercase
- Replace all
\or/with.
Examples
| Module Path | useApiSpecsAlignedName |
Result |
|---|---|---|
/recovery-services/vault |
true |
microsoft.recoveryservices.vault |
false |
recovery-services.vault |
|
/recovery-services\vault\replication-fabric\replication-protection-container\replication-protection-container-mapping |
true |
microsoft.recoveryservices.vault.replicationfabric.replicationprotectioncontainer.replicationprotectioncontainermapping |
false |
recovery-services.vault.replication-fabric.replication-protection-container.replication-protection-container-mapping |
The publishing works as follows:
- The script
utilities/pipelines/resourcePublish/Get-ModulesToPublish.ps1gets all changed module files, including child modules, and handles the logic of propagating the appropriate module version to be used:- The major (
x.0) and minor (0.x) version are set based on theversion.jsonfile in the module folder. - The patch (
0.0.x) version is calculated based on the number of commits on theHEADref (aka. git height). This will cause the patch version to never reset to 0 with major and/or minor increment, as specified for semver. - The module is published with a
major.minor.patchversion (x.y.z). For Template Specs and Bicep Registry only, amajorversion (x), amajor.minorversion (x.y) and alatestversion are also updated, allowing a consumer to:- Reference the latest version of a major, i.e., the latest minor and patch of a major version.
Example: Using Template Specs, the reference to a
majorcould look like:ts/modules:resources.resource-group:1which means that the template will always consume whatever the potentially overwritten/updated version1contains. - Reference the latest version of a minor, i.e., the latest patch of a minor version.
Example: Using the Bicep registry, the reference to a
major.minorcould look like:br/modules:resources.resource-group:0.4which means that the template will always consume whatever the potentially overwritten/updated version0.4contains.
- Reference the latest version of a major, i.e., the latest minor and patch of a major version.
- For a changed child module, the direct parent hierarchy is also registered for an update, following the same procedure as above.
- The list of module files paths and their versions are passed on as a array list.
- The major (
- The Get-ModulesMissingFrom*.ps1 scripts further check if a given module is missing from the corresponding target location (e.g., Azure Container Registry) and adds each missing entry to to aforementioned array - using the version specified in the module's
version.jsonfile. - The different publishing scripts run (Artifact, Template Spec or Bicep Registry) and publish the module to the respective target location for each item on the list.
Let's look at an example run where we would do a patch change on the fileShares module:
- A new branch is created for further development of the
fileSharemodule. Let's assume the new branch started from commit500on the default branch and theversion.jsonof thefileSharemodule contains major and minor0.3. - Bug-fixes, documentation, and security updates are added to the
fileSharemodule by the author. Theversion.jsonfile is not changed in either the child or parent module folders. - The author runs a manual workflow based on their development branch, with the 'Publish prerelease module' option enabled.
- A prerelease run of publishing triggers after test and validation of the module.
- For the child and parent modules, the module version's major and minor version is read from the
version.jsonfile in the module folder respectively. Being unchanged, it still contains the version0.3. - The patch is calculated based on the total number of commits in history on the branch (independent on the module). The new branch started from commit
500on the default branch and 1 commit has been pushed, so the total number of commits on the new branch is501. - As the pipeline is not running based on the 'default branch', a prerelease segment (
-prerelease) is added to the version. - The version results in being
0.3.501-prerelease. The child and parent modules may have different major and minor versions, but the patch version will be the same in this case. Other unmodified child modules ofstorageAccountwill not be republished and remain with the existing version.
- For the child and parent modules, the module version's major and minor version is read from the
- Sequential commits on the branch and runs of the module pipeline, with the 'publish prerelease' option enabled results in the following versions being published:
0.3.502-prerelease0.3.503-prerelease- ...
0.3.506-prerelease
- When the branch is merged to the default branch, the only thing that changes is the patch version and the removal of the
-prereleasesegment.- The number of commits will at this point be calculated based on the number of commits on the default branch.
- Assuming the development branch started from commit 500 on the default branch, and the author added 6 commits on the development branch, the prerelease versions will reach
0.3.506-prerelease. - Meanwhile, there can be changes (let's say 2 squashed PR merges) on the default branch that is pushing its number of commits in history further.
- If the PR for the changes to
fileShareis squash merged as commit number 503, the patch version on the child and parent module is then503, resulting in a version0.3.503being published.
- The merge triggers cascading updates in the following way:
- The module is published with a
major.minor.patchversion. In addition, only for Template Specs and Bicep Registry, the module is also published withmajor.minorandmajorversion updates, allowing consumers to target the latest major or latest minor version respectively. - All parent module are published following the steps mentioned above.
- The module is published with a
\ \
C499 -> C500 ---> C501 ---> C502 ---> C503 (503)
\ /
D1 --> D2 --> D3 ... --> D6
(501) (502) (503) (506)
Cx - Commits on main,
Dx - Commits on development branch,
(x) - Calculated patch version