Skip to content
This repository was archived by the owner on Sep 10, 2024. It is now read-only.

Commit e6e24bf

Browse files
stijnmoreelsstijnmoreels
authored
docs: pubslish v0.3 (#91)
1 parent c535648 commit e6e24bf

File tree

8 files changed

+377
-0
lines changed

8 files changed

+377
-0
lines changed

‎docs/versioned_docs/version-v0.3/01-index.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
title: "Arcus Background Jobs"
3+
layout: default
4+
permalink: /
5+
slug: /
6+
sidebar_label: Welcome
7+
---
8+
9+
[![NuGet Badge](https://buildstats.info/nuget/Arcus.BackgroundJobs.CloudEvents?packageVersion=0.3.0)](https://www.nuget.org/packages/Arcus.BackgroundJobs.CloudEvents/0.3.0)
10+
11+
# Installation
12+
13+
The Arcus BackgroundJobs can be installed via NuGet:
14+
15+
```shell
16+
PM > Install-Package Arcus.BackgroundJobs.CloudEvents -Version 0.3.0
17+
```
18+
19+
For more granular packages we recommend reading the documentation.
20+
21+
# Features
22+
23+
- **General**
24+
- [Securely Receive CloudEvents](features/general/receive-cloudevents-job)
25+
- **Security**
26+
- [Automatically invalidate cached secrets from Azure Key Vault](features/security/auto-invalidate-secrets)
27+
- **Databricks**
28+
- [Measure Databricks job run outcomes as metric](features/databricks/job-metrics)
29+
- [Interact with Databricks to gain insights](features/databricks/gain-insights)
30+
31+
# License
32+
This is licensed under The MIT License (MIT). Which means that you can use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the web application. But you always need to state that Codit is the original author of this web application.
33+
34+
*[Full license here](https://github.com/arcus-azure/arcus.backgroundjobs/blob/master/LICENSE)*

‎docs/versioned_docs/version-v0.3/02-Features/01-General/receive-cloudevents-job.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
title: "Securely Receive CloudEvents"
3+
layout: default
4+
---
5+
6+
# Securely Receive CloudEvents
7+
8+
The `Arcus.BackgroundJobs.CloudEvents` library provides a collection of background jobs to securely receive [CloudEvents](https://github.com/cloudevents/spec).
9+
10+
This allows workloads to asynchronously process event from other components without exposing a public endpoint.
11+
12+
## How does it work?
13+
14+
An Azure Service Bus Topic resource is required to receive CloudEvents on. CloudEvent messages on this Topic will be processed by a background job.
15+
16+
![Automatically Invalidate Azure Key Vault Secrets](/media/CloudEvents-Job.png)
17+
18+
## Usage
19+
20+
The CloudEvents background job uses the [Arcus Messaging](https://github.com/arcus-azure/arcus.messaging) functionality to receive messages.
21+
Make sure you take a look at the [documentation on message handlers](https://messaging.arcus-azure.net/features/message-pumps/service-bus) to fully grasp the possibilities.
22+
23+
The CloudEvent background job itself can be easily registered:
24+
25+
```csharp
26+
using Microsoft.Extensions.DependencyInjection;
27+
28+
public class Startup
29+
{
30+
public void ConfigureServices(IServiceCollection services)
31+
{
32+
// Add CloudEvents background job with an namespace-scoped connection string.
33+
services.AddCloudEventBackgroundJob(
34+
topicName: "<your-topic>",
35+
subscriptionNamePrefix: "Sub-",
36+
serviceBusNamespaceConnectionStringSecretKey: "<secret-key-name-for-servicebus-namespace-connection-string>");
37+
38+
// Add CloudEvent background job with an entity-scoped connection string.
39+
services.AddCloudEventBackgroundJob(
40+
subscriptionNamePrefix: "Sub-",
41+
serviceBusTopicConnectionStringSecretKey: "<secret-key-name-for-servicebus-topic-connection-string>");
42+
}
43+
}
44+
```
45+
46+
To handle the incoming CloudEvents messages, you can register an custom `IAzureServiceBusMessageHandler<CloudEvent>` message handler instance:
47+
48+
```csharp
49+
using CloudNative.CloudEvents;
50+
using Microsoft.Extensions.DependencyInjection;
51+
52+
public class Startup
53+
{
54+
public void ConfigureServices(IServiceCollection services)
55+
{
56+
services.AddCloudEventEventBackgroundJob(...)
57+
.WithServiceBusMessageHandler<MyCloudEventMessageHandler, CloudEvent>();
58+
}
59+
}
60+
```
61+
62+
Such an custom implementation could look like this:
63+
64+
```csharp
65+
using Arcus.Messaging.Abstractions;
66+
using Arcus.Messaging.Abstractions.ServiceBus;
67+
using Arcus.Messaging.Abstractions.ServiceBus.MessageHandling;
68+
using CloudNative.CloudEvents;
69+
using Microsoft.Extensions.Logging;
70+
71+
public class MyCloudEventMessageHandler : IAzureServiceBusMessageHandler<CloudEvent>
72+
{
73+
private readonly ILogger _logger;
74+
75+
public MyCloudEventMessageHandler(ILogger<MyCloudEventMessageHandler> logger)
76+
{
77+
_logger = logger;
78+
}
79+
80+
public async Task ProcessMessageAsync(
81+
CloudEvent message,
82+
AzureServiceBusMessageContext messageContext,
83+
MessageCorrelationInfo correlationInfo,
84+
CancellationToken cancellationToken)
85+
{
86+
_logger.LogInformation("Processing CloudEvent message");
87+
}
88+
}
89+
```
90+
91+
[&larr; back](/)

‎docs/versioned_docs/version-v0.3/02-Features/02-Security/auto-invalidate-secrets.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
title: "Automatically Invalidate Azure Key Vault Secrets"
3+
layout: default
4+
---
5+
6+
# Automatically Invalidate Azure Key Vault Secrets
7+
8+
The `Arcus.BackgroundJobs.KeyVault` library provides a background job to automatically invalidate cached Azure Key Vault secrets from an `ICachedSecretProvider` instance of your choice.
9+
10+
## How does it work?
11+
12+
<a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Farcus-azure%2Farcus.backgroundjobs%2Fmaster%2Fdeploy%2Farm%2Fazure-key-vault-job.json" target="_blank">
13+
<img src="https://azuredeploy.net/deploybutton.png"/>
14+
</a>
15+
16+
17+
This automation works by subscribing on the `SecretNewVersionCreated` event of an Azure Key Vault resource and placing those events on a Azure Service Bus Topic; which we process in our background job.
18+
19+
![Automatically Invalidate Azure Key Vault Secrets](/media/Azure-Key-Vault-Job.png)
20+
21+
To make this automation opperational, following Azure Resources has to be used:
22+
* Azure Key Vault instance
23+
* Azure Service Bus Topic
24+
* Azure Event Grid subscription for `SecretNewVersionCreated` events that are sent to the Azure Service Bus Topic
25+
26+
## Usage
27+
28+
Our background job has to be configured in `ConfigureServices` method:
29+
30+
```csharp
31+
using Arcus.Security.Core;
32+
using Arcus.Security.Core.Caching;
33+
using Microsoft.Extensions.DependencyInjection;
34+
35+
public class Startup
36+
{
37+
public void ConfigureServices(IServiceCollection services)
38+
{
39+
// An 'ISecretProvider' implementation (see: https://security.arcus-azure.net/) to access the Azure Service Bus Topic resource;
40+
// this will get the 'serviceBusTopicConnectionStringSecretKey' string (configured below) and has to retrieve the connection string for the topic.
41+
services.AddSingleton<ISecretProvider>(serviceProvider => ...);
42+
43+
// An `ICachedSecretProvider` implementation which secret keys will automatically be invalidated.
44+
services.AddSingleton<ICachedSecretProvider>(serviceProvider => new CachedSecretProvider(mySecretProvider));
45+
46+
services.AddAutoInvalidateKeyVaultSecretBackgroundJob(
47+
// Prefix of the Azure Service Bus Topic subscription;
48+
// this allows the background jobs to support applications that are running multiple instances, processing the same type of events, without conflicting subscription names.
49+
subscriptionNamePrefix: "MyPrefix"
50+
51+
// Connection string secret key to a Azure Service Bus Topic.
52+
serviceBusTopicConnectionStringSecretKey: "MySecretKeyToServiceBusTopicConnectionString");
53+
}
54+
}
55+
```
56+
57+
[&larr; back](/)

‎docs/versioned_docs/version-v0.3/02-Features/02-Security/auto-restart-servicebus-messagepump-on-rotated-credentials.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: "Automatically restart Azure Service Bus message pump on rotated credentials"
3+
layout: default
4+
---
5+
6+
# Automatic Azure Key Vault credentials rotation
7+
8+
The library `Arcus.BackgroundJobs.KeyVault` provides an extension on the message pump to restart the pump automatically when the credentials of the pump stored in Azure Key Vault are changed.
9+
This feature allows more reliable restarting instead of relying on authentication exceptions that may be throwed during the lifetime of the message pump.
10+
11+
## How does this work?
12+
13+
A background job is polling for `SecretNewVersionCreated` events on an Azure Service Bus Topic for the secret that stores the connection string.
14+
15+
That way, when the background job receives a new Key Vault event, it will get the latest connection string, restart the message pump and authenticate with the latest credentials.
16+
17+
### Installation
18+
19+
This features requires to install our NuGet package:
20+
21+
```shell
22+
PM > Install-Package Arcus.BackgroundJobs.KeyVault -Version 0.3.0
23+
```
24+
25+
### Usage
26+
27+
When the package is installed, you'll be able to use the extension in your application:
28+
29+
```csharp
30+
using Microsoft.Extensions.DependencyInjection;
31+
32+
public class Startup
33+
{
34+
public void ConfigureServices(IServiceCollection services)
35+
{
36+
// You should have a unique Job ID to identity the message pump so the automatic process knows which pump to restart.
37+
string jobId = Guid.NewGuid().ToString();
38+
39+
string secretName = hostContext.Configuration["ARCUS_KEYVAULT_CONNECTIONSTRINGSECRETNAME"];
40+
services.AddServiceBusQueueMessagePump(secretName, options => options.JobId =
41+
.WithServiceBusMessageHandler<OrdersAzureServiceBusMessageHandler, Order>();
42+
43+
// This extension will be available to you once you installed the package.
44+
services.AddAutoRestartServiceBusMessagePumpOnRotatedCredentialsBackgroundJob(
45+
jobId: jobId,
46+
subscriptionNamePrefix: "TestSub",
47+
48+
// The secret key where the Azure Service Bus Topic connection string is located that the background job will use to receive the Azure Key vault events.
49+
serviceBusTopicConnectionStringSecretKey: "ARCUS_KEYVAULT_SECRETNEWVERSIONCREATED_CONNECTIONSTRING",
50+
51+
// The secret key where the Azure Service Bus connection string is located that your target message pump uses.
52+
// This secret key name will be used to check if the received Azure Key Vault event is from this secret or not.
53+
messagePumpConnectionStringKey: secretName,
54+
55+
// The maximum amount of thrown unauthorized exceptions that your message pump should allow before it should restart either way.
56+
// This amount can be used to either wait for an Azure Key Vault event or rely on the thrown unauthorized exceptions.
57+
maximumUnauthorizedExceptionsBeforeRestart: 5)
58+
}
59+
}
60+
```

‎docs/versioned_docs/version-v0.3/02-Features/03-Databricks/gain-insights.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
title: "Interact with Databricks to gain insights"
3+
layout: default
4+
---
5+
6+
# Interact with Databricks to gain insights
7+
8+
## Installation
9+
10+
To use these features, you have to install the following package:
11+
12+
```shell
13+
PM > Install-Package Arcus.BackgroundJobs.Databricks -Version 0.3.0
14+
```
15+
16+
> :bulb: With using our [Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights](https://www.nuget.org/packages/Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights/), you can report these Databricks reports as metrics in Application Insights.
17+
18+
## Usage
19+
We provide a `DatabricksInfoProvider` which allows you to interact with Databricks clusters to gain insights on your workloads, such as measuring job run outcomes.
20+
21+
It can be easily setup and used anywhere such as .NET Core workers, Azure Functions and more. We are using this ourselves for our [job metrics](./job-metrics).
22+
23+
```csharp
24+
using Arcus.BackgroundJobs.Databricks;
25+
using Microsoft.Azure.Databricks.Client;
26+
using Microsoft.Extensions.Logging;
27+
28+
ILogger logger = ...
29+
using (var client = DatabricksClient.CreateClient("https://databricks.base.url", "security.token"))
30+
using (var provider = new DatabricksInfoProvider(client, logger))
31+
{
32+
}
33+
```
34+
35+
### Getting finished job run information
36+
Gets all the finished job runs within a given time window.
37+
```csharp
38+
using Arcus.BackgroundJobs.Databricks;
39+
40+
DatabricksInfoProvider provider = ...
41+
var startOfWindow = DateTimeOffset.UtcNow.AddDays(-1);
42+
var endOfWindow = DateTimeOffset.UtcNow;
43+
IEnumerable<JobRun> finishedJobRuns = await provider.GetFinishedJobRunsAsync(startOfWindow, endOfWindow);
44+
```
45+
46+
It provides following information about the job that was executated, such as name & id, along with [details about a single job run](https://github.com/Azure/azure-databricks-client/blob/master/csharp/Microsoft.Azure.Databricks.Client/Run.cs).
47+
48+
### Measure finished job outcomes
49+
Measures the finished job runs by reporting the results as (multi-dimensional) metrics.
50+
51+
This method is an combination of the previously defined method (**Getting finished jobs**) and calling an `ILogger` extension provided in this package (`ILogger.LogMetricFinishedJobOutcome`) which will write the finished job runs `JobRun` instances as metrics.
52+
53+
```csharp
54+
using Arcus.BackgroundJobs.Databricks;
55+
56+
DatabricksInfoProvider provider = ...
57+
var metricName = "Databricks Job Completed";
58+
var startOfWindow = DateTimeOffset.UtcNow.AddDays(-1);
59+
var endOfWindow = DateTimeOffset.UtcNow;
60+
await provider.MeasureJobOutcomesAsync(metricName, startOfWindow, endOfWindow);
61+
// Logs > Metric Databricks Job Completed: 1 {UtcNow} (Context: {[Run Id] = my.run.id, [Job Id] = my.job.id, [Job Name] = my.job.name, [Outcome] = Success})
62+
```
63+
64+
> Note: you can always call **Getting finished jobs** yourself and pass along the finished jobs to the available `ILogger.LogMetricFinishedJobOutcome` extension.
65+
> That way, you can pass along additional contextual properties
66+
67+
[&larr; back](/)

‎docs/versioned_docs/version-v0.3/02-Features/03-Databricks/job-metrics.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
title: "Measure Databricks job run outcomes as metric"
3+
layout: default
4+
---
5+
6+
# Measure Databricks job run outcomes as metric
7+
8+
The `Arcus.BackgroundJobs.Databricks` library provides a background job to repeatedly query for Databricks **finished** job runs, and reports them as metrics.
9+
10+
> :bulb: With using our [Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights](https://www.nuget.org/packages/Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights/), you can report these Databricks reports as metrics in Application Insights.
11+
12+
## Installation
13+
14+
To use these features, you have to install the following package:
15+
16+
```shell
17+
PM > Install-Package Arcus.BackgroundJobs.Databricks -Version 0.3.0
18+
```
19+
20+
## Usage
21+
22+
Our background job has to be configured in `ConfigureServices` method:
23+
24+
```csharp
25+
using Arcus.Security.Core;
26+
using Microsoft.Extensions.DependencyInjection;
27+
28+
public class Startup
29+
{
30+
public void ConfigureServices(IServiceCollection services)
31+
{
32+
// An 'ISecretProvider' implementation (see: https://security.arcus-azure.net/) to access the Azure Service Bus Topic resource;
33+
// this will get the 'tokenSecretKey' string (configured below) and has to retrieve the connection token for the Databricks instance.
34+
services.AddSingleton<ISecretProvider>(serviceProvider => ...);
35+
36+
// Simplest registration of the scheduler job:
37+
services.AddDatabricksJobMetricsJob(
38+
baseUrl: "https://url.to.databricks.instance/"
39+
// Token secret key to connect to the Databricks token.
40+
tokenSecretKey: "Databricks.Token");
41+
42+
// Customized registration of the scheduler job:
43+
services.AddDatabricksJobMetricsJob(
44+
baseUrl: "https://url.to.databricks.instance/"
45+
// Token secret key to connect to the Databricks token.
46+
tokenSecretKey: "Databricks.Token",
47+
options =>
48+
{
49+
// Setting the name which will be used when reporting the metric for finished Databricks job runs (default: "Databricks Job Completed").
50+
options.MetricName = "MyDatabricksJobMetric";
51+
52+
// Settings the time interval (in minutes) in which the scheduler job should run (default: 5 minutes).
53+
options.IntervalInMinutes = 6;
54+
});
55+
}
56+
}
57+
```
58+
59+
[&larr; back](/)

‎docs/versioned_sidebars/version-v0.3-sidebars.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"version-v0.3/tutorialSidebar": [
3+
{
4+
"type": "autogenerated",
5+
"dirName": "."
6+
}
7+
]
8+
}

‎docs/versions.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
[
2+
"v0.3",
23
"v0.2",
34
"v0.1"
45
]

0 commit comments

Comments
 (0)