Docs - publish feature docs v0.2

docs/features/databricks/gain-insights.md

@@ -0,0 +1,59 @@
+---
+title: "Interact with Databricks to gain insights"
+layout: default
+---
+
+# Interact with Databricks to gain insights
+
+## Installation
+
+To use these features, you have to install the following package:
+
+```shell
+PM > Install-Package Arcus.BackgroundJobs.Databricks
+```
+
+> :bulb: With using our [Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights](https://www.nuget.org/packages/Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights/), you can report these Databricks reports as metrics in Application Insights.
+
+## Usage
+We provide a  `DatabricksInfoProvider` which allows you to interact with Databricks clusters to gain insights on your workloads, such as measuring job run outcomes.
+
+It can be easily setup and used anywhere such as .NET Core workers, Azure Functions and more. We are using this ourselves for our [job metrics](./job-metrics).
+
+```csharp
+ILogger logger = ...
+using (var client = DatabricksClient.CreateClient("https://databricks.base.url", "security.token"))
+using (var provider = new DatabricksInfoProvider(client, logger))
+{
+}
+```
+
+### Getting finished job run information
+Gets all the finished job runs within a given time window.
+```csharp
+DatabricksInfoProvider provider = ...
+var startOfWindow = DateTimeOffset.UtcNow.AddDays(-1);
+var endOfWindow = DateTimeOffset.UtcNow;
+IEnumerable<JobRun> finishedJobRuns = await provider.GetFinishedJobRunsAsync(startOfWindow, endOfWindow);
+```
+
+It provides following information about the job that was executated, such as name & id, along with [details about a single job run](https://github.com/Azure/azure-databricks-client/blob/master/csharp/Microsoft.Azure.Databricks.Client/Run.cs).
+
+### Measure finished job outcomes
+Measures the finished job runs by reporting the results as (multi-dimensional) metrics.
+
+This method is an combination of the previously defined method (**Getting finished jobs**) and calling an `ILogger` extension provided in this package (`ILogger.LogMetricFinishedJobOutcome`) which will write the finished job runs `JobRun` instances as metrics.
+
+```csharp
+DatabricksInfoProvider provider = ...
+var metricName = "Databricks Job Completed";
+var startOfWindow = DateTimeOffset.UtcNow.AddDays(-1);
+var endOfWindow = DateTimeOffset.UtcNow;
+await provider.MeasureJobOutcomesAsync(metricName, startOfWindow, endOfWindow);
+// Logs > Metric Databricks Job Completed: 1 {UtcNow} (Context: {[Run Id] = my.run.id, [Job Id] = my.job.id, [Job Name] = my.job.name, [Outcome] = Success})
+```
+
+> Note: you can always call **Getting finished jobs** yourself and pass along the finished jobs to the available `ILogger.LogMetricFinishedJobOutcome` extension. 
+> That way, you can pass along additional contextual properties
+
+[&larr; back](/)

docs/features/databricks/job-metrics.md

@@ -0,0 +1,53 @@
+---
+title: "Measure Databricks job run outcomes as metric"
+layout: default
+---
+
+# Measure Databricks job run outcomes as metric
+
+The `Arcus.BackgroundJobs.Databricks` library provides a background job to repeatedly query for Databricks **finished** job runs, and reports them as metrics.
+
+> :bulb: With using our [Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights](https://www.nuget.org/packages/Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights/), you can report these Databricks reports as metrics in Application Insights.
+
+## Installation
+
+To use these features, you have to install the following package:
+
+```shell
+PM > Install-Package Arcus.BackgroundJobs.Databricks
+```
+
+## Usage
+
+Our background job has to be configured in `ConfigureServices` method:
+
+```csharp
+public void ConfigureServices(IServiceCollection services)
+{
+    // An 'ISecretProvider' implementation (see: https://security.arcus-azure.net/) to access the Azure Service Bus Topic resource;
+    //     this will get the 'tokenSecretKey' string (configured below) and has to retrieve the connection token for the Databricks instance.
+    services.AddSingleton<ISecretProvider>(serviceProvider => ...);
+
+    // Simplest registration of the scheduler job:
+    services.AddDatabricksJobMetricsJob(
+        baseUrl: "https://url.to.databricks.instance/" 
+        // Token secret key to connect to the Databricks token.
+        tokenSecretKey: "Databricks.Token");
+
+    // Customized registration of the scheduler job:
+    services.AddDatabricksJobMetricsJob(
+        baseUrl: "https://url.to.databricks.instance/" 
+        // Token secret key to connect to the Databricks token.
+        tokenSecretKey: "Databricks.Token",
+        options =>
+        {
+            // Setting the name which will be used when reporting the metric for finished Databricks job runs (default: "Databricks Job Completed").
+            options.MetricName = "MyDatabricksJobMetric";
+
+            // Settings the time interval (in minutes) in which the scheduler job should run (default: 5 minutes). 
+            options.IntervalInMinutes = 6;
+        });
+}
+```
+
+[&larr; back](/)

docs/features/security/auto-invalidate-secrets.md

@@ -9,7 +9,7 @@ The `Arcus.BackgroundJobs.KeyVault` library provides a background job to automat
 
 ## How does it work?
 
-<a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Farcus.azure%2Farcus.backgroundjobs%2Fmaster%2Fdeploy%2Farm%2Fazure-key-vault-job.json" target="_blank">
+<a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Farcus-azure%2Farcus.backgroundjobs%2Fmaster%2Fdeploy%2Farm%2Fazure-key-vault-job.json" target="_blank">
     <img src="https://azuredeploy.net/deploybutton.png"/>
 </a>
 

docs/index.md

@@ -24,6 +24,13 @@ For more granular packages we recommend reading the documentation.
     - [Securely Receive CloudEvents](features/cloudevent/receive-cloudevents-job)
 - **Security**
     - [Automatically invalidate cached secrets from Azure Key Vault](features/security/auto-invalidate-secrets)
+- **Databricks**
+    - [Measure Databricks job run outcomes as metric](features/databricks/job-metrics)
+    - [Interact with Databricks to gain insights](features/databricks/gain-insights)
+
+# Older versions
+
+* [v0.1](v0.1)
 
 # License
 This is licensed under The MIT License (MIT). Which means that you can use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the web application. But you always need to state that Codit is the original author of this web application.

docs/v0.2/features/cloudevent/receive-cloudevents-job.md

@@ -0,0 +1,46 @@
+---
+title: "Securely Receive CloudEvents"
+layout: default
+---
+
+# Securely Receive CloudEvents
+
+The `Arcus.BackgroundJobs.CloudEvents` library provides a collection of background jobs to securely receive [CloudEvents](https://github.com/cloudevents/spec).
+
+This allows workloads to asynchronously process event from other components without exposing a public endpoint.
+
+## How does it work?
+
+An Azure Service Bus Topic resource is required to receive CloudEvents on. CloudEvent messages on this Topic will be processed by a background job.
+
+![Automatically Invalidate Azure Key Vault Secrets](/media/CloudEvents-Job.png)
+
+You can write your own background job(s) by deriving from `CloudEventBackgroundJob` which already takes care of topic subscription creation/deletion on start/stop of the job.
+
+## Usage
+
+You can easily implement your own job by implementing the `ProcessMessageAsync` method to prcocess new CloudEvents.
+
+```csharp
+public class MyBackgroundJob : CloudEventBackgroundJob
+{
+    public MyBackgroundJob(
+        IConfiguration configuration,
+        IServiceProvider serviceProvider,
+        ILogger<CloudEventBackgroundJob> logger) : base(configuration, serviceProvider, logger)
+    {
+
+    }
+
+    protected override async Task ProcessMessageAsync(
+        CloudEvent message,
+        AzureServiceBusMessageContext messageContext,
+        MessageCorrelationInfo correlationInfo,
+        CancellationToken cancellationToken)
+        {
+            // Process the CloudEvent message...
+    }
+}
+```
+
+[&larr; back](/)

docs/v0.2/features/databricks/gain-insights.md

@@ -0,0 +1,59 @@
+---
+title: "Interact with Databricks to gain insights"
+layout: default
+---
+
+# Interact with Databricks to gain insights
+
+## Installation
+
+To use these features, you have to install the following package:
+
+```shell
+PM > Install-Package Arcus.BackgroundJobs.Databricks
+```
+
+> :bulb: With using our [Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights](https://www.nuget.org/packages/Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights/), you can report these Databricks reports as metrics in Application Insights.
+
+## Usage
+We provide a  `DatabricksInfoProvider` which allows you to interact with Databricks clusters to gain insights on your workloads, such as measuring job run outcomes.
+
+It can be easily setup and used anywhere such as .NET Core workers, Azure Functions and more. We are using this ourselves for our [job metrics](./job-metrics).
+
+```csharp
+ILogger logger = ...
+using (var client = DatabricksClient.CreateClient("https://databricks.base.url", "security.token"))
+using (var provider = new DatabricksInfoProvider(client, logger))
+{
+}
+```
+
+### Getting finished job run information
+Gets all the finished job runs within a given time window.
+```csharp
+DatabricksInfoProvider provider = ...
+var startOfWindow = DateTimeOffset.UtcNow.AddDays(-1);
+var endOfWindow = DateTimeOffset.UtcNow;
+IEnumerable<JobRun> finishedJobRuns = await provider.GetFinishedJobRunsAsync(startOfWindow, endOfWindow);
+```
+
+It provides following information about the job that was executated, such as name & id, along with [details about a single job run](https://github.com/Azure/azure-databricks-client/blob/master/csharp/Microsoft.Azure.Databricks.Client/Run.cs).
+
+### Measure finished job outcomes
+Measures the finished job runs by reporting the results as (multi-dimensional) metrics.
+
+This method is an combination of the previously defined method (**Getting finished jobs**) and calling an `ILogger` extension provided in this package (`ILogger.LogMetricFinishedJobOutcome`) which will write the finished job runs `JobRun` instances as metrics.
+
+```csharp
+DatabricksInfoProvider provider = ...
+var metricName = "Databricks Job Completed";
+var startOfWindow = DateTimeOffset.UtcNow.AddDays(-1);
+var endOfWindow = DateTimeOffset.UtcNow;
+await provider.MeasureJobOutcomesAsync(metricName, startOfWindow, endOfWindow);
+// Logs > Metric Databricks Job Completed: 1 {UtcNow} (Context: {[Run Id] = my.run.id, [Job Id] = my.job.id, [Job Name] = my.job.name, [Outcome] = Success})
+```
+
+> Note: you can always call **Getting finished jobs** yourself and pass along the finished jobs to the available `ILogger.LogMetricFinishedJobOutcome` extension. 
+> That way, you can pass along additional contextual properties
+
+[&larr; back](/)

docs/v0.2/features/databricks/job-metrics.md

@@ -0,0 +1,53 @@
+---
+title: "Measure Databricks job run outcomes as metric"
+layout: default
+---
+
+# Measure Databricks job run outcomes as metric
+
+The `Arcus.BackgroundJobs.Databricks` library provides a background job to repeatedly query for Databricks **finished** job runs, and reports them as metrics.
+
+> :bulb: With using our [Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights](https://www.nuget.org/packages/Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights/), you can report these Databricks reports as metrics in Application Insights.
+
+## Installation
+
+To use these features, you have to install the following package:
+
+```shell
+PM > Install-Package Arcus.BackgroundJobs.Databricks
+```
+
+## Usage
+
+Our background job has to be configured in `ConfigureServices` method:
+
+```csharp
+public void ConfigureServices(IServiceCollection services)
+{
+    // An 'ISecretProvider' implementation (see: https://security.arcus-azure.net/) to access the Azure Service Bus Topic resource;
+    //     this will get the 'tokenSecretKey' string (configured below) and has to retrieve the connection token for the Databricks instance.
+    services.AddSingleton<ISecretProvider>(serviceProvider => ...);
+
+    // Simplest registration of the scheduler job:
+    services.AddDatabricksJobMetricsJob(
+        baseUrl: "https://url.to.databricks.instance/" 
+        // Token secret key to connect to the Databricks token.
+        tokenSecretKey: "Databricks.Token");
+
+    // Customized registration of the scheduler job:
+    services.AddDatabricksJobMetricsJob(
+        baseUrl: "https://url.to.databricks.instance/" 
+        // Token secret key to connect to the Databricks token.
+        tokenSecretKey: "Databricks.Token",
+        options =>
+        {
+            // Setting the name which will be used when reporting the metric for finished Databricks job runs (default: "Databricks Job Completed").
+            options.MetricName = "MyDatabricksJobMetric";
+
+            // Settings the time interval (in minutes) in which the scheduler job should run (default: 5 minutes). 
+            options.IntervalInMinutes = 6;
+        });
+}
+```
+
+[&larr; back](/)

docs/v0.2/features/security/auto-invalidate-secrets.md

@@ -0,0 +1,50 @@
+---
+title: "Automatically Invalidate Azure Key Vault Secrets"
+layout: default
+---
+
+# Automatically Invalidate Azure Key Vault Secrets
+
+The `Arcus.BackgroundJobs.KeyVault` library provides a background job to automatically invalidate cached Azure Key Vault secrets from an `ICachedSecretProvider` instance of your choice.
+
+## How does it work?
+
+<a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Farcus-azure%2Farcus.backgroundjobs%2Fmaster%2Fdeploy%2Farm%2Fazure-key-vault-job.json" target="_blank">
+    <img src="https://azuredeploy.net/deploybutton.png"/>
+</a>
+
+
+This automation works by subscribing on the `SecretNewVersionCreated` event of an Azure Key Vault resource and placing those events on a Azure Service Bus Topic; which we process in our background job.
+
+![Automatically Invalidate Azure Key Vault Secrets](/media/Azure-Key-Vault-Job.png)
+
+To make this automation opperational, following Azure Resources has to be used:
+* Azure Key Vault instance
+* Azure Service Bus Topic
+* Azure Event Grid subscription for `SecretNewVersionCreated` events that are sent to the Azure Service Bus Topic
+
+## Usage
+
+Our background job has to be configured in `ConfigureServices` method:
+
+```csharp
+public void ConfigureServices(IServiceCollection services)
+{
+    // An 'ISecretProvider' implementation (see: https://security.arcus-azure.net/) to access the Azure Service Bus Topic resource;
+    //     this will get the 'serviceBusTopicConnectionStringSecretKey' string (configured below) and has to retrieve the connection string for the topic.
+    services.AddSingleton<ISecretProvider>(serviceProvider => ...);
+
+    // An `ICachedSecretProvider` implementation which secret keys will automatically be invalidated.
+    services.AddSingleton<ICachedSecretProvider>(serviceProvider => new CachedSecretProvider(mySecretProvider));
+
+    services.AddAutoInvalidateKeyVaultSecretBackgroundJob(
+        // Prefix of the Azure Service Bus Topic subscription;
+        //    this allows the background jobs to support applications that are running multiple instances, processing the same type of events, without conflicting subscription names.
+        subscriptionNamePrefix: "MyPrefix"
+
+        // Connection string secret key to a Azure Service Bus Topic.
+        serviceBusTopicConnectionStringSecretKey: "MySecretKeyToServiceBusTopicConnectionString");
+}
+```
+
+[&larr; back](/)

docs/v0.2/index.md

@@ -0,0 +1,35 @@
+---
+title: "Home"
+layout: default
+---
+
+[![NuGet Badge](https://buildstats.info/nuget/Arcus.BackgroundJobs.CloudEvents?packageVersion=0.2)](https://www.nuget.org/packages/Arcus.BackgroundJobs.CloudEvents/0.2.0)
+
+# Installation
+
+The Arcus BackgroundJobs can be installed via NuGet:
+
+```shell
+PM > Install-Package Arcus.BackgroundJobs.CloudEvents -Version 0.2.0
+```
+
+For more granular packages we recommend reading the documentation.
+
+# Features
+
+- **General**
+    - [Securely Receive CloudEvents](features/cloudevent/receive-cloudevents-job)
+- **Security**
+    - [Automatically invalidate cached secrets from Azure Key Vault](features/security/auto-invalidate-secrets)
+- **Databricks**
+    - [Measure Databricks job run outcomes as metric](features/databricks/job-metrics)
+    - [Interact with Databricks to gain insights](features/databricks/gain-insights)
+
+# Older versions
+
+* [v0.1](../v0.1)
+
+# License
+This is licensed under The MIT License (MIT). Which means that you can use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the web application. But you always need to state that Codit is the original author of this web application.
+
+*[Full license here](https://github.com/arcus-azure/arcus.backgroundjobs/blob/master/LICENSE)*