Automated Event Grid integration for rotated secrets

[gverstraete]

Because of the annoucement by Microsoft to integrate KeyVault & EventGrid. It might be a good solution to introduce a way to receive those notifications for keyvault rotation in web api's/applications so old cache is actively removed/refreshed.

For web-api's: we can expose one operation & let pass through to arcus to refresh (& renew) cache.
If it's not an API, we can work with a Backgroundthread listening on Service bus topics in the background.

[tomkerkhove]

That's certainly possible, the question is - Do you really want to do that?

In theory you can already do this today by doing a Get("", skipCache: true) (pseudo code).

[tomkerkhove]

The reason why I'm saying that is that if you have a secret that is rolled, you'll get unauthorized exceptions anyway on which you can retry with Polly as shown here:https://github.com/tomkerkhove/demo-azure-key-vault-key-rotation/blob/master/src/TomKerkhove.Demos.KeyVault.API/Controllers/SecretsWithCachingController.cs#L81

In theory you can already achieve your scenario with Arcus by using Arcus.EventGrid to deserialize the events and handle the handshake afterwhich you can use Arcus.Security to get the latest version of the secret (although a RefreshAsync(<key>) might be a good addition.

[gverstraete]

you won't get this on a primary/secondary key approach just as API-M/ blob storage etc... And if you do a skipCache you bypass all the benefits of caching...

Yes probably, with some code in your solution you might be able to solve this. But it's not out of the box. I think having this in kv is a best practice, which we should use as much as possible

[tomkerkhove]

you won't get this on a primary/secondary key approach just as API-M/ blob storage etc..

Why is that?

And if you do a skipCache you bypass all the benefits of caching...

Not if you only do it when you get UnauthorizedException

Yes probably, with some code in your solution you might be able to solve this. But it's not out of the box. I think having this in kv is a best practice, which we should use as much as possible

Since we already provide the fundamentals I think that's as much as we can/should do.

If you want to wire the events in, it should definately be a must and rather something for Arcus.WebApi where you configure it.

Why am I skeptical?

• You need a new endpoint which can be consumed by Azure Event Grid which potentially require you to use a different auth mechanism
• There can be a lot of events flowing in, if it's misconfigured you can potentially create your own DDOS
• It's already possible via Polly as shown in the sample for most scenarios

Does that mean it's a bad idea? Not perse.

What is it that you would need to have to be satisfied? Would you be happy if Arcus.WebApi gives you the foundation to expose an endpoint for which you opt-in?

/cc @ToonVanhoutte in case he wants to provide feedback as well

[gverstraete]

you won't get this on a primary/secondary key approach just as API-M/ blob storage etc..

Why is that?

I mean you won't get an 'UnauthorizedException' in that case. And you want to avoid UnauthorizedException's right?

And if you do a skipCache you bypass all the benefits of caching...

Not if you only do it when you get UnauthorizedException

But that's again custom logic

Yes probably, with some code in your solution you might be able to solve this. But it's not out of the box. I think having this in kv is a best practice, which we should use as much as possible

Since we already provide the fundamentals I think that's as much as we can/should do.

If you want to wire the events in, it should definately be a must and rather something for Arcus.WebApi where you configure it.

Then I still think this is a very common thing that is currently not done because of the extra work on the project. It might still be very good to have it in the template

Why am I skeptical?

• You need a new endpoint which can be consumed by Azure Event Grid which potentially require you to use a different auth mechanism
• There can be a lot of events flowing in, if it's misconfigured you can potentially create your own DDOS
• It's already possible via Polly as shown in the sample for most scenarios

I agree with all the stuff you say here, that's why I proposed also the Service Bus Namespaces. Depending on the security you need to consider a different approach.

Does that mean it's a bad idea? Not perse.

What is it that you would need to have to be satisfied? Would you be happy if Arcus.WebApi gives you the foundation to expose an endpoint for which you opt-in?

/cc @ToonVanhoutte in case he wants to provide feedback as well

I just think automated key rotation is an underestimated thing, this suggestion was a suggestion to fix this. And as we have these EventGrid feature now, it might help us for future.

If you consider this too big for Arcus. Fine, but then this idea is a good one to create a template & quickstart for within Codit.

[tomkerkhove]

We took this offline and discussed the following:

• We'll do a small POC to verify the scenario
• No public endpoint will be exposed but use a Service Bus Topic instead (good suggestion by @gverstraete)
  • Every instance can then create a subscription so that all nodes are updated
• Feature will be opt-in

Things to keep in mind:

• This will be only for Key Vault for now, which is fine
• A good location for this should be found (security, event grid, [...])

Thanks for the idea! We'll have to flesh it out a bit but will be a good addition.

[tomkerkhove]

Specs

Quickly fleshed this out with @gverstraete on what we need:

• We'll provide an ARM template that will create:
  • A Service Bus namespace and topic
  • An Event Grid subscription on a given Key Vault instance for Secret New Version Created events that forward cloud events to the created topic
  • Topic should auto-delete idle subscriptions longer older than 1d
• A KeyVaultSecretManagementJob that will automatically create a subscription on a Service Bus Topic and poll for new events. (name is terrible and subject to change)
  • Once an event occurs, it will use Arcus Event Grid to deserialize it to the MSFT schema (docs) and update the secret in out ISecretProvider
  • A log message should be written when a secret was updated (info)
  • Once the down stops, it should delete the subscription

Here is a quick visual representation:

This approach is done because that way we don't have to expose open ports and use an outbound-only principle. Topics are used in case we have multiple nodes of the same API so that they don't interfere with each other.

ServiceBusEventConsumerHost uses a similar approach as mentioned above so we might want to re-use the principles and refactor it later on base on Arcus Messaging.

I'm happy to pick this one up unless somebody else wants to pick this up.

[tomkerkhove]

@gverstraete As this is more Web API releated we're going to move this as we can use the same mechanism for other things like distributed memory-cache revocations etc.