Allow publishers to specify a different subject for "Nats-Expected-Last-Subject-Sequence"

[cchamplin]

Proposed change

I'd like to request a change to allow publishers to provide a specific subject for use when nats-server enforced "Nats-Expected-Last-Subject-Sequence". Adding an additional header field that allows publishers to optionally change the subject used for "Nats-Expected-Last-Subject-Sequence" from the subject of the message being published to to a subject provided in the header.

The proposal is for a "Nats-Expected-Last-Subject-Sequence-Subject" header that when set with a string value becomes the subject used during enforcement of the last sequence. The header does not necessarily need to be that string exactly, the functionality is the actual proposal.

Use case

The specific use case I am trying to solve here is to allow publishing to complex and dynamic subjects while still enabling optimistic concurrency control.

For example in an event sourcing application an aggregate might be defined against the subject:

• orders.[order_id]
  • orders.1
  • orders.2
  • orders.3
  • ...

Today we can use "Nats-Expected-Last-Subject-Sequence" to ensure that when we publish to orders.2 we are publishing against the known last sequence for messages on the orders.2 subject.

However, it is often desirable to have more complex subject patterns. For example image a pattern that looks like this:

• orders.[order_id].[event_type].[visibility]
  • orders.1.created.public
  • orders.1.approved.private
  • orders.2.created.public
  • ...

Today we cannot use "Nats-Expected-Last-Subject-Sequence" with this pattern and still have OCC for the aggregate orders.2 when publishing. If a publisher attempts to set the expected sequence and then publish to order.2.rejected.private the sequence for that subject is likely 0 as opposed to sequence for the last event published to order.2.>

By allowing publishers to override and customize the subject used in the expect last subject sequence check OCC can be enforced for the aggregate itself versus the subject being published to. For example by setting the proposed "Nats-Expected-Last-Subject-Sequence-Subject" to "order.2.>"

Relevant Slack Conversations:

• https://natsio.slack.com/archives/C055AB5M1KM/p1710872807434689
• https://natsio.slack.com/archives/CM3T6T7JQ/p1687313617160579

Contribution

Possibly

[ripienaar]

Looping in some people who know more about event sourcing than me @bruth @codegangsta

[cchamplin]

If we can get this change accepted or have some discussion around it we'd also like to propose an additional set of headers:

	JSExpectedLastSubjHeader     = "Nats-Expected-Last-Subject-Header"
	JSExpectedLastSubjHeaderSubj = "Nats-Expected-Last-Subject-Header-Subject"

That would allow publishers to specify an expected header and value on the last message for a message subject or an overriding subject.

For example Nats-Expected-Last-Subject-Header: "Aggregate-Version=10"

In this case nats server would grab the last message for the subject on the stream, extract the "Aggregate-Version" header from that message and compare it against 10 and rejecting the publish accordingly. This I think offers the most flexibility and would allow event sourcing implementations to leverage their own versioning for an aggregate root rather than worrying about the intricacies of stream sequences, especially since my understanding is that sequences can potentially change when events are mirrored or sources which makes them pretty undesirable as a source of truth for a version or sequence number in an event sources aggregate.

Flexible on the exact details, pattern, and names of things here, but hoping the idea gets some traction.

[derekcollison]

Mirrors are bit for bit copies, so sequence and timestamp stay the same. For sources they change, but we preserve the originals in headers.

[bruth]

@cchamplin This is a great addition and straightforward PR. I will want to play with this a bit to see how much this could be abused (subject length/number of tokens), but I am very supportive of this. Solves a very concrete problem for ES.

[codegangsta]

Yeah I really like this addition. like @bruth I'd like the opportunity to play with it this week