[N-Region Synchronous Commit] ConsistencyWriter - Apply Barrier on Writes for <Strong Consistency, when `EnableNRegionSynchrousCommit` flag and Header is pPresent

[kundadebdatta]

Requirement

• Update the StoreResponse class to add the new GlobalNRegionCommittedGLSN headers and make them available in the ConsistencyWriter class.

• Refactor the ConsistencyWriter barrier implementation to work with any header value. At the moment it is implemented with GlobalCommittedDataLSN (GCLSN). In the future it should work to wait for any LSN in our case the GlobalNRegionCommittedGLSN.

• Enforce the write barrier when the consistency level !=strong and EnableNRegionSynchrousCommit flag = True.

Acceptance Criteria:

• Add tests to validate the barrier calls when n-region synchronous commit header is present.

Notes and Follow-Ups:

In the n-region spec backend team has decided that when all regions are down, then BE will have to prevent failovers when NRegionSynchronousCommit is enabled. This also applies for N=1. Write region fails, we have to stop failover since N=0. Now this requirement can be relaxed since we are promising best effort. In that case, no additional will be required in the backend.

We might have to check the logic here what happens when PPAF min durability = N = 1. To summarize, there's going to be a change to make the barrier stick to the write region and not jump around. So if there's failover the barrier will keep polling the old region and time out. On the next write it will be redirected on the new write region and the barrier will be enforced there.

The issue is if the write is performed on region A and the barrier enforced on region B. This will be problematic if there's false progress and it has rolled back changes. In this case the GlobalNREgionCommittedGLSN > LSN.