Policy equivalence assessment based on their execution results for the same data #1886
Description
Problem description
Policy comparison capability in Guardian enables policy authors to compare the structure of two policies, and where the tool highlights the differences it is left to the author to draw conclusions about their significance. While this is useful in many respects there are use-cases where it would be more beneficial to assess the equivalence of policies by their effect (results of their actions) irrespective of (or additionally to) their internal structure.
Requirements
- Introduce the capability into the Guardian codebase to easily run 'test dataset' on a policy instance such that the results of the 'run' are also captured, saved and can be compared between runs. Such test runs should not have effect on the external persistent storage (Hedera, IPFS), i.e. this should be performed in the 'test-run' mode.
- Introduce the capability to capture the 'input' into the policy during its execution, in both the 'test-run' and the production operation mode. The 'input' should be saved into the DB, from which it can then be used for running the policy test as per above.
- It should be possible to export saved inputs and results.
Definition of done
- Functionality is implemented as per above
- Documentation is updated accordingly
- At least a single dataset is created to demonstrate the functionality on at least one demo policy
Acceptance criteria
- Ability to capture inputs
- Ability to run test scenarios and compare results