Auto-testing community submitted policies

[anvabr]

Problem description

Guardian repo contain a growing number of policies produced/released by the community of Guardian users. While care is taken to ensure every new version of Guardian maintains backward compatibility, there is a risk that some issues will slip through and some of these policies will become stale and outdated.

Requirements

• Relying on the Policy equivalence assessment based on their execution results for the same data #1886 and Full project data comparison as produced/captured by policies #2704 introduce capability to automatically and repeatably test policies
• Introduce a hook into the new policy merge and release build events which triggers execution of the community policies regression test cycle

Definition of done

Automated policy testing is implemented as described above
At the pre-release stage the full list of community policies is validated, and any problems/deviations are dealt with for the release

Acceptance criteria

• Tests triggers a warning when an incompatible policy version is merged into the repository
• Tests trigger errors when an existing policy failed execution on the new version of Guardian