feature(sdcm/sct_events): Real-time Argus Events

[k0machi]

feature(sdcm/sct_events): Real-time Argus Events

This commit adds new sct_events related module, which allows posting
events as they come from events device into Argus. It is based on the
previously available grafana pipeline. To support earlier (and
always-available) posting, the argus utils module was extended to
provide a singleton argus instance class upon first time initialization
of argus client.

Task: scylladb/argus#787

Testing

• Argus
• Jenkins

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[soyacz]

Run also test with Argus disabled (e.g. locally) so we're sure it does not regress local tries. Add a comment that you tested it (or in description)

[soyacz]

maybe 1 is enough?

[k0machi]

These values were taken from grafana implementation, we could reduce it, I'll ask people who decided on this number

[soyacz]

I think we should start it along with other devices in sdcm.sct_events.setup.start_events_device
so we're not starting it before main events device is there

[k0machi]

We aren't - this is to release the lock on the poster, which would either be started or not already by main events device - the reason this exception handler is there is to guard against situations where events device isn't initialized, for example inside the create_argus_run cli command

[soyacz]

start_events_device also is not run when create_argus_run is fired - is it?
We start other devices there - so this place looks natural.
Also this enable_argus_posting would be redundant then

[k0machi]

There are currently 3 steps that need to be taken to enable argus events:

• Events device, done in start_events_device, which initializes the argus events pipeline
• enable_posting_posting, which sets the client for the postman
• start_posting_argus_events, which releases the lock on the postman to start submitting the events

Latter two can be combined into one, but the split here is warranted as test_config logic decides whether or not argus is enabled and does not enable argus events if they aren't, the pipeline in this case just idles - same as grafana.

[soyacz]

why do we need that? can we create argus client when enabling posting in ArgusEventPostman?

[k0machi]

why do we need that? can we create argus client when enabling posting in ArgusEventPostman?

This means having to instantiate SCTConfiguration (circular dependency) or reading env vars and then also having to manage logic for when argus isn't available (which again, necessitates importing SCTConfiguration), where this singleton class should help other parts of SCT to get already initialized argus client without either resolving the dependency chain or having to manually initialize it.

[soyacz]

we have TestConfig class which should have been initialized when trying to send event to Argus.
Can we use sdcm.test_config.TestConfig.argus_client?

[k0machi]

Circular dependency as well here as importing TestConfig into argus.py means we can't import enablement functions into the test config (NOT the initialization function, those are imported into the events setup.py).

[soyacz]

Generally, I dislike this algorithm: it delays event by time_window and in case of burst of 1000 events it will send anyway 200 of them (with 5 max duplicates set).

I think it would be better to fire off event straight away and for some time period silence the other events with the same event_key.
Something like this (needs testing with e.g. unit test):

event_key = self.unique_key(event)
if time.perf_counter() - time_window_counters.get(event_key, 0) > TIME_WINDOW  # time window e.g. 30 seconds
    # not seen event from sometime or ever 
    time_window_counters[event_key] = time.perf_counter()
else:
        # recently seen
        continue

This solution will bring some memory burden (proportional to no of distinct event_keys) - but assuming even 1M events, should fit memory anyway.

@k0machi can you try this approach?

[k0machi]

Hmm, sounds better, I'll try it (probably should also figure out a way to add argus events to unit_tests)

[soyacz]

see how other events devices are tested

[k0machi]

Before merge: Needs rebase after #12091 is merged

[soyacz]

these are not needed anymore.
@k0machi can you clean up the code a bit and paste links to runs with this new approach?

[k0machi]

Removed.

New approach runs:

https://argus.scylladb.com/tests/scylla-cluster-tests/5eec5069-2332-42ea-9c4b-5d686db45555

[soyacz]

try again, with views-with-tablets experimental feature