Integrate with Drupal's queueing mechanisms

[dannylamb]

Drupal has solid queue support, but it's very much database based and expects consumers to be running within Drupal. If we latched onto the core queue system, or perhaps the advancedqueue module, we could get a better UI/Dashboard for the queue jobs. We also then would have the possibility to move our consumers from Java/Camel and into Drupal as Drush/Cron type stuff instead. This would help a lot when dealing with jobs that fail, b/c we'd have direct access to Drupal's logging mechanisms. It would also mean we could drop the few REST endpoints we have, since we can manipulate entities directly without the need for REST.

[joecorall]

I think this is a great idea. This wouldn't necessarily require running code within Drupal to subscribe to the queue.

We could just make an event bus (e.g. alpaca or isle-event-bus) that has credentials to read/write to drupal_default.{queue}, and write the small piece of logic we'd need to interact with the queue table as Drupal expects.

Then Drupal could write its events right into the {queue} instead of using STOMP to send/write to ActiveMQ.

Our event bus could pop the event from {queue}, generate the JWT, and pass to the given index/derivative service, then maybe deadletter or put the event back on failure.

This approach could unlock the ability to make ActiveMQ optional for the stack.

[kayakr]

@joecorall I have been experimenting with using actions to add queueable items via Drupal queues that are then processed via Symfony Messenger https://symfony.com/doc/current/messenger.html and https://www.drupal.org/project/sm; at least on the small scale I've tried it, items were dequeued in near-realtime. I like the idea of a PHP-based event bus (easier to debug rather than having to jump into Java). I prefer Symfony MS over Advanced Queue as I feel the Symfony approach is more PHP-mainstream than AQ. E.g. symfony ms can be backed by non-db transports, including ActiveMQ if you want to go full-circle...

I haven't had the chance to try retries, error handling etc. per https://www.drupal.org/project/sm/issues/3403929.

SM as queue runner for Drupal legacy queues. See https://www.previousnext.com.au/blog/symfony-messenger/post-5-intercepting-queue-worker-plugins ( the whole blog series is worth reading).

[github-actions]

This issue is being marked as stale from inactivity and will be automatically closed in 2 weeks unless further action is taken. If this issue is still relevant please comment. Please also consider attending the weekly Tech Call to discuss the issue

[joecorall]

still relevant