FITS derivatives can trigger infinite loops

[dannylamb]

Despite being explicitly told to only attempt 10 retries, Camel is infinitely looping on FITS derivatives on occasion. I've yet to narrow things down as to what specifically is triggering this, but it does appear to manifest itself on a variety of file types/sizes etc. You can tell it's happening because you'll see versions of the FITS file being created over and over again.

If you disable the alpaca container, the loop stops. Turn it back on and the infinite loop starts all over again.

I've confirmed this behaviour on 1.0.7 containers, on both the born digital platform and the community offering. I'm thinking it would be prudent to update our Alpaca containers to run the newest Alpaca that has updated Camel and runs as an executable jar file. If that doesn't magically fix it, then we'll need to file an issue against Alpaca and fix the code/configuration set up.

[dannylamb]

It would be great to know if folks have run into this with non-ISLE setups.

[nigelgbanks]

Related ticket: #225

[nigelgbanks]

I've seen two fits media entities get created for a single item today, not sure if related, also see discussion here: Islandora-Devops/islandora-starter-site#19

[nigelgbanks]

Not just fits, seems like this can happen to all the queues, and it happens often.

[nigelgbanks]

Even when successful, I've often seen the queues going up and up and repeating work successfully.

[nigelgbanks]

My issue from before was caused by having both the Service File and Original File set on a media entity, which caused a derivative to be generated for the item, then the system would see that Original File had changed, and trigger another derivative and so on forever.

[dannylamb]

🤔 very interesting. There has to be a way to safeguard against that.

[rosiel]

There are a few issues here.

1. infinite derivatives, if you set something as Original File and Service File then you update it in a loop forever.
2. Around the release of Filehash 2, filehashes became fields on files (not properties). because of that, our 'skip derivatives if the filehash is the same' never kicks in, so infinite multifile derivatives.
3. Also, the same failed check is causing "2" things to be run because derivatives get executed once during hook_file_update and once during hook_media_update/insert.

[nigelgbanks]

With @rosiel fixes, I cannot reproduce infinite derivatives. Additionally, in the upcoming 2.x release of isle-buildkit we've switched to using alpaca as a stand-alone jar (which might have been related to the FITS issue... unsure). I'm going to close this issue.