refactor(db-scheduler): optimize all_as_schedule query

[alirafiei75]

Summary

This PR optimizes the all_as_schedule function to improve performance by filtering unnecessary tasks from the database query.

Changes

• Exclude clocked tasks not due within 5 minutes.
• Exclude crontab tasks outside the relevant hour range.

fixes #675
fixes #786

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.42%. Comparing base (17d87f4) to head (7b9b0d2).
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #835      +/-   ##
==========================================
+ Coverage   87.20%   87.42%   +0.21%     
==========================================
  Files          32       32              
  Lines         938      954      +16     
  Branches       76       76              
==========================================
+ Hits          818      834      +16     
  Misses        102      102              
  Partials       18       18              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alirafiei75]

This is cpu usage of our celery-beat server. We depend highly on cronjobs and clockjobs and as you see, optimizing the scheduler query, led to reduction in cpu usage considerably (see 11:53 in the picture)

[auvipy]

The changes look good from the first review. I will thoroughly Review it again in a couple of days before merging

[daydaychen]

I have also encountered performance issues when there are a large number of scheduled tasks. I addressed this in a straightforward manner by adding a last_pull and pull_interval attribute. Within the schedule_changed method, when a change is detected, I check whether the conditions between last_pull and pull_interval are met. This reduces the number of queries and avoids repeated pulls during frequent updates.

However, it is evident that alirafiei75's solution is more ingenious. It not only addresses the performance issues but also effectively reduces the pressure on database queries.

[auvipy]

thanks for your work. feel free to contribute back if you find any better improvement or regression.

[auvipy]

hey, we got report about this PR #875 (comment) can you please check and if possible come with a possible fix? otherwise we have to revert this

[alirafiei75]

Oh man, just saw this. Thanks for pointing it out. I've read the comments, and it looks like there's an issue with this PR. I'll definitely look into it and try to come up with a fix. However, I'm currently on a short trip, so it might take until tomorrow. If it's too urgent to wait, feel free to revert for now—I'll make sure to fix it later.

[auvipy]

I can wait for your pr

[alirafiei75]

I have made the PR @auvipy . please review it whenever it is convenient

• Refactor / all_as_schedule crontab query optimization #879