Programmatically Defer Sidekiq Jobs
Published: July 31, 2020
Why Would You Want This?
Let's say you have a Rails app processing a bunch of jobs of different types - most of them reach out to some 3rd party API. One day, that API breaks for some reason and the dead queue starts filling up. Perhaps you wish to move into some brief pseudo-maintenance period for your app and want jobs to "hold up" until such a period has passed. Or, maybe you really can't deal with Heroku's mandatory dyno once a day restarts and would rather pause jobs, restart the dyno yourself, and then continue on once a day during "off hours."
Here are three options for setting up Sidekiq job deferrals through code.
Option One: Use the Sidekiq API
Sidekiq PRO offers an API to access a given queue to pause and unpause it. The code is rather simple and straightforward...
q = Sidekiq::Queue.new('queue_name') q.pause! q.paused? # => true q.unpause!
This is great... except there are two issues:
- This is a Sidekiq PRO feature. While I 100% think Sidekiq PRO is worth the price if just to support Mike Perham's work, purchasing a PRO licence may not be in the cards.
- This code applies to an entire queue. Your needs may involve targeting certain job classes to defer rather than everything going into a queue. Granted, you could work around this by putting such classes in their own queue.
Option Two: Sidekiq Middleware
Sidekiq supports adding middleware to both the client and server allowing you to append data to jobs, make decisions, and report errors (this is what the Honeybadger gem uses for its Sidekiq integration). For this option, we'll make a very simple server middleware. Why a server middleware? The Sidekiq server is what processes jobs while the client is what adds jobs to the queues.
Assuming this is a relatively up-to-date Rails app and jobs are using ActiveJob...
# lib/sidekiq_middleware/defer_jobs.rb class DeferJobs def call(worker, job, queue) if defer_job?(job) Rails.logger.info("Deferring #{job['class']} with arguments (#{job['args'].inspect})") job['class'].constantize.set(wait: 3.minutes).perform_later(*job['args']) return end yield end def defer_job?(job) # Your logic to determine if the job should be deferred. end end
When the Sidekiq server pulls a job off of the queue, it will eventually hit the #call method on this file. At this point, the job was pulled off the queue, but has not been executed yet (ie, #perform has not been called on the job class). Three arguments are passed in. We are most interested in the job argument. This hash is the standard "job payload" which Sidekiq works with for pretty much the entire library. The full breakdown of what's in this hash can be found on this wiki page. What we are most interested in are the "class" and "args" keys which is a string of the job class name along with an array of positional arguments to pass into the class's #perform method.
Instead of returning like a normal method, you call yield. This will pass the job arguments to the next middleware until there are no more middleware to run and #perform is called on the job class. Returning early without calling yield will result in #perform never being called - effectively sending the job to /dev/null.
In this case, we are calling #defer_job in our middleware and pass in the job hash. At this point, you can do whatever business logic you wish to determine if the job should be deferred. You could look at a database table for a flag that's set, a redis key, the presence of a file, if the job class starts with the letter "B", the current temperature in Intercourse, Pennsylvania, ... whatever you wish. Once you determine to defer the job, the middleware will take the job class and arguments and effectively re-enqueues it for later. In this case, we use the built in Sidekiq job scheduling to run the job in three minutes. After three minutes, the server will pull the job off and run through this middleware again. Repeat until the job is allowed to run.
Finally, we add the middleware in the Sidekiq server setup file...
# config/initializers/sidekiq.rb require 'sidekiq_middleware/defer_jobs' Sidekiq.configure_client do |config| # ... end Sidekiq.configure_server do |config| # ... config.server_middleware do |chain| chain.add DeferJobs end end
... and we're done. This method is relatively straightforward, allows for fully control over what and when to defer, and integrates nicely into Sidekiq's flow.
One possible downside to this option is it could be a little too out of the way. I don't see custom middlewares in Rails apps very often, and it's possible a senior developer may write something like this and it works great for the longest time, but everyone forgets about it. Later in the app's life, other developers may be perplexed as to how this "deferral logic" works and where it happens since it slips itself so nicely into Sidekiq.
Option Three: Good Old Fashioned Prepended Module
This is a rather "rustic" but simple approach to solving this. Instead of hooking into Sidekiq directly, take control of your own code and stop #perform from doing anything!
# app/jobs/job_deferral.rb module JobDeferral def perform(*args) if defer_job? Rails.logger.info "Deferring '#{self.class}' (with args: #{args.inspect})" self.class.set(wait: 3.minutes).perform_later(*args) return end super end def defer_job? # Logic here... end end # app/jobs/my_worker.rb class MyWorker < ApplicationJob queue_as :default prepend JobDeferral def perform(id) # Your worker code... end end
This is very similar to the Sidekiq middleware, except we are intercepting the call to #perform via Module.prepend with our own method. The method goes through whatever logic may want to determine deferrals or not and schedules the job for later. Instead of yield, we call super which is the original #perform method of the job class. You may pick and choose which job classes to prepend module instead of every class gaining the extra logic. This allows for any developer to look at any job class code, see a module being prepended, and easily jump to said code to see what it does.
There is one rather annoying side effect. To sidekiq, #perform is called. Deferred or not, Sidekiq sees #perform called and completing without issue. So if you do defer the job, your Rails log will still include the "enqueued job" and "completed job" log entries from Sidekiq. This could lead to confusion while viewing logs to debug issues.
Wrapping It Up
All three options are viable and offer their own pros and cons. I personally prefer the first option where some code can trigger a pause on queues and then unpause them later. The other two options have the benefit of being 100% free and gives you more fine-grained control over logic, but comes with the price of added complexity.
One More Note...
The astute reader may recall seeing a "Quiet" button in the non-pro Sidekiq UI for each worker process. This is NOT the same as pause. Putting a process in "quiet" mode is typically reserved for prepping Sidekiq for restarts. Once you quite a process, you cannot "unquiet" it until Sidekiq is restarted.
