On cron and cron management DSLs
The typical approach when using whenever for scheduling is to use deployment hooks to update the crontab from whenever's configuration during application deployment. However this approach falls short when the schedule is expected to be configurable at run time and especially if we want to give administrators fine grained controls over what is being scheduled and when.
Database driven scheduling
The post outlines an alternative solution using the library clockwork that makes it easy to make event scheduling run time configurable through database models managed using familiar ruby ORMs.
ActiveJob and standardized APIs for background processing
Rather than coupling our code to a specific background processor which might in turn may be coupled with a specific transport system (like Sidekiq and Redis) it is advisable to rely instead (as much as possible) on ActiveJob API which provides a standardized API for background processing in Rails ecosystem.
Integration with Admin interfaces
The benefit of the event system being manageable through ActiveRecord is that integrated admin interfaces like ActiveAdmin and Administrate (which we might already have integrated in our existing applications) work out of the box and we get a complete schedule management interface with minimal extraneous boilerblate.
Getting started with our app
The rest of the tutorial walks through the creation of a Postgres backed Rails 5 application that illustrates the concepts outlined above.
We start off with familiar rails application generation steps:
$ gem install rails --pre --no-rdoc --no-ri $ rails -v Rails 5.0.0.beta3
Note that while skipping rdoc and ri is purely a matter of convenience (I did not need them at the time) - the
--pre flag is required, as of this writing, for installing Rails 5 as it is still in beta.
Note the line
create app/jobs/application_job.rb above. Rails 5 comes pre-integrated with ActiveJob.
Next we add clockwork and sidekiq to our Gemfile.
gem 'clockwork' gem 'sidekiq'
It may be tempting to just leave the in-memory adapter of ActiveJob in place but it should not be used in any production application. Rails guides explain it well enough:
Rails itself only provides an in-process queuing system, which only keeps the jobs in RAM. If the process crashes or the machine is reset, then all outstanding jobs are lost with the default async back-end. This may be fine for smaller apps or non-critical jobs, but most production apps will need to pick a persistent backend.
Implementing the Event model
Next we need to define our models for persisting our schedules.
rails g model event name:string frequency:integer at:string job_name:string job_arguments:jsonb
Here the first three columns correspond to accessors mandated by Clockwork. Name is primary for descriptive logging purposes (more on this below).
frequency specifies the recurrance frequency in seconds.
at signifies point of occurance within the recurrance span. Following are the valid formats:
HH:MM H:MM **:MM HH:** (Mon|mon|Monday|monday) HH:MM
The last two columns
job_arguments identify the job to be triggered. As would become obvious below, we did not need to provide the job name through the database - it could be inferred at the runtime through any custom logic expressed in ruby. But having it in database leads to a straightforward and transparent implementation and management.
It may be tempting to just reuse the
name field as the job_name as well, but it may obscure debugging when same job is being invoked as part of multiple events for different use cases. It is recommended to keep the
name as something representative of the use case - eg.
Our model would look something like below:
class Event < ApplicationRecord validates :name, :frequency, :job_name, presence: true end
The entry point of clockwork is the file
clock.rb. This is the file that tells clockwork to
poll the events table and execute the inferred job.
require 'clockwork' require 'clockwork/database_events' require_relative './config/boot' require_relative './config/environment' module Clockwork # required to enable database syncing support Clockwork.manager = DatabaseEvents::Manager.new sync_database_events model: ::Event, every: 1.minute do |event| event.job_name.constantize.perform_later(event.job_arguments) end end
Configuring ActiveJob to use sidekiq
We had added sidekiq as the persistence backend for ActiveJob but we have not configured ActiveJob to use it.
That is one additional line of code in
module Rails5ClockworkDemo class Application < Rails::Application # Configure ActiveJob to use sidekiq config.active_job.queue_adapter = :sidekiq end end
Clockwork can be executed by running
clockwork clock.rb at project root. However we can not expect something exciting yet because we simply have no entries in the table:
$ clockwork clock.rb I, [2016-04-01T02:02:54.032058 #57534] INFO -- : Starting clock for 1 events: [ sync_database_events_for_model_Event ] I, [2016-04-01T02:02:54.032150 #57534] INFO -- : Triggering 'sync_database_events_for_model_Event'
Adding administrative interface:
We would be using ActiveAdmin for our admin interface for managing events. As of this writing to use ActiveAdmin along with Rails 5 we need to use the master branch of ActiveAdmin.
While we will not go into elaboration of the ActiveAdmin DSL, most of the ideas should be applicable to alternative admin builders as well.
# Gemfile.rb gem 'activeadmin', github: 'activeadmin'
$ rails g active_admin:install
For now we skip authorization as well as authentication entirely - integrating ActiveAdmin with authentication systems is covered here and usage in conjugation with authorization systems is covered here in the official docs.
To present our Event model through ActiveAdmin we need to generate an ActiveAdmin Event resource.
$ rails g active_admin:resource Event
The above command creates the following file for us:
# app/admin/event.rb ActiveAdmin.register Event do # See permitted parameters documentation: # https://github.com/activeadmin/activeadmin/blob/master/docs/2-resource-customization.md#setting-up-strong-parameters # # permit_params :list, :of, :attributes, :on, :model # # or # # permit_params do # permitted = [:permitted, :attributes] # permitted << :other if params[:action] == 'create' && current_user.admin? # permitted # end end
Once we configure permitted parameters as elaborated in the comments we have something like below:
ActiveAdmin.register Event do permit_params :name, :job_name, :job_arguments, :frequency, :at end
Now after we can run our migrations and booted up our server:
=> Booting Puma => Rails 5.0.0.beta3 application starting in development on http://localhost:3000 => Run `rails server -h` for more startup options => Ctrl-C to shutdown server Puma starting in single mode... * Version 3.2.0 (ruby 2.3.0-p0), codename: Spring Is A Heliocentric Viewpoint * Min threads: 5, max threads: 5 * Environment: development * Listening on tcp://localhost:3000
If this is the first time you are using Rails 5 the new default home page as well as puma as the default server can be a pleasant surprise.
Our admin panel available at
/admin provides us with the means to edit our events.
However when we attempt to create an event we would be faced with an error because formtastic does not know out of the box how to handle jsonb field which we used for job arguments.
While it is not difficult to create custom form inputs for formtastic and I have outlined an alternative approach before, to keep the example simple let us simply use a textarea for the arguments field:
form do |f| f.inputs do f.input :name f.input :job_name f.input :frequency f.input :at f.input :job_arguments, as: :text end f.actions end
Now we can create an event to trigger a dummy job:
Defining our jobs:
Of course, for this job to be runnable, we need to define the job as well. We can use ActiveJob's generators for the same:
$ rails g job dummy Running via Spring preloader in process 63269 invoke test_unit create test/jobs/dummy_job_test.rb create app/jobs/dummy_job.rb
Our job implementation itself is fairly mundane but serves the purpose of illustration:
class DummyJob < ApplicationJob queue_as :default def perform(*args) puts "Dummy Job Executed" end end
Now once our clockwork process synchronizes with the databases, it will pickup the dummy job and keep executing every one second:
I, [2016-04-01T02:30:54.003679 #57534] INFO -- : Triggering 'sync_database_events_for_model_Event' I, [2016-04-01T02:30:54.005132 #57534] INFO -- : Triggering 'execute_dummy_job' I, [2016-04-01T02:30:55.005873 #57534] INFO -- : Triggering 'execute_dummy_job' I, [2016-04-01T02:30:56.001670 #57534] INFO -- : Triggering 'execute_dummy_job' I, [2016-04-01T02:30:57.001761 #57534] INFO -- : Triggering 'execute_dummy_job'
In the rails development log we can see that ActiveRecord is queuing this job:
[ActiveJob] Enqueued DummyJob (Job ID: 64813900-c70f-4b40-9dd7-452c4cac6b73) to Sidekiq(default) [ActiveJob] Enqueued DummyJob (Job ID: 0d0a66d1-a46b-4427-92db-83a301b38c1c) to Sidekiq(default) [ActiveJob] Enqueued DummyJob (Job ID: 1de45b36-7cc9-4a9a-9bb7-363591f64dd5) to Sidekiq(default) [ActiveJob] Enqueued DummyJob (Job ID: 483fe8a2-5acf-4830-80fb-6c6883f0f7c2) to Sidekiq(default)
Now if we run sidekiq, our background processor - we should see the enqued job getting executed in the
2016-03-31T21:01:00.064Z 64128 TID-ox54l2v7s DummyJob JID-fc59a048d2d1fe2aeecc3452 INFO: start Dummy Job Executed
This concludes our introductory post on database driven scheduling with clockwork. Please share any issues you might have faced or any suggestions for improvement in the comments.
The source code for this post is available in this github repo.