Ruby
This post has not been updated in quite some time and the content here may be out of date
or not reflect my current my recommedation in the matter.On cron and cron management DSLs
The cron utility is the typical goto scheduling solution in unix/linux systems. Utilities like whenever allow us to take advantage of cron through an elegant and declarative pure ruby DSL.
The typical approach when using whenever for scheduling is to use deployment hooks to update the crontab from whenever's configuration during application deployment. However this approach falls short when the schedule is expected to be configurable at run time and especially if we want to give administrators fine grained controls over what is being scheduled and when.
Database driven scheduling
The post outlines an alternative solution using the library clockwork that makes it easy to make event scheduling run time configurable through database models managed using familiar ruby ORMs.
While clockwork handles the scheduling aspect, the responsibility of actual execution of the jobs is expected to be delegated to a background processor like sidekiq or delayed job.
ActiveJob and standardized APIs for background processing
Rather than coupling our code to a specific background processor which might in turn may be coupled with a specific transport system (like Sidekiq and Redis) it is advisable to rely instead (as much as possible) on ActiveJob API which provides a standardized API for background processing in Rails ecosystem.
Integration with Admin interfaces
The benefit of the event system being manageable through ActiveRecord is that integrated admin interfaces like ActiveAdmin and Administrate (which we might already have integrated in our existing applications) work out of the box and we get a complete schedule management interface with minimal extraneous boilerblate.
Getting started with our app
The rest of the tutorial walks through the creation of a Postgres backed Rails 5 application that illustrates the concepts outlined above.
We start off with familiar rails application generation steps:
$ gem install rails --pre --no-rdoc --no-ri
$ rails -v
Rails 5.0.0.beta3
Note that while skipping rdoc and ri is purely a matter of convenience (I did not need them at the time) - the --pre flag is required, as of this writing, for installing Rails 5 as it is still in beta.
$ rails new rails5-clockwork-demo --database=postgresql
create
create README.md
create Rakefile
create config.ru
create .gitignore
create Gemfile
create app
create app/assets/config/manifest.js
create app/assets/javascripts/application.js
create app/assets/javascripts/cable.coffee
create app/assets/stylesheets/application.css
create app/channels/application_cable/channel.rb
create app/channels/application_cable/connection.rb
create app/controllers/application_controller.rb
create app/helpers/application_helper.rb
create app/jobs/application_job.rb
create app/mailers/application_mailer.rb
create app/models/application_record.rb
create app/views/layouts/application.html.erb
create app/views/layouts/mailer.html.erb
create app/views/layouts/mailer.text.erb
create app/assets/images/.keep
create app/assets/javascripts/channels
create app/assets/javascripts/channels/.keep
create app/controllers/concerns/.keep
create app/models/concerns/.keep
create bin
create bin/bundle
create bin/rails
create bin/rake
create bin/setup
create bin/update
create config
create config/routes.rb
create config/application.rb
create config/environment.rb
create config/secrets.yml
create config/cable.yml
create config/puma.rb
create config/environments
create config/environments/development.rb
create config/environments/production.rb
create config/environments/test.rb
create config/initializers
create config/initializers/active_record_belongs_to_required_by_default.rb
create config/initializers/application_controller_renderer.rb
create config/initializers/assets.rb
create config/initializers/backtrace_silencers.rb
create config/initializers/callback_terminator.rb
create config/initializers/cookies_serializer.rb
create config/initializers/cors.rb
create config/initializers/filter_parameter_logging.rb
create config/initializers/inflections.rb
create config/initializers/mime_types.rb
create config/initializers/per_form_csrf_tokens.rb
create config/initializers/request_forgery_protection.rb
create config/initializers/session_store.rb
create config/initializers/wrap_parameters.rb
create config/locales
create config/locales/en.yml
create config/boot.rb
create config/database.yml
create db
create db/seeds.rb
create lib
create lib/tasks
create lib/tasks/.keep
create lib/assets
create lib/assets/.keep
create log
create log/.keep
create public
create public/404.html
create public/422.html
create public/500.html
create public/apple-touch-icon-precomposed.png
create public/apple-touch-icon.png
create public/favicon.ico
create public/robots.txt
create test/fixtures
create test/fixtures/.keep
create test/fixtures/files
create test/fixtures/files/.keep
create test/controllers
create test/controllers/.keep
create test/mailers
create test/mailers/.keep
create test/models
create test/models/.keep
create test/helpers
create test/helpers/.keep
create test/integration
create test/integration/.keep
create test/test_helper.rb
create tmp
create tmp/.keep
create tmp/cache
create tmp/cache/assets
create vendor/assets/javascripts
create vendor/assets/javascripts/.keep
create vendor/assets/stylesheets
create vendor/assets/stylesheets/.keep
remove config/initializers/cors.rb
Note the line create app/jobs/application_job.rb above. Rails 5 comes pre-integrated with ActiveJob.
Next we add clockwork and sidekiq to our Gemfile.
gem 'clockwork'
gem 'sidekiq'
It may be tempting to just leave the in-memory adapter of ActiveJob in place but it should not be used in any production application. Rails guides explain it well enough:
Rails itself only provides an in-process queuing system, which only keeps the jobs in RAM. If the process crashes or the machine is reset, then all outstanding jobs are lost with the default async back-end. This may be fine for smaller apps or non-critical jobs, but most production apps will need to pick a persistent backend.
Implementing the Event model
Next we need to define our models for persisting our schedules.
rails g model event name:string frequency:integer at:string job_name:string job_arguments:jsonb
Here the first three columns correspond to accessors mandated by Clockwork. Name is primary for descriptive logging purposes (more on this below). frequency specifies the recurrance frequency in seconds. at signifies point of occurance within the recurrance span. Following are the valid formats:
HH:MM
H:MM
**:MM
HH:**
(Mon|mon|Monday|monday) HH:MM
The last two columns job_name and job_arguments identify the job to be triggered. As would become obvious below, we did not need to provide the job name through the database - it could be inferred at the runtime through any custom logic expressed in ruby. But having it in database leads to a straightforward and transparent implementation and management.
It may be tempting to just reuse the name field as the job_name as well, but it may obscure debugging when same job is being invoked as part of multiple events for different use cases. It is recommended to keep the name as something representative of the use case - eg. enterprise_plan_customers_sales_aggregation_trigger.
Our model would look something like below:
class Event < ApplicationRecord
validates :name, :frequency, :job_name, presence: true
end
clock.rb file:
The entry point of clockwork is the file clock.rb. This is the file that tells clockwork to
poll the events table and execute the inferred job.
require 'clockwork'
require 'clockwork/database_events'
require_relative './config/boot'
require_relative './config/environment'
module Clockwork
# required to enable database syncing support
Clockwork.manager = DatabaseEvents::Manager.new
sync_database_events model: ::Event, every: 1.minute do |event|
event.job_name.constantize.perform_later(event.job_arguments)
end
end
Configuring ActiveJob to use sidekiq
We had added sidekiq as the persistence backend for ActiveJob but we have not configured ActiveJob to use it.
That is one additional line of code in config/application.rb
module Rails5ClockworkDemo
class Application < Rails::Application
# Configure ActiveJob to use sidekiq
config.active_job.queue_adapter = :sidekiq
end
end
Running clockwork:
Clockwork can be executed by running clockwork clock.rb at project root. However we can not expect something exciting yet because we simply have no entries in the table:
$ clockwork clock.rb
I, [2016-04-01T02:02:54.032058 #57534] INFO -- : Starting clock for 1 events: [ sync_database_events_for_model_Event ]
I, [2016-04-01T02:02:54.032150 #57534] INFO -- : Triggering 'sync_database_events_for_model_Event'
Adding administrative interface:
We would be using ActiveAdmin for our admin interface for managing events. As of this writing to use ActiveAdmin along with Rails 5 we need to use the master branch of ActiveAdmin.
While we will not go into elaboration of the ActiveAdmin DSL, most of the ideas should be applicable to alternative admin builders as well.
# Gemfile.rb
gem 'activeadmin', github: 'activeadmin'
$ rails g active_admin:install
For now we skip authorization as well as authentication entirely - integrating ActiveAdmin with authentication systems is covered here and usage in conjugation with authorization systems is covered here in the official docs.
$ rails g active_admin:install --skip-users
Running via Spring preloader in process 59785
create config/initializers/active_admin.rb
create app/admin
create app/admin/dashboard.rb
route ActiveAdmin.routes(self)
generate active_admin:assets
Running via Spring preloader in process 59787
create app/assets/javascripts/active_admin.js.coffee
create app/assets/stylesheets/active_admin.scss
create db/migrate/20160331204250_create_active_admin_comments.rb
To present our Event model through ActiveAdmin we need to generate an ActiveAdmin Event resource.
$ rails g active_admin:resource Event
The above command creates the following file for us:
# app/admin/event.rb
ActiveAdmin.register Event do
# See permitted parameters documentation:
# https://github.com/activeadmin/activeadmin/blob/master/docs/2-resource-customization.md#setting-up-strong-parameters
#
# permit_params :list, :of, :attributes, :on, :model
#
# or
#
# permit_params do
# permitted = [:permitted, :attributes]
# permitted << :other if params[:action] == 'create' && current_user.admin?
# permitted
# end
end
Once we configure permitted parameters as elaborated in the comments we have something like below:
ActiveAdmin.register Event do
permit_params :name, :job_name, :job_arguments, :frequency, :at
end
Now after we can run our migrations and booted up our server:
=> Booting Puma
=> Rails 5.0.0.beta3 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
Puma starting in single mode...
* Version 3.2.0 (ruby 2.3.0-p0), codename: Spring Is A Heliocentric Viewpoint
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://localhost:3000
If this is the first time you are using Rails 5 the new default home page as well as puma as the default server can be a pleasant surprise.
Our admin panel available at /admin provides us with the means to edit our events.
However when we attempt to create an event we would be faced with an error because formtastic does not know out of the box how to handle jsonb field which we used for job arguments.
While it is not difficult to create custom form inputs for formtastic and I have outlined an alternative approach before, to keep the example simple let us simply use a textarea for the arguments field:
form do |f|
f.inputs do
f.input :name
f.input :job_name
f.input :frequency
f.input :at
f.input :job_arguments, as: :text
end
f.actions
end
Now we can create an event to trigger a dummy job:
Defining our jobs:
Of course, for this job to be runnable, we need to define the job as well. We can use ActiveJob's generators for the same:
$ rails g job dummy
Running via Spring preloader in process 63269
invoke test_unit
create test/jobs/dummy_job_test.rb
create app/jobs/dummy_job.rb
Our job implementation itself is fairly mundane but serves the purpose of illustration:
class DummyJob < ApplicationJob
queue_as :default
def perform(*args)
puts "Dummy Job Executed"
end
end
Now once our clockwork process synchronizes with the databases, it will pickup the dummy job and keep executing every one second:
I, [2016-04-01T02:30:54.003679 #57534] INFO -- : Triggering 'sync_database_events_for_model_Event'
I, [2016-04-01T02:30:54.005132 #57534] INFO -- : Triggering 'execute_dummy_job'
I, [2016-04-01T02:30:55.005873 #57534] INFO -- : Triggering 'execute_dummy_job'
I, [2016-04-01T02:30:56.001670 #57534] INFO -- : Triggering 'execute_dummy_job'
I, [2016-04-01T02:30:57.001761 #57534] INFO -- : Triggering 'execute_dummy_job'
In the rails development log we can see that ActiveRecord is queuing this job:
[ActiveJob] Enqueued DummyJob (Job ID: 64813900-c70f-4b40-9dd7-452c4cac6b73) to Sidekiq(default)
[ActiveJob] Enqueued DummyJob (Job ID: 0d0a66d1-a46b-4427-92db-83a301b38c1c) to Sidekiq(default)
[ActiveJob] Enqueued DummyJob (Job ID: 1de45b36-7cc9-4a9a-9bb7-363591f64dd5) to Sidekiq(default)
[ActiveJob] Enqueued DummyJob (Job ID: 483fe8a2-5acf-4830-80fb-6c6883f0f7c2) to Sidekiq(default)
Now if we run sidekiq, our background processor - we should see the enqued job getting executed in the log/sidekiq.log:
2016-03-31T21:01:00.064Z 64128 TID-ox54l2v7s DummyJob JID-fc59a048d2d1fe2aeecc3452 INFO: start
Dummy Job Executed
This concludes our introductory post on database driven scheduling with clockwork. Please share any issues you might have faced or any suggestions for improvement in the comments.
The source code for this post is available in this github repo.