Ramblings on Web Development and Software Architecture

Posted  5 years ago


Database driven scheduling with Clockwork and ActiveJob
This post has not been updated in quite some time and the content here may be out of date or not reflect my current my recommedation in the matter.

On cron and cron management DSLs

The cron utility is the typical goto scheduling solution in unix/linux systems. Utilities like whenever allow us to take advantage of cron through an elegant and declarative pure ruby DSL.

The typical approach when using whenever for scheduling is to use deployment hooks to update the crontab from whenever's configuration during application deployment. However this approach falls short when the schedule is expected to be configurable at run time and especially if we want to give administrators fine grained controls over what is being scheduled and when.

Database driven scheduling

The post outlines an alternative solution using the library clockwork that makes it easy to make event scheduling run time configurable through database models managed using familiar ruby ORMs.

While clockwork handles the scheduling aspect, the responsibility of actual execution of the jobs is expected to be delegated to a background processor like sidekiq or delayed job.

ActiveJob and standardized APIs for background processing

Rather than coupling our code to a specific background processor which might in turn may be coupled with a specific transport system (like Sidekiq and Redis) it is advisable to rely instead (as much as possible) on ActiveJob API which provides a standardized API for background processing in Rails ecosystem.

Integration with Admin interfaces

The benefit of the event system being manageable through ActiveRecord is that integrated admin interfaces like ActiveAdmin and Administrate (which we might already have integrated in our existing applications) work out of the box and we get a complete schedule management interface with minimal extraneous boilerblate.

Getting started with our app

The rest of the tutorial walks through the creation of a Postgres backed Rails 5 application that illustrates the concepts outlined above.

We start off with familiar rails application generation steps:

$ gem install rails --pre --no-rdoc --no-ri

$ rails -v
Rails 5.0.0.beta3

Note that while skipping rdoc and ri is purely a matter of convenience (I did not need them at the time) - the --pre flag is required, as of this writing, for installing Rails 5 as it is still in beta.

$ rails new rails5-clockwork-demo --database=postgresql

      create  README.md
      create  Rakefile
      create  config.ru
      create  .gitignore
      create  Gemfile
      create  app
      create  app/assets/config/manifest.js
      create  app/assets/javascripts/application.js
      create  app/assets/javascripts/cable.coffee
      create  app/assets/stylesheets/application.css
      create  app/channels/application_cable/channel.rb
      create  app/channels/application_cable/connection.rb
      create  app/controllers/application_controller.rb
      create  app/helpers/application_helper.rb
      create  app/jobs/application_job.rb
      create  app/mailers/application_mailer.rb
      create  app/models/application_record.rb
      create  app/views/layouts/application.html.erb
      create  app/views/layouts/mailer.html.erb
      create  app/views/layouts/mailer.text.erb
      create  app/assets/images/.keep
      create  app/assets/javascripts/channels
      create  app/assets/javascripts/channels/.keep
      create  app/controllers/concerns/.keep
      create  app/models/concerns/.keep
      create  bin
      create  bin/bundle
      create  bin/rails
      create  bin/rake
      create  bin/setup
      create  bin/update
      create  config
      create  config/routes.rb
      create  config/application.rb
      create  config/environment.rb
      create  config/secrets.yml
      create  config/cable.yml
      create  config/puma.rb
      create  config/environments
      create  config/environments/development.rb
      create  config/environments/production.rb
      create  config/environments/test.rb
      create  config/initializers
      create  config/initializers/active_record_belongs_to_required_by_default.rb
      create  config/initializers/application_controller_renderer.rb
      create  config/initializers/assets.rb
      create  config/initializers/backtrace_silencers.rb
      create  config/initializers/callback_terminator.rb
      create  config/initializers/cookies_serializer.rb
      create  config/initializers/cors.rb
      create  config/initializers/filter_parameter_logging.rb
      create  config/initializers/inflections.rb
      create  config/initializers/mime_types.rb
      create  config/initializers/per_form_csrf_tokens.rb
      create  config/initializers/request_forgery_protection.rb
      create  config/initializers/session_store.rb
      create  config/initializers/wrap_parameters.rb
      create  config/locales
      create  config/locales/en.yml
      create  config/boot.rb
      create  config/database.yml
      create  db
      create  db/seeds.rb
      create  lib
      create  lib/tasks
      create  lib/tasks/.keep
      create  lib/assets
      create  lib/assets/.keep
      create  log
      create  log/.keep
      create  public
      create  public/404.html
      create  public/422.html
      create  public/500.html
      create  public/apple-touch-icon-precomposed.png
      create  public/apple-touch-icon.png
      create  public/favicon.ico
      create  public/robots.txt
      create  test/fixtures
      create  test/fixtures/.keep
      create  test/fixtures/files
      create  test/fixtures/files/.keep
      create  test/controllers
      create  test/controllers/.keep
      create  test/mailers
      create  test/mailers/.keep
      create  test/models
      create  test/models/.keep
      create  test/helpers
      create  test/helpers/.keep
      create  test/integration
      create  test/integration/.keep
      create  test/test_helper.rb
      create  tmp
      create  tmp/.keep
      create  tmp/cache
      create  tmp/cache/assets
      create  vendor/assets/javascripts
      create  vendor/assets/javascripts/.keep
      create  vendor/assets/stylesheets
      create  vendor/assets/stylesheets/.keep
      remove  config/initializers/cors.rb

Note the line create app/jobs/application_job.rb above. Rails 5 comes pre-integrated with ActiveJob.

Next we add clockwork and sidekiq to our Gemfile.

gem 'clockwork'
gem 'sidekiq'

It may be tempting to just leave the in-memory adapter of ActiveJob in place but it should not be used in any production application. Rails guides explain it well enough:

Rails itself only provides an in-process queuing system, which only keeps the jobs in RAM. If the process crashes or the machine is reset, then all outstanding jobs are lost with the default async back-end. This may be fine for smaller apps or non-critical jobs, but most production apps will need to pick a persistent backend.

Implementing the Event model

Next we need to define our models for persisting our schedules.

rails g model event name:string frequency:integer at:string job_name:string job_arguments:jsonb

Here the first three columns correspond to accessors mandated by Clockwork. Name is primary for descriptive logging purposes (more on this below). frequency specifies the recurrance frequency in seconds. at signifies point of occurance within the recurrance span. Following are the valid formats:

(Mon|mon|Monday|monday) HH:MM

The last two columns job_name and job_arguments identify the job to be triggered. As would become obvious below, we did not need to provide the job name through the database - it could be inferred at the runtime through any custom logic expressed in ruby. But having it in database leads to a straightforward and transparent implementation and management.

It may be tempting to just reuse the name field as the job_name as well, but it may obscure debugging when same job is being invoked as part of multiple events for different use cases. It is recommended to keep the name as something representative of the use case - eg. enterprise_plan_customers_sales_aggregation_trigger.

Our model would look something like below:

class Event < ApplicationRecord
  validates :name, :frequency, :job_name, presence: true

clock.rb file:

The entry point of clockwork is the file clock.rb. This is the file that tells clockwork to poll the events table and execute the inferred job.

require 'clockwork'
require 'clockwork/database_events'
require_relative './config/boot'
require_relative './config/environment'

module Clockwork

  # required to enable database syncing support
  Clockwork.manager = DatabaseEvents::Manager.new

  sync_database_events model: ::Event, every: 1.minute do |event|


Configuring ActiveJob to use sidekiq

We had added sidekiq as the persistence backend for ActiveJob but we have not configured ActiveJob to use it.

That is one additional line of code in config/application.rb

module Rails5ClockworkDemo
  class Application < Rails::Application

    # Configure ActiveJob to use sidekiq
    config.active_job.queue_adapter = :sidekiq


Running clockwork:

Clockwork can be executed by running clockwork clock.rb at project root. However we can not expect something exciting yet because we simply have no entries in the table:

$ clockwork clock.rb
I, [2016-04-01T02:02:54.032058 #57534]  INFO -- : Starting clock for 1 events: [ sync_database_events_for_model_Event ]
I, [2016-04-01T02:02:54.032150 #57534]  INFO -- : Triggering 'sync_database_events_for_model_Event'

Adding administrative interface:

We would be using ActiveAdmin for our admin interface for managing events. As of this writing to use ActiveAdmin along with Rails 5 we need to use the master branch of ActiveAdmin.

While we will not go into elaboration of the ActiveAdmin DSL, most of the ideas should be applicable to alternative admin builders as well.

# Gemfile.rb

gem 'activeadmin', github: 'activeadmin'
$ rails g active_admin:install

For now we skip authorization as well as authentication entirely - integrating ActiveAdmin with authentication systems is covered here and usage in conjugation with authorization systems is covered here in the official docs.

$ rails g active_admin:install --skip-users
Running via Spring preloader in process 59785
      create  config/initializers/active_admin.rb
      create  app/admin
      create  app/admin/dashboard.rb
       route  ActiveAdmin.routes(self)
    generate  active_admin:assets
Running via Spring preloader in process 59787
      create  app/assets/javascripts/active_admin.js.coffee
      create  app/assets/stylesheets/active_admin.scss
      create  db/migrate/20160331204250_create_active_admin_comments.rb

To present our Event model through ActiveAdmin we need to generate an ActiveAdmin Event resource.

$ rails g active_admin:resource Event

The above command creates the following file for us:

# app/admin/event.rb

ActiveAdmin.register Event do

# See permitted parameters documentation:
# https://github.com/activeadmin/activeadmin/blob/master/docs/2-resource-customization.md#setting-up-strong-parameters
# permit_params :list, :of, :attributes, :on, :model
# or
# permit_params do
#   permitted = [:permitted, :attributes]
#   permitted << :other if params[:action] == 'create' && current_user.admin?
#   permitted
# end


Once we configure permitted parameters as elaborated in the comments we have something like below:

ActiveAdmin.register Event do

  permit_params :name, :job_name, :job_arguments, :frequency, :at


Now after we can run our migrations and booted up our server:

=> Booting Puma
=> Rails 5.0.0.beta3 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
Puma starting in single mode...
* Version 3.2.0 (ruby 2.3.0-p0), codename: Spring Is A Heliocentric Viewpoint
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://localhost:3000

If this is the first time you are using Rails 5 the new default home page as well as puma as the default server can be a pleasant surprise.

Our admin panel available at /admin provides us with the means to edit our events.

However when we attempt to create an event we would be faced with an error because formtastic does not know out of the box how to handle jsonb field which we used for job arguments.

While it is not difficult to create custom form inputs for formtastic and I have outlined an alternative approach before, to keep the example simple let us simply use a textarea for the arguments field:

  form do |f|
    f.inputs do
      f.input :name
      f.input :job_name
      f.input :frequency
      f.input :at
      f.input :job_arguments, as: :text

Now we can create an event to trigger a dummy job:

Defining our jobs:

Of course, for this job to be runnable, we need to define the job as well. We can use ActiveJob's generators for the same:

$ rails g job dummy
Running via Spring preloader in process 63269
      invoke  test_unit
      create    test/jobs/dummy_job_test.rb
      create  app/jobs/dummy_job.rb

Our job implementation itself is fairly mundane but serves the purpose of illustration:

class DummyJob < ApplicationJob
  queue_as :default

  def perform(*args)
    puts "Dummy Job Executed"

Now once our clockwork process synchronizes with the databases, it will pickup the dummy job and keep executing every one second:

I, [2016-04-01T02:30:54.003679 #57534]  INFO -- : Triggering 'sync_database_events_for_model_Event'
I, [2016-04-01T02:30:54.005132 #57534]  INFO -- : Triggering 'execute_dummy_job'
I, [2016-04-01T02:30:55.005873 #57534]  INFO -- : Triggering 'execute_dummy_job'
I, [2016-04-01T02:30:56.001670 #57534]  INFO -- : Triggering 'execute_dummy_job'
I, [2016-04-01T02:30:57.001761 #57534]  INFO -- : Triggering 'execute_dummy_job'

In the rails development log we can see that ActiveRecord is queuing this job:

[ActiveJob] Enqueued DummyJob (Job ID: 64813900-c70f-4b40-9dd7-452c4cac6b73) to Sidekiq(default)
[ActiveJob] Enqueued DummyJob (Job ID: 0d0a66d1-a46b-4427-92db-83a301b38c1c) to Sidekiq(default)
[ActiveJob] Enqueued DummyJob (Job ID: 1de45b36-7cc9-4a9a-9bb7-363591f64dd5) to Sidekiq(default)
[ActiveJob] Enqueued DummyJob (Job ID: 483fe8a2-5acf-4830-80fb-6c6883f0f7c2) to Sidekiq(default)

Now if we run sidekiq, our background processor - we should see the enqued job getting executed in the log/sidekiq.log:

2016-03-31T21:01:00.064Z 64128 TID-ox54l2v7s DummyJob JID-fc59a048d2d1fe2aeecc3452 INFO: start
Dummy Job Executed

This concludes our introductory post on database driven scheduling with clockwork. Please share any issues you might have faced or any suggestions for improvement in the comments.

The source code for this post is available in this github repo.