Skip to content

DelegationError in BlockedExecution release path when a job's class_name no longer resolves #784

Description

@julik

Summary

solid_queue (verified in 1.4.0 and still present in 1.6.0) raises ActiveSupport::DelegationError: concurrency_limit delegated to job_class, but job_class is nil from the dispatcher's concurrency-maintenance tick whenever a SolidQueue::BlockedExecution row exists for a SolidQueue::Job whose class_name no longer safe_constantizes (e.g. the ActiveJob class was renamed, removed, or namespaced differently between deploys).

There is already a guard for this condition (Job#concurrency_limited? returns false when job_class.blank?), but the release path called by the dispatcher does not consult it — it goes straight to Semaphore.wait(job), which then dereferences the missing delegate target.

Result: the dispatcher's unblock_blocked_executions timer fires every tick, hits the same orphan rows, and reports the exception on every attempt indefinitely. The blocked rows are never released or cleaned up on their own.

Environment

  • solid_queue: reproduced on 1.4.0, still present in 1.6.0 (source verified — the relevant files are byte-identical between 1.4.0 and 1.6.0).
  • Ruby: 3.4.2
  • Rails / ActiveRecord: 8.1.3.1
  • Backend: PostgreSQL (not adapter-specific — the bug is model-layer).

Reproduction

  1. Enqueue an ActiveJob that uses limits_concurrency to: N, key: ... and that gets blocked (its concurrency semaphore is at cap), producing at least one solid_queue_blocked_executions row.
  2. In the next deploy, delete or rename the job class so that class_name.safe_constantize returns nil for the row created in step 1.
  3. Wait for SolidQueue::Dispatcher::ConcurrencyMaintenance#unblock_blocked_executions to fire (it runs on a timer).
  4. Every tick raises ActiveSupport::DelegationError and, because the release transaction rolls back, the same orphan row is picked up again on the next tick — the error will recur forever.

Alternative repro without a real code change: manually update a solid_queue_jobs.class_name value to a string that does not resolve, e.g.

SolidQueue::Job.find(<blocked job id>).update_columns(class_name: "GoneJob")

Then wait for the dispatcher tick.

Stack trace (from a real production hit)

ActiveSupport::DelegationError: concurrency_limit delegated to job_class, but job_class is nil
NoMethodError: undefined method 'concurrency_limit' for nil

solid_queue (1.4.0) app/models/solid_queue/job/concurrency_controls.rb:11:in 'SolidQueue::Job#concurrency_limit'
solid_queue (1.4.0) app/models/solid_queue/semaphore.rb:86:in 'SolidQueue::Semaphore::Proxy#limit'
solid_queue (1.4.0) app/models/solid_queue/semaphore.rb:58:in 'SolidQueue::Semaphore::Proxy#attempt_creation'
solid_queue (1.4.0) app/models/solid_queue/semaphore.rb:46:in 'SolidQueue::Semaphore::Proxy#wait'
solid_queue (1.4.0) app/models/solid_queue/semaphore.rb:10:in 'SolidQueue::Semaphore.wait'
solid_queue (1.4.0) app/models/solid_queue/blocked_execution.rb:65:in 'SolidQueue::BlockedExecution#acquire_concurrency_lock'
solid_queue (1.4.0) app/models/solid_queue/blocked_execution.rb:49:in 'block (2 levels) in SolidQueue::BlockedExecution#release'
...
solid_queue (1.4.0) app/models/solid_queue/blocked_execution.rb:28:in 'SolidQueue::BlockedExecution.release_one'
solid_queue (1.4.0) app/models/solid_queue/blocked_execution.rb:24:in 'SolidQueue::BlockedExecution.release_many'
solid_queue (1.4.0) app/models/solid_queue/blocked_execution.rb:14:in 'SolidQueue::BlockedExecution.unblock'
solid_queue (1.4.0) lib/solid_queue/dispatcher/concurrency_maintenance.rb:40:in 'block in SolidQueue::Dispatcher::ConcurrencyMaintenance#unblock_blocked_executions'
solid_queue (1.4.0) lib/solid_queue/app_executor.rb:7:in 'SolidQueue::AppExecutor#wrap_in_app_executor'
solid_queue (1.4.0) lib/solid_queue/dispatcher/concurrency_maintenance.rb:39:in 'SolidQueue::Dispatcher::ConcurrencyMaintenance#unblock_blocked_executions'
solid_queue (1.4.0) lib/solid_queue/dispatcher/concurrency_maintenance.rb:17:in 'block in SolidQueue::Dispatcher::ConcurrencyMaintenance#start'

Root cause

SolidQueue::Job::ConcurrencyControls sets up a delegate:

# app/models/solid_queue/job/concurrency_controls.rb
delegate :concurrency_limit, :concurrency_duration, to: :job_class

def job_class
  @job_class ||= class_name.safe_constantize
end

def concurrency_limited?
  concurrency_key.present? && job_class.present?
end

Job#acquire_concurrency_lock correctly gates on concurrency_limited?:

def acquire_concurrency_lock
  return true unless concurrency_limited?
  Semaphore.wait(self)
end

But BlockedExecution#acquire_concurrency_lock (a different method on a different class, invoked from BlockedExecution#release in the dispatcher-driven release path) does not:

# app/models/solid_queue/blocked_execution.rb
def acquire_concurrency_lock
  Semaphore.wait(job)
end

Semaphore.wait calls Semaphore::Proxy#limit, which calls job.concurrency_limit, which is the delegate — and blows up when job_class is nil.

Because this happens inside BlockedExecution#release's transaction, the row is never destroyed and never promoted; the dispatcher rediscovers it on every subsequent tick.

set_expires_at in the same file (self.expires_at = job.concurrency_duration.from_now, called on before_create) has the same latent problem, but only bites at create time — new orphans wouldn't be created here in practice, since the class exists when the block is initially recorded.

Impact

  • Every SolidQueue::Dispatcher::ConcurrencyMaintenance tick raises and reports an exception for as long as the orphaned rows exist — days, weeks, indefinitely.
  • Ops noise: floods exception dashboards from a source that is entirely internal state, unrelated to any user request.
  • Real blocked jobs that share a concurrency_key with an orphan may also be starved, because release_one locks one row per key inside a transaction that then rolls back.
  • Any deploy that removes or renames a concurrency-controlled ActiveJob class is a landmine.

Suggested fix

Two small guards, symmetric with the existing Job#acquire_concurrency_lock gate:

  1. BlockedExecution#release — short-circuit and destroy the row when the job class no longer resolves, so the dispatcher stops retrying forever.
  2. BlockedExecution#set_expires_at — fall back to SolidQueue.default_concurrency_control_period when the class does not resolve.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions