Sidekiq Background Jobs for Rails on DomainIndia VPS

# Sidekiq Background Jobs for Rails on DomainIndia VPS ## Why Sidekiq (not Solid Queue, DelayedJob, Resque)? Rails 8 ships with **Solid Queue** (DB-backed queue), which is great for simple workloads. But for anything serious, **Sidekiq** still wins: Sidekiq's multi-thread model + Redis backend make it 10× faster per process than DB-backed queues. It's the standard for production Rails. ## Architecture on DomainIndia VPS ``` [Rails web (Puma)] ──► enqueues job ──► [Redis] │ [Sidekiq worker 1] ◄──── picks jobs ─────┤ [Sidekiq worker 2] ◄──── picks jobs ─────┘ ``` - Rails web process (Puma) accepts HTTP, enqueues jobs - Redis holds the queue - Sidekiq worker processes pull jobs, run them - All three run as systemd services on the same VPS (or different VPS for scale) ## Step 1 — Install Redis On the VPS: ```bash # AlmaLinux sudo dnf install -y redis sudo systemctl enable --now redis # Ubuntu sudo apt install -y redis-server sudo systemctl enable --now redis-server redis-cli ping # → PONG ``` Enable persistence so queued jobs survive server restart: ```bash sudo sed -i 's/^appendonly no/appendonly yes/' /etc/redis/redis.conf sudo systemctl restart redis ``` ## Step 2 — Add Sidekiq to your Rails app `Gemfile`: ```ruby gem 'sidekiq', '~> 7.2' ``` ```bash bundle install ``` `config/initializers/sidekiq.rb`: ```ruby Sidekiq.configure_server do |config| config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') } end Sidekiq.configure_client do |config| config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') } end ``` `config/application.rb`: ```ruby config.active_job.queue_adapter = :sidekiq ``` ## Step 3 — Write a job `app/jobs/send_welcome_email_job.rb`: ```ruby class SendWelcomeEmailJob < ApplicationJob queue_as :default retry_on Net::SMTPError, wait: :exponentially_longer, attempts: 5 def perform(user_id) user = User.find(user_id) UserMailer.welcome(user).deliver_now end end ``` Enqueue: ```ruby SendWelcomeEmailJob.perform_later(user.id) ``` ## Step 4 — Systemd service for Sidekiq `/etc/systemd/system/sidekiq-yourapp.service`: ```ini [Unit] Description=Sidekiq for YourApp After=network.target redis.service postgresql.service [Service] Type=simple User=deploy Group=deploy WorkingDirectory=/home/deploy/rails_app Environment="RAILS_ENV=production" Environment="PATH=/home/deploy/.rbenv/shims:/usr/bin:/bin" ExecStart=/home/deploy/.rbenv/shims/bundle exec sidekiq -e production -C /home/deploy/rails_app/config/sidekiq.yml ExecReload=/bin/kill -TSTP $MAINPID Restart=on-failure RestartSec=5 # Graceful shutdown — let Sidekiq finish current jobs TimeoutStopSec=30 KillSignal=SIGTERM EnvironmentFile=-/home/deploy/rails_app/.env StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target ``` `config/sidekiq.yml`: ```yaml :concurrency: 10 # threads per worker process :queues: - [critical, 3] # weight 3 — critical queue serviced 3× more - [default, 2] - [mailers, 1] - [low, 1] :timeout: 25 # jobs killed after 25s if still running when SIGTERM ``` Enable: ```bash sudo systemctl daemon-reload sudo systemctl enable --now sidekiq-yourapp sudo journalctl -u sidekiq-yourapp -f ``` ## Step 5 — Sidekiq web UI Mount the dashboard for monitoring: `config/routes.rb`: ```ruby require 'sidekiq/web' Rails.application.routes.draw do authenticate :user, ->(u) { u.admin? } do mount Sidekiq::Web => '/sidekiq' end # ... other routes end ``` Visit `https://yourcompany.com/sidekiq` — see queued, retried, scheduled, dead jobs. ## Job patterns ### Scheduled / delayed jobs ```ruby ReminderJob.set(wait: 1.hour).perform_later(booking_id) ReminderJob.set(wait_until: 2.days.from_now).perform_later(booking_id) ``` ### Recurring jobs (Sidekiq-cron or whenever gem) ```ruby # Gemfile gem 'sidekiq-cron' # config/schedule.yml cleanup_old_sessions: cron: "0 2 * * *" # 2 AM daily class: "CleanupOldSessionsJob" reindex_search: cron: "0 */6 * * *" # every 6 hours class: "ReindexSearchJob" ``` ### Unique jobs (prevent duplicates) ```ruby # Gemfile gem 'sidekiq-unique-jobs' class ImportantJob < ApplicationJob sidekiq_options lock: :until_executed def perform(id) # Only one concurrent execution per id end end ``` ## Scaling patterns **Single VPS, multiple workers:** Add more worker processes on the same VPS — 2–4 is typical before RAM becomes the limit. ``` /etc/systemd/system/[email protected] (templated service) ``` Start `[email protected]`, `@2.service` etc. **Multi-VPS (horizontal scale):** Put Redis on a dedicated VPS. Point all web + worker VPS at it: ``` REDIS_URL=redis://redis.internal.yourcompany.com:6379/0 ``` For HA Redis: Sentinel (free) or Redis Cloud (managed). ## Monitoring and alerts **Sidekiq stats endpoint:** ```ruby Sidekiq::Stats.new.enqueued # waiting to run Sidekiq::Stats.new.retry_size # failed, waiting retry Sidekiq::DeadSet.new.size # permanently failed ``` Expose as `/health/sidekiq` and alert when queue >1000 or dead >10. **Sidekiq Pro/Enterprise** (commercial): - Batches (run many jobs, callback when all done) - Rate limiting - Quiet time, reliable fetch, encryption $179/mo. Worth it if Sidekiq is mission-critical. ## Common pitfalls ## FAQ