Stack Overflow
Stack Overflow

Questions tagged [shoryuken]

AWS SQS thread based message processor for Ruby

AWS SQS thread based message processor for Ruby. Available on GitHub.

20 questions
15
votes
2 answers

How to determine concurrency (threads) while using shoryuken for background jobs?

In my Ruby on Rails application, I'm using shoryouken for background processing. I've many sqs queues (6-7) in my application. One of the queue has 2000-3000 jobs and it takes around 3 hours for the worker to process these 2-3k jobs with a default…
TarunJadhwani
  • 1,151
  • 7
  • 21
5
votes
1 answer

Setting up Shoryuken, Active Job and Amazon SQS in a Rails project

As a relative newcomer to AWS, it has been a little bit of curve getting Shoryuken gem (with Active Job and Active Record) to work based on the setup documentation. Middleware: I wasn't sure if the middleware was a requisite part of setting up…
geoboy
  • 1,172
  • 1
  • 11
  • 25
2
votes
0 answers

Manually retry an sqs failed job with shoryuken

I'm currently building an admin dashboard to monitor our SQS queues, inspired by this medium post. I managed to have everything working fine with this code in my custom shoryuken middelware : class DashboardMiddleware def call(worker_instance,…
Vanessa
  • 209
  • 1
  • 11
2
votes
2 answers

Environment specific shoryuken configuration

I'm using the shoryuken gem by running bundle exec shoryuken -R -C config/shoryuken.yml and shoryuken.yml is concurrency: 25 delay: 0 queues: - [development_high, 30] - [development_mid, 6] - [development_low, 2] This works fine. However, I'd…
dimid
  • 7,285
  • 1
  • 46
  • 85
2
votes
2 answers

Can't run rails shoryuken gem in production environment that reads from SQS

The shoryuken gem is a background worker for rails applications that reads from aws SQS. I can run the shoryuken worker in my local and it's working fine. When I run it in production environment in AWS it does not work. How do you run shoryuken in…
tungsten_carbide
  • 525
  • 5
  • 18
2
votes
1 answer

Customizing retry delays with Amazon SQS from Shoryuken (ruby)

I am migrating a background job processing service made with sidekiq to shoryuken, which is based on Amazon SQS. With sidekiq you can customize the retries pattern by using sidekiq_retry_in: class WorkerWithCustomRetry include Sidekiq::Worker …
JPG
  • 545
  • 3
  • 19
1
vote
1 answer

SQS + Shoryuken: Large Receive Count in FIFO despite auto_delete=true

I have an AWS SQS FIFO queue configured to deduplicate messages based on content. My rails app uses Shoryuken worker to get messages from SQS. Here is the worker code: class MyJob include Shoryuken::Worker shoryuken_options queue:…
MikeMarsian
  • 608
  • 1
  • 7
  • 21
1
vote
1 answer

shoryuken error to `validate_queues': The specified queue(s)

I am using rails "Shoryuken" gem but I am getting errors for validation on queues on my development environment when I started rails server below is the error:- gems/shoryuken-2.0.11/lib/shoryuken/environment_loader.rb:172:in `validate_queues': The…
Ravindra Yadav
  • 619
  • 2
  • 6
  • 15
0
votes
1 answer

Elastic beanstalk not streaming custom log to cloudwatch [ amazon linux 2 ]

I am unable to send custom logs to Cloudwatch. I have created /opt/aws/amazon-cloudwatch-agent/etc/shoryuken.json which contains { "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path":…
pritam
  • 102
  • 1
  • 1
  • 4
0
votes
1 answer

Shoryuken taking very long to enqueue jobs

We are trying to switch to shoryuken as our ActiveJob backend. But while testing we found that there is a lot of latency while trying to enqueue a job. this was the code which was written to test the latency. class ApplicationController <…
codecopy
  • 1
0
votes
0 answers

Getting bundler: failed to load command: shoryuken (/usr/local/bundle/bin/shoryuken)

I'm getting below error in my QA logs bundler: failed to load command: shoryuken (/usr/local/bundle/bin/shoryuken) Below is my setup information. Ruby version: ruby 3.0.3p15 Rails version: Rails 7.0.3 Bundler version: Bundler version 2.3.6 Shoryuken…
VINAY VKK
  • 23
  • 6
0
votes
0 answers

Active::Job and Shoryuken do action on SIGTERM

In my rails app, I have a Shoryuken runner as a runner on a Kubernetes Stack. I have some long running job, and I want to do some action when the worker pod is stopped. As far as I know, Kubernetes send a SIGTERM to the worker. I wish to catch this…
GPif
  • 543
  • 2
  • 6
  • 22
0
votes
0 answers

Shoryuken appears attempts to reprocess inflight SQS message

I currently have a rails 6 app using Shoryuken (5.2.3) with SQS to asynchronously handle user initiated image processing and uploading. Everything appears to be working as expected as far as the side effects of the job (which includes the…
humbledev7000
  • 275
  • 1
  • 4
  • 8
0
votes
0 answers

Worker directory for Shoryuken and Sidekiq

I have a rails app where we want to use both sidekiq (redis queue) and shoryuken (sqs queue) for different use cases. So now , all the sidekiq workers are defined under app/workers. To separate out shoryuken workers, we thought to define and keep…
Vipul Kumar
  • 259
  • 1
  • 2
  • 12
0
votes
0 answers

How can Shoryuken Process read aws credentials from IAM profile

We have a shoryuken worker process setup in rails application (ruby 2.6.6, rails > 5). this is the shoryuken.yml.erb file :aws: :access_key_id: ? :secret_access_key: ? :region: <%= Settings.aws.region %> :concurrency: 10 :daemon:…
Vipul Kumar
  • 259
  • 1
  • 2
  • 12
1
2