In a project I tried to implemented circuit breaker pattern on Redis service. I found a project at shopify that has an implementation
https://github.com/Shopify/semian https://github.com/Shopify/semian/blob/master/lib/semian/redis.rb
I followed its instructions and tried to deploy on my production environment and I got lot of Connection Timeout error at Sidekiq initialization.
It looks like the redis/adapter used by semian are conflicting with the redis calls used at sidekiq.
Redis::TimeoutError: [redis_x.y.z.k:port/x] Connection timed out
> -74 non-project frames
1
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb" line 56 in block in _read_from_socket
2
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb" line 52 in loop
3
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb" line 52 in _read_from_socket
4
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb" line 45 in gets
5
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb" line 378 in read
6
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 280 in block in read
7
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 268 in io
8
File "/usr/local/bundle/gems/semian-0.16.0/lib/semian/redis.rb" line 95 in block in io
9
File "/usr/local/bundle/gems/semian-0.16.0/lib/semian/adapter.rb" line 74 in mark_resource_as_acquired
10
File "/usr/local/bundle/gems/semian-0.16.0/lib/semian/adapter.rb" line 38 in block in acquire_semian_resource
11
File "/usr/local/bundle/gems/semian-0.16.0/lib/semian/unprotected_resource.rb" line 27 in acquire
12
File "/usr/local/bundle/gems/semian-0.16.0/lib/semian/adapter.rb" line 37 in acquire_semian_resource
13
File "/usr/local/bundle/gems/semian-0.16.0/lib/semian/redis.rb" line 94 in io
14
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 279 in read
15
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 131 in block in call
16
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 248 in block (2 levels) in process
17
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 389 in ensure_connected
18
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 238 in block in process
19
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 325 in logging
20
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 237 in process
21
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 131 in call
22
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/contrib/redis/instrumentation.rb" line 30 in block in call
23
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/trace_operation.rb" line 172 in block in measure
24
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/span_operation.rb" line 153 in measure
25
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/trace_operation.rb" line 172 in measure
26
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/tracer.rb" line 376 in start_span
27
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/tracer.rb" line 159 in block in trace
28
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/context.rb" line 45 in activate!
29
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/tracer.rb" line 158 in trace
30
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing.rb" line 18 in trace
31
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/contrib/redis/instrumentation.rb" line 24 in call
32
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 226 in block in call_with_timeout
33
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 300 in with_socket_timeout
34
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 225 in call_with_timeout
35
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 1284 in block in brpoplpush
36
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 69 in block in synchronize
37
File "/usr/local/lib/ruby/2.7.0/monitor.rb" line 202 in synchronize
38
File "/usr/local/lib/ruby/2.7.0/monitor.rb" line 202 in mon_synchronize
39
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 69 in synchronize
40
File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 1281 in brpoplpush
41
File "/usr/local/bundle/gems/sidekiq-pro-5.2.1/lib/sidekiq/pro/super_fetch.rb" line 337 in block in strict
42
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq.rb" line 98 in block in redis
43
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 63 in block (2 levels) in with
44
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 62 in handle_interrupt
45
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 62 in block in with
46
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 59 in handle_interrupt
47
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 59 in with
48
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq.rb" line 95 in redis
49
File "/usr/local/bundle/gems/sidekiq-pro-5.2.1/lib/sidekiq/pro/super_fetch.rb" line 323 in strict
50
File "/usr/local/bundle/gems/sidekiq-pro-5.2.1/lib/sidekiq/pro/super_fetch.rb" line 300 in call
51
File "/usr/local/bundle/gems/sidekiq-pro-5.2.1/lib/sidekiq/pro/super_fetch.rb" line 300 in get_job
52
File "/usr/local/bundle/gems/sidekiq-pro-5.2.1/lib/sidekiq/pro/super_fetch.rb" line 288 in block in retrieve_work
53
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 63 in block (2 levels) in with
54
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 62 in handle_interrupt
55
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 62 in block in with
56
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 59 in handle_interrupt
57
File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 59 in with
58
File "/usr/local/bundle/gems/sidekiq-pro-5.2.1/lib/sidekiq/pro/super_fetch.rb" line 287 in retrieve_work
59
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq/processor.rb" line 83 in get_one
60
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq/processor.rb" line 95 in fetch
61
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/contrib/sidekiq/server_internal_tracer/job_fetch.rb" line 26 in block in fetch
62
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/trace_operation.rb" line 172 in block in measure
63
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/span_operation.rb" line 153 in measure
64
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/trace_operation.rb" line 172 in measure
65
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/tracer.rb" line 376 in start_span
66
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/tracer.rb" line 159 in block in trace
67
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/context.rb" line 45 in activate!
68
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/tracer.rb" line 158 in trace
69
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing.rb" line 18 in trace
70
File "/usr/local/bundle/gems/ddtrace-1.0.0/lib/datadog/tracing/contrib/sidekiq/server_internal_tracer/job_fetch.rb" line 15 in fetch
71
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq/processor.rb" line 77 in process_one
72
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq/processor.rb" line 68 in run
73
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq/util.rb" line 15 in watchdog
74
File "/usr/local/bundle/gems/sidekiq-6.1.3/lib/sidekiq/util.rb" line 24 in block in safe_thread
Do anyone tried it before using Semian + Redis/Adapter and Sidekiq?
Any clues ? Thank you in advance
