ruby on rails - How to debug/fix random occurring Redis::TimeoutError? -

i have rails app running using redis quite lot - - i'm seeing quite few redis::timeouterror occurring here , there, time time. there no pattern in circumstances. occurs both in web app , in background jobs (which being processed using sidekiq) - not time time.

now have no idea how track down root cause of , hence no idea how fix it.

here little background on setup:

the redis instance running on separate physical server connected both web server , background server in private local 1gbit network. servers running ubuntu 12.04. redis version 2.6.10. i'm connecting rails app (which 3.2) using initializer so:

require 'redis' require 'redis/objects' redis = redis.new(:url => app_config['redis_url']) redis.current = redis 

this output of redis-cli info:

# server redis_version:2.6.10 redis_git_sha1:00000000 redis_git_dirty:0 redis_mode:standalone os:linux 3.2.0-38-generic x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.6.3 process_id:28475 run_id:d89bbb1b81d3169c4228cf23c0988ae437d496a1 tcp_port:6379 uptime_in_seconds:14913365 uptime_in_days:172 lru_clock:1507056  # clients connected_clients:233 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:19  # memory used_memory:801637360 used_memory_human:764.50m used_memory_rss:594706432 used_memory_peak:4295394784 used_memory_peak_human:4.00g used_memory_lua:31744 mem_fragmentation_ratio:0.74 mem_allocator:jemalloc-3.3.0  # persistence loading:0 rdb_changes_since_last_save:23166 rdb_bgsave_in_progress:0 rdb_last_save_time:1378219310 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:4 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok  # stats total_connections_received:932395 total_commands_processed:3088408103 instantaneous_ops_per_sec:837 rejected_connections:0 expired_keys:31428 evicted_keys:3007 keyspace_hits:124093049 keyspace_misses:53060192 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:17651  # replication role:master connected_slaves:1 slave0:192.168.0.2,6379,online  # cpu used_cpu_sys:54000.21 used_cpu_user:73692.52 used_cpu_sys_children:36229.79 used_cpu_user_children:420655.84  # keyspace db0:keys=1498962,expires=1310 

in redis config have following set:

\fidaemonize yes pidfile /var/run/redis/redis-server.pid timeout 0 loglevel notice databases 1 save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /var/lib/redis slave-serve-stale-data yes slave-read-only yes slave-priority 100 maxclients 1000 maxmemory 4gb maxmemory-policy volatile-lru appendonly no appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-entries 512 list-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60  

that come many issues :

• because use save command (it setup in conf) generating lot of i/o , hammering server, if use ebs volumes on amazon.
• because have redis slave (same before, doing save before mirroring).
• because use key * slow on lot of indexes.

i wrote article on these issues, see here.