spring boot redis operation throw broken pipe error

Question

We use redis in spring boot project. After running a period of time, the redis operation MAY throw a broken pipe error, but sometimes it will succeed. Restarting the service will resolve this problem, but it's not a good idea.

I can't tell the reason why it happens. It seems that some redis connections in the pool are unusable, but aren't closed and evicted from the pool.

my questions are:

possible reason causing the broken pipe error?
if there isn't redis operation for a long period, will the idle connection in the pool become unusable?
will the connection be closed and evicted from pool when broken pipe error happen?

pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

application.yml:

spring:
  redis:
    database: 0
    host: ${REDIS_HOST:127.0.0.1}
    password: ${REDIS_PASSWORD:password}
    port: ${REDIS_PORT:6379}
    timeout: ${REDIS_TIMEOUT:1000}
    pool:
      max-active: ${REDIS_MAX_ACTIVE:100}
      max-wait: ${REDIS_MAX_WAIT:500}
      max-idle: ${REDIS_MAX_IDLE:20}
      min-idle: ${REDIS_MIN_IDLE:5}

error message:

org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketException: Broken pipe (Write failed); nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe (Write failed)
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:212) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisConnection.hSet(JedisConnection.java:2810) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.DefaultHashOperations$9.doInRedis(DefaultHashOperations.java:173) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:204) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:166) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:88) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.DefaultHashOperations.put(DefaultHashOperations.java:170) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe (Write failed)
    at redis.clients.jedis.Connection.flush(Connection.java:291) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.jedis.Connection.getIntegerReply(Connection.java:220) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.jedis.BinaryJedis.hset(BinaryJedis.java:749) ~[jedis-2.8.2.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisConnection.hSet(JedisConnection.java:2808) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    ... 115 common frames omitted
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_111]
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_111]
    at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_111]
    at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.jedis.Connection.flush(Connection.java:288) ~[jedis-2.8.2.jar!/:na]
    ... 118 common frames omitted

score 3 · Answer 1 · answered Mar 31 '17 at 00:51

Answer my question:

why the broken pipe error happen?

TransactionSynchronizationManager will hold the RedisConnection in thread, and wont close it or return it to pool, see RedisTemplate.java and RedisConnectionUtils.java. After restarting the redis server, operation on the holded RedisConnection in thread will throw broken pipe error.

how to resolve it?

Adding try/catch for all redis operation, if the error happens , unbinding it from thread, and could get a new connection from pool and execute redis operation again.

private static final ExceptionTranslationStrategy EXCEPTION_TRANSLATION =
        new FallbackExceptionTranslationStrategy(JedisConverters.exceptionConverter());

public Object req(RedisRequest req) {
    try {
        return req.request();
    } catch (Exception ex) {
        if (ex instanceof NullPointerException) {
            throw ex;
        }
        DataAccessException exception = EXCEPTION_TRANSLATION.translate(ex);
        if (exception instanceof RedisConnectionFailureException) {
            RedisConnectionUtils.unbindConnection(factory);
            /** retry again */
            return req.request();
        } else {
            throw ex;
        }
    }
}

did setting TestOnBorrow resolve this issue for you ? – madcolonel10 Aug 21 '19 at 09:30 — madcolonel10, Aug 21 '19 at 09:30

score 2 · Accepted Answer · edited Mar 13 '21 at 02:03

This can happen for n number of reasons, one of them could be when you use a long-living connection (e. g. connect to Redis on application start and then use the connection over and over).

Some of the things to do are:

Reconnect if the connection is broken (needs some try/catch magic to prevent that errors are propagated to your application logic) or a better was is to use
TestOnBorrow - Sends a PING request when you ask for the resource.
TestOnReturn - Sends a PING when you return a resource to the pool.
TestWhileIdle - Sends periodic PINGS from idle resources in the pool.
Connect at the moment you need the connection and disconnect afterwards

Regarding
if there isn't redis operation for a long period, will the idle connection in the pool become unusable?

maxidle means at any given time the system allows 'maxIdle' many number of connections to be idle the rest will be constantly checked, closed and returned to the pool. I dont know a reason when an idle connection be unusable. Anyways this can be got rid of by using ways mentioned above.

there isn't TestOnBorrow config in spring @ConfigurationProperties RedisProperties, does that means it's need to create a customer JedisConnectionFactory and add TestOnBorrow in JedisPoolConfig ? — Geln Yang, Mar 29 '17 at 05:43
@geln yes, you have to create a custom JedisConnectionFactory and pass that in — StormeHawke, Nov 29 '18 at 14:40

spring boot redis operation throw broken pipe error

2 Answers2

why the broken pipe error happen?

how to resolve it?