0

Seeing RST_STREAM with CANCEL being sent from client to server for some streams. Have added two stream data where in one case seeing RST_STREAM and in another not.

HTTP2 Stream where RST_STREAM is seen

No.     Time                          Source                Destination           Protocol Length Info
     87 2020-05-13 16:38:33.051377    192.168.1.1           192.168.2.2           HTTP2    131    HEADERS[25]: PUT /resource/2

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 25, Length 56, PUT /resource/2
        Length: 56
        Type: HEADERS (1)
        Flags: 0x04
            .... ...0 = End Stream: False
            .... .1.. = End Headers: True
            .... 0... = Padded: False
            ..0. .... = Priority: False
            00.0 ..0. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0001 1001 = Stream Identifier: 25
        [Pad Length: 0]
        Header Block Fragment: cecd45af62ab64a56b4a4a58ee160d4a0cb3210321004406…
        [Header Length: 279]
        [Header Count: 8]
        Header: :authority: 192.168.2.2:8000
            Name Length: 10
            Name: :authority
            Value Length: 19
            Value: 192.168.2.2:8000
            :authority: 192.168.2.2:8000
            [Unescaped: 192.168.2.2:8000]
            Representation: Indexed Header Field
            Index: 78
        Header: :method: PUT
            Name Length: 7
            Name: :method
            Value Length: 3
            Value: PUT
            :method: PUT
            [Unescaped: PUT]
            Representation: Indexed Header Field
            Index: 77
        Header: :path: /resource/2
            Name Length: 5
            Name: :path
            Value Length: 68
            Value: /resource/2
            :path: /resource/2
            [Unescaped: /resource/2]
            Representation: Literal Header Field with Incremental Indexing - Indexed Name
            Index: 5
        Header: :scheme: http
            Name Length: 7
            Name: :scheme
            Value Length: 4
            Value: http
            :scheme: http
            [Unescaped: http]
            Representation: Indexed Header Field
            Index: 6
        Header: content-type: application/json
            Name Length: 12
            Name: content-type
            Value Length: 16
            Value: application/json
            content-type: application/json
            [Unescaped: application/json]
            Representation: Indexed Header Field
            Index: 76
        Header: content-length: 259
            Name Length: 14
            Name: content-length
            Value Length: 3
            Value: 259
            content-length: 259
            [Unescaped: 259]
            Representation: Indexed Header Field
            Index: 75
        Header: accept-encoding: gzip
            Name Length: 15
            Name: accept-encoding
            Value Length: 4
            Value: gzip
            accept-encoding: gzip
            [Unescaped: gzip]
            Representation: Indexed Header Field
            Index: 74
        Header: user-agent: Go-http-client/2.0
            Name Length: 10
            Name: user-agent
            Value Length: 18
            Value: Go-http-client/2.0
            user-agent: Go-http-client/2.0
            [Unescaped: Go-http-client/2.0]
            Representation: Indexed Header Field
            Index: 73

No.     Time                          Source                Destination           Protocol Length Info
     89 2020-05-13 16:38:33.051399    192.168.1.1           192.168.2.2           HTTP2    334    DATA[25] (application/json)

HyperText Transfer Protocol 2
    Stream: DATA, Stream ID: 25, Length 259
        Length: 259
        Type: DATA (0)
        Flags: 0x01
            .... ...1 = End Stream: True
            .... 0... = Padded: False
            0000 .00. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0001 1001 = Stream Identifier: 25
        [Pad Length: 0]
        Data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx…
    JavaScript Object Notation: application/json
        Object

No.     Time                          Source                Destination           Protocol Length Info
    299 2020-05-13 16:38:33.192347    192.168.2.2           192.168.1.1           HTTP2    742    DATA[15] (application/json), HEADERS[47]: 200 OK, HEADERS[25]: 201 Created, DATA[25] (application/json)

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 25, Length 107, 201 Created
        Length: 107
        Type: HEADERS (1)
        Flags: 0x04
            .... ...0 = End Stream: False
            .... .1.. = End Headers: True
            .... 0... = Padded: False
            ..0. .... = Priority: False
            00.0 ..0. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0001 1001 = Stream Identifier: 25
        [Pad Length: 0]
        Header Block Fragment: ce0f0d03323539d40f1fd99d29aee30c040b85d797085e5c…
        [Header Length: 396]
        [Header Count: 8]
        Header: :status: 201 Created
            Name Length: 7
            Name: :status
            Value Length: 3
            Value: 201
            :status: 201
            [Unescaped: 201]
            Representation: Indexed Header Field
            Index: 78
        Header: content-length: 259
            Name Length: 14
            Name: content-length
            Value Length: 3
            Value: 259
            content-length: 259
            [Unescaped: 259]
            Representation: Literal Header Field without Indexing - Indexed Name
            Index: 28
        Header: content-type: application/json
            Name Length: 12
            Name: content-type
            Value Length: 16
            Value: application/json
            content-type: application/json
            [Unescaped: application/json]
            Representation: Indexed Header Field
            Index: 84
        Header: location: http://192.168.2.2:8000/192.168.3.3:8080~odxdp~api~/resource/2
            Name Length: 8
            Name: location
            Value Length: 124
            Value: http://192.168.2.2:8000/192.168.3.3:8080~odxdp~api~/resource/2
            location: http://192.168.2.2:8000/192.168.3.3:8080~odxdp~api~/resource/2
            [Unescaped: http://192.168.2.2:8000/192.168.3.3:8080~odxdp~api~/resource/2]
            Representation: Literal Header Field without Indexing - Indexed Name
            Index: 46
        Header: x-envoy-upstream-service-time: 82
            Name Length: 29
            Name: x-envoy-upstream-service-time
            Value Length: 2
            Value: 82
            [Unescaped: 82]
            Representation: Literal Header Field with Incremental Indexing - Indexed Name
            Index: 62
        Header: date: Wed, 13 May 2020 16:38:33 GMT
            Name Length: 4
            Name: date
            Value Length: 29
            Value: Wed, 13 May 2020 16:38:33 GMT
            date: Wed, 13 May 2020 16:38:33 GMT
            [Unescaped: Wed, 13 May 2020 16:38:33 GMT]
            Representation: Indexed Header Field
            Index: 76
        Header: server: istio-envoy
            Name Length: 6
            Name: server
            Value Length: 11
            Value: istio-envoy
            server: istio-envoy
            [Unescaped: istio-envoy]
            Representation: Indexed Header Field
            Index: 82
        Header: via: 2.0 worker.com
            Name Length: 3
            Name: via
            Value Length: 61
            Value: 2.0 worker.com
            via: 2.0 worker.com
            [Unescaped: 2.0 worker.com]
            Representation: Indexed Header Field
            Index: 81
    Stream: DATA, Stream ID: 25, Length 259
        Length: 259
        Type: DATA (0)
        Flags: 0x01
            .... ...1 = End Stream: True
            .... 0... = Padded: False
            0000 .00. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0001 1001 = Stream Identifier: 25
        [Pad Length: 0]
        Data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx…
    JavaScript Object Notation: application/json
        Object


No.     Time                          Source                Destination           Protocol Length Info
    303 2020-05-13 16:38:33.192550    192.168.1.1           192.168.2.2           HTTP2    79     RST_STREAM[25]

HyperText Transfer Protocol 2
    Stream: RST_STREAM, Stream ID: 25, Length 4
        Length: 4
        Type: RST_STREAM (3)
        Flags: 0x00
            0000 0000 = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0001 1001 = Stream Identifier: 25
        Error: CANCEL (8)

HTTP2 stream where RST_STREAM is not seen

No.     Time                          Source                Destination           Protocol Length Info
     27 2020-05-13 16:38:33.000206    192.168.1.1           192.168.2.2           HTTP2    131    HEADERS[7]: PUT /resource/1

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 7, Length 56, PUT /resource/1
        Length: 56
        Type: HEADERS (1)
        Flags: 0x04
            .... ...0 = End Stream: False
            .... .1.. = End Headers: True
            .... 0... = Padded: False
            ..0. .... = Priority: False
            00.0 ..0. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0111 = Stream Identifier: 7
        [Pad Length: 0]
        Header Block Fragment: c5c445af62ab64a56b4a4a58ee160d4a0cb3210321004406…
        [Header Length: 279]
        [Header Count: 8]
        Header: :authority: 192.168.2.2:8000
            Name Length: 10
            Name: :authority
            Value Length: 19
            Value: 192.168.2.2:8000
            :authority: 192.168.2.2:8000
            [Unescaped: 192.168.2.2:8000]
            Representation: Indexed Header Field
            Index: 69
        Header: :method: PUT
            Name Length: 7
            Name: :method
            Value Length: 3
            Value: PUT
            :method: PUT
            [Unescaped: PUT]
            Representation: Indexed Header Field
            Index: 68
        Header: :path: /resource/1
            Name Length: 5
            Name: :path
            Value Length: 68
            Value: /resource/1
            :path: /resource/1
            [Unescaped: /resource/1]
            Representation: Literal Header Field with Incremental Indexing - Indexed Name
            Index: 5
        Header: :scheme: http
            Name Length: 7
            Name: :scheme
            Value Length: 4
            Value: http
            :scheme: http
            [Unescaped: http]
            Representation: Indexed Header Field
            Index: 6
        Header: content-type: application/json
            Name Length: 12
            Name: content-type
            Value Length: 16
            Value: application/json
            content-type: application/json
            [Unescaped: application/json]
            Representation: Indexed Header Field
            Index: 67
        Header: content-length: 259
            Name Length: 14
            Name: content-length
            Value Length: 3
            Value: 259
            content-length: 259
            [Unescaped: 259]
            Representation: Indexed Header Field
            Index: 66
        Header: accept-encoding: gzip
            Name Length: 15
            Name: accept-encoding
            Value Length: 4
            Value: gzip
            accept-encoding: gzip
            [Unescaped: gzip]
            Representation: Indexed Header Field
            Index: 65
        Header: user-agent: Go-http-client/2.0
            Name Length: 10
            Name: user-agent
            Value Length: 18
            Value: Go-http-client/2.0
            user-agent: Go-http-client/2.0
            [Unescaped: Go-http-client/2.0]
            Representation: Indexed Header Field
            Index: 64

No.     Time                          Source                Destination           Protocol Length Info
     29 2020-05-13 16:38:33.000227    192.168.1.1           192.168.2.2           HTTP2    334    DATA[7] (application/json)

HyperText Transfer Protocol 2
    Stream: DATA, Stream ID: 7, Length 259
        Length: 259
        Type: DATA (0)
        Flags: 0x01
            .... ...1 = End Stream: True
            .... 0... = Padded: False
            0000 .00. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0111 = Stream Identifier: 7
        [Pad Length: 0]
        Data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx…
    JavaScript Object Notation: application/json
        Object

No.     Time                          Source                Destination           Protocol Length Info
    185 2020-05-13 16:38:33.131058    192.168.2.2           192.168.1.1           HTTP2    1149   HEADERS[3]: 200 OK, DATA[3] (application/json), HEADERS[11]: 200 OK, DATA[11] (application/json), HEADERS[7]: 201 Created, DATA[7] (application/json)

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 7, Length 111, 201 Created
        Length: 111
        Type: HEADERS (1)
        Flags: 0x04
            .... ...0 = End Stream: False
            .... .1.. = End Headers: True
            .... 0... = Padded: False
            ..0. .... = Priority: False
            00.0 ..0. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0111 = Stream Identifier: 7
        [Pad Length: 0]
        Header Block Fragment: 488210030f0d03323539c40f1fd99d29aee30c040b85d797…
        [Header Length: 397]
        [Header Count: 8]
        Header: :status: 201 Created
            Name Length: 7
            Name: :status
            Value Length: 3
            Value: 201
            :status: 201
            [Unescaped: 201]
            Representation: Literal Header Field with Incremental Indexing - Indexed Name
            Index: 8
        Header: content-length: 259
            Name Length: 14
            Name: content-length
            Value Length: 3
            Value: 259
            content-length: 259
            [Unescaped: 259]
            Representation: Literal Header Field without Indexing - Indexed Name
            Index: 28
        Header: content-type: application/json
            Name Length: 12
            Name: content-type
            Value Length: 16
            Value: application/json
            content-type: application/json
            [Unescaped: application/json]
            Representation: Indexed Header Field
            Index: 68
        Header: location: http://192.168.2.2:8000/192.168.3.4:8080~odxdp~api~/resource/1
            Name Length: 8
            Name: location
            Value Length: 124
            Value: http://192.168.2.2:8000/192.168.3.4:8080~odxdp~api~/resource/1
            location: http://192.168.2.2:8000/192.168.3.4:8080~odxdp~api~/resource/1
            [Unescaped: http://192.168.2.2:8000/192.168.3.4:8080~odxdp~api~/resource/1]
            Representation: Literal Header Field without Indexing - Indexed Name
            Index: 46
        Header: x-envoy-upstream-service-time: 119
            Name Length: 29
            Name: x-envoy-upstream-service-time
            Value Length: 3
            Value: 119
            [Unescaped: 119]
            Representation: Literal Header Field with Incremental Indexing - Indexed Name
            Index: 63
        Header: date: Wed, 13 May 2020 16:38:32 GMT
            Name Length: 4
            Name: date
            Value Length: 29
            Value: Wed, 13 May 2020 16:38:32 GMT
            date: Wed, 13 May 2020 16:38:32 GMT
            [Unescaped: Wed, 13 May 2020 16:38:32 GMT]
            Representation: Indexed Header Field
            Index: 67
        Header: server: istio-envoy
            Name Length: 6
            Name: server
            Value Length: 11
            Value: istio-envoy
            server: istio-envoy
            [Unescaped: istio-envoy]
            Representation: Indexed Header Field
            Index: 66
        Header: via: 2.0 worker.com
            Name Length: 3
            Name: via
            Value Length: 61
            Value: 2.0 worker.com
            via: 2.0 worker.com
            [Unescaped: 2.0 worker.com]
            Representation: Indexed Header Field
            Index: 65
    Stream: DATA, Stream ID: 7, Length 259
        Length: 259
        Type: DATA (0)
        Flags: 0x01
            .... ...1 = End Stream: True
            .... 0... = Padded: False
            0000 .00. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0111 = Stream Identifier: 7
        [Pad Length: 0]
        Data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx…
    JavaScript Object Notation: application/json
        Object

I am not understanding the behaviour here. As already the END_STREAM is received in the response data, why the http2 stack is sending RST_STREAM.

Note1: Using http2 package from golang golang.org/x/net/http2

Note2: There is no difference in the data content of both the streams

swamydkv
  • 27
  • 1
  • 11
  • You should enable some debug/verbose option on the client to know why it is sending a RST_STREAM. Because you don't post the actual messages, it may also be possible that what you posted above is not what it has been sent/received. For example, a compliant HTTP/2 client will never send the `host` header, and the request lacks the `:authority` field that is mandatory. Post the real evidence of the traffic. – sbordet May 20 '20 at 13:30
  • yes ```:authority``` field is present which contains the server ip and port. Collecting logs is not feasible as 1000 request responses exchanged in a second. – swamydkv May 20 '20 at 15:01
  • added the actual http information. IP address and few things are masked. – swamydkv May 20 '20 at 18:00
  • The traffic for stream=25 seems correct. The client should not send a RST_STREAM unless it's due to the client application. For example, the client application can parse the JSON in the DATA frame that is arrived, find that the JSON is malformed or otherwise invalid, raise an error that is converted to a RST_STREAM even if that stream is already terminated. The only way to know is to debug the client. That RST_STREAM, when arriving to the server, will be dropped because for the server stream=25 does not exist anymore (it has been completed successfully already). – sbordet May 20 '20 at 19:33
  • Thank you @sbordet.I have analysed the json content it is valid and not malformed, but anyways will check again. Would like to know why the ErrorCause is set to CANCEL because RFC says `CANCEL (0x8): Used by the endpoint to indicate that the stream is no longer needed.` – swamydkv May 21 '20 at 04:42
  • @sbordet, I have added a traffic where we are not seeing RST_STREAM. The RST_STREAM is going for some cases. – swamydkv May 21 '20 at 07:45

0 Answers0