payload: {"action":"created","issue":{"url":"https://api.github.com/repos/aspnet/AspNetCore/issues/12887","repository_url":"https://api.github.com/repos/aspnet/AspNetCore","labels_url":"https://api.github.com/repos/aspnet/AspNetCore/issues/12887/labels{/name}","comments_url":"https://api.github.com/repos/aspnet/AspNetCore/issues/12887/comments","events_url":"https://api.github.com/repos/aspnet/AspNetCore/issues/12887/events","html_url":"https://github.com/aspnet/AspNetCore/issues/12887","id":477029554,"node_id":"MDU6SXNzdWU0NzcwMjk1NTQ=","number":12887,"title":"Kestrel/HTTP2: Getting errors when cancelling slow requests","user":{"login":"eiriktsarpalis","id":2813363,"node_id":"MDQ6VXNlcjI4MTMzNjM=","avatar_url":"https://avatars2.githubusercontent.com/u/2813363?v=4","gravatar_id":"","url":"https://api.github.com/users/eiriktsarpalis","html_url":"https://github.com/eiriktsarpalis","followers_url":"https://api.github.com/users/eiriktsarpalis/followers","following_url":"https://api.github.com/users/eiriktsarpalis/following{/other_user}","gists_url":"https://api.github.com/users/eiriktsarpalis/gists{/gist_id}","starred_url":"https://api.github.com/users/eiriktsarpalis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eiriktsarpalis/subscriptions","organizations_url":"https://api.github.com/users/eiriktsarpalis/orgs","repos_url":"https://api.github.com/users/eiriktsarpalis/repos","events_url":"https://api.github.com/users/eiriktsarpalis/events{/privacy}","received_events_url":"https://api.github.com/users/eiriktsarpalis/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"jkotalik","id":8302101,"node_id":"MDQ6VXNlcjgzMDIxMDE=","avatar_url":"https://avatars0.githubusercontent.com/u/8302101?v=4","gravatar_id":"","url":"https://api.github.com/users/jkotalik","html_url":"https://github.com/jkotalik","followers_url":"https://api.github.com/users/jkotalik/followers","following_url":"https://api.github.com/users/jkotalik/following{/other_user}","gists_url":"https://api.github.com/users/jkotalik/gists{/gist_id}","starred_url":"https://api.github.com/users/jkotalik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkotalik/subscriptions","organizations_url":"https://api.github.com/users/jkotalik/orgs","repos_url":"https://api.github.com/users/jkotalik/repos","events_url":"https://api.github.com/users/jkotalik/events{/privacy}","received_events_url":"https://api.github.com/users/jkotalik/received_events","type":"User","site_admin":false},"assignees":[{"login":"jkotalik","id":8302101,"node_id":"MDQ6VXNlcjgzMDIxMDE=","avatar_url":"https://avatars0.githubusercontent.com/u/8302101?v=4","gravatar_id":"","url":"https://api.github.com/users/jkotalik","html_url":"https://github.com/jkotalik","followers_url":"https://api.github.com/users/jkotalik/followers","following_url":"https://api.github.com/users/jkotalik/following{/other_user}","gists_url":"https://api.github.com/users/jkotalik/gists{/gist_id}","starred_url":"https://api.github.com/users/jkotalik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkotalik/subscriptions","organizations_url":"https://api.github.com/users/jkotalik/orgs","repos_url":"https://api.github.com/users/jkotalik/repos","events_url":"https://api.github.com/users/jkotalik/events{/privacy}","received_events_url":"https://api.github.com/users/jkotalik/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2019-08-05T19:53:25Z","updated_at":"2019-08-09T19:33:09Z","closed_at":"2019-08-05T22:59:02Z","author_association":"NONE","body":"I've been able to consistently reproduce an issue with Kestrel when stress testing HTTP2. The issue seems to manifest itself in cases where we trigger cancellation on the client side for requests that send request content slowly, by flushing one byte at a time.\r\n\r\n### To Reproduce\r\nInstall the nightly build (off master):\r\n```\r\n$ dotnet --info\r\n.NET Core SDK (reflecting any global.json):\r\n Version:   5.0.100-alpha1-013662\r\n Commit:    7031ddb3f6\r\n\r\nRuntime Environment:\r\n OS Name:     Windows\r\n OS Version:  10.0.18362\r\n OS Platform: Windows\r\n RID:         win10-x64\r\n Base Path:   C:\\Users\\eitsarpa\\AppData\\Local\\Temp\\dotnet-nightly-190805\\sdk\\5.0.100-alpha1-013662\\\r\n\r\nHost (useful for support):\r\n  Version: 3.0.0-preview8-28379-02\r\n  Commit:  3a87ac0079\r\n```\r\nRun the http stress suite from [this branch](https://github.com/eiriktsarpalis/corefx/tree/96f3a66878af3e1eb4098c0904cd64c751e3f851/src/System.Net.Http/tests/StressTests/HttpStress) using the following arguments:\r\n```\r\ndotnet run -c Release -- -ops 13 -n 2 -cancelRate 0.5 -maxContentLength 5\r\n```\r\nThis runs the stress suite using two client workers, sending a single request type, a PUT whose content is being [written to the stream one byte at a time](https://github.com/eiriktsarpalis/corefx/blob/96f3a66878af3e1eb4098c0904cd64c751e3f851/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs#L524-L545). Requests are being cancelled while in-flight with a probability of 50%.\r\n\r\nI'm seeing an assortment of connection errors, on the client side:\r\n```\r\nSystem.Net.Http.HttpRequestException: Error while copying content to a stream.\r\n ---> System.IO.IOException: The request was aborted.\r\n ---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.\r\n   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)\r\n   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)\r\n   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)\r\n   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()\r\n   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)\r\n   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)\r\n   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)\r\n   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)\r\n   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 333\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 109\r\n```\r\n```\r\nSystem.Net.Http.HttpRequestException: Error while copying content to a stream.\r\n ---> System.IO.IOException: The request was aborted.\r\n ---> System.IO.IOException: Unable to read data from the transport connection: Broken pipe.\r\n ---> System.Net.Sockets.SocketException (32): Broken pipe\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)\r\n   at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)\r\n   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync()\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()\r\n   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)\r\n   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)\r\n   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)\r\n   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)\r\n   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 333\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 109\r\n```\r\nOn the server side:\r\n```\r\nMicrosoft.AspNetCore.Server.Kestrel[13]\r\n      Connection id \"0HLOPPQ1U3CIT\", Request id \"0HLOPPQ1U3CIT:0000001D\": An unhandled exception was thrown by the application.\r\nSystem.IO.IOException: The client reset the request stream.\r\n   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()\r\n   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)\r\n   at System.IO.Pipelines.Pipe.GetReadAsyncResult()\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)\r\n   at HttpStress.StressServer.<>c.<<MapRoutes>b__6_9>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressServer.cs:line 209\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)\r\n```\r\n```\r\nMicrosoft.AspNetCore.Server.Kestrel[13]\r\n      Connection id \"0HLOPPQ1U3CIR\", Request id \"0HLOPPQ1U3CIR:000000A5\": An unhandled exception was thrown by the application.\r\nSystem.IO.IOException: The request stream was aborted.\r\n ---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted.\r\n ---> Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2ConnectionErrorException: HTTP/2 connection error (STREAM_CLOSED): A frame of type RST_STREAM was received after stream 163 was reset or aborted.\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRstStreamFrameAsync()\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()\r\n   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)\r\n   at System.IO.Pipelines.Pipe.GetReadAsyncResult()\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)\r\n   at HttpStress.StressServer.<>c.<<MapRoutes>b__6_9>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressServer.cs:line 209\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\r\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)\r\n```\r\n\r\nThe frequency of errors is low, with approximately 0.02% of requests failing:\r\n```\r\n[8/5/2019 7:45:32 PM] Total: 161,579 Runtime: 00:01:04\r\n        13: PUT Slow                  Success: 107,726  Canceled: 53,812        Fail: 41\r\n            TOTAL                     Success: 107,726  Canceled: 53,812        Fail: 41\r\n```\r\n\r\n### Expected behavior\r\n\r\nIt's not clear to me at this point if the issue is a bug with the client or the server. That being said I haven't been able to reproduce any errors when hosting the stress suite using http.sys (adding the `-httpSys` argument to the above command).\r\n\r\n### Additional context\r\n\r\n* Reproduces on Windows & Linux\r\n* Does not reproduce using http/1.1 (adding `-http 1.1` to the above command).\r\n* Setting cancellation rate to zero (`-cancelRate 0`) makes the issue go away.\r\n* Errors are thrown both by cancelled and not cancelled requests.\r\n\r\ncc @geoffkizer "},"comment":{"url":"https://api.github.com/repos/aspnet/AspNetCore/issues/comments/520038064","html_url":"https://github.com/aspnet/AspNetCore/issues/12887#issuecomment-520038064","issue_url":"https://api.github.com/repos/aspnet/AspNetCore/issues/12887","id":520038064,"node_id":"MDEyOklzc3VlQ29tbWVudDUyMDAzODA2NA==","user":{"login":"eiriktsarpalis","id":2813363,"node_id":"MDQ6VXNlcjI4MTMzNjM=","avatar_url":"https://avatars2.githubusercontent.com/u/2813363?v=4","gravatar_id":"","url":"https://api.github.com/users/eiriktsarpalis","html_url":"https://github.com/eiriktsarpalis","followers_url":"https://api.github.com/users/eiriktsarpalis/followers","following_url":"https://api.github.com/users/eiriktsarpalis/following{/other_user}","gists_url":"https://api.github.com/users/eiriktsarpalis/gists{/gist_id}","starred_url":"https://api.github.com/users/eiriktsarpalis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eiriktsarpalis/subscriptions","organizations_url":"https://api.github.com/users/eiriktsarpalis/orgs","repos_url":"https://api.github.com/users/eiriktsarpalis/repos","events_url":"https://api.github.com/users/eiriktsarpalis/events{/privacy}","received_events_url":"https://api.github.com/users/eiriktsarpalis/received_events","type":"User","site_admin":false},"created_at":"2019-08-09T19:33:09Z","updated_at":"2019-08-09T19:33:09Z","author_association":"NONE","body":"@jkotalik just FYI we're still seeing a tiny number of errors of the first kind (`The response ended prematurely, with at least 9 additional bytes expected.`) when stress testing against kestrel. Approximately 10 occurrences in approximately 40 million requests."}}
other: {"actor":{"display_login":"eiriktsarpalis"}}