payload: {"action":"created","issue":{"url":"https://api.github.com/repos/grpc/grpc/issues/18554","repository_url":"https://api.github.com/repos/grpc/grpc","labels_url":"https://api.github.com/repos/grpc/grpc/issues/18554/labels{/name}","comments_url":"https://api.github.com/repos/grpc/grpc/issues/18554/comments","events_url":"https://api.github.com/repos/grpc/grpc/issues/18554/events","html_url":"https://github.com/grpc/grpc/issues/18554","id":426231862,"node_id":"MDU6SXNzdWU0MjYyMzE4NjI=","number":18554,"title":"Client channel unusable after a network reset","user":{"login":"Anton1123","id":11339201,"node_id":"MDQ6VXNlcjExMzM5MjAx","avatar_url":"https://avatars3.githubusercontent.com/u/11339201?v=4","gravatar_id":"","url":"https://api.github.com/users/Anton1123","html_url":"https://github.com/Anton1123","followers_url":"https://api.github.com/users/Anton1123/followers","following_url":"https://api.github.com/users/Anton1123/following{/other_user}","gists_url":"https://api.github.com/users/Anton1123/gists{/gist_id}","starred_url":"https://api.github.com/users/Anton1123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Anton1123/subscriptions","organizations_url":"https://api.github.com/users/Anton1123/orgs","repos_url":"https://api.github.com/users/Anton1123/repos","events_url":"https://api.github.com/users/Anton1123/events{/privacy}","received_events_url":"https://api.github.com/users/Anton1123/received_events","type":"User","site_admin":false},"labels":[{"id":155504842,"node_id":"MDU6TGFiZWwxNTU1MDQ4NDI=","url":"https://api.github.com/repos/grpc/grpc/labels/kind/bug","name":"kind/bug","color":"93732c","default":false},{"id":166332908,"node_id":"MDU6TGFiZWwxNjYzMzI5MDg=","url":"https://api.github.com/repos/grpc/grpc/labels/lang/c++","name":"lang/c++","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2019-03-27T23:49:14Z","updated_at":"2019-04-04T03:01:35Z","closed_at":null,"author_association":"NONE","body":"### Summary: If a client channel is in a READY state and the network is disconnected, the channel becomes unusable and the client will not attempt to reconnect to the server once the network connection is re-established.\r\n\r\n### What version of gRPC and what language are you using?\r\n1.17.2\r\nSame issue experience in version 1.11.x\r\nC++\r\n \r\n### What operating system (Linux, Windows,...) and version?\r\nClient running on Ubuntu 16.04.\r\nServer running Windows Enterprise.\r\n \r\n### What runtime / compiler are you using (e.g. python version or version of gcc)\r\nN/A\r\n \r\n### What did you do?\r\nServer and client are both started on a connected network. I can successfully make calls and receive responses from the server. When the network is turned off, the server receives a \"Disconnected client - Endpoint read failed\" error. Some other relevant fields in this debug message - \"grpc_status\":14 (UNAVAILABLE), \"occured_during_write\":0, \"description\":\"An established connection was aborted by the software in your host machine\".\r\n\r\nAt the time of network disconnect, the client does not print out any logs at all (using GRPC_TRACE=connectivity_state,call_error,op_failure,server_channel,client_channel,channel GRPC_VERBOSITY=DEBUG).\r\n\r\nOnce the network is turned on again there are no logs experienced on neither the server nor the client. Attempting to make a call using the client (send a launch request) results in a repeated DEADLINE_EXCEEDED error. Turning off the network connection at this time does not result in a server side \"Disconnected client\" error.\r\n\r\nThe client context is set to use a deadline (tested with 2 and 10 seconds). Synchronous calls are used in this case.\r\n\r\nCode sniplets:\r\n/rpc_service.proto\r\nsyntax = \"proto3\";\r\n\r\nimport \"google/rpc/status.proto\";\r\n\r\nmessage RpcRequest {\r\n}\r\n\r\nmessage RpcResponse {\r\n}\r\n\r\nservice RpcService{\r\n  rpc Call(RpcRequest) returns (RpcResponse);\r\n}\r\n\r\n\r\n/client.cc\r\nInitialization:\r\nstd::unique_ptr<RpcService::Stub> stub_ = RpcService::NewStub(::grpc::CreateChannel(\r\n      server_endpoint, ::grpc::InsecureChannelCredentials()));\r\n\r\nSending a rpc request:\r\n  ::grpc::ClientContext context;\r\n  context.set_deadline(\r\n      gpr_time_from_micros(call_timeout_.InMicroseconds(), GPR_TIMESPAN));\r\n  RpcRequest request;\r\n  RpcResponse response;\r\n  ::grpc::Status grpc_status = stub_->Call(&context, request, &response);\r\n\r\n/server.cc\r\ngrpc::ServerBuilder builder;\r\nbuilder.AddListeningPort(endpoint, ::grpc::InsecureServerCredentials());\r\nbuilder.RegisterService(&rpc_service);\r\nstd::unique_ptr<grpc::Server> grpc_server_ = builder.BuildAndStart();\r\n\r\n### What did you expect to see?\r\nClient should make a successful call after a network reset. \r\n \r\n### What did you see instead?\r\nClient fails to receive a response from the server. \r\n\r\n### Anything else we should know about your project / environment?\r\nWhen the network connection is re-established and the client fails to receive a response from the server, tcpdump captures the client sending out some packets.\r\nStarting up both client and server with network ON, and then unplugging the network does not result in any error messages until a call is attempted. This is the same result as when starting both client and server with the network disconnected. Once a call is attempted the client will transition from IDLE to CONNECTING and then begin to bounce back and forth between CONNECTING and TRANSIENT_FAILURE states (attempting to reconnect with exponential back-off) until the connection is re-established.\r\nIf the client is started with the network connected, but doesn't send a request and the network is disconnected the server doesn't get a disconnected client error. Until a call is made, the client stays in a \"IDLE\".\r\nIf a client is initialized and a call is made on a disconnected network, then the client will enter a CONNECTING state (with exponential backoff up to a max of 2 min where the client will be in a TRANSIENT_FAILURE state). Once the network is connected, the connection will be re-established the next time the channel will enter a CONNECTING state and the client will enter the READY state. After this, each call will succeed until the network is reset.\r\nDisconnecting the network after the client is in a READY state will not transition the client out of a READY state.\r\nIn summary: Until a call is made, the client will stay in an \"IDLE\" state no matter the network status. Once a call is made, the client will attempt to make a connection by entering the CONNECTING state. If no connection is found, it will transition bounce in-between CONNECTING and TRANSIENT_FAILURE states. Once a connection is found, the client will go into a READY state. From here, if a connection is lost, the client will not attempt to enter a CONNECTING state again.\r\n\r\nSimilar issue (closed) to the one I’m having:\r\nhttps://github.com/grpc/grpc/issues/16974\r\n\r\n### Known fix\r\nCreate a new channel on each call. \r\n\r\n### Failed fix attempts\r\nSet GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA = 0\r\n\r\n### Questions\r\nShould the client be able to use the already created channel after a network reset?\r\nDoes the channel have to be restarted when a network is reset?\r\n"},"comment":{"url":"https://api.github.com/repos/grpc/grpc/issues/comments/479731705","html_url":"https://github.com/grpc/grpc/issues/18554#issuecomment-479731705","issue_url":"https://api.github.com/repos/grpc/grpc/issues/18554","id":479731705,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3OTczMTcwNQ==","user":{"login":"rmstar","id":2873805,"node_id":"MDQ6VXNlcjI4NzM4MDU=","avatar_url":"https://avatars3.githubusercontent.com/u/2873805?v=4","gravatar_id":"","url":"https://api.github.com/users/rmstar","html_url":"https://github.com/rmstar","followers_url":"https://api.github.com/users/rmstar/followers","following_url":"https://api.github.com/users/rmstar/following{/other_user}","gists_url":"https://api.github.com/users/rmstar/gists{/gist_id}","starred_url":"https://api.github.com/users/rmstar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rmstar/subscriptions","organizations_url":"https://api.github.com/users/rmstar/orgs","repos_url":"https://api.github.com/users/rmstar/repos","events_url":"https://api.github.com/users/rmstar/events{/privacy}","received_events_url":"https://api.github.com/users/rmstar/received_events","type":"User","site_admin":false},"created_at":"2019-04-04T03:01:35Z","updated_at":"2019-04-04T03:01:35Z","author_association":"CONTRIBUTOR","body":"Can you post complete client and server code to reproduce this issue? Also, can you please check if you're still seeing this with the latest grpc code on master?"}}
other: {"actor":{"display_login":"rmstar"}}