payload: {"action":"created","issue":{"url":"https://api.github.com/repos/uswitch/kiam/issues/217","repository_url":"https://api.github.com/repos/uswitch/kiam","labels_url":"https://api.github.com/repos/uswitch/kiam/issues/217/labels{/name}","comments_url":"https://api.github.com/repos/uswitch/kiam/issues/217/comments","events_url":"https://api.github.com/repos/uswitch/kiam/issues/217/events","html_url":"https://github.com/uswitch/kiam/issues/217","id":411570564,"node_id":"MDU6SXNzdWU0MTE1NzA1NjQ=","number":217,"title":"kiam server HA. Context deadline exceeded while rotating servers.","user":{"login":"max-lobur","id":5246026,"node_id":"MDQ6VXNlcjUyNDYwMjY=","avatar_url":"https://avatars0.githubusercontent.com/u/5246026?v=4","gravatar_id":"","url":"https://api.github.com/users/max-lobur","html_url":"https://github.com/max-lobur","followers_url":"https://api.github.com/users/max-lobur/followers","following_url":"https://api.github.com/users/max-lobur/following{/other_user}","gists_url":"https://api.github.com/users/max-lobur/gists{/gist_id}","starred_url":"https://api.github.com/users/max-lobur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/max-lobur/subscriptions","organizations_url":"https://api.github.com/users/max-lobur/orgs","repos_url":"https://api.github.com/users/max-lobur/repos","events_url":"https://api.github.com/users/max-lobur/events{/privacy}","received_events_url":"https://api.github.com/users/max-lobur/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"rhysemmas","id":44974642,"node_id":"MDQ6VXNlcjQ0OTc0NjQy","avatar_url":"https://avatars0.githubusercontent.com/u/44974642?v=4","gravatar_id":"","url":"https://api.github.com/users/rhysemmas","html_url":"https://github.com/rhysemmas","followers_url":"https://api.github.com/users/rhysemmas/followers","following_url":"https://api.github.com/users/rhysemmas/following{/other_user}","gists_url":"https://api.github.com/users/rhysemmas/gists{/gist_id}","starred_url":"https://api.github.com/users/rhysemmas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhysemmas/subscriptions","organizations_url":"https://api.github.com/users/rhysemmas/orgs","repos_url":"https://api.github.com/users/rhysemmas/repos","events_url":"https://api.github.com/users/rhysemmas/events{/privacy}","received_events_url":"https://api.github.com/users/rhysemmas/received_events","type":"User","site_admin":false},"assignees":[{"login":"rhysemmas","id":44974642,"node_id":"MDQ6VXNlcjQ0OTc0NjQy","avatar_url":"https://avatars0.githubusercontent.com/u/44974642?v=4","gravatar_id":"","url":"https://api.github.com/users/rhysemmas","html_url":"https://github.com/rhysemmas","followers_url":"https://api.github.com/users/rhysemmas/followers","following_url":"https://api.github.com/users/rhysemmas/following{/other_user}","gists_url":"https://api.github.com/users/rhysemmas/gists{/gist_id}","starred_url":"https://api.github.com/users/rhysemmas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhysemmas/subscriptions","organizations_url":"https://api.github.com/users/rhysemmas/orgs","repos_url":"https://api.github.com/users/rhysemmas/repos","events_url":"https://api.github.com/users/rhysemmas/events{/privacy}","received_events_url":"https://api.github.com/users/rhysemmas/received_events","type":"User","site_admin":false}],"milestone":null,"comments":25,"created_at":"2019-02-18T16:59:42Z","updated_at":"2019-09-25T12:40:17Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"In my test AWS lab I run 3 masters with KIAM servers (`quay.io/uswitch/kiam:v3.0`) and 6 nodes with KIAM agents (same images). I'm getting these \r\n```\r\n{\"addr\":\"100.112.154.80:56850\",\"level\":\"error\",\"method\":\"GET\",\"msg\":\"error processing request: rpc error: code = DeadlineExceeded desc = context deadline exceeded\",\"path\":\"/latest/meta-data/iam/security-credentials/\",\"status\":500,\"time\":\"2019-02-18T16:48:17Z\"}\r\n```\r\nevery time I'm rotating masters. The error starts to appear on every one of 6 agents, once I remove at least 1 master. To stop it I need to restart all agents, when they restart they start to work normally. This does not happen if I rotate kiam server itself (delete the pod), only when I rotate the whole master instance.\r\n\r\nI'm using this chart https://github.com/helm/charts/tree/master/stable/kiam to run kiam. No special tuning, except for I'm setting `gatewayTimeoutCreation: 500ms` on both agent and server. \r\n\r\nAm I missing smth important in docs, some flag? Please suggest"},"comment":{"url":"https://api.github.com/repos/uswitch/kiam/issues/comments/535002577","html_url":"https://github.com/uswitch/kiam/issues/217#issuecomment-535002577","issue_url":"https://api.github.com/repos/uswitch/kiam/issues/217","id":535002577,"node_id":"MDEyOklzc3VlQ29tbWVudDUzNTAwMjU3Nw==","user":{"login":"mehstg","id":4677843,"node_id":"MDQ6VXNlcjQ2Nzc4NDM=","avatar_url":"https://avatars3.githubusercontent.com/u/4677843?v=4","gravatar_id":"","url":"https://api.github.com/users/mehstg","html_url":"https://github.com/mehstg","followers_url":"https://api.github.com/users/mehstg/followers","following_url":"https://api.github.com/users/mehstg/following{/other_user}","gists_url":"https://api.github.com/users/mehstg/gists{/gist_id}","starred_url":"https://api.github.com/users/mehstg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mehstg/subscriptions","organizations_url":"https://api.github.com/users/mehstg/orgs","repos_url":"https://api.github.com/users/mehstg/repos","events_url":"https://api.github.com/users/mehstg/events{/privacy}","received_events_url":"https://api.github.com/users/mehstg/received_events","type":"User","site_admin":false},"created_at":"2019-09-25T12:40:17Z","updated_at":"2019-09-25T12:40:17Z","author_association":"NONE","body":"Ha. Ok, so all on different CNI's\n\nOn Wed, 25 Sep 2019 at 10:21, David Dyball <notifications@github.com> wrote:\n\n> @mehstg <https://github.com/mehstg> I'm using Calcio CNI\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/uswitch/kiam/issues/217?email_source=notifications&email_token=ABDWBU6WB5P4QCJY6LQ7EPDQLMURVA5CNFSM4GYFD7D2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RG75Q#issuecomment-534933494>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ABDWBU6C7B4DEGKPPN3345TQLMURVANCNFSM4GYFD7DQ>\n> .\n>\n"}}
other: {"actor":{"display_login":"mehstg"}}