payload: {"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60987","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60987/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60987/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60987/events","html_url":"https://github.com/kubernetes/kubernetes/issues/60987","id":303911453,"node_id":"MDU6SXNzdWUzMDM5MTE0NTM=","number":60987,"title":"Orphaned pod found - but volume paths are still present on disk","user":{"login":"patrickstjohn","id":7337041,"node_id":"MDQ6VXNlcjczMzcwNDE=","avatar_url":"https://avatars1.githubusercontent.com/u/7337041?v=4","gravatar_id":"","url":"https://api.github.com/users/patrickstjohn","html_url":"https://github.com/patrickstjohn","followers_url":"https://api.github.com/users/patrickstjohn/followers","following_url":"https://api.github.com/users/patrickstjohn/following{/other_user}","gists_url":"https://api.github.com/users/patrickstjohn/gists{/gist_id}","starred_url":"https://api.github.com/users/patrickstjohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patrickstjohn/subscriptions","organizations_url":"https://api.github.com/users/patrickstjohn/orgs","repos_url":"https://api.github.com/users/patrickstjohn/repos","events_url":"https://api.github.com/users/patrickstjohn/events{/privacy}","received_events_url":"https://api.github.com/users/patrickstjohn/received_events","type":"User","site_admin":false},"labels":[{"id":173493665,"node_id":"MDU6TGFiZWwxNzM0OTM2NjU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false},{"id":169428334,"node_id":"MDU6TGFiZWwxNjk0MjgzMzQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":57,"created_at":"2018-03-09T17:04:45Z","updated_at":"2019-06-25T04:24:56Z","closed_at":null,"author_association":"NONE","body":"<!-- This form is for bug reports and feature requests ONLY! \r\n\r\nIf you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?**:\r\nBUG\r\n\r\n**What happened**:\r\nKubelet is periodically going into an error state and causing errors with our storage layer (ceph, shared filesystem). Upon cleaning out the orphaned pod directory things eventually right themselves.\r\n* Workaround: `rmdir /var/lib/kubelet/pods/*/volumes/*rook/*`\r\n\r\n**What you expected to happen**:\r\nKubelet should intelligently deal with orphaned pods. Cleaning a stale directory manually should not be required.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nUsing rook-0.7.0 (this isn't a rook problem as far as I can tell but this is how we're reproducing):\r\nkubectl create -f rook-operator.yaml\r\nkubectl create -f rook-cluster.yaml\r\nkubectl create -f rook-filesystem.yaml\r\n\r\nMount/write to the shared filesystem and monitor /var/log/messages for the following:\r\n`\r\nkubelet: E0309 16:46:30.429770    3112 kubelet_volumes.go:128] Orphaned pod \"2815f27a-219b-11e8-8a2a-ec0d9a3a445a\" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.\r\n`\r\n\r\n**Anything else we need to know?**:\r\nThis looks identical to the following: https://github.com/kubernetes/kubernetes/issues/45464 but for a different plugin.\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n`\r\nClient Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.3\", GitCommit:\"d2835416544f298c919e2ead3be3d0864b52323b\", GitTreeState:\"clean\", BuildDate:\"2018-02-07T12:22:21Z\", GoVersion:\"go1.9.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.3\", GitCommit:\"d2835416544f298c919e2ead3be3d0864b52323b\", GitTreeState:\"clean\", BuildDate:\"2018-02-07T11:55:20Z\", GoVersion:\"go1.9.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n`\r\n\r\n- Cloud provider or hardware configuration:\r\nBare-metal private cloud\r\n- OS (e.g. from /etc/os-release):\r\nRed Hat Enterprise Linux Server release 7.4 (Maipo)\r\n- Kernel (e.g. `uname -a`):\r\nLinux 4.4.115-1.el7.elrepo.x86_64 #1 SMP Sat Feb 3 20:11:41 EST 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n- Install tools:\r\nkubeadm\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/505276790","html_url":"https://github.com/kubernetes/kubernetes/issues/60987#issuecomment-505276790","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60987","id":505276790,"node_id":"MDEyOklzc3VlQ29tbWVudDUwNTI3Njc5MA==","user":{"login":"dongwangdw","id":2303377,"node_id":"MDQ6VXNlcjIzMDMzNzc=","avatar_url":"https://avatars3.githubusercontent.com/u/2303377?v=4","gravatar_id":"","url":"https://api.github.com/users/dongwangdw","html_url":"https://github.com/dongwangdw","followers_url":"https://api.github.com/users/dongwangdw/followers","following_url":"https://api.github.com/users/dongwangdw/following{/other_user}","gists_url":"https://api.github.com/users/dongwangdw/gists{/gist_id}","starred_url":"https://api.github.com/users/dongwangdw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongwangdw/subscriptions","organizations_url":"https://api.github.com/users/dongwangdw/orgs","repos_url":"https://api.github.com/users/dongwangdw/repos","events_url":"https://api.github.com/users/dongwangdw/events{/privacy}","received_events_url":"https://api.github.com/users/dongwangdw/received_events","type":"User","site_admin":false},"created_at":"2019-06-25T04:24:56Z","updated_at":"2019-06-25T04:24:56Z","author_association":"NONE","body":"Is there any progress on this issue ?"}}
other: {"actor":{"display_login":"dongwangdw"}}