payload: {"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/26440","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/26440/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/26440/comments","events_url":"https://api.github.com/repos/apache/spark/issues/26440/events","html_url":"https://github.com/apache/spark/pull/26440","id":520238088,"node_id":"MDExOlB1bGxSZXF1ZXN0MzM4ODYyMjIz","number":26440,"title":"[SPARK-20628][CORE][K8S] Start to improve Spark decommissioning & preemption support","user":{"login":"holdenk","id":59893,"node_id":"MDQ6VXNlcjU5ODkz","avatar_url":"https://avatars1.githubusercontent.com/u/59893?v=4","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"labels":[{"id":1405801482,"node_id":"MDU6TGFiZWwxNDA1ODAxNDgy","url":"https://api.github.com/repos/apache/spark/labels/SPARK%20CORE","name":"SPARK CORE","color":"ededed","default":false,"description":null}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":29,"created_at":"2019-11-08T21:17:41Z","updated_at":"2020-01-10T23:19:59Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/26440","html_url":"https://github.com/apache/spark/pull/26440","diff_url":"https://github.com/apache/spark/pull/26440.diff","patch_url":"https://github.com/apache/spark/pull/26440.patch"},"body":"This PR is based on an existing/previou PR - https://github.com/apache/spark/pull/19045\r\n\r\n### What changes were proposed in this pull request?\r\n\r\nThis changes adds a decommissioning state that we can enter when the cloud provider/scheduler lets us know we aren't going to be removed immediately but instead will be removed soon. This concept fits nicely in K8s and also with spot-instances on AWS / preemptible instances all of which we can get a notice that our host is going away. For now we simply stop scheduling jobs, in the future we could perform some kind of migration of data during scale-down, or at least stop accepting new blocks to cache.\r\n\r\nThere is a design document at https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing\r\n\r\n### Why are the changes needed?\r\n\r\nWith more move to preemptible multi-tenancy, serverless environments, and spot-instances better handling of node scale down is required.\r\n\r\n### Does this PR introduce any user-facing change?\r\n\r\nThere is no API change, however an additional configuration flag is added to enable/disable this behaviour.\r\n\r\n### How was this patch tested?\r\n\r\nNew integration tests in the Spark K8s integration testing. Extension of the AppClientSuite to test decommissioning seperate from the K8s."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/573243478","html_url":"https://github.com/apache/spark/pull/26440#issuecomment-573243478","issue_url":"https://api.github.com/repos/apache/spark/issues/26440","id":573243478,"node_id":"MDEyOklzc3VlQ29tbWVudDU3MzI0MzQ3OA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2020-01-10T23:19:59Z","updated_at":"2020-01-10T23:19:59Z","author_association":"NONE","body":"Kubernetes integration test status failure\nURL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21293/\n"}}
other: {"actor":{"display_login":"SparkQA"}}