payload: {"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/26294","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/26294/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/26294/comments","events_url":"https://api.github.com/repos/apache/spark/issues/26294/events","html_url":"https://github.com/apache/spark/pull/26294","id":513903400,"node_id":"MDExOlB1bGxSZXF1ZXN0MzMzNjA0NzEx","number":26294,"title":"[SPARK-28477] [SQL] Rewrite CaseWhen with single branch to If","user":{"login":"davidvrba","id":45728981,"node_id":"MDQ6VXNlcjQ1NzI4OTgx","avatar_url":"https://avatars1.githubusercontent.com/u/45728981?v=4","gravatar_id":"","url":"https://api.github.com/users/davidvrba","html_url":"https://github.com/davidvrba","followers_url":"https://api.github.com/users/davidvrba/followers","following_url":"https://api.github.com/users/davidvrba/following{/other_user}","gists_url":"https://api.github.com/users/davidvrba/gists{/gist_id}","starred_url":"https://api.github.com/users/davidvrba/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidvrba/subscriptions","organizations_url":"https://api.github.com/users/davidvrba/orgs","repos_url":"https://api.github.com/users/davidvrba/repos","events_url":"https://api.github.com/users/davidvrba/events{/privacy}","received_events_url":"https://api.github.com/users/davidvrba/received_events","type":"User","site_admin":false},"labels":[{"id":1406605297,"node_id":"MDU6TGFiZWwxNDA2NjA1Mjk3","url":"https://api.github.com/repos/apache/spark/labels/OPTIMIZER","name":"OPTIMIZER","color":"ededed","default":false},{"id":1405794576,"node_id":"MDU6TGFiZWwxNDA1Nzk0NTc2","url":"https://api.github.com/repos/apache/spark/labels/SQL","name":"SQL","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2019-10-29T13:06:13Z","updated_at":"2019-11-07T08:33:50Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/26294","html_url":"https://github.com/apache/spark/pull/26294","diff_url":"https://github.com/apache/spark/pull/26294.diff","patch_url":"https://github.com/apache/spark/pull/26294.patch"},"body":"### What changes were proposed in this pull request?\r\nSpark org.apache.spark.sql.functions do not have `if` function so conditions are expressed using `when-otherwise` function. However `If` (which is available in SQL) has more efficient code gen. This pr rewrites `when-otherwise` conditions to `If` if it is possible (`when-otherwise` with single branch)\r\n\r\n### Why are the changes needed?\r\nIt is an optimization enhancement. Here is a simple performance comparison (tested in local mode (with 4 cores)):\r\n```\r\nval df = spark.range(10000000000L).withColumn(\"x\", rand)\r\nval resultA = df.withColumn(\"r\", when($\"x\" < 0.5, lit(1)).otherwise(lit(0))).agg(sum($\"r\"))\r\nval resultB = df.withColumn(\"r\", expr(\"if(x < 0.5, 1, 0)\")).agg(sum($\"r\"))\r\n\r\nresultA.collect() // takes 56s to finish\r\nresultB.collect() // takes 30s to finish\r\n```\r\n### Does this PR introduce any user-facing change?\r\nNo\r\n\r\n### How was this patch tested?\r\nNew test is added.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/550977477","html_url":"https://github.com/apache/spark/pull/26294#issuecomment-550977477","issue_url":"https://api.github.com/repos/apache/spark/issues/26294","id":550977477,"node_id":"MDEyOklzc3VlQ29tbWVudDU1MDk3NzQ3Nw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2019-11-07T08:33:50Z","updated_at":"2019-11-07T08:33:50Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18272/\nTest PASSed."}}
other: {"actor":{"display_login":"AmplabJenkins"}}