payload: {"action":"created","issue":{"url":"https://api.github.com/repos/apache/beam/issues/9556","repository_url":"https://api.github.com/repos/apache/beam","labels_url":"https://api.github.com/repos/apache/beam/issues/9556/labels{/name}","comments_url":"https://api.github.com/repos/apache/beam/issues/9556/comments","events_url":"https://api.github.com/repos/apache/beam/issues/9556/events","html_url":"https://github.com/apache/beam/pull/9556","id":492674622,"node_id":"MDExOlB1bGxSZXF1ZXN0MzE2ODAwNDUx","number":9556,"title":"[BEAM-3200, BEAM-3772] Fix: Honor create and write disposition with dynamic destinations","user":{"login":"bcbradle","id":16803325,"node_id":"MDQ6VXNlcjE2ODAzMzI1","avatar_url":"https://avatars1.githubusercontent.com/u/16803325?v=4","gravatar_id":"","url":"https://api.github.com/users/bcbradle","html_url":"https://github.com/bcbradle","followers_url":"https://api.github.com/users/bcbradle/followers","following_url":"https://api.github.com/users/bcbradle/following{/other_user}","gists_url":"https://api.github.com/users/bcbradle/gists{/gist_id}","starred_url":"https://api.github.com/users/bcbradle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcbradle/subscriptions","organizations_url":"https://api.github.com/users/bcbradle/orgs","repos_url":"https://api.github.com/users/bcbradle/repos","events_url":"https://api.github.com/users/bcbradle/events{/privacy}","received_events_url":"https://api.github.com/users/bcbradle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2019-09-12T08:57:34Z","updated_at":"2019-09-14T20:42:10Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/beam/pulls/9556","html_url":"https://github.com/apache/beam/pull/9556","diff_url":"https://github.com/apache/beam/pull/9556.diff","patch_url":"https://github.com/apache/beam/pull/9556.patch"},"body":"If writing directly to the destination, then the table is created on the first write, but only when the first pane actually contains elements that are destined for that table. If the first pane does not contain such elements, the CreateDisposition is currently inadvertently set to CreateNever, which produces production issues for me:\r\n```\r\njava.lang.RuntimeException: Failed to create job with prefix beam_load_iottimeseriesdatapipeline_399f52a8d4c249c387948d3472b516b4_a9bbce63e9d84f698e587084f1402bd1_00001_00345, reached max retries: 1000, last failed job: {\r\n  \"configuration\" : {\r\n    \"jobType\" : \"LOAD\",\r\n    \"load\" : {\r\n      \"clustering\" : {\r\n        \"fields\" : [ \"device_id\" ]\r\n      },\r\n      \"createDisposition\" : \"CREATE_NEVER\",\r\n      \"destinationTable\" : {\r\n        \"datasetId\" : \"device_metrics_raw\",\r\n        \"projectId\" : \"awair-stg\",\r\n        \"tableId\" : \"diskused_awairgateway\"\r\n      },\r\n      \"ignoreUnknownValues\" : false,\r\n      \"sourceFormat\" : \"NEWLINE_DELIMITED_JSON\",\r\n      \"timePartitioning\" : {\r\n        \"field\" : \"timestamp\",\r\n        \"requirePartitionFilter\" : true,\r\n        \"type\" : \"DAY\"\r\n      },\r\n      \"writeDisposition\" : \"WRITE_APPEND\"\r\n    }\r\n  },\r\n```\r\n\r\nwhich shows up despite explicitly setting:\r\n```\r\n.to(\r\n        new BigqueryDestinationFunc(project, datasetName)\r\n      ).withClustering(\r\n      ).optimizedWrites(\r\n      ).withWriteDisposition(\r\n        WriteDisposition.WRITE_APPEND\r\n      ).withCreateDisposition(\r\n        CreateDisposition.CREATE_IF_NEEDED\r\n      ).withMethod(\r\n        BigQueryIO.Write.Method.FILE_LOADS\r\n      ).withTriggeringFrequency(\r\n        Duration.standardMinutes(5)\r\n      )\r\n```\r\n------------------------------------------------------------------------------------------------\r\n\r\nLang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark\r\n--- | --- | --- | --- | --- | --- | --- | ---\r\nGo | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)\r\nJava | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)\r\nPython | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)\r\nXLang | --- | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---\r\n\r\nPre-Commit Tests Status (on master branch)\r\n------------------------------------------------------------------------------------------------\r\n\r\n--- |Java | Python | Go | Website\r\n--- | --- | --- | --- | ---\r\nNon-portable | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) \r\nPortable | --- | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---\r\n\r\nSee [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/beam/issues/comments/531512134","html_url":"https://github.com/apache/beam/pull/9556#issuecomment-531512134","issue_url":"https://api.github.com/repos/apache/beam/issues/9556","id":531512134,"node_id":"MDEyOklzc3VlQ29tbWVudDUzMTUxMjEzNA==","user":{"login":"reuvenlax","id":21071061,"node_id":"MDQ6VXNlcjIxMDcxMDYx","avatar_url":"https://avatars1.githubusercontent.com/u/21071061?v=4","gravatar_id":"","url":"https://api.github.com/users/reuvenlax","html_url":"https://github.com/reuvenlax","followers_url":"https://api.github.com/users/reuvenlax/followers","following_url":"https://api.github.com/users/reuvenlax/following{/other_user}","gists_url":"https://api.github.com/users/reuvenlax/gists{/gist_id}","starred_url":"https://api.github.com/users/reuvenlax/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reuvenlax/subscriptions","organizations_url":"https://api.github.com/users/reuvenlax/orgs","repos_url":"https://api.github.com/users/reuvenlax/repos","events_url":"https://api.github.com/users/reuvenlax/events{/privacy}","received_events_url":"https://api.github.com/users/reuvenlax/received_events","type":"User","site_admin":false},"created_at":"2019-09-14T20:42:10Z","updated_at":"2019-09-14T20:42:10Z","author_association":"CONTRIBUTOR","body":"I'm not entirely sure I understand the bug reported. The pane should be per key, and table destination is part of the input key. Therefore I think it should be impossible for there to be no elements for that table - in that case there would be no pane triggering at all.\r\n\r\nthis PR appears to break triggered loads, if I'm reading it correctly. e.g. if the write disposition is set to WRITE_TRUNCATE, then each trigger will delete the data written by the previous trigger, which is incorrect behavior. You want the first trigger for this table to truncate, but not the subsequent ones."}}
other: {"actor":{"display_login":"reuvenlax"}}