payload: {"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/gatk/issues/1524","repository_url":"https://api.github.com/repos/broadinstitute/gatk","labels_url":"https://api.github.com/repos/broadinstitute/gatk/issues/1524/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/gatk/issues/1524/comments","events_url":"https://api.github.com/repos/broadinstitute/gatk/issues/1524/events","html_url":"https://github.com/broadinstitute/gatk/issues/1524","id":136530010,"node_id":"MDU6SXNzdWUxMzY1MzAwMTA=","number":1524,"title":"fix NegativeArraySizeException in ReadsSparkPipeline","user":{"login":"lbergelson","id":4700332,"node_id":"MDQ6VXNlcjQ3MDAzMzI=","avatar_url":"https://avatars3.githubusercontent.com/u/4700332?v=4","gravatar_id":"","url":"https://api.github.com/users/lbergelson","html_url":"https://github.com/lbergelson","followers_url":"https://api.github.com/users/lbergelson/followers","following_url":"https://api.github.com/users/lbergelson/following{/other_user}","gists_url":"https://api.github.com/users/lbergelson/gists{/gist_id}","starred_url":"https://api.github.com/users/lbergelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbergelson/subscriptions","organizations_url":"https://api.github.com/users/lbergelson/orgs","repos_url":"https://api.github.com/users/lbergelson/repos","events_url":"https://api.github.com/users/lbergelson/events{/privacy}","received_events_url":"https://api.github.com/users/lbergelson/received_events","type":"User","site_admin":false},"labels":[{"id":226129596,"node_id":"MDU6TGFiZWwyMjYxMjk1OTY=","url":"https://api.github.com/repos/broadinstitute/gatk/labels/Spark","name":"Spark","color":"FFD700","default":false},{"id":153552584,"node_id":"MDU6TGFiZWwxNTM1NTI1ODQ=","url":"https://api.github.com/repos/broadinstitute/gatk/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"lbergelson","id":4700332,"node_id":"MDQ6VXNlcjQ3MDAzMzI=","avatar_url":"https://avatars3.githubusercontent.com/u/4700332?v=4","gravatar_id":"","url":"https://api.github.com/users/lbergelson","html_url":"https://github.com/lbergelson","followers_url":"https://api.github.com/users/lbergelson/followers","following_url":"https://api.github.com/users/lbergelson/following{/other_user}","gists_url":"https://api.github.com/users/lbergelson/gists{/gist_id}","starred_url":"https://api.github.com/users/lbergelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbergelson/subscriptions","organizations_url":"https://api.github.com/users/lbergelson/orgs","repos_url":"https://api.github.com/users/lbergelson/repos","events_url":"https://api.github.com/users/lbergelson/events{/privacy}","received_events_url":"https://api.github.com/users/lbergelson/received_events","type":"User","site_admin":false},"assignees":[{"login":"lbergelson","id":4700332,"node_id":"MDQ6VXNlcjQ3MDAzMzI=","avatar_url":"https://avatars3.githubusercontent.com/u/4700332?v=4","gravatar_id":"","url":"https://api.github.com/users/lbergelson","html_url":"https://github.com/lbergelson","followers_url":"https://api.github.com/users/lbergelson/followers","following_url":"https://api.github.com/users/lbergelson/following{/other_user}","gists_url":"https://api.github.com/users/lbergelson/gists{/gist_id}","starred_url":"https://api.github.com/users/lbergelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbergelson/subscriptions","organizations_url":"https://api.github.com/users/lbergelson/orgs","repos_url":"https://api.github.com/users/lbergelson/repos","events_url":"https://api.github.com/users/lbergelson/events{/privacy}","received_events_url":"https://api.github.com/users/lbergelson/received_events","type":"User","site_admin":false}],"milestone":null,"comments":12,"created_at":"2016-02-25T22:23:20Z","updated_at":"2019-03-08T17:39:18Z","closed_at":null,"author_association":"COLLABORATOR","body":"We have multiple reports of `NegativeArraySizeException` being thrown during kryo serialization while running `ReadsSparkPipeline`\n\nan example\n\n```\n./gatk/gatk-launch \\\nReadsPipelineSpark \\\n-I $bam \\\n-R $ref \\\n--programName ${name} \\\n-O $bamout \\\n--bamPartitionSize 134217728 \\\n--knownSites $dbsnp \\\n--shardedOutput true \\\n--emit_original_quals \\\n--duplicates_scoring_strategy SUM_OF_BASE_QUALITIES \\\n-- \\\n--sparkRunner LOCAL\n```\n\nresulted in\n\n```\ncom.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException\nSerialization trace:\nvs (org.broadinstitute.hellbender.utils.collections.IntervalsSkipListOneContig)\nintervals (org.broadinstitute.hellbender.utils.collections.IntervalsSkipList)\nat com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)\nat com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)\nat com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:549)\nat com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:82)\n```\n"},"comment":{"url":"https://api.github.com/repos/broadinstitute/gatk/issues/comments/471013025","html_url":"https://github.com/broadinstitute/gatk/issues/1524#issuecomment-471013025","issue_url":"https://api.github.com/repos/broadinstitute/gatk/issues/1524","id":471013025,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3MTAxMzAyNQ==","user":{"login":"danking","id":106194,"node_id":"MDQ6VXNlcjEwNjE5NA==","avatar_url":"https://avatars2.githubusercontent.com/u/106194?v=4","gravatar_id":"","url":"https://api.github.com/users/danking","html_url":"https://github.com/danking","followers_url":"https://api.github.com/users/danking/followers","following_url":"https://api.github.com/users/danking/following{/other_user}","gists_url":"https://api.github.com/users/danking/gists{/gist_id}","starred_url":"https://api.github.com/users/danking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danking/subscriptions","organizations_url":"https://api.github.com/users/danking/orgs","repos_url":"https://api.github.com/users/danking/repos","events_url":"https://api.github.com/users/danking/events{/privacy}","received_events_url":"https://api.github.com/users/danking/received_events","type":"User","site_admin":false},"created_at":"2019-03-08T17:39:18Z","updated_at":"2019-03-08T17:39:18Z","author_association":"NONE","body":"FYI, Spark 2.4.0 upgraded to Kryo 4.0.0 in [3e033035](https://github.com/apache/spark/commit/3e033035a3c0b7d46c2ae18d0d322d4af3808711). There does not appear to be a back port to older spark versions.\r\n\r\nDid y'all happen to also try [this](https://github.com/EsotericSoftware/kryo/pull/527/files):\r\n\r\n```java\r\nKryo kryo = new Kryo();\r\nkryo.setReferences(false);\r\n```\r\n\r\nI'd like to avoid asking Hail users to set JVM options, so this approach is appealing to me. Curious if you all had experience trying it out."}}
other: {"actor":{"display_login":"danking"}}