payload: {"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/25172","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/25172/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/25172/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/25172/events","html_url":"https://github.com/pytorch/pytorch/issues/25172","id":485108158,"node_id":"MDU6SXNzdWU0ODUxMDgxNTg=","number":25172,"title":"test_det_logdet_slogdet_batched (in test_cuda.py) fails on ppc64le","user":{"login":"branfosj","id":4967,"node_id":"MDQ6VXNlcjQ5Njc=","avatar_url":"https://avatars1.githubusercontent.com/u/4967?v=4","gravatar_id":"","url":"https://api.github.com/users/branfosj","html_url":"https://github.com/branfosj","followers_url":"https://api.github.com/users/branfosj/followers","following_url":"https://api.github.com/users/branfosj/following{/other_user}","gists_url":"https://api.github.com/users/branfosj/gists{/gist_id}","starred_url":"https://api.github.com/users/branfosj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/branfosj/subscriptions","organizations_url":"https://api.github.com/users/branfosj/orgs","repos_url":"https://api.github.com/users/branfosj/repos","events_url":"https://api.github.com/users/branfosj/events{/privacy}","received_events_url":"https://api.github.com/users/branfosj/received_events","type":"User","site_admin":false},"labels":[{"id":1410522473,"node_id":"MDU6TGFiZWwxNDEwNTIyNDcz","url":"https://api.github.com/repos/pytorch/pytorch/labels/module:%20POWER","name":"module: POWER","color":"f7e101","default":false},{"id":1300933336,"node_id":"MDU6TGFiZWwxMzAwOTMzMzM2","url":"https://api.github.com/repos/pytorch/pytorch/labels/module:%20tests","name":"module: tests","color":"f7e101","default":false},{"id":1301397902,"node_id":"MDU6TGFiZWwxMzAxMzk3OTAy","url":"https://api.github.com/repos/pytorch/pytorch/labels/topic:%20flaky-tests","name":"topic: flaky-tests","color":"0e8a16","default":false},{"id":1301347167,"node_id":"MDU6TGFiZWwxMzAxMzQ3MTY3","url":"https://api.github.com/repos/pytorch/pytorch/labels/triaged","name":"triaged","color":"006b75","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2019-08-26T08:14:42Z","updated_at":"2019-08-29T09:38:52Z","closed_at":null,"author_association":"NONE","body":"## 🐛 Bug\r\n\r\n```\r\n    @unittest.skipIf(not TEST_MAGMA, \"no MAGMA library detected\")\r\n    def test_det_logdet_slogdet_batched(self):\r\n        _TestTorchMixin._test_det_logdet_slogdet_batched(self, 'cuda')\r\n```\r\n\r\nhttps://github.com/pytorch/pytorch/blob/v1.2.0/test/test_cuda.py#L2224 fails for me on ppc64le.\r\n\r\nOn following the test through to https://github.com/pytorch/pytorch/blob/v1.2.0/test/test_torch.py#L6558 and then testing all the different possible combinations, the failures are seen for all combinations of `matsize` and `batchdims` in both `run_test(matsize, batchdims, mat_chars=['sing'])` and `run_test(matsize, batchdims, mat_chars=['sing', 'non_sing'])` (the other tests in there are fine).\r\n\r\nI went through and commented out the assert statements and printed out the tensors that were being compared for each of the combinations:\r\n```\r\n3\r\n(3,)\r\n    run_test(matsize, batchdims, mat_chars=['sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([-2.5175e-35, -1.5868e-38, -9.0737e-36], device='cuda:0')\r\ntensor([-4.0940e-35,  0.0000e+00,  0.0000e+00], device='cuda:0')\r\n\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([nan, nan, nan], device='cuda:0')\r\ntensor([nan, -inf, -inf], device='cuda:0')\r\n\r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([-1.,  0.,  0.], device='cuda:0')\r\ntensor([-1., -1., -1.], device='cuda:0')\r\n\r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([-79.1810,     -inf,     -inf], device='cuda:0')\r\ntensor([-79.6672, -87.0365, -80.6877], device='cuda:0')\r\n\r\n    run_test(matsize, batchdims, mat_chars=['sing', 'non_sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([ 3.8190e-34, -3.9619e-01, -3.5083e-33], device='cuda:0')\r\ntensor([ 6.4256e-34, -3.9619e-01, -1.0143e-32], device='cuda:0')\r\n\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([-76.9479,      nan,      nan], device='cuda:0')\r\ntensor([-76.4276,      nan,      nan], device='cuda:0')\r\n\r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([ 1., -1., -1.], device='cuda:0')\r\ntensor([ 1., -1., -1.], device='cuda:0')\r\n\r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([-76.4276,  -0.9259, -73.6685], device='cuda:0')\r\ntensor([-76.9479,  -0.9259, -74.7302], device='cuda:0')\r\n\r\n\r\n3 \r\n(5, 3)\r\n    run_test(matsize, batchdims, mat_chars=['sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[ 1.7500e-33, -3.9282e-36,  4.5183e-36],\r\n        [ 9.6257e-34,  8.1698e-35, -1.6681e-40],\r\n        [ 1.2369e-34,  1.6281e-35, -1.5470e-33],\r\n        [ 2.4720e-33, -2.6346e-33, -6.8148e-35],\r\n        [ 3.7382e-35, -1.8944e-33,  5.6313e-35]], device='cuda:0')\r\ntensor([[ 0.0000e+00,  0.0000e+00, -0.0000e+00],\r\n        [ 1.0046e-33,  6.8543e-35, -0.0000e+00],\r\n        [-0.0000e+00, -0.0000e+00,  0.0000e+00],\r\n        [ 2.3483e-33,  0.0000e+00,  0.0000e+00],\r\n        [ 0.0000e+00, -2.2742e-33,  0.0000e+00]], device='cuda:0')\r\n  \r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[-75.4257,      nan, -81.3849],\r\n        [-76.0235, -78.4900,      nan],\r\n        [-78.0753, -80.1031,      nan],\r\n        [-75.0803,      nan,      nan],\r\n        [-79.2719,      nan, -78.8621]], device='cuda:0')\r\ntensor([[    -inf,     -inf,     -inf],\r\n        [-75.9808, -78.6656,     -inf],\r\n        [    -inf,     -inf,     -inf],\r\n        [-75.1316,     -inf,     -inf],\r\n        [    -inf,      nan,     -inf]], device='cuda:0')\r\n  \r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([[ 0.,  0., -0.],\r\n        [ 1.,  1., -0.],\r\n        [-0., -0.,  0.],\r\n        [ 1.,  0.,  0.],\r\n        [ 0., -1.,  0.]], device='cuda:0')\r\ntensor([[ 1., -1.,  1.],\r\n        [ 1.,  1., -1.],\r\n        [ 1.,  1., -1.],\r\n        [ 1., -1., -1.],\r\n        [ 1., -1.,  1.]], device='cuda:0')\r\n  \r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([[    -inf,     -inf,     -inf],\r\n        [-75.9808, -78.6656,     -inf],\r\n        [    -inf,     -inf,     -inf],\r\n        [-75.1316,     -inf,     -inf],\r\n        [    -inf, -75.1637,     -inf]], device='cuda:0')\r\ntensor([[-75.4257, -81.5249, -81.3849],\r\n        [-76.0235, -78.4900, -91.5917],\r\n        [-78.0753, -80.1031, -75.5490],\r\n        [-75.0803, -75.0166, -78.6714],\r\n        [-79.2719, -75.3464, -78.8621]], device='cuda:0')\r\n\r\n    run_test(matsize, batchdims, mat_chars=['sing', 'non_sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[-7.2073e-35, -9.6858e-01,  2.1245e-34],\r\n        [-2.1929e-02, -3.3690e-33,  8.4981e-02],\r\n        [ 2.3438e-36,  8.2995e-01, -4.4813e-33],\r\n        [ 2.1107e+00,  3.5198e-34, -1.5583e+00],\r\n        [ 1.3380e-36,  2.1991e+00, -2.2806e-34]], device='cuda:0')\r\ntensor([[-0.0000e+00, -9.6858e-01,  0.0000e+00],\r\n        [-2.1929e-02, -2.5940e-33,  8.4981e-02],\r\n        [-0.0000e+00,  8.2995e-01, -0.0000e+00],\r\n        [ 2.1107e+00,  3.0731e-34, -1.5583e+00],\r\n        [ 2.6372e-36,  2.1991e+00, -0.0000e+00]], device='cuda:0')\r\n\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[     nan,      nan, -77.5343],\r\n        [     nan,      nan,  -2.4653],\r\n        [-82.0413,  -0.1864,      nan],\r\n        [  0.7470, -77.0295,      nan],\r\n        [-82.6019,   0.7880,      nan]], device='cuda:0')\r\ntensor([[    -inf,      nan,     -inf],\r\n        [     nan,      nan,  -2.4653],\r\n        [    -inf,  -0.1864,     -inf],\r\n        [  0.7470, -77.1652,      nan],\r\n        [-81.9234,   0.7880,     -inf]], device='cuda:0')\r\n\r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([[-0., -1.,  0.],\r\n        [-1., -1.,  1.],\r\n        [-0.,  1., -0.],\r\n        [ 1.,  1., -1.],\r\n        [ 1.,  1., -0.]], device='cuda:0')\r\ntensor([[-1., -1.,  1.],\r\n        [-1., -1.,  1.],\r\n        [ 1.,  1., -1.],\r\n        [ 1.,  1., -1.],\r\n        [ 1.,  1., -1.]], device='cuda:0')\r\n\r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([[       -inf, -3.1922e-02,        -inf],\r\n        [-3.8199e+00, -7.5032e+01, -2.4653e+00],\r\n        [       -inf, -1.8638e-01,        -inf],\r\n        [ 7.4703e-01, -7.7165e+01,  4.4361e-01],\r\n        [-8.1923e+01,  7.8803e-01,        -inf]], device='cuda:0')\r\ntensor([[-7.8615e+01, -3.1922e-02, -7.7534e+01],\r\n        [-3.8199e+00, -7.4771e+01, -2.4653e+00],\r\n        [-8.2041e+01, -1.8638e-01, -7.4485e+01],\r\n        [ 7.4703e-01, -7.7029e+01,  4.4361e-01],\r\n        [-8.2602e+01,  7.8803e-01, -7.7463e+01]], device='cuda:0')\r\n\r\n\r\n5\r\n(3,)\r\n    run_test(matsize, batchdims, mat_chars=['sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([ 1.0570e-49, -1.6987e-49,  9.0382e-51], device='cuda:0')\r\ntensor([ 0.0000e+00, -4.2903e-49,  3.6495e-50], device='cuda:0')\r\n\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([-112.7712,       nan, -115.2304], device='cuda:0')\r\ntensor([     -inf,       nan, -113.8347], device='cuda:0')\r\n\r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([ 0., -1.,  1.], device='cuda:0')\r\ntensor([ 1., -1.,  1.], device='cuda:0')\r\n  \r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([     -inf, -111.3703, -113.8347], device='cuda:0')\r\ntensor([-112.7712, -112.2968, -115.2304], device='cuda:0')\r\n  \r\n    run_test(matsize, batchdims, mat_chars=['sing', 'non_sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([-7.0089e-49, -2.0986e+00,  6.3015e-51], device='cuda:0')\r\ntensor([-6.6740e-49, -2.0986e+00,  1.0697e-49], device='cuda:0')\r\n  \r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([      nan,       nan, -115.5911], device='cuda:0')\r\ntensor([      nan,       nan, -112.7593], device='cuda:0')\r\n  \r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([-1., -1.,  1.], device='cuda:0')\r\ntensor([-1., -1.,  1.], device='cuda:0')\r\n  \r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([-110.9285,    0.7413, -112.7593], device='cuda:0')\r\ntensor([-110.8795,    0.7413, -115.5911], device='cuda:0')\r\n\r\n\r\n5 \r\n(5, 3)\r\n    run_test(matsize, batchdims, mat_chars=['sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[-5.4639e-49,  1.0809e-48, -1.4355e-48],\r\n        [ 4.0162e-50, -1.6792e-49, -1.4588e-49],\r\n        [ 4.5016e-49, -6.4845e-49,  4.9028e-49],\r\n        [ 8.8669e-49, -1.1956e-50, -9.0934e-50],\r\n        [ 1.0180e-50, -2.3492e-49,  2.9704e-50]], device='cuda:0')\r\ntensor([[-0.0000e+00,  1.0751e-48,  5.9039e-49],\r\n        [ 0.0000e+00, -4.9508e-50,  0.0000e+00],\r\n        [ 5.8240e-49, -1.3868e-48,  0.0000e+00],\r\n        [ 8.6981e-49, -1.3410e-49,  0.0000e+00],\r\n        [-6.8240e-51, -1.2646e-48, -3.6321e-49]], device='cuda:0')\r\n  \r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[      nan, -110.4463,       nan],\r\n        [-113.7389,       nan,       nan],\r\n        [-111.3222,       nan, -111.2369],\r\n        [-110.6443,       nan,       nan],\r\n        [-115.1115,       nan, -114.0406]], device='cuda:0')\r\ntensor([[     -inf, -110.4516, -111.0511],\r\n        [     -inf,       nan,      -inf],\r\n        [-111.0647,       nan,      -inf],\r\n        [-110.6636,       nan,      -inf],\r\n        [      nan,       nan,       nan]], device='cuda:0')\r\n  \r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([[-0.,  1.,  1.],\r\n        [ 0., -1.,  0.],\r\n        [ 1., -1.,  0.],\r\n        [ 1., -1.,  0.],\r\n        [-1., -1., -1.]], device='cuda:0')\r\ntensor([[-1.,  1., -1.],\r\n        [ 1., -1., -1.],\r\n        [ 1., -1.,  1.],\r\n        [ 1., -1., -1.],\r\n        [ 1., -1.,  1.]], device='cuda:0')\r\n  \r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([[     -inf, -110.4516, -111.0511],\r\n        [     -inf, -113.5297,      -inf],\r\n        [-111.0647, -110.1971,      -inf],\r\n        [-110.6636, -112.5332,      -inf],\r\n        [-115.5114, -110.2894, -111.5369]], device='cuda:0')\r\ntensor([[-111.1285, -110.4463, -110.1625],\r\n        [-113.7389, -112.3083, -112.4491],\r\n        [-111.3222, -110.9573, -111.2369],\r\n        [-110.6443, -114.9506, -112.9217],\r\n        [-115.1115, -111.9726, -114.0406]], device='cuda:0')\r\n  \r\n    run_test(matsize, batchdims, mat_chars=['sing', 'non_sing'])\r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[ 2.0255e-49, -4.7865e-01,  5.8494e-50],\r\n        [-3.8510e+00,  2.4657e-49,  4.6330e-01],\r\n        [-9.1485e-51,  2.4230e-02, -5.0224e-49],\r\n        [ 7.1769e-01, -1.7340e-49, -4.0999e+00],\r\n        [-1.5233e-48, -5.1991e-01,  1.5723e-49]], device='cuda:0')\r\ntensor([[ 4.0698e-49, -4.7865e-01,  0.0000e+00],\r\n        [-3.8510e+00, -4.9829e-49,  4.6330e-01],\r\n        [ 0.0000e+00,  2.4230e-02,  0.0000e+00],\r\n        [ 7.1769e-01,  0.0000e+00, -4.0999e+00],\r\n        [ 0.0000e+00, -5.1991e-01, -2.2120e-49]], device='cuda:0')\r\n  \r\n        self.assertEqual(actual_value, expected_value, allow_inf=True)\r\ntensor([[-112.1209,       nan, -113.3629],\r\n        [      nan, -111.9242,   -0.7694],\r\n        [      nan,   -3.7202,       nan],\r\n        [  -0.3317,       nan,       nan],\r\n        [      nan,       nan, -112.3741]], device='cuda:0')\r\ntensor([[-111.4231,       nan,      -inf],\r\n        [      nan,       nan,   -0.7694],\r\n        [     -inf,   -3.7202,      -inf],\r\n        [  -0.3317,      -inf,       nan],\r\n        [     -inf,       nan,       nan]], device='cuda:0')\r\n  \r\n        self.assertEqual(sign_value, actual_value[0], allow_inf=True)\r\ntensor([[ 1., -1.,  0.],\r\n        [-1., -1.,  1.],\r\n        [ 0.,  1.,  0.],\r\n        [ 1.,  0., -1.],\r\n        [ 0., -1., -1.]], device='cuda:0')\r\ntensor([[ 1., -1.,  1.],\r\n        [-1.,  1.,  1.],\r\n        [-1.,  1., -1.],\r\n        [ 1., -1., -1.],\r\n        [-1., -1.,  1.]], device='cuda:0')\r\n  \r\n        self.assertEqual(expected_value, actual_value[1], allow_inf=True)\r\ntensor([[-111.4231,   -0.7368,      -inf],\r\n        [   1.3483, -111.2207,   -0.7694],\r\n        [     -inf,   -3.7202,      -inf],\r\n        [  -0.3317,      -inf,    1.4110],\r\n        [     -inf,   -0.6541, -112.0328]], device='cuda:0')\r\ntensor([[-112.1209,   -0.7368, -113.3629],\r\n        [   1.3483, -111.9242,   -0.7694],\r\n        [-115.2183,   -3.7202, -111.2128],\r\n        [  -0.3317, -112.2762,    1.4110],\r\n        [-110.1032,   -0.6541, -112.3741]], device='cuda:0')\r\n```\r\n\r\n\r\n## To Reproduce\r\n\r\nSteps to reproduce the behavior:\r\n\r\n1. Build from source on ppc64le\r\n1. Run the tests\r\n\r\n## Expected behavior\r\n\r\nThe test should pass\r\n\r\n## Environment\r\n\r\nPyTorch version: 1.2.0\r\nIs debug build: No\r\nCUDA used to build PyTorch: 10.1.105\r\n\r\nOS: Red Hat Enterprise Linux\r\nGCC version: (GCC) 8.2.0\r\nCMake version: version 3.13.3\r\n\r\nPython version: 3.7\r\nIs CUDA available: Yes\r\nCUDA runtime version: 10.1.105\r\nGPU models and configuration: \r\nGPU 0: Tesla V100-SXM2-16GB\r\nGPU 1: Tesla V100-SXM2-16GB\r\n\r\nNvidia driver version: 418.67\r\ncuDNN version: 7.4.2.24\r\nmagma version: 7.5.1\r\n\r\nVersions of relevant libraries:\r\n[pip3] numpy==1.16.2\r\n[pip3] torch==1.2.0\r\n[conda] Could not collect\r\n\r\nPyTorch built from source\r\nBuild command: PYTORCH_BUILD_VERSION=1.2.0 PYTORCH_BUILD_NUMBER=1 VERBOSE=1 LDFLAGS=\"$LDFLAGS -ldl\" CFLAGS=\"\" CXXFLAGS=\"\" USE_FFMPEG=ON USE_GLOO_IBVERBS=1 USE_GFLAGS=ON USE_GLOG=ON CUDNN_LIB_DIR=$EBROOTCUDNN/lib64 CUDNN_INCLUDE_DIR=$EBROOTCUDNN/include TORCH_CUDA_ARCH_LIST=\"3.5 6.0 7.0\"  /rds/bear-apps/devel/2019a/branfosj-eb-pytorch/EL7/EL7-power9/software/Python/3.7.2-GCCcore-8.2.0/bin/python setup.py build\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/526108686","html_url":"https://github.com/pytorch/pytorch/issues/25172#issuecomment-526108686","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/25172","id":526108686,"node_id":"MDEyOklzc3VlQ29tbWVudDUyNjEwODY4Ng==","user":{"login":"branfosj","id":4967,"node_id":"MDQ6VXNlcjQ5Njc=","avatar_url":"https://avatars1.githubusercontent.com/u/4967?v=4","gravatar_id":"","url":"https://api.github.com/users/branfosj","html_url":"https://github.com/branfosj","followers_url":"https://api.github.com/users/branfosj/followers","following_url":"https://api.github.com/users/branfosj/following{/other_user}","gists_url":"https://api.github.com/users/branfosj/gists{/gist_id}","starred_url":"https://api.github.com/users/branfosj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/branfosj/subscriptions","organizations_url":"https://api.github.com/users/branfosj/orgs","repos_url":"https://api.github.com/users/branfosj/repos","events_url":"https://api.github.com/users/branfosj/events{/privacy}","received_events_url":"https://api.github.com/users/branfosj/received_events","type":"User","site_admin":false},"created_at":"2019-08-29T09:38:52Z","updated_at":"2019-08-29T09:38:52Z","author_association":"NONE","body":"The determinants of those three are: -1.0408340855860843e-17, -2.168404344971009e-19, and 0. So two are very close to being singular and one is. This means that there'll be particularly susceptible to accumulation of numerical errors during the decomposition (apply_lu) process and differences in pivot choices is a strong possibility for the differences we are seeing in results."}}
other: {"actor":{"display_login":"branfosj"}}