payload: {"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/nccl/issues/246","repository_url":"https://api.github.com/repos/NVIDIA/nccl","labels_url":"https://api.github.com/repos/NVIDIA/nccl/issues/246/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/nccl/issues/246/comments","events_url":"https://api.github.com/repos/NVIDIA/nccl/issues/246/events","html_url":"https://github.com/NVIDIA/nccl/issues/246","id":481959742,"node_id":"MDU6SXNzdWU0ODE5NTk3NDI=","number":246,"title":"the mismatch of GPU and NIC's topology shown by NCCL runtime INFO level log and `nvidia-smi topo -m`","user":{"login":"Keepmoving-ZXY","id":19542656,"node_id":"MDQ6VXNlcjE5NTQyNjU2","avatar_url":"https://avatars2.githubusercontent.com/u/19542656?v=4","gravatar_id":"","url":"https://api.github.com/users/Keepmoving-ZXY","html_url":"https://github.com/Keepmoving-ZXY","followers_url":"https://api.github.com/users/Keepmoving-ZXY/followers","following_url":"https://api.github.com/users/Keepmoving-ZXY/following{/other_user}","gists_url":"https://api.github.com/users/Keepmoving-ZXY/gists{/gist_id}","starred_url":"https://api.github.com/users/Keepmoving-ZXY/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Keepmoving-ZXY/subscriptions","organizations_url":"https://api.github.com/users/Keepmoving-ZXY/orgs","repos_url":"https://api.github.com/users/Keepmoving-ZXY/repos","events_url":"https://api.github.com/users/Keepmoving-ZXY/events{/privacy}","received_events_url":"https://api.github.com/users/Keepmoving-ZXY/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2019-08-18T06:28:36Z","updated_at":"2019-08-20T15:49:59Z","closed_at":null,"author_association":"NONE","body":"Hi, nccl team:\r\n\tThese days I much pay attention to the topology of GPU servers in our compays becasuse we find something wrong about the train speed of deep learning and our gpu server has 8 pieces of  v100 GPUs and a Mellanox RDMA NIC. And yesterday I run distributed train with support of `horovod` and `horovod` is compiled to use NCCL to do reduction,  And I run `horovod` with command line below:\r\n```\r\n/usr/mpi/gcc/openmpi-4.0.2a1/bin/mpirun -np 16 --host 10.0.0.1:8,10.0.0.2:8 \\\r\n\t-v --allow-run-as-root \\\r\n\t-x NCCL_DEBUG=INFO \\\r\n\t-x NCCL_IB_DISABLE=0 \\\r\n\t-x NCCL_IB_HCA=mlx5_0:1 \\\r\n\t-x HOROVOD_MPI_THREADS_DISABLE=1 \\\r\n\t-x UCX_NET_DEVICES=mlx5_0:1 \\\r\n\tpython ./tf_cnn_benchmarks.py --num_gpus=1 --batch_size=256 \\\r\n\t\t\t--model=resnet50 --variable_update=horovod --device=gpu \r\n```\r\nThe `openmpi` is installed by mellanox NIC driver, so I directly run it instead of compile `openmpi` from source code, and I want to see NCCL's debug log, so I add `-x NCCL_DEBUG_INFO` to command line. And during run , I can see NCCL do check the topology of  all v100 GPUs and NIC, and output is:\r\n\r\n\tgpu0:189432:189623 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance :  PIX\r\n\tgpu0:189433:189664 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance :  PIX\r\n\tgpu0:189435:189654 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance :  PIX\r\n\tgpu0:189434:189628 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance :  PIX\r\n\tgpu0:189436:189661 [4] NCCL INFO CUDA Dev 4[4], IB NIC distance :  PXB\r\n\tgpu0:189437:189653 [5] NCCL INFO CUDA Dev 5[5], IB NIC distance :  PXB\r\n\tgpu0:189438:189665 [6] NCCL INFO CUDA Dev 6[6], IB NIC distance :  PXB\r\n\tgpu0:189439:189671 [7] NCCL INFO CUDA Dev 7[7], IB NIC distance :  PXB\r\n\tgpu1:66170:66310 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance :  PIX\r\n\tgpu1:66171:66292 [4] NCCL INFO CUDA Dev 4[4], IB NIC distance :  PXB\r\n\tgpu1:66173:66312 [6] NCCL INFO CUDA Dev 6[6], IB NIC distance :  PXB\r\n\tgpu1:66174:66338 [7] NCCL INFO CUDA Dev 7[7], IB NIC distance :  PXB\r\n\tgpu1:66172:66293 [5] NCCL INFO CUDA Dev 5[5], IB NIC distance :  PXB\r\n\tgpu1:66167:66298 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance :  PIX\r\n\tgpu1:66168:66336 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance :  PIX\r\n\tgpu1:66169:66426 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance :  PIX\r\n\r\nNCCL think the there are two different connection of  all GPUs and NIC: `PIX` and `PXB` but the connection of all GPUs and NIC output by `nvidia-smi topo -m` is:\r\n```\r\n[root@gpu0 ~]# nvidia-smi topo -m\r\n\t   GPU0\tGPU1\tGPU2\tGPU3\tGPU4\tGPU5\tGPU6\tGPU7\tmlx5_0\tmlx5_1\tCPU Affinity\r\nGPU0\t X \tPIX\tPIX\tPIX\tPHB\tPHB\tPHB\tPHB\tPIX\tPIX\t0-11,24-35\r\nGPU1\tPIX\t X \tPIX\tPIX\tPHB\tPHB\tPHB\tPHB\tPIX\tPIX\t0-11,24-35\r\nGPU2\tPIX\tPIX\t X \tPIX\tPHB\tPHB\tPHB\tPHB\tPIX\tPIX\t0-11,24-35\r\nGPU3\tPIX\tPIX\tPIX\t X \tPHB\tPHB\tPHB\tPHB\tPIX\tPIX\t0-11,24-35\r\nGPU4\tPHB\tPHB\tPHB\tPHB\t X \tPIX\tPIX\tPIX\tPHB\tPHB\t0-11,24-35\r\nGPU5\tPHB\tPHB\tPHB\tPHB\tPIX\t X \tPIX\tPIX\tPHB\tPHB\t0-11,24-35\r\nGPU6\tPHB\tPHB\tPHB\tPHB\tPIX\tPIX\t X \tPIX\tPHB\tPHB\t0-11,24-35\r\nGPU7\tPHB\tPHB\tPHB\tPHB\tPIX\tPIX\tPIX\t X \tPHB\tPHB\t0-11,24-35\r\nmlx5_0\tPIX\tPIX\tPIX\tPIX\tPHB\tPHB\tPHB\tPHB\t X \tPIX\t\r\nmlx5_1\tPIX\tPIX\tPIX\tPIX\tPHB\tPHB\tPHB\tPHB\tPIX\t X \t\r\n\r\nLegend:\r\n\r\n  X    = Self\r\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\r\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\r\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\r\n  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)\r\n  PIX  = Connection traversing a single PCIe switch\r\n  NV#  = Connection traversing a bonded set of # NVLinks\r\n[root@gpu0 ~]# \r\n\r\n```\r\nand I notice that connection of some GPU  like GPU 4 and NIC is `PHB`  but nccl think  connection of GPU4 and NIC is `PXB`, there is a mismatch which made me very confused.  So what the real connection between GPU4 and NIC, and I also do not understand the meaning of `PHB` and `PXB`,  could you also explain them for me, thank you.\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/nccl/issues/comments/523076601","html_url":"https://github.com/NVIDIA/nccl/issues/246#issuecomment-523076601","issue_url":"https://api.github.com/repos/NVIDIA/nccl/issues/246","id":523076601,"node_id":"MDEyOklzc3VlQ29tbWVudDUyMzA3NjYwMQ==","user":{"login":"sjeaugey","id":12857445,"node_id":"MDQ6VXNlcjEyODU3NDQ1","avatar_url":"https://avatars1.githubusercontent.com/u/12857445?v=4","gravatar_id":"","url":"https://api.github.com/users/sjeaugey","html_url":"https://github.com/sjeaugey","followers_url":"https://api.github.com/users/sjeaugey/followers","following_url":"https://api.github.com/users/sjeaugey/following{/other_user}","gists_url":"https://api.github.com/users/sjeaugey/gists{/gist_id}","starred_url":"https://api.github.com/users/sjeaugey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sjeaugey/subscriptions","organizations_url":"https://api.github.com/users/sjeaugey/orgs","repos_url":"https://api.github.com/users/sjeaugey/repos","events_url":"https://api.github.com/users/sjeaugey/events{/privacy}","received_events_url":"https://api.github.com/users/sjeaugey/received_events","type":"User","site_admin":false},"created_at":"2019-08-20T15:49:59Z","updated_at":"2019-08-20T15:49:59Z","author_association":"CONTRIBUTOR","body":"`dmidecode` (run as root, e.g. with `sudo`) should give you the motherboard reference under the \"System Information\" section, which should help to find the spec sheet.\r\n\r\nAlso, `lspci -tv` would also give us an idea of the PCI topology."}}
other: {"actor":{"display_login":"sjeaugey"}}