payload: {"action":"opened","issue":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/5243","repository_url":"https://api.github.com/repos/ofiwg/libfabric","labels_url":"https://api.github.com/repos/ofiwg/libfabric/issues/5243/labels{/name}","comments_url":"https://api.github.com/repos/ofiwg/libfabric/issues/5243/comments","events_url":"https://api.github.com/repos/ofiwg/libfabric/issues/5243/events","html_url":"https://github.com/ofiwg/libfabric/issues/5243","id":489028834,"node_id":"MDU6SXNzdWU0ODkwMjg4MzQ=","number":5243,"title":"[PSM2] Unhandled exception [PSM2_MQ_SENDREQS_MAX & PSM2_MQ_RECVREQS_MAX]","user":{"login":"fsalem","id":11193987,"node_id":"MDQ6VXNlcjExMTkzOTg3","avatar_url":"https://avatars3.githubusercontent.com/u/11193987?v=4","gravatar_id":"","url":"https://api.github.com/users/fsalem","html_url":"https://github.com/fsalem","followers_url":"https://api.github.com/users/fsalem/followers","following_url":"https://api.github.com/users/fsalem/following{/other_user}","gists_url":"https://api.github.com/users/fsalem/gists{/gist_id}","starred_url":"https://api.github.com/users/fsalem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fsalem/subscriptions","organizations_url":"https://api.github.com/users/fsalem/orgs","repos_url":"https://api.github.com/users/fsalem/repos","events_url":"https://api.github.com/users/fsalem/events{/privacy}","received_events_url":"https://api.github.com/users/fsalem/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-09-04T08:54:29Z","updated_at":"2019-09-04T08:54:29Z","closed_at":null,"author_association":"NONE","body":"While sending RDMA messages as fast as possible between client and server, I get one of these two exceptions (which are because of the same reason):\r\n- Exhausted 1048576 MQ isend request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM2_MQ_SENDREQS_MAX=1048576)\r\nlibfabric:135750:psm2:core:psmx2_fabric_close():48<info> refcnt=3\r\nterminate called without an active exception\r\n\r\n- Exhausted 1048576 MQ irecv request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM2_MQ_RECVREQS_MAX=1048576)\r\nlibfabric:135750:psm2:core:psmx2_fabric_close():48<info> refcnt=3\r\nterminate called without an active exception\r\n\r\n\r\nThe implementation is based on sending as many RDMA messages as possible from a client to a server. The client collects the completion events (FI_COMPLETION) for time to time, which limits the injection rate. But unlike the GNI Provider, it seems that the completion events are generated when the message left the source message and not delivered to the destination. \r\n\r\n- Is there any way in Libfabric to check the current isend or irecv counter in order to limit the injection rate?\r\n- Does PSM2 provider handle this limitation internally?"}}
other: {"actor":{"display_login":"fsalem"}}