payload: {"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/14496","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/14496/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/14496/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/14496/events","html_url":"https://github.com/apache/incubator-mxnet/issues/14496","id":423893087,"node_id":"MDU6SXNzdWU0MjM4OTMwODc=","number":14496,"title":"performance degradation from 1.3.1 to 1.4.0","user":{"login":"samskalicky","id":1541993,"node_id":"MDQ6VXNlcjE1NDE5OTM=","avatar_url":"https://avatars2.githubusercontent.com/u/1541993?v=4","gravatar_id":"","url":"https://api.github.com/users/samskalicky","html_url":"https://github.com/samskalicky","followers_url":"https://api.github.com/users/samskalicky/followers","following_url":"https://api.github.com/users/samskalicky/following{/other_user}","gists_url":"https://api.github.com/users/samskalicky/gists{/gist_id}","starred_url":"https://api.github.com/users/samskalicky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samskalicky/subscriptions","organizations_url":"https://api.github.com/users/samskalicky/orgs","repos_url":"https://api.github.com/users/samskalicky/repos","events_url":"https://api.github.com/users/samskalicky/events{/privacy}","received_events_url":"https://api.github.com/users/samskalicky/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2019-03-21T19:00:53Z","updated_at":"2019-03-21T19:01:35Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"There appears to be some performance degradation between the 1.3.1 and 1.4.0 releases. So far we know of imdecode and tranpose operators having reduced performance.\r\n\r\nWe've tracked the responsible PRs for these operators to:\r\nTranspose:\r\nhttps://github.com/dmlc/mshadow/pull/359/files\r\nhttps://github.com/apache/incubator-mxnet/pull/11742\r\n\r\nImdecode:\r\nhttps://github.com/apache/incubator-mxnet/pull/8757\r\n\r\nIm using the cat.jpg from here as input data:\r\nhttps://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg\r\n\r\nHeres the current performance benchmarking script for imdecode:\r\n\r\n```\r\nimport time\r\nimport mxnet as mx\r\nfrom mxnet import image as img\r\nimport numpy as np\r\n\r\nflag=1\r\nto_rgb=True\r\nout=None\r\n\r\ndtimes = []\r\nfor i in range(1000):\r\n    buf = open(\"cat.jpg\", 'rb').read()\r\n\r\n    start = time.time()\r\n    img.imdecode(buf, flag, to_rgb, out)\r\n    decode = (time.time() - start)*1000\r\n    dtimes.append(decode)\r\n\r\nprint('mxnet version: %s' % mx.__version__)\r\nprint('decode:---------')\r\nprint('p50: %4.2f ms' % np.percentile(dtimes,50))\r\nprint('p90: %4.2f ms' % np.percentile(dtimes,90))\r\nprint('p99: %4.2f ms' % np.percentile(dtimes,99))\r\n```\r\n\r\nAnd here are the performance results using mxnet-cu90mkl for 1.4.0 and 1.3.1:\r\n\r\nmxnet version: 1.4.0\r\ndecode:--------- \r\np50: 13.75 ms \r\np90: 15.74 ms \r\np99: 19.89 ms \r\n\r\nmxnet version: 1.3.1 \r\ndecode:--------- \r\np50: 12.82 ms \r\np90: 13.59 ms \r\np99: 13.88 ms \r\n\r\nSetting the flag = 1 + 128 (instead of just 1) as an argument to imdecode results in the following results: \r\n\r\nmxnet version: 1.4.0 \r\nload:--------- \r\np50: 0.10 ms \r\np90: 0.16 ms \r\np99: 0.18 ms \r\ndecode:--------- \r\np50: 13.27 ms \r\np90: 13.83 ms \r\np99: 15.63 ms \r\n\r\nSo there is some additional work thats going on that makes the imdecode take longer. This is a \"feature\" of the v1.4 release and changed the defaults which is where some performance degradation is happening at least in the imdecode function. \r\n\r\nHeres the current performance benchmarking script for transpose:\r\n\r\n```\r\nimport mxnet as mx\r\nimport time\r\nimport numpy as np\r\n\r\nsizes = [10, 50, 100,200,500]\r\niters = [10000,1000,500,200,20]\r\ntimes = []\r\nfor size in range(len(sizes)):\r\n    data = []\r\n    s = sizes[size]\r\n    print(s)\r\n    for i in range(iters[size]):\r\n        x = mx.nd.ones((s,s,s))\r\n        mx.nd.waitall()\r\n        start = time.time()\r\n        y = mx.nd.transpose(x,(2,0,1))\r\n        mx.nd.waitall()\r\n        data.append((time.time() - start)*1000)\r\n        #print(data[-1])                                                                                                                                                                            \r\n    times.append(data)\r\n\r\nprint('mxnet version: %s' % mx.__version__)\r\nfor s in range(len(sizes)):\r\n    print('--------------------')\r\n    print('size: %s' % str(sizes[s]))\r\n    print('p50: %4.2f ms' % np.percentile(times[s],50))\r\n    print('p90: %4.2f ms' % np.percentile(times[s],90))\r\n    print('p99: %4.2f ms' % np.percentile(times[s],99))\r\n```\r\n\r\nAnd here are the performance results using mxnet-cu90mkl for 1.4.0 and 1.3.1:\r\n\r\nmxnet version: 1.4.0 \r\n-------------------- \r\nsize: 10 \r\np50: 0.04 ms \r\np90: 0.04 ms \r\np99: 0.05 ms \r\n-------------------- \r\nsize: 50 \r\np50: 1.08 ms \r\np90: 1.09 ms \r\np99: 1.13 ms \r\n-------------------- \r\nsize: 100 \r\np50: 12.50 ms \r\np90: 12.53 ms \r\np99: 13.52 ms \r\n-------------------- \r\nsize: 200 \r\np50: 123.06 ms \r\np90: 125.63 ms \r\np99: 125.95 ms \r\n-------------------- \r\nsize: 500 \r\np50: 2768.49 ms \r\np90: 2797.46 ms \r\np99: 2809.45 ms \r\n\r\n\r\nmxnet version: 1.3.1 \r\n-------------------- \r\nsize: 10 \r\np50: 0.03 ms \r\np90: 0.04 ms \r\np99: 0.04 ms \r\n-------------------- \r\nsize: 50 \r\np50: 0.36 ms \r\np90: 0.37 ms \r\np99: 0.37 ms \r\n-------------------- \r\nsize: 100 \r\np50: 2.79 ms \r\np90: 2.90 ms \r\np99: 3.97 ms \r\n-------------------- \r\nsize: 200 \r\np50: 46.05 ms \r\np90: 50.07 ms \r\np99: 50.11 ms \r\n-------------------- \r\nsize: 500 \r\np50: 1094.50 ms \r\np90: 1095.89 ms \r\np99: 1096.44 ms \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments/475362261","html_url":"https://github.com/apache/incubator-mxnet/issues/14496#issuecomment-475362261","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/14496","id":475362261,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3NTM2MjI2MQ==","user":{"login":"samskalicky","id":1541993,"node_id":"MDQ6VXNlcjE1NDE5OTM=","avatar_url":"https://avatars2.githubusercontent.com/u/1541993?v=4","gravatar_id":"","url":"https://api.github.com/users/samskalicky","html_url":"https://github.com/samskalicky","followers_url":"https://api.github.com/users/samskalicky/followers","following_url":"https://api.github.com/users/samskalicky/following{/other_user}","gists_url":"https://api.github.com/users/samskalicky/gists{/gist_id}","starred_url":"https://api.github.com/users/samskalicky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samskalicky/subscriptions","organizations_url":"https://api.github.com/users/samskalicky/orgs","repos_url":"https://api.github.com/users/samskalicky/repos","events_url":"https://api.github.com/users/samskalicky/events{/privacy}","received_events_url":"https://api.github.com/users/samskalicky/received_events","type":"User","site_admin":false},"created_at":"2019-03-21T19:01:35Z","updated_at":"2019-03-21T19:01:35Z","author_association":"CONTRIBUTOR","body":"So question for the community, is are others seeing performance issues moving from 1.3.1 to 1.4.0?\r\n\r\nAnd for what operators do you see issue?"}}
other: {"actor":{"display_login":"samskalicky"}}