payload: {"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/26572","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/26572/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/26572/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/26572/events","html_url":"https://github.com/tensorflow/tensorflow/issues/26572","id":419440606,"node_id":"MDU6SXNzdWU0MTk0NDA2MDY=","number":26572,"title":"Cuda 10: Failed to get device properties, error code: 3","user":{"login":"huljar","id":24304044,"node_id":"MDQ6VXNlcjI0MzA0MDQ0","avatar_url":"https://avatars2.githubusercontent.com/u/24304044?v=4","gravatar_id":"","url":"https://api.github.com/users/huljar","html_url":"https://github.com/huljar","followers_url":"https://api.github.com/users/huljar/followers","following_url":"https://api.github.com/users/huljar/following{/other_user}","gists_url":"https://api.github.com/users/huljar/gists{/gist_id}","starred_url":"https://api.github.com/users/huljar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huljar/subscriptions","organizations_url":"https://api.github.com/users/huljar/orgs","repos_url":"https://api.github.com/users/huljar/repos","events_url":"https://api.github.com/users/huljar/events{/privacy}","received_events_url":"https://api.github.com/users/huljar/received_events","type":"User","site_admin":false},"labels":[{"id":1097546578,"node_id":"MDU6TGFiZWwxMDk3NTQ2NTc4","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/comp:keras","name":"comp:keras","color":"0052cc","default":false},{"id":404586594,"node_id":"MDU6TGFiZWw0MDQ1ODY1OTQ=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"f4b400","default":false},{"id":473172988,"node_id":"MDU6TGFiZWw0NzMxNzI5ODg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"159b2e","default":false}],"state":"open","locked":false,"assignee":{"login":"jch1","id":1811449,"node_id":"MDQ6VXNlcjE4MTE0NDk=","avatar_url":"https://avatars2.githubusercontent.com/u/1811449?v=4","gravatar_id":"","url":"https://api.github.com/users/jch1","html_url":"https://github.com/jch1","followers_url":"https://api.github.com/users/jch1/followers","following_url":"https://api.github.com/users/jch1/following{/other_user}","gists_url":"https://api.github.com/users/jch1/gists{/gist_id}","starred_url":"https://api.github.com/users/jch1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jch1/subscriptions","organizations_url":"https://api.github.com/users/jch1/orgs","repos_url":"https://api.github.com/users/jch1/repos","events_url":"https://api.github.com/users/jch1/events{/privacy}","received_events_url":"https://api.github.com/users/jch1/received_events","type":"User","site_admin":false},"assignees":[{"login":"jch1","id":1811449,"node_id":"MDQ6VXNlcjE4MTE0NDk=","avatar_url":"https://avatars2.githubusercontent.com/u/1811449?v=4","gravatar_id":"","url":"https://api.github.com/users/jch1","html_url":"https://github.com/jch1","followers_url":"https://api.github.com/users/jch1/followers","following_url":"https://api.github.com/users/jch1/following{/other_user}","gists_url":"https://api.github.com/users/jch1/gists{/gist_id}","starred_url":"https://api.github.com/users/jch1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jch1/subscriptions","organizations_url":"https://api.github.com/users/jch1/orgs","repos_url":"https://api.github.com/users/jch1/repos","events_url":"https://api.github.com/users/jch1/events{/privacy}","received_events_url":"https://api.github.com/users/jch1/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2019-03-11T12:20:14Z","updated_at":"2019-05-12T12:23:24Z","closed_at":null,"author_association":"NONE","body":"I am using [keras-retinanet](https://github.com/fizyr/keras-retinanet) (0.5.0) to perform object detection on images using its TensorFlow backend. When I `load_model` their pre-trained COCO model, the GPU is initialized as expected:\r\n```\r\nUsing TensorFlow backend.\r\nWARNING:tensorflow:From .../virtualenv/lib64/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\r\nInstructions for updating:\r\nColocations handled automatically by placer.\r\n2019-03-11 11:26:25.063507: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-03-11 11:26:25.064403: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55c2e09b7bb0 executing computations on platform CUDA. Devices:\r\n2019-03-11 11:26:25.064432: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1\r\n2019-03-11 11:26:25.066831: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2993245000 Hz\r\n2019-03-11 11:26:25.067138: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55c2e06ce580 executing computations on platform Host. Devices:\r\n2019-03-11 11:26:25.067163: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\r\n2019-03-11 11:26:25.068038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:\r\nname: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 10.92GiB freeMemory: 10.77GiB\r\n2019-03-11 11:26:25.068063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\r\n2019-03-11 11:26:25.069446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2019-03-11 11:26:25.069467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0\r\n2019-03-11 11:26:25.069477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N\r\n2019-03-11 11:26:25.070214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10481 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)\r\n.../virtualenv/lib64/python3.6/site-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\r\n  warnings.warn('No training configuration found in save file: '\r\n```\r\n\r\nHowever, when calling `predict_on_batch` on this model afterwards, the GPU cannot be initialized any more:\r\n```\r\n2019-03-11 11:26:52.699096: E tensorflow/core/grappler/clusters/utils.cc:83] Failed to get device properties, error code: 3\r\n2019-03-11 11:26:52.947157: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_2. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:52.947225: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_3. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:52.947245: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_4. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:52.947271: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_5. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:53.215964: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_2. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:53.216024: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_3. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:53.216042: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_4. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:53.216068: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes/strided_slice_5. Error: Pack node (boxes/stack) axis attribute is out of bounds: 2 \r\n2019-03-11 11:26:53.354507: E tensorflow/stream_executor/cuda/cuda_driver.cc:1131] failed to enqueue async memcpy from host to device: CUDA_ERROR_NOT_INITIALIZED: initialization error; GPU dst: 0x7fc4f4b62900; host src: 0x7fc77b201a00; size: 4=0x4\r\n```\r\n\r\n**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no custom code, but TensorFlow is called through keras-retinanet\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Gentoo (Base System release 2.6)\r\n- TensorFlow installed from (source or binary): binary from PyPi\r\n- TensorFlow version: 1.13.1\r\n- Python version: 3.6.5\r\n- CUDA/cuDNN version: 10.0.130 / 7.4.2.24\r\n- GPU model and memory: Nvidia GeForce GTX 1080 Ti (11 GB memory)\r\n- Nvidia driver version: 418.43\r\n\r\n**Describe the current behavior**\r\nsee above\r\n\r\n**Describe the expected behavior**\r\nThere should be no error initializing the GPU when using `predict_on_batch`. On other machines I have used older versions (TF 1.10.0 with CUDA 9.0) and it worked fine.\r\n\r\n**Code to reproduce the issue**\r\n* Install CUDA 10, cuDNN 7.4.2\r\n* Install tensorflow-gpu 1.13.1 (or 1.13.0rc1/rc2) and keras-retinanet 0.5.0 into a virtualenv from PyPi\r\n* Download pre-trained COCO model from [here](https://github.com/fizyr/keras-retinanet/releases/download/0.5.0/resnet50_coco_best_v2.1.0.h5)\r\n* Load the model using `model = keras_retinanet.models.load_model('path/to/model.h5')` (will generate first blob of output above)\r\n* Predict bounding boxes on an image via `model.predict_on_batch(np.expand_dims(my_image, axis=0))` (will generate second blob of output above)\r\n\r\n**Other info / logs**\r\nI am reporting this here and not at the keras-retinanet project because it seems to be a TensorFlow internal issue. Furthermore, it has worked fine with the same version of keras-retinanet when using older versions of TF and CUDA (see above).\r\n\r\nThank you!"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/491591046","html_url":"https://github.com/tensorflow/tensorflow/issues/26572#issuecomment-491591046","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/26572","id":491591046,"node_id":"MDEyOklzc3VlQ29tbWVudDQ5MTU5MTA0Ng==","user":{"login":"shaulimaya","id":15960536,"node_id":"MDQ6VXNlcjE1OTYwNTM2","avatar_url":"https://avatars0.githubusercontent.com/u/15960536?v=4","gravatar_id":"","url":"https://api.github.com/users/shaulimaya","html_url":"https://github.com/shaulimaya","followers_url":"https://api.github.com/users/shaulimaya/followers","following_url":"https://api.github.com/users/shaulimaya/following{/other_user}","gists_url":"https://api.github.com/users/shaulimaya/gists{/gist_id}","starred_url":"https://api.github.com/users/shaulimaya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shaulimaya/subscriptions","organizations_url":"https://api.github.com/users/shaulimaya/orgs","repos_url":"https://api.github.com/users/shaulimaya/repos","events_url":"https://api.github.com/users/shaulimaya/events{/privacy}","received_events_url":"https://api.github.com/users/shaulimaya/received_events","type":"User","site_admin":false},"created_at":"2019-05-12T12:23:24Z","updated_at":"2019-05-12T12:23:24Z","author_association":"NONE","body":"I have the same issue while running  keras-retinanet after upgraded to cuda 10 t.f. 1.13.1 python 3.6.8"}}
other: {"actor":{"display_login":"shaulimaya"}}