payload: {"action":"created","issue":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/19628","repository_url":"https://api.github.com/repos/PaddlePaddle/Paddle","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/19628/labels{/name}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/19628/comments","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/19628/events","html_url":"https://github.com/PaddlePaddle/Paddle/issues/19628","id":489020212,"node_id":"MDU6SXNzdWU0ODkwMjAyMTI=","number":19628,"title":"1.5和更高版本模型GPU报错","user":{"login":"kahitomi","id":2569549,"node_id":"MDQ6VXNlcjI1Njk1NDk=","avatar_url":"https://avatars3.githubusercontent.com/u/2569549?v=4","gravatar_id":"","url":"https://api.github.com/users/kahitomi","html_url":"https://github.com/kahitomi","followers_url":"https://api.github.com/users/kahitomi/followers","following_url":"https://api.github.com/users/kahitomi/following{/other_user}","gists_url":"https://api.github.com/users/kahitomi/gists{/gist_id}","starred_url":"https://api.github.com/users/kahitomi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kahitomi/subscriptions","organizations_url":"https://api.github.com/users/kahitomi/orgs","repos_url":"https://api.github.com/users/kahitomi/repos","events_url":"https://api.github.com/users/kahitomi/events{/privacy}","received_events_url":"https://api.github.com/users/kahitomi/received_events","type":"User","site_admin":false},"labels":[{"id":651040732,"node_id":"MDU6TGFiZWw2NTEwNDA3MzI=","url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels/User","name":"User","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-09-04T08:36:59Z","updated_at":"2019-09-05T02:13:18Z","closed_at":null,"author_association":"NONE","body":"模型是一个带copy的lstm生成模型\r\n模型在1.3，1.4版本下GPU和CPU都没有问题。\r\n在使用paddle 1.5和lastest版本时候，GPU版本报错，CPU版本正常。GPU报错如下~：\r\n![image](https://user-images.githubusercontent.com/2569549/64237527-18eb5800-cf2f-11e9-8f36-89a879936779.png)\r\n\r\n提示中没有错误的具体位置~\r\n根据提示，寻找sum相关的OP，未果~\r\n最后定位的代码，在DynamicRNN的block()中，如下\r\n```\r\n......\r\n......\r\nwith drnn.block():\r\n......\r\n......\r\n    current_h_expand_seq = pd.reshape(\r\n                current_h, [-1, 1, self.decoder_size])\r\n    current_h_expand_seq = pd.expand(\r\n                current_h_expand_seq, [1, self.max_length, 1])\r\n\r\n    copy_score_sub = pd.elementwise_mul(\r\n                copy_score_weight, current_h_expand_seq, axis=0)\r\n    copy_score = pd.reduce_sum(copy_score_sub, dim=2)\r\n......\r\n......\r\n```\r\n去掉这段代码就不报错了\r\n其中self.decoder_size是解码的lstm size，self.max_length是生成的最大步长\r\n其中current_h是本时间步的LSTM输出，shape为[batch_size, self.decoder_size]\r\n其中copy_score_weight来自encoder端的所有单词，shape为[batch_size, self.max_length, self.decoder_size]\r\n\r\n同时实验去掉了reduce_sum的op，代码是\r\n```\r\n......\r\n......\r\nwith drnn.block():\r\n......\r\n......\r\n    current_h_expand_seq = pd.reshape(\r\n                current_h, [-1, 1, self.decoder_size])\r\n    current_h_expand_seq = pd.expand(\r\n                current_h_expand_seq, [1, self.max_length, 1])\r\n\r\n    copy_score_in = pd.concat([copy_score_weight, current_h_expand_seq], axis=2)\r\n    copy_score_in = pd.reshape(\r\n                copy_score_in, [-1, self.decoder_size * 2])\r\n    copy_score = fluid.layers.fc(input=copy_score_in,\r\n                    act='tanh',\r\n                    size=1,\r\n                    bias_attr=False,\r\n                    param_attr=fluid.ParamAttr(name=\"copy_score_combine_weight_w\"))\r\n    copy_score = pd.reshape(\r\n                copy_score, [-1, self.max_length])\r\n......\r\n......\r\n```\r\n结果报的错和开始是一致的，还是报sum to lod tensor~\r\n\r\n完整报错如下：\r\n```\r\nTraceback (most recent call last):\r\nFile \"train.py\", line 528, in <module>\r\ntrain()\r\nFile \"train.py\", line 467, in train\r\nreturn_numpy=False)\r\nFile \"/home/slurm/job/tmp/job-128277/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/executor.py\", line 651, in run\r\nuse_program_cache=use_program_cache)\r\nFile \"/home/slurm/job/tmp/job-128277/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/executor.py\", line 749, in _run\r\nexe.run(program.desc, scope, 0, True, True, fetch_var_name)\r\npaddle.fluid.core_avx.EnforceNotMet: holder_ should not be null\r\nTensor holds no memory. Call Tensor::mutable_data first. at [/paddle/paddle/fluid/framework/tensor.cc:23]\r\nPaddlePaddle Call Stacks: \r\n0 0x7fc4e021aff8p void paddle::platform::EnforceNotMet::Init<std::string>(std::string, char const*, int) + 360\r\n1 0x7fc4e021b347p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 87\r\n2 0x7fc4e21d8f09p paddle::framework::Tensor::check_memory_size() const + 185\r\n3 0x7fc4e0221c59p float const* paddle::framework::Tensor::data<float>() const + 25\r\n4 0x7fc4e06c38cbp void paddle::operators::SumToLoDTensor<float>(paddle::framework::ExecutionContext const&) + 763\r\n5 0x7fc4e06cca38p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, int>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, long>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 248\r\n6 0x7fc4e2183037p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 375\r\n7 0x7fc4e2183411p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529\r\n8 0x7fc4e2180a0cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332\r\n9 0x7fc4e03a746ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 382\r\n10 0x7fc4e1c2139dp paddle::operators::WhileGradOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 1869\r\n11 0x7fc4e2180a0cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332\r\n12 0x7fc4e03a746ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 382\r\n13 0x7fc4e03aa50fp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocator<std::string> > const&, bool) + 143\r\n14 0x7fc4e020bf8dp\r\n15 0x7fc4e024d936p\r\n16 0x7fc58a29fcc8p PyEval_EvalFrameEx + 28264\r\n17 0x7fc58a2a235dp PyEval_EvalCodeEx + 2061\r\n18 0x7fc58a29fd50p PyEval_EvalFrameEx + 28400\r\n19 0x7fc58a2a235dp PyEval_EvalCodeEx + 2061\r\n20 0x7fc58a29fd50p PyEval_EvalFrameEx + 28400\r\n21 0x7fc58a2a235dp PyEval_EvalCodeEx + 2061\r\n22 0x7fc58a29fd50p PyEval_EvalFrameEx + 28400\r\n23 0x7fc58a2a235dp PyEval_EvalCodeEx + 2061\r\n24 0x7fc58a2a2492p PyEval_EvalCode + 50\r\n25 0x7fc58a2cc1a2p PyRun_FileExFlags + 146\r\n26 0x7fc58a2cd539p PyRun_SimpleFileExFlags + 217\r\n27 0x7fc58a2e31bdp Py_Main + 3149\r\n28 0x7fc5894e0bd5p __libc_start_main + 245\r\n29 0x4007a1p\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments/528165782","html_url":"https://github.com/PaddlePaddle/Paddle/issues/19628#issuecomment-528165782","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/19628","id":528165782,"node_id":"MDEyOklzc3VlQ29tbWVudDUyODE2NTc4Mg==","user":{"login":"wanghaoshuang","id":7534971,"node_id":"MDQ6VXNlcjc1MzQ5NzE=","avatar_url":"https://avatars0.githubusercontent.com/u/7534971?v=4","gravatar_id":"","url":"https://api.github.com/users/wanghaoshuang","html_url":"https://github.com/wanghaoshuang","followers_url":"https://api.github.com/users/wanghaoshuang/followers","following_url":"https://api.github.com/users/wanghaoshuang/following{/other_user}","gists_url":"https://api.github.com/users/wanghaoshuang/gists{/gist_id}","starred_url":"https://api.github.com/users/wanghaoshuang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wanghaoshuang/subscriptions","organizations_url":"https://api.github.com/users/wanghaoshuang/orgs","repos_url":"https://api.github.com/users/wanghaoshuang/repos","events_url":"https://api.github.com/users/wanghaoshuang/events{/privacy}","received_events_url":"https://api.github.com/users/wanghaoshuang/received_events","type":"User","site_admin":false},"created_at":"2019-09-05T02:13:18Z","updated_at":"2019-09-05T02:13:18Z","author_association":"CONTRIBUTOR","body":"@zhaoyuchen2018  帮忙看下sum op相关的问题？"}}
other: {"actor":{"display_login":"wanghaoshuang"}}