Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation #21

Open
kriscpu opened this issue Dec 5, 2023 · 4 comments

Comments

@kriscpu
Copy link

kriscpu commented Dec 5, 2023

Getting below error under pytorch version >1.4 . but it's good at pytorch 1.4.

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

tried to found and solve all inplace operation I can locate, but CANNOT get around still. Any ideas?

@kriscpu
Copy link
Author

kriscpu commented Dec 5, 2023

D:\Anaconda\envs\pt21\Lib\site-packages\torch\autograd_init_.py:251: UserWarning: Error detected in ConvolutionBackward0. Traceback of forward call that caused the error:
File "D:\AI\EraseNet-master\train_STE.py", line 110, in
G_loss = criterion(imgs, masks, x_o1, x_o2, x_o3, fake_images, mm, gt, count, i)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AI\EraseNet-master\loss\Loss.py", line 54, in forward
D_fake = self.discriminator(output, mask)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AI\EraseNet-master\models\discriminator.py", line 40, in forward
return self.fusion(concat_feat).view(input.size()[0], -1)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\container.py", line 215, in forward
input = module(input)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\nn\modules\conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
(Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\autograd\python_anomaly_mode.cpp:119.)
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
File "D:\AI\EraseNet-master\train_STE.py", line 113, in
G_loss.backward()
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch_tensor.py", line 492, in backward
torch.autograd.backward(
File "D:\Anaconda\envs\pt21\Lib\site-packages\torch\autograd_init
.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
(pt21) PS D:\AI\EraseNet-master>

@hustliujian
Copy link

@kriscpu I find the matter is self.D_optimizer.step() modifies the weights, but D_fake is used in GLoss computation later.
So delete D_fake = -torch.mean(D_fake), and compute it like D_fake = -self.discriminator(output, mask).mean() # SN-Patch-GAN loss after function step() is ok.

@ScooperCC
Copy link

@kriscpu I find the matter is self.D_optimizer.step() modifies the weights, but D_fake is used in GLoss computation later. So delete D_fake = -torch.mean(D_fake), and compute it like D_fake = -self.discriminator(output, mask).mean() # SN-Patch-GAN loss after function step() is ok.

it's quite useful!

@Suncheng2022
Copy link

@kriscpu I find the matter is self.D_optimizer.step() modifies the weights, but D_fake is used in GLoss computation later. So delete D_fake = -torch.mean(D_fake), and compute it like D_fake = -self.discriminator(output, mask).mean() # SN-Patch-GAN loss after function step() is ok.

single GPU is useful, nn.DataParallel is error:
RuntimeError: Output 15 of BroadcastBackward is a view and its base or another view of its base has been modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants