I have a simple NN:
import torch
import torch.nn as nn
import torch.optim as optim
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.fc1 = nn.Linear(1, 5)
self.fc2 = nn.Linear(5, 10)
self.fc3 = nn.Linear(10, 1)
def forward(self, x):
x = self.fc1(x)
x = torch.relu(x)
x = torch.relu(self.fc2(x))
x = self.fc3(x)
return x
net = Model()
opt = optim.Adam(net.parameters())
I also have some input features:
features = torch.rand((3,1))
I can train it normally with a simple MSE loss function:
for i in range(10):
opt.zero_grad()
out = net(features)
loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
print('loss:', loss)
loss.backward()
opt.step()
However, I'm trying to create a loss function that takes the actual weight values into account:
loss = 1 - torch.mean(torch.tensor([torch.sum(w_arr) for w_arr in net.parameters()]))
But I'm getting an error:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
The goal here is to get each weight's value closest to 1 (or any other value) as possible.
CodePudding user response:
A quick error fix will be to include requires_grad = True
while creating tensor. This way -
loss = 1 - torch.mean(torch.tensor([torch.sum(w_arr) for w_arr in net.parameters()], requires_grad=True))
But when converting list of weights to tensor, torch doesn't know origin of that tensor so the loss doesn't decrease. One way to do it is
for i in range(500):
opt.zero_grad()
out = net(features)
loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
len_w = 0
for w_arr in net.parameters():
loss = torch.mean(torch.abs(1 - w_arr))
len_w = 1
loss /= len_w
print('loss:', loss)
loss.backward()
opt.step()
In this way of loss computation, it makes sure that all the weights are close to 1
.