I am trying to implement Neural Network from scratch. By default, it works as i expected, however, now i am trying to add L2 regularization to my model. To do so, I need to change three methods-
cost() #which calculate cost, cost_derivative , backward_prop # propagate networt backward
You can see below that, I have L2_regularization = None as an input to the init function
def __init__(self,sizes, activations = None , cost_function = 'binary_cross_entropy' ,param_init_type = None, L2_regularization=None,dropout = None):
self.sizes = sizes
self.num_layers = len(sizes)
self.caches = dict()
self.cost_function = cost_function
if activations == None: self.layer_activations = self.default_layer_activations_init(sizes)
else: self.layer_activations = activations
if param_init_type == None: self.param_init_type = 'default'
else: self.param_init_type = param_init_type
self.parameters_initializer()
So, if L2_regularization is True i need slight changes in abovementioned methods.
I can copy all three functions and change them and while training ask about:
if self.regularization: cost = self.cost_reg(input) # as if i'm overriding the cost function
as well as for other ones
However,
There are two problems with this way
I don't see this way as a really pythonic way of doing it. So, it doesn't look good when copying one method and giving it another name with slight changes.
I don't want to check whether
self.regularization is Trueorself.regularization is None: in each iteration. I think it could slow down the model and there is a better way instead. If i am wrong let me know about this problem.
What I want from model is to be aware of regularization beforehand.
For example, I have self.regualrization==True
when i call backprop method in train function, it returns back propagation with regularization expression
Code.
It would be too complicated for you to read the whole code and advise me a way of doing things as i want. Hence, i wrote simpler code with same scenario actually
class Network():
def __init__(self,sizes, regularization = None):
self.sizes = sizes
self.expression = 5
self.regularization = regularization
def compute_cost(self):
count = 0
for i in self.sizes:
count =i
return count
def compute_cost_regularized(self):
count = 0
for i in self.sizes:
count =i
#as if self.expression is value of regularization expression
count = count self.expression
return count
def cost_value(self):
if self.regularization:
return self.compute_cost_regularized()
else:
return self.compute_cost()
net_default = Network([3,3,4])
net_regularized= Network([3,3,4],regularization=True)
print('This is the answer from net_default ',net_default.cost_value())
print('This is the answer from net_regularized ',net_regularized.cost_value())
The output is : This is the answer from net_default 10 This is the answer from net_regularized 15
It doesn't solve none of my problems.
I wrote one method 2 times with 1 line change and i used if statement while computing.
How can i write it without doing this. And do i need to avoid using if statement in each iteration
I also tried to override the method
class Network():
def __init__(self,sizes, regularization = None):
self.sizes = sizes
self.expression = 5
self.regularization = regularization
def compute_cost(self):
count = 0
for i in self.sizes:
count =i
return count
def cost_value(self):
if self.regularization:
return regularized(self).compute_cost()
else:
return self.compute_cost()
class regularized(Network):
def __init__(self, sizes, regularization=None):
super().__init__(sizes, regularization)
def compute_cost(self):
return super().compute_cost() self.expression
net_default = Network([3,3,4])
net_regularized= Network([3,3,4],regularization=True)
print('This is the answer from net_default ',net_default.cost_value())
print('This is the answer from net_regularized ',net_regularized.cost_value())
However, I got an error TypeError : 'Network' object is not iterable
Here is the actual train() and cost() function if you have modification idea for implementing regularization
def cost(self,X,Y):
#TODO L2 reg ll change cost function
"""param X : Input that will be given to network , Function itself does forward propagation steps and compute cost
param Y : Wanted output corresponds to given input data. Cost will be computed by This Y and Y_hat which is output of NN for X input"""
Y_hat = self.feed_forward(X)
m = Y.shape[1]
if self.cost_function == 'binary_cross_entropy':
cost = (-1/m)*np.sum( np.multiply(Y,np.log(Y_hat)) np.multiply( (1-Y) , np.log(1-Y_hat) )) ; cost = np.squeeze(cost)
return cost
elif self.cost_function == 'mse':
cost = (1/m)*np.sum(np.square(Y-Y_hat)) ; cost = np.squeeze(cost)
return cost
else:
raise Exception('No such cost function yet')
def train(self,X,Y,lr = 0.0001,epoch=1000 , X_test = None , Y_test = None , regularization = None , dropout = False):
assert (X.shape[1] == Y.shape[1]) , "Unmatched In out batch size"
self.caches['A0'] = X
for iter in range(epoch):
A_l = self.feed_forward(X)
dA_l = self.cost_derivative(A_l,Y)
for layer_num in reversed(range(1,self.num_layers)):
grad_w,grad_b,dA_l = self.backward_prop(dA_l,layer_num)
self.update_param(grad_w,grad_b,layer_num, lr = lr)
if iter% (epoch/10) ==0:
print('\n COST:::',self.cost(X,Y),end=' ')
self.score(X,Y)
if X_test is not None:
self.score(X_test,Y_test)
#Saving parameters dictionary to file
a_file = open("parameters.pkl", "wb")
pickle.dump(self.parameters, a_file)
a_file.close()
Just in case if you are interested, i have downloaded full code here.
https://github.com/IlkinKarimli0/Neural-Network-from-scratch/blob/main/NeuralNetwork.py
CodePudding user response：
General
Overall you should not create an object inside an object for the purpose of overriding a single method, instead you can just do
class Network():
def __init__(self, sizes):
self.sizes = sizes
self.expression = 5
def compute_cost(self):
count = 0
for i in self.sizes:
count =i
return count
def cost_value(self):
return self.compute_cost()
class RegularizedNetwork(Network):
def __init__(self, sizes):
super().__init__(sizes)
def compute_cost(self):
return super().compute_cost() self.expression
net_default = Network([3,3,4])
net_regularized= RegularizedNetwork([3,3,4])
print('This is the answer from net_default ',net_default.cost_value())
print('This is the answer from net_regularized ',net_regularized.cost_value())
In other words you actually create an instance of your child class, which overrides a specific function (here: compute_cost), and inherits all the remaining ones. Now when cost_value() is called, it will call the corresponding compute_cost. In fact you do not need compute_cost either.
class Network():
def __init__(self, sizes):
self.sizes = sizes
self.expression = 5
def cost_value(self):
count = 0
for i in self.sizes:
count =i
return count
class RegularizedNetwork(Network):
def __init__(self, sizes):
super().__init__(sizes)
def cost_value(self):
return super().cost_value() self.expression
net_default = Network([3,3,4])
net_regularized= RegularizedNetwork([3,3,4])
print('This is the answer from net_default ',net_default.cost_value())
print('This is the answer from net_regularized ',net_regularized.cost_value())
Code issue
If for some reason you would like to continue with your own code, the issue is that here
def cost_value(self):
if self.regularization:
return regularized(self).compute_cost()
else:
return self.compute_cost()
you are passing reference to "self" to a constructor of regularized which is expecting sizes, it should be
def cost_value(self):
if self.regularization:
return regularized(self.sizes).compute_cost()
else:
return self.compute_cost()