Tensorflow general methods

 #method to get shape of tensor flow element

#method to apply a method to all elements

#defining variables, constants in Tensorflow for a matrix of elements.

# Casting, activation functions

Synchronously shuffle X,Y

    import numpy as np

    np.random.seed(seed) 

    m = X.shape[1]                  # number of training examples    

    permutation = list(np.random.permutation(m))

    shuffled_X = X[:, permutation]

    shuffled_Y = Y[:, permutation].reshape((1, m))

Dropout

DROPOUT is a widely used regularization technique that is specific to deep learning. 

It randomly shuts down some neurons in each iteration. 

At each iteration, you shut down (= set to zero) each neuron of a layer with probability 1−keep_prob or keep it with probability  keep_prob (50% here). 

The dropped neurons don't contribute to the training in both the forward and backward propagations of the iteration.

When you shut some neurons down, you actually modify your model. The idea behind drop-out is that at each iteration, you train a different model that uses only a subset of your neurons. With dropout, your neurons thus become less sensitive to the activation of one other specific neuron, because that other neuron might be shut down at any time.

• Dropout is a regularization technique.
• You only use dropout during training. Don't use dropout (randomly eliminate nodes) during test time.
• Apply dropout both during forward and backward propagation.
• During training time, divide each dropout layer by keep_prob to keep the same expected value for the activations. For example, if keep_prob is 0.5, then we will on average shut down half the nodes, so the output will be scaled by 0.5 since only the remaining half are contributing to the solution. Dividing by 0.5 is equivalent to multiplying by 2. Hence, the output now has the same expected value. You can check that this works even when keep_prob is other values than 0.5.

L2 Regulerization

 

m=  # of training examples

l= layer

k , j=shape of weight matrix 

python - Initialization of weights

 The main difference between Gaussian variable (numpy.random.randn()) and uniform random variable is the distribution of the generated random numbers:

• numpy.random.rand() produces numbers in a uniform distribution.
• and numpy.random.randn() produces numbers in a normal distribution.

When used for weight initialization, randn() helps most the weights to Avoid being close to the extremes, allocating most of them in the center of the range.

An intuitive way to see it is, for example, if you take the sigmoid() activation function.

You’ll remember that the slope near 0 or near 1 is extremely small, so the weights near those extremes will converge much more slowly to the solution, and having most of them near the center will speed the convergence.

Initialization of weights

 

• The weights 𝑊[𝑙]${\mathrm{<br>}}^{}$ should be initialized randomly to break symmetry.
• However, it's okay to initialize the biases 𝑏[𝑙]${\mathrm{<br>}}^{<br>}$ to zeros. Symmetry is still broken so long as 𝑊[𝑙]${\mathrm{<br>}}^{}$ is initialized randomly.
• Initializing weights to very large random values doesn't work well.
• Initializing with small random values should do better.

python code to plot cost

import matplotlib.pyplot as plt

%matplotlib inline 

def plot_costs(costs, learning_rate=0.0075):

    plt.plot(np.squeeze(costs))

    plt.ylabel('cost')

    plt.xlabel('iterations (per hundreds)')

    plt.title("Learning rate =" + str(learning_rate))

    plt.show()

#Assuming "costs" is a list of costs obtained during training iterations per hundred

#calling the method with some learning rate

plot_costs(costs, learning_rate)

output:

Deep Learning methodology using gradient descent

Usual Deep Learning methodology to build the model:

1. Initialize parameters / Define hyperparameters
2. Loop for num_iterations:   

                       a. Forward propagation 

                       b. Compute cost function 

                       c. Backward propagation 

                       d. Update parameters (using parameters, and grads from backprop)

    3. Use trained parameters to predict labels

split data set to train, cross validation and test sets

print(f"the shape of the original set (input) is: {x.shape}")

print(f"the shape of the original set (target) is: {y.shape}\n")

from sklearn.model_selection import train_test_split

# Get 60% of the dataset as the training set. Put the remaining 40% in temporary variables.

x_train, x_, y_train, y_ = train_test_split(x, y, test_size=0.40, random_state=1)

# Split the 40% subset above into two: one half for cross validation and the other for the test set

x_cv, x_test, y_cv, y_test = train_test_split(x_, y_, test_size=0.50, random_state=1)

# Delete temporary variables

del x_, y_

print(f"the shape of the training set (input) is: {x_train.shape}")

print(f"the shape of the training set (target) is: {y_train.shape}\n")

print(f"the shape of the cross validation set (input) is: {x_cv.shape}")

print(f"the shape of the cross validation set (target) is: {y_cv.shape}\n")

print(f"the shape of the test set (input) is: {x_test.shape}")

print(f"the shape of the test set (target) is: {y_test.shape}")

Epochs and batches

We provide epoch value while fitting/training the model as below.

Example: model.fit(X,Y,epoch=100) 

Epochs and batches

In the fit statement above, the number of epochs was set to 100. This specifies that the entire data set 

should be applied during training 100 times. During training, you see output describing the progress of 

training that looks like this:

Epoch 1/100
157/157 [==============================] - 0s 1ms/step - loss: 2.2770

The first line, Epoch 1/100, describes which epoch the model is currently running. For efficiency,

the training data set is broken into 'batches'. The default size of a batch in Tensorflow is 32. 

if given an model has are 5000 examples(X_train) it will set or roughly to 157 batches. 

The notation on the 2nd line 157/157 [==== is describing which batch has been executed.

Loss (cost)

Ideally, the cost will decrease as the number of iterations of the algorithm increases. Tensorflow refers to 

the cost as loss. 

Derivative using python

Libraries for derivative

 from sympy import symbols, diff

Let's try this out. Let's look at the derivative of the function 

𝐽(𝑤)=𝑤2its  derivative is, 

∂𝐽(𝑤)∂𝑤=2𝑤

SparseCategorialCrossentropy or CategoricalCrossEntropy

 

Tensorflow has two potential formats for target values and the selection of the loss defines which is expected.

• SparseCategorialCrossentropy: expects the target to be an integer corresponding to the index. For example, if there are 10 potential target values, y would be between 0 and 9.
• CategoricalCrossEntropy: Expects the target value of an example to be one-hot encoded where the value at the target index is 1 while the other N-1 entries are zero. An example with 10 potential target values, where the target is 2 would be [0,0,1,0,0,0,0,0,0,0].

Get the output of each layer in Neural network

Lets Consider following simple neural network

import keras.backend as K

from keras.models import Model
from keras.layers import Input, Dense

input_layer = Input((10,))

layer_1 = Dense(10)(input_layer)
layer_2 = Dense(20)(layer_1)
layer_3 = Dense(5)(layer_2)

output_layer = Dense(1)(layer_3)

model = Model(inputs=input_layer, outputs=output_layer)

# some random input
import numpy as np
features = np.random.rand(100,10)

and consider this model is trained

# With a Keras function get the ouputs of all the layers
get_all_layer_outputs = K.function([model.layers[0].input],
                                  [l.output for l in model.layers[0:]])

layer_output = get_all_layer_outputs([features]) # return the same thing

#layer_output is a list of all layers outputs

#if the model is trained you will get the output for input with trained weights other wise 
it will give outout with initial weights