Coursera深度学习系列课程

RNN环节编程作业复习

# 1. Building a Recurrent Neural Network

implement Recurrent Neural Network in numpy.

Notation:

• Superscript $[l]$ denotes an object associated with the $l^{th}$ layer.

• Example: $a^{[4]}$ is the $4^{th}$ layer activation. $W^{[5]}$ and $b^{[5]}$ are the $5^{th}$ layer parameters.
• Superscript $(i)$ denotes an object associated with the $i^{th}$ example.

• Example: $x^{(i)}$ is the $i^{th}$ training example input.
• Superscript $\langle t \rangle$ denotes an object at the $t^{th}$ time-step.

• Example: $x^{\langle t \rangle}$ is the input x at the $t^{th}$ time-step. $x^{(i)\langle t \rangle}$ is the input at the $t^{th}$ timestep of example $i$.
• Lowerscript $i$ denotes the $i^{th}$ entry of a vector.

• Example: $a^{[l]}_i$ denotes the $i^{th}$ entry of the activations in layer $l$.

## 1.1 RNN cell

Instructions:

1. Compute the hidden state with tanh activation: $a^{\langle t \rangle} = \tanh(W_{aa} a^{\langle t-1 \rangle} + W_{ax} x^{\langle t \rangle} + b_a)$.
2. Using your new hidden state $a^{\langle t \rangle}$, compute the prediction $\hat{y}^{\langle t \rangle} = softmax(W_{ya} a^{\langle t \rangle} + b_y)$. We provided you a function: softmax.
3. Store $(a^{\langle t \rangle}, a^{\langle t-1 \rangle}, x^{\langle t \rangle}, parameters)$ in cache
4. Return $a^{\langle t \rangle}$ , $y^{\langle t \rangle}$ and cache

We will vectorize over $m$ examples. Thus, $x^{\langle t \rangle}$ will have dimension $(n_x,m)$, and $a^{\langle t \rangle}$ will have dimension $(n_a,m)$.

## 1.2 RNN forward pass

Instructions:

1. Create a vector of zeros ($a$) that will store all the hidden states computed by the RNN.
2. Initialize the “next” hidden state as $a_0$ (initial hidden state).
3. Start looping over each time step, your incremental index is $t$ :
• Update the “next” hidden state and the cache by running rnn_cell_forward
• Store the “next” hidden state in $a$ ($t^{th}$ position)
• Store the prediction in y
• Add the cache to the list of caches
4. Return $a$, $y$ and caches

RNN will work well enough for some applications, but it suffers from vanishing gradient problems. So it works best when each output $y^{\langle t \rangle}$ can be estimated using mainly “local” context (meaning information from inputs $x^{\langle t’ \rangle}$ where $t’$ is not too far from $t$).

In the next part, you will build a more complex LSTM model, which is better at addressing vanishing gradients. The LSTM will be better able to remember a piece of information and keep it saved for many timesteps.

# 2. Building a Long Short-Term Memory (LSTM) network

LSTM-cell. This tracks and updates a “cell state” or memory variable $c^{\langle t \rangle}$ at every time-step, which can be different from $a^{\langle t \rangle}$.

### - Forget gate

For the sake of this illustration, lets assume we are reading words in a piece of text, and want use an LSTM to keep track of grammatical structures, such as whether the subject is singular or plural. If the subject changes from a singular word to a plural word, we need to find a way to get rid of our previously stored memory value of the singular/plural state. In an LSTM, the forget gate lets us do this:

$$\Gamma_f^{\langle t \rangle} = \sigma(W_f[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_f)\tag{1}$$
Here, $W_f​$ are weights that govern the forget gate’s behavior. We concatenate $[a^{\langle t-1 \rangle}, x^{\langle t \rangle}]​$ and multiply by $W_f​$. The equation above results in a vector $\Gamma_f^{\langle t \rangle}​$ with values between 0 and 1. This forget gate vector will be multiplied element-wise by the previous cell state $c^{\langle t-1 \rangle}​$. So if one of the values of $\Gamma_f^{\langle t \rangle}​$ is 0 (or close to 0) then it means that the LSTM should remove that piece of information (e.g. the singular subject) in the corresponding component of $c^{\langle t-1 \rangle}​$. If one of the values is 1, then it will keep the information.

### - Update gate

Once we forget that the subject being discussed is singular, we need to find a way to update it to reflect that the new subject is now plural. Here is the formulat for the update gate:

$$\Gamma_u^{\langle t \rangle} = \sigma(W_u[a^{\langle t-1 \rangle}, x^] + b_u)\tag{2}$$
Similar to the forget gate, here $\Gamma_u^{\langle t \rangle}$ is again a vector of values between 0 and 1. This will be multiplied element-wise with $\tilde{c}^{\langle t \rangle}$, in order to compute $c^{\langle t \rangle}$.

$c^{\langle t \rangle}​$是需要传递的记忆单元，forget gate： $\Gamma_f^{\langle t \rangle}$为0时(或接近0)，表示可以将当前记忆的丢$c^{\langle t \rangle}$弃，如果此时$\Gamma_u^{\langle t \rangle}$接近1，就可以将将内容更新为$\tilde{c}^{\langle t \rangle}​$。

### - Updating the cell

To update the new subject we need to create a new vector of numbers that we can add to our previous cell state. The equation we use is:

$$\tilde{c}^{\langle t \rangle} = \tanh(W_c[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_c)\tag{3}$$

Finally, the new cell state is:

$$c^{\langle t \rangle} = \Gamma_f^{\langle t \rangle}*c^{\langle t-1 \rangle} + \Gamma_u^{\langle t \rangle}*\tilde{c}^{\langle t \rangle}\tag{4}$$

### - Output gate

To decide which outputs we will use, we will use the following two formulas:

$$\Gamma_o^{\langle t \rangle}= \sigma(W_o[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_o)\tag{5}$$

$$a^{\langle t \rangle} = \Gamma_o^{\langle t \rangle}* \tanh(c^{\langle t \rangle})\tag{6}$$

Where in equation 5 you decide what to output using a sigmoid function and in equation 6 you multiply that by the $\tanh$ of the previous state.

## 2.2 LSTM cell

Instructions:

1. Concatenate $a^{\langle t-1 \rangle}$ and $x^{\langle t \rangle}$ in a single matrix: $concat = \begin{bmatrix} a^{\langle t-1 \rangle} \ x^{\langle t \rangle} \end{bmatrix}$
注意：根据公式，把$a^{\langle t-1 \rangle}、x^{\langle t \rangle}$拼成一个矩阵，作为一个整体！即把之前时间步传递过来的信息也当做当前时间步的输入。
2. Compute all the formulas 1-6. You can use sigmoid() (provided) and np.tanh().
3. Compute the prediction $y^{\langle t \rangle}$. You can use softmax() (provided).

## 2.3 LSTM forward pass

LSTM over multiple time-steps.

Exercise: Implement lstm_forward() to run an LSTM over $T_x$ time-steps.

Note: $c^{\langle 0 \rangle}$ is initialized with zeros.

# 3. Backpropagation in recurrent neural networks

In modern deep learning frameworks, you only have to implement the forward pass, and the framework takes care of the backward pass, so most deep learning engineers do not need to bother with the details of the backward pass. If however you are an expert in calculus and want to see the details of backprop in RNNs, you can work through this optional portion of the notebook.

When in an earlier course you implemented a simple (fully connected) neural network, you used backpropagation to compute the derivatives with respect to the cost to update the parameters. Similarly, in recurrent neural networks you can to calculate the derivatives with respect to the cost in order to update the parameters. The backprop equations are quite complicated and we did not derive them in lecture. However, we will briefly present them below.

## 3.1 Basic RNN backward pass

We will start by computing the backward pass for the basic RNN-cell.

RNN-cell’s backward pass. Just like in a fully-connected neural network, the derivative of the cost function $J$ backpropagates through the RNN by following the chain-rule from calculas. The chain-rule is also used to calculate $(\frac{\partial J}{\partial W_{ax}},\frac{\partial J}{\partial W_{aa}},\frac{\partial J}{\partial b})$ to update the parameters $(W_{ax}, W_{aa}, b_a)$.

### Deriving the one step backward functions:

To compute the rnn_cell_backward you need to compute the following equations. It is a good exercise to derive them by hand.

The derivative of $\tanh$ is $1-\tanh(x)^2$. You can find the complete proof here. Note that: $\text{sech}(x)^2 = 1 - \tanh(x)^2$

Similarly for $\frac{ \partial a^{\langle t \rangle} } {\partial W_{ax}}, \frac{ \partial a^{\langle t \rangle} } {\partial W_{aa}}, \frac{ \partial a^{\langle t \rangle} } {\partial b}$, the derivative of $\tanh(u)$ is $(1-\tanh(u)^2)du$.

The final two equations also follow same rule and are derived using the $\tanh$ derivative. Note that the arrangement is done in a way to get the same dimensions to match.

$\frac{ \partial J } {\partial a^{\langle t \rangle}}$是函数的传入参数da_next（反向传播，next cell传到当前cell中），dxt的计算公式是$\frac{ \partial J } {\partial a^{\langle t \rangle}}\frac{ \partial a^{\langle t \rangle} } {\partial x^{\langle t \rangle}}$，可以写成
$$d_{xt}=da_{next}\cdot\frac{ \partial a^{\langle t \rangle} } {\partial x^{\langle t \rangle}}$$

$$a^{\langle t \rangle} = \tanh(W_{aa} a^{\langle t-1 \rangle} + W_{ax} x^{\langle t \rangle} + b_a)$$

$$\frac{ \partial a^{\langle t \rangle} } {\partial \tanh(u)}\cdot\frac{ \partial \tanh(u)} {\partial u}\cdot \frac{ \partial u} {\partial x^{\langle t \rangle}}=(1- \tanh(u)^2)\cdot W_{ax}=(1-{a^{\langle t \rangle}}^2)\cdot W_{ax}$$

$$d_{xt}=\frac{ \partial J } {\partial a^{\langle t \rangle}}\frac{ \partial a^{\langle t \rangle} } {\partial x^{\langle t \rangle}}=da_{next}\cdot\frac{ \partial a^{\langle t \rangle} } {\partial x^{\langle t \rangle}}=da_{next}\cdot\frac{ \partial a^{\langle t \rangle} } {\partial \tanh(u)}\cdot\frac{ \partial \tanh(u)} {\partial u}\cdot \frac{ \partial u} {\partial x^{\langle t \rangle}}=da_{next}\cdot(1-{a^{\langle t \rangle}}^2)\cdot\frac{ \partial u} {\partial x^{\langle t \rangle}}$$

$$d_{W_{ax}}=\frac{\partial J}{\partial W_{ax}}=\frac{ \partial J } {\partial a^{\langle t \rangle}}\frac{ \partial a^{\langle t \rangle} } {\partial W_{ax}}=da_{next}\cdot(1-{a^{\langle t \rangle}}^2)\cdot\frac{ \partial u} {\partial x^{\langle t \rangle}}=da_{next}\cdot(1-{a^{\langle t \rangle}}^2)\cdot x^{\langle t \rangle}=d_{tanh}\cdot x^{\langle t \rangle}$$

## 3.2 Backward pass through the RNN

Computing the gradients of the cost with respect to $a^{\langle t \rangle}$ at every time-step $t$ is useful because it is what helps the gradient backpropagate to the previous RNN-cell. To do so, you need to iterate through all the time steps starting at the end, and at each step, you increment the overall $db_a$, $dW_{aa}$, $dW_{ax}$ and you store $dx$.

Instructions:

Implement the rnn_backward function. Initialize the return variables with zeros first and then loop through all the time steps while calling the rnn_cell_backward at each time timestep, update the other variables accordingly.

# 4. LSTM backward pass

## 4.1 One Step backward

The LSTM backward pass is slighltly more complicated than the forward one. We have provided you with all the equations for the LSTM backward pass below. (If you enjoy calculus exercises feel free to try deriving these from scratch yourself.)

### - gate derivatives

$$d \Gamma_o^{\langle t \rangle} = da_{next}*\tanh(c_{next}) * \Gamma_o^{\langle t \rangle}*(1-\Gamma_o^{\langle t \rangle})\tag{7}$$

$$d\tilde c^{\langle t \rangle} = dc_{next}*\Gamma_u^{\langle t \rangle}+ \Gamma_o^{\langle t \rangle} (1-\tanh(c_{next})^2) * i_t * da_{next} * \tilde c^{\langle t \rangle} * (1-\tanh(\tilde c)^2) \tag{8}$$

$$d\Gamma_u^{\langle t \rangle} = dc_{next}*\tilde c^{\langle t \rangle} + \Gamma_o^{\langle t \rangle} (1-\tanh(c_{next})^2) * \tilde c^{\langle t \rangle} * da_{next}*\Gamma_u^{\langle t \rangle}*(1-\Gamma_u^{\langle t \rangle})\tag{9}$$

$$d\Gamma_f^{\langle t \rangle} = dc_{next}*\tilde c_{prev} + \Gamma_o^{\langle t \rangle} (1-\tanh(c_{next})^2) * c_{prev} * da_{next}*\Gamma_f^{\langle t \rangle}*(1-\Gamma_f^{\langle t \rangle})\tag{10}$$

### - parameter derivatives

$$dW_f = d\Gamma_f^{\langle t \rangle} * \begin{pmatrix} a_{prev} \ x_t\end{pmatrix}^T \tag{11}$$

$$dW_u = d\Gamma_u^{\langle t \rangle} * \begin{pmatrix} a_{prev} \ x_t\end{pmatrix}^T \tag{12}$$

$$dW_c = d\tilde c^{\langle t \rangle} * \begin{pmatrix} a_{prev} \ x_t\end{pmatrix}^T \tag{13}$$

$$dW_o = d\Gamma_o^{\langle t \rangle} * \begin{pmatrix} a_{prev} \ x_t\end{pmatrix}^T \tag{14}$$
To calculate $db_f, db_u, db_c, db_o$ you just need to sum across the horizontal (axis= 1) axis on $d\Gamma_f^{\langle t \rangle}, d\Gamma_u^{\langle t \rangle}, d\tilde c^{\langle t \rangle}, d\Gamma_o^{\langle t \rangle}$ respectively. Note that you should have the keep_dims = True option.

Finally, you will compute the derivative with respect to the previous hidden state, previous memory state, and input.
$$da_{prev} = W_f^T*d\Gamma_f^{\langle t \rangle} + W_u^T * d\Gamma_u^{\langle t \rangle}+ W_c^T * d\tilde c^{\langle t \rangle} + W_o^T * d\Gamma_o^{\langle t \rangle} \tag{15}$$
Here, the weights for equations 13 are the first n_a, (i.e. $W_f = W_f[:n_a,:]$ etc…)
$$dc_{prev} = dc_{next}\Gamma_f^{\langle t \rangle} + \Gamma_o^{\langle t \rangle} * (1- \tanh(c_{next})^2)*\Gamma_f^{\langle t \rangle}*da_{next} \tag{16}$$

$$dx^{\langle t \rangle} = W_f^T*d\Gamma_f^{\langle t \rangle} + W_u^T * d\Gamma_u^{\langle t \rangle}+ W_c^T * d\tilde c_t + W_o^T * d\Gamma_o^{\langle t \rangle}\tag{17}$$
where the weights for equation 15 are from n_a to the end, (i.e. $W_f = W_f[n_a:,:]$ etc…)

Exercise: Implement lstm_cell_backward by implementing equations $7-17$ below. Good luck! :)

## 4.2 Backward pass through the LSTM RNN

This part is very similar to the rnn_backward function you implemented above. You will first create variables of the same dimension as your return variables. You will then iterate over all the time steps starting from the end and call the one step function you implemented for LSTM at each iteration. You will then update the parameters by summing them individually. Finally return a dictionary with the new gradients.

Instructions: Implement the lstm_backward function. Create a for loop starting from $T_x$ and going backward. For each step call lstm_cell_backward and update the your old gradients by adding the new gradients to them. Note that dxt is not updated but is stored.

# 关于LSTM的理解

1. $c^{\langle t \rangle}$代表了长时记忆，$a^{\langle t \rangle}$则代表了工作记忆或短时记忆
2. cell 的权重是共享的，这是指LSTM forward过程中，不管有多少个LSTM cell 连接在一起，但是实际上，它只是代表了一个 cell 在不同时序时候的状态，所有的数据只会通过一个 cell，然后不断更新它的权重。
3. 一层的 LSTM 的参数有多少个？这里只有一个 cell，所以参数的数量就是这个 cell 里面用到的参数个数。假设 num_units 是128，输入数据是28位的，cell中4个运算单元的参数一共有$(128+28)*(128*4)​$个，也就是$(156 * 512)​$个，加上输出的时候的激活函数的参数，假设是10个类的话，就是128*10的 W 参数和10个bias 参数。
4. cell中每一个处理单元代表一个前馈网络层（共4个处理单元，因为有4个不同的权重矩阵，他们的维度都相同），就是经典的神经网络的结构，tensorflow定义LSTM cell中使用的参数num_units就是每一层的隐藏神经元个数

input和target错开一位

batch的划分，假设数据总长度是12

# 5. Peephole Connection LSTM

K. Greff, R. K. Srivastava, J. Koutnı́k, B. R. Steunebrink, and J. Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neuralnetworks and learning systems, 2016.

Re3：Real-Time Recurrent Regression Networks for Visual Tracking of Generic Objects 中，用的就是这种LSTM。

﻿
----------over----------
﻿