DL-01

1.2 What is a Neural Network?

An Example

Housing Price Prediction

Size ($x$) → (a circle) (neuron) → Price ($\hat{y}$)

ReLU function (rectified linear unit): “rectified” here means taking the maximum of zero and the input.

$$\mathrm{ReLU}(z)=max(0,z)$$

This is a small neural network.

Adding Other Features

Like bedrooms, zip code (postal code, maybe as a feature that tells you about walkability), and wealth.

Then you have a neural network with 4 inputs, including size.

These circles in the hidden layer are called hidden units in a neural network.

2.1 Binary Classification & Notation Convention

$y$: $1$ (cat) vs. $0$ (non-cat).

How your computer stores an image:

It stores 3 separate matrices: red, green, and blue.

For a $64\times 64$ RGB image, flatten these matrices into an input feature vector $x\in {\mathbb{R}}^{n_x}$, where

$$n_x=64\times 64\times 3=\mathrm{12,288.}$$

A single training example is represented by a pair $(x,y)$, where $x$ is an $n_x$-dimensional vector and $y$, the label, is either $0$ or $1$.

Your training set will comprise $m$ training examples:

$$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\dots ,(x^{(m)},y^{(m)}).$$

$m=m_{\mathrm{train}}$ is the number of training examples, while $m_{\mathrm{test}}$ is the number of test examples. The superscript $(i)$ indexes the example.

Stack the training examples as columns:

$$X=\left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]\in {\mathbb{R}}^{n_x\times m},$$$$Y=\left[\begin{array}{cccc}{y}^{(1)}& y^{(2)}& \cdots & y^{(m)}\end{array}\right]\in {\mathbb{R}}^{1\times m}.$$

2.2 Logistic Regression

$$z=w^{\mathsf{T}}x+b,\hat{y}=a=\sigma (z)=\frac{1}{1+e^{-z}}.$$

Here, $w\in {\mathbb{R}}^{n_x}$, $b\in \mathbb{R}$, and $\hat{y}$ estimates $P(y=1\mid x)$.

2.3 Logistic Regression: Loss Function

Loss is for a single example; cost is the average loss over the training set.

$$\mathcal{L}(\hat{y},y)=-[y\log \hat{y}+(1-y)\log (1-\hat{y})].$$

• If $y=1$: $\mathcal{L}=-\log \hat{y}$; we want $\hat{y}$ large.
• If $y=0$: $\mathcal{L}=-\log (1-\hat{y})$; we want $\hat{y}$ small.

$$J(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)}).$$

2.4 Gradient Descent

Repeat the updates to minimize $J(w,b)$:

$$w\leftarrow w-\alpha {\mathrm{\nabla }}_{w}J(w,b),b\leftarrow b-\alpha \frac{\partial J(w,b)}{\partial b}.$$

$\alpha$ is the learning rate. Compute both gradients using the current parameters before updating them.

2.9 Gradient Descent in Logistic Regression

For a single example, with $a=\sigma (z)$:

$$\begin{array}{rl}dz& =\frac{\partial \mathcal{L}}{\partial z}=a-y,\\ d{w}_j& =\frac{\partial \mathcal{L}}{\partial w_j}=x_{j}dz,\\ db& =\frac{\partial \mathcal{L}}{\partial b}=dz.\end{array}$$

Here, $dz$, $d{w}_j$, and $db$ are shorthand for derivatives.

2.10 Logistic Regression on $m$ Examples

Average the gradients over all $m$ examples:

$$\begin{array}{rl}d{w}_j& =\frac{\partial J}{\partial w_j}=\frac{1}{m}\sum _{i=1}^m{x}_j^{(i)}(a^{(i)}-y^{(i)}),\\ db& =\frac{\partial J}{\partial b}=\frac{1}{m}\sum _{i=1}^m(a^{(i)}-y^{(i)}).\end{array}$$

Then update $w_j\leftarrow w_j-\alpha d{w}_j$ and $b\leftarrow b-\alpha db$.

2.11 Vectorization

What Is Vectorization?

In $z=w^{\mathsf{T}}x+b$, $w$ and $x$ are both $n_x$-dimensional column vectors.

Maybe they're very large vectors if you have a lot of features. Using a for loop over $j=1,\dots ,n_x$ to calculate $d{w}_1,d{w}_2,\dots ,d{w}_{n_x}$ can take a lot of time. Vectorization uses array operations instead of explicit loops.

A non-vectorized example for calculating $z$:

Instead, use (with w and x stored as one-dimensional NumPy arrays):

z = np.dot(w, x) + b

This means $z=w^{\mathsf{T}}x+b$, and it's usually much faster. For explicit column arrays with shape (n_x, 1), use w.T @ x + b.