大家在國高中生物課程中學過生物的神經，有樹突作為輸入端，而當輸入的強度超過閾值時，就會啟動輸出，將訊號傳遞到下一個神經元；而類神經網路，則是比照生物學的原理，以人工的方式來模擬神經元的行為。我們依據模擬的程度，依序說明如下：

In middle school and high school biology, you learned about neurons. A neuron has dendrites as its input. When the input is strong enough to pass a threshold, the neuron fires an output and passes the signal to the next neuron. An artificial neural network follows the same idea, and uses math to imitate how neurons behave. We'll explain it step by step, from simple to more realistic:

• 在最簡單的狀況，假設一個神經元只包含一個輸入端（對照到機器學習的術語，就是特徵只有一個維度）和一個輸出端的情況中，我們可以用數學將其表示成 ŷ = f( x * w + b)，其中：
  • x 是輸入值，w 和 b 分別是我們需要訓練來代表神經元行為的權重(weight)和偏差(bias)值。
  • f 稱為激勵函數(activation function)，通常是以一個非線性的函數，來模擬生物上的神經元，在超過強度閾值時才會有輸出的這種行為。
  • ŷ 為神經元的輸出值。
• 若考慮到生物的一個神經元中，樹突會有多個輸入端，但假設輸出端還是只有一個的情況，則前面的式子會變成 ŷ = f(x^Tw + b)，此時的 x 和 w 都各自是一個 column vector，而 b 和經過運算的 ŷ 還是單一數值。
• 進一步再考慮一層中有多個神經元，也就是輸出端有多個的情況，則算式會再進一步變成 ŷ = f(x^TW + b)，此時的輸入 x、偏差 b 和輸出 ŷ 都是向量，而 W 是矩陣。

• In the simplest case, a neuron has only one input (in machine learning terms, the feature has only one dimension) and one output. We can write this as ŷ = f( x * w + b), where:
  • x is the input value. w and b are the weight and the bias, which we need to train so they can represent how the neuron behaves.
  • f is called the activation function. It is usually a nonlinear function, used to imitate how a real neuron only gives an output when the input passes a threshold.
  • ŷ is the output value of the neuron.
• In real life, a neuron has many inputs on its dendrites, but still only one output. In this case, the formula above becomes ŷ = f(x^Tw + b). Now x and w are both column vectors, while b and the output ŷ are still single numbers.
• Next, let's say one layer has many neurons, which means many outputs. Then the formula becomes ŷ = f(x^TW + b). Now the input x, the bias b, and the output ŷ are all vectors, and W is a matrix.

把上述最後一種模擬的神經元堆疊多層，就是所謂的多層感知器(Multilayer perceptron, MLP)，並且由於我們通常會讓每一層的每個輸出都是完全連接到下一層的每個輸入，所以這樣的每一層叫做全連接層(fully connected layer)。如果有 n 層的話，這個網路寫成算式會是這樣：

If we stack many layers like the last case above, we get what we call a multilayer perceptron (MLP). Usually, every output of one layer is connected to every input of the next layer, so each layer like this is called a fully connected layer. If there are n layers, we can write the network like this:

h₁ = f(x^TW₁ + b₁),

h₂ = f(h₁^TW₂ + b₂),

...,

ŷ = f(h_n-1^TW_n + b_n)

因此，若以一個恰好有一個隱藏層，且輸入、隱藏、輸出層都是維度 2 的神經網路為例，可以繪製示意圖如下，其中 w、b 以及 h 的上標代表其所屬為第幾層，w 的下標代表由前一層的第幾維度連接到後一層的第幾維度，其餘的下標代表其為第幾維度：

So, let's take a network with exactly one hidden layer, where the input, hidden, and output layers all have 2 dimensions. We can draw it like the figure below. The superscript of w, b, and h tells you which layer it belongs to. The subscript of w tells you which dimension in the previous layer connects to which dimension in the next layer. The other subscripts tell you which dimension it is:

假設一路上的每個 W 和 b 已經給定，則對於一個輸入 x，要計算 ŷ 是相當簡單的事情，就是高中學過的矩陣運算而已，此步驟稱為向前傳播(forward propagation, forward pass)。

Suppose every W and b is already given. Then, for an input x, computing ŷ is pretty easy. It's just the matrix math you learned in high school. This step is called forward propagation (or forward pass).

---

而要訓練這個類神經網路，也就是說要更新每個 W 和 b，則需要先使用損失函數(loss function)評估網路算出來的 ŷ 和標準答案的 y 之間的差異，接著以損失函數對每個參數進行偏微分求取梯度，最後再利用各種最佳化（求極值）的方法(optimizer)來更新每個 W 和 b。其中，由於求梯度的步驟是由算出來的 loss 倒回去進行，所以通稱為反向傳播(backpropagation)。如此反覆的向前傳播、反向傳播以及最佳化更新參數以後，通常就可以訓練出恰當的類神經網路。

To train this neural network, which means updating every W and b, we first need to use a loss function to measure the difference between the network's output ŷ and the correct answer y. Then, we take the partial derivative of the loss function with respect to each parameter to get the gradients. Finally, we use an optimizer to update every W and b. Since the gradients are calculated going backward from the loss, this step is called backpropagation. If we repeat forward propagation, backpropagation, and parameter updates again and again, we can usually train a good neural network.

以前述的那個恰好有一個隱藏層，且輸入、隱藏、輸出層都是維度 2 的神經網路為例，把向前傳播的矩陣形式展開回數值時，可以寫為以下方式：

Let's use the same example again: one hidden layer, with 2 dimensions in the input, hidden, and output layers. If we expand the matrix form of forward propagation back into plain numbers, we can write it like this:

h¹₁ = f(x₁w¹_1,1 + x₂w¹_2,1 + b¹₁),

h¹₂ = f(x₁w¹_1,2 + x₂w¹_2,2 + b¹₂),

ŷ₁ = f(h¹₁w²_1,1 + h¹₂w²_2,1 + b²₁),

ŷ₂ = f(h¹₁w²_1,2 + h¹₂w²_2,2 + b²₂)

此時，對於 w² 和 b² 來說，由於各自都只有一條路徑通往 ŷ，因此以損失函數 L 對其偏微分相對簡單，依照連鎖律(chain rule)直接展開即可，如下：

Now, for w² and b², each of them has only one path to ŷ. So taking the partial derivative of the loss function L is pretty simple. We just expand it with the chain rule, like this:

∂L / ∂b²₁ = (∂L / ∂ŷ₁)(∂ŷ₁ / ∂b²₁),

∂L / ∂b²₂ = (∂L / ∂ŷ₂)(∂ŷ₂ / ∂b²₂),

∂L / ∂w²_1,1 = (∂L / ∂ŷ₁)(∂ŷ₁ / ∂w²_1,1),

∂L / ∂w²_1,2 = (∂L / ∂ŷ₂)(∂ŷ₂ / ∂w²_1,2),

...

而對於 w¹ 和 b¹ 來說，由於各自有兩條路徑通往 ŷ，因此以損失函數 L 對其偏微分時，需要同時考慮這兩條路徑的影響，亦即將其相加。以 w¹_1,1 和 b¹₁ 作為範例，對應的路徑和微分算式如下，其中圖上的橘色代表路徑重疊處：

For w¹ and b¹, each of them has two paths to ŷ. So when we take the partial derivative of the loss function L, we need to consider both paths at the same time, which means adding them together. Let's use w¹_1,1 and b¹₁ as examples. The paths and the derivative formulas are shown below. In the figure, the orange color shows where the paths overlap:

∂L / ∂b¹₁ = (∂L / ∂ŷ₁)(∂ŷ₁ / ∂h¹₁)(∂h¹₁ / ∂b¹₁) + (∂L / ∂ŷ₂)(∂ŷ₂ / ∂h¹₁)(∂h¹₁ / ∂b¹₁),

∂L / ∂w¹_1,1 = (∂L / ∂ŷ₁)(∂ŷ₁ / ∂h¹₁)(∂h¹₁ / ∂w¹_1,1) + (∂L / ∂ŷ₂)(∂ŷ₂ / ∂h¹₁)(∂h¹₁ / ∂w¹_1,1)

在上述算式中，你可以發現有一些相同的部份，例如 ∂L / ∂ŷ₁ 以及 ∂ŷ₁ / ∂h¹₁ 等等，這些部份在各家工具的實作中，通常會將重複的部份保留起來，避免重複計算，以節省資源。而在預測階段這類不需要計算反向傳播的時候，以 PyTorch 為例，則有「torch.no_grad()」等方法來略去梯度的計算。

In the formulas above, you can see that some parts are the same, such as ∂L / ∂ŷ₁ and ∂ŷ₁ / ∂h¹₁. In most tools, these repeated parts are saved so they don't need to be calculated again, which saves resources. And when you don't need backpropagation, such as at the prediction stage, you can skip the gradient calculation. In PyTorch, for example, you can use methods like "torch.no_grad()".

此外，由於在反向傳播的算式中，我們除了對損失函數微分以外，在反向傳播時也需要多次對 activation function 在某些點進行微分並連續相乘，因此若 activation function 在那些點的微分值太大（實務上較少碰到僅由 activation function 造成的此問題，通常與權重初始化或網路架構等方面有關）或太小（例如 sigmoid function 在 y 非常接近 0 或 1 的地方），連續相乘起來就會讓梯度爆炸或者消失；目前已有多項技巧用於緩解或繞開相關問題，較經典的方式有梯度剪切(gradient clipping)、殘差連接(residual connection)，以及 batch normalization 等，或者你可以直接選用不易造成梯度問題的 activation function，如 ReLU 系列。

Also, in backpropagation, besides taking the derivative of the loss function, we also need to take the derivative of the activation function at certain points many times, and multiply them together. If the derivative of the activation function at those points is too large (in practice, this problem is rarely caused by the activation function alone, and it is usually related to things like weight initialization or network architecture) or too small (for example, a sigmoid function where the output is very close to 0 or 1), then multiplying them together makes the gradient explode or vanish. There are already many tricks to ease or avoid these problems. Some classic ones are gradient clipping, residual connection, and batch normalization. Or, you can simply choose an activation function that is less likely to cause gradient problems, such as the ReLU family.

而更新參數的方法，最基本的是梯度下降法(gradient descent)；而由於通常訓練類神經網路時，是將資料分成隨機切分的小批次餵進去，因此在這種情境下，會特別稱為隨機梯度下降法(stochastic gradient descent, SGD)。SGD 的公式是 p^t+1 = p^t - r * ∂L / ∂p^t，其中的 p^t 和 p^t+1 分別代表目前的和更新後的參數；而 r 則稱為學習率(learning rate)，代表每次更新的時候你「大膽」的程度。而為了比 SGD 更快找到好的參數，或者找到更好的參數，還有其他各種最佳化方法被開發出來，此處就不一一介紹。

For updating parameters, the most basic method is gradient descent. When we train neural networks, we usually split the data randomly into small batches and feed them in. In this case, the method is specifically called stochastic gradient descent (SGD). The formula of SGD is p^t+1 = p^t - r * ∂L / ∂p^t, where p^t and p^t+1 are the current and the updated parameters. r is called the learning rate, and it shows how "bold" you are with each update. To find good parameters faster than SGD, or to find even better parameters, many other optimizers have been developed. We won't go through them one by one here.

而關於 loss function 的選擇，則會視你要處理的問題而定。一般來說，若是處理回歸問題，則我們可以選用 MSE (mean squared error)；而若是處理分類問題，則通常會使用 cross entropy；而在對比式學習(contrastive learning)或多模態的學習中，甚至會以 triplet loss，或者以 cosine similarity 為基礎的 loss function 等方法，來拉近相似的樣本的網路輸出，並推遠相異樣本的網路輸出。

How to choose a loss function depends on the problem you want to solve. Generally speaking, for regression problems, we can use MSE (mean squared error). For classification problems, we usually use cross entropy. In contrastive learning or multimodal learning, we may even use triplet loss, or loss functions based on cosine similarity, to pull the network outputs of similar samples closer, and push the network outputs of different samples farther apart.