線性回歸(linear regression)的目標是找出一條直線，使得所有點到直線的距離（y 軸方向）之平方和為最小；即你有一堆資料點 (x₁, y₁), (x₂, y₂), ..., (x_n, y_n)，並想找出一條 ŷ = wx+b，使得下列式子為最小：

The goal of linear regression is to find a straight line that minimizes the sum of squared distances (in the y-axis direction) from all points to the line. That is, given a set of data points (x₁, y₁), (x₂, y₂), ..., (x_n, y_n), you want to find a line ŷ = wx+b that minimizes the following expression:

Σ(y_i−ŷ_i)² = Σ(y_i−(wx_i+b))²

這個式子有正規的解析解，也可以用梯度下降來解。實作上，我們通常可以使用 scikit-learn 的 linear_model.LinearRegression 來幫我們進行線性回歸的求解，以下範例將會隨機的產生一些資料點來示範：

This expression has a standard closed-form solution, and can also be solved using gradient descent. In practice, we can usually use scikit-learn's linear_model.LinearRegression to solve linear regression for us. The example below randomly generates some data points to demonstrate this:

import numpy as np
from sklearn.linear_model import LinearRegression

x_train = np.random.rand(1000) * 10
y_train = 3 * x_train + 2 + np.random.randn(1000) * 0.1

x_test = np.random.rand(100) * 10
y_test = 3 * x_test + 2 + np.random.randn(100) * 0.1

model = LinearRegression()
model.fit(x_train[:, np.newaxis], y_train)
print(model.coef_[0], model.intercept_)
y_pred = model.predict(x_test[:, np.newaxis])
y_pred_2 = model.coef_ * x_test + model.intercept_
print('MAE (by model.predict):', np.mean(np.abs(y_test - y_pred)))
print('MAE (by model.coef_ * x_test + model.intercept_):', np.mean(np.abs(y_test - y_pred_2)))

x_train_2 = np.hstack([x_train[:, np.newaxis], np.ones((x_train.size, 1))])
(coef, intercept), _, _, _ = np.linalg.lstsq(x_train_2, y_train, rcond=None)
print(coef, intercept)
y_pred_2 = coef * x_test + intercept
print('MAE (by np.linalg.lstsq):', np.mean(np.abs(y_test - y_pred_2)))

在上述範例中：

In the example above:

• 進行訓練時，x 的 shape 必須是「(資料點數, 每筆輸入資料的維度)」，因此需要新增一個軸。身為訓練目標的 y，則因為此例中剛好只有單變數，所以可以不必新增一個軸。
• 訓練完畢後，model.coef_ 的內容會是每個輸入維度的係數，model.intercept_ 則是常數項。
• 由於範例中的資料是亂數產生，所以每次執行時，模型學出的係數可能不太一樣。
• 除了使用 sklearn 以外，也同時示範了使用 np.linalg.lstsq 來求取結果，以便讓各位稍加了解內部的演算方式。

• When training, the shape of x must be "(number of data points, dimension of each input sample)," so an extra axis needs to be added. Since y, the training target, happens to be a single variable in this example, it does not need an extra axis.
• After training, the content of model.coef_ will be the coefficient for each input dimension, and model.intercept_ will be the constant term.
• Since the data in the example is randomly generated, the coefficients learned by the model may differ slightly each time you run it.
• Besides using sklearn, the example also demonstrates using np.linalg.lstsq to obtain the result, so you can get a bit of insight into how the computation works internally.

如果要利用 LinearRegression 來擬合多項式，那麼你可以建立一個包含 x 的各個次方的矩陣，然後用該矩陣來對 y 做線性回歸。以下範例是一個擬合三次多項式的範例：

If you want to use LinearRegression to fit a polynomial, you can build a matrix containing the various powers of x, and then perform linear regression on y using that matrix. The example below fits a cubic polynomial:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression

x = np.linspace(-2, 2, 1000)
y_raw = x ** 3 - 2 * x ** 2 + 3 * x - 4
y = y_raw + np.random.randn(1000)

x2 = np.hstack([x[:, np.newaxis] ** 3, x[:, np.newaxis] ** 2, x[:, np.newaxis]])
model = LinearRegression()
model.fit(x2, y[:, np.newaxis])
print(model.coef_, model.intercept_)
y_fit = model.predict(x2)

plt.plot(x, y, '.', label='Random data')
plt.plot(x, y_raw, '-', label='Raw line')
plt.plot(x, y_fit, '-', label='Fitted line')
plt.legend()
plt.show()

在上述範例中：

In the example above:

• 只展示了訓練部分。
• 因為 LinearRegression 會自動幫你加入常數項，因此你不必像使用 np.linalg.lstsq 時一樣去自己加入。
• 由於範例中的資料依然是亂數產生，所以每次執行時，模型學出的係數可能不太一樣。

• Only the training part is shown.
• Since LinearRegression automatically adds the constant term for you, you do not need to add it manually as you would when using np.linalg.lstsq.
• Since the data in the example is still randomly generated, the coefficients learned by the model may differ slightly each time you run it.

當你的特徵之中，有幾個是高度相關的時候，普通的 linear regression 可能會產生不太穩定的結果，亦即係數過大。這時候，我們可以透過 Lasso 或者 Ridge regression，來限制係數的絕對值或者平方不要太大，以讓求解係數的過程比較穩定，即：

When several of your features are highly correlated, ordinary linear regression may produce unstable results, that is, coefficients that are too large. In this case, we can use Lasso or Ridge regression to constrain the absolute value or the square of the coefficients from becoming too large, making the process of solving for the coefficients more stable, that is:

L(θ) = Σ(y_i−(x_i^Tθ))² + α Σ | θ_j |、

L(θ) = Σ(y_i−(x_i^Tθ))² + α Σ θ_j²。

上述關於 linear、Lasso、Ridge 三者的比較如下：

The comparison among linear, Lasso, and Ridge regression described above is shown below:

import numpy as np
from sklearn.linear_model import LinearRegression, Lasso, Ridge

np.random.seed(42)
n = 100
X1 = np.random.randn(n)
X2 = X1 + np.random.randn(n) * 0.0001

X = np.column_stack([X1, X2])
y = 3 * X1 + np.random.randn(n) * 0.1

lr = LinearRegression()
lr.fit(X, y)
print("Linear Regression:")
print(f"  w1 = {lr.coef_[0]:.2f}")
print(f"  w2 = {lr.coef_[1]:.2f}")
print(f"  w1 + w2 = {lr.coef_[0] + lr.coef_[1]:.2f}")

lasso = Lasso(alpha=1.0)
lasso.fit(X, y)
print("\nLasso Regression:")
print(f"  w1 = {lasso.coef_[0]:.2f}")
print(f"  w2 = {lasso.coef_[1]:.2f}")

# Ridge
ridge = Ridge(alpha=1.0)
ridge.fit(X, y)
print("\nRidge Regression:")
print(f"  w1 = {ridge.coef_[0]:.2f}")
print(f"  w2 = {ridge.coef_[1]:.2f}")