一般來說，我們會將整個資料集切成訓練、驗證與測試集，而交叉驗證(cross validation)則是把整個資料集切分成訓練與測試集，再將訓練集分成 k 個小份，每次拿一份作為驗證使用，另外 k - 1 份做為訓練使用，如此輪流 k 次的整個過程。交叉驗證的好處是可以更有效的利用資料，讓訓練模型的過程，有機會以不同的資料來選擇超參數等。

Generally speaking, we split the whole dataset into training, validation, and test sets. Cross validation instead splits the whole dataset into a training set and a test set, and then splits the training set into k small parts. Each time, one part is used as validation, and the other k - 1 parts are used as training. This whole process is repeated k times, taking turns. The benefit of cross validation is that it can use the data more efficiently, giving the training process a chance to choose hyperparameters based on different portions of the data.

下列是一個交叉驗證的範例。此範例使用了 wine dataset 的 80% 作為訓練集，並將這個訓練集分成三份進行交叉驗證。你可以試著改變模型超參數，看看平均的 accuracy 是否會變化：

Below is an example of cross validation. This example uses 80% of the wine dataset as the training set, and splits this training set into three parts for cross validation. You can try changing the model's hyperparameters to see whether the average accuracy changes:

import numpy as np
from sklearn.datasets import load_wine
from sklearn.neighbors import KNeighborsClassifier

dataset = load_wine()
total_data_num = dataset.data.shape[0]
print('Data shapes:', dataset.data.shape, dataset.target.shape)

idx_tr = np.where(np.arange(total_data_num) % 5 != 0)[0]
x_train = dataset.data[idx_tr]
y_train = dataset.target[idx_tr]
tr_data_num = x_train.shape[0]
print('Training data shapes:', x_train.shape, y_train.shape)

N_FOLD = 3
accuracies = []
for i in range(N_FOLD):
	idx_fold_tr = np.where(np.arange(tr_data_num) % N_FOLD != i)[0]
	idx_fold_va = np.where(np.arange(tr_data_num) % N_FOLD == i)[0]
	model = KNeighborsClassifier(n_neighbors=1)
	model.fit(x_train[idx_fold_tr], y_train[idx_fold_tr])
	pred = model.predict(x_train[idx_fold_va])
	acc = 100 * np.mean(pred == y_train[idx_fold_va])
	accuracies.append(acc)
print('Average acc:', np.mean(accuracies))

Scikit-learn 的 model_selection.cross_validate 也可以幫你進行交叉驗證與超參數搜尋，若將前一個範例改為用 sklearn.model_selection.cross_validate 來進行，則會變為如下：

Scikit-learn's model_selection.cross_validate can also help you perform cross validation and hyperparameter search. If we rewrite the previous example using sklearn.model_selection.cross_validate, it becomes as follows:

import numpy as np
from sklearn.datasets import load_wine
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_validate

dataset = load_wine()
total_data_num = dataset.data.shape[0]
print('Data shapes:', dataset.data.shape, dataset.target.shape)

idx_tr = np.where(np.arange(total_data_num) % 5 != 0)[0]
x_train = dataset.data[idx_tr]
y_train = dataset.target[idx_tr]
print('Training data shapes:', x_train.shape, y_train.shape)

N_FOLD = 3
model = KNeighborsClassifier(n_neighbors=1)
cv_results = cross_validate(model, x_train, y_train, cv=N_FOLD, scoring='accuracy')

accuracies = cv_results['test_score'] * 100
print('Average acc:', np.mean(accuracies))

在切分資料的時候，有件事情是必須留意的，也就是每個 fold 的各類資料的比率，要盡量跟整個資料集的原始分布相同，稱為 class-wise stratified K-Fold。你當然可以自己確保如何切分，也可以透過 sklearn.model_selection.StratifiedKFold 來幫你處理，如下：

When splitting the data, there is something you must pay attention to: the class ratio in each fold should be kept as close as possible to the original distribution of the whole dataset. This is called class-wise stratified K-Fold. You can of course make sure of this yourself, or you can let sklearn.model_selection.StratifiedKFold handle it for you, as shown below:

import numpy as np
from sklearn.datasets import load_wine
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_validate, StratifiedKFold

dataset = load_wine()
total_data_num = dataset.data.shape[0]
print('Data shapes:', dataset.data.shape, dataset.target.shape)

idx_tr = np.where(np.arange(total_data_num) % 5 != 0)[0]
x_train = dataset.data[idx_tr]
y_train = dataset.target[idx_tr]
print('Training data shapes:', x_train.shape, y_train.shape)

N_FOLD = 3
model = KNeighborsClassifier(n_neighbors=1)
cv_results = cross_validate(model, x_train, y_train, cv=StratifiedKFold(n_splits=N_FOLD), scoring='accuracy')

accuracies = cv_results['test_score'] * 100
print('Average acc:', np.mean(accuracies))

在前面幾個範例中，我們都是固定使用一組超參數，來執行 cross validation 的效果。然而，cross validation 的用處之一，就是以平均的模型效果來決定超參數。若要達成此目的，你可以反覆手動修改超參數並執行程式、自行撰寫迴圈來暴力搜尋，或者使用 scikit-learn 的 model_selection.GridSearchCV 函式庫來進行，如下：

In the previous examples, we always used one fixed set of hyperparameters to run cross validation. However, one use of cross validation is to decide the hyperparameters based on the average model performance. To do this, you can repeatedly change the hyperparameters by hand and rerun the program, write your own loop to brute-force search, or use scikit-learn's model_selection.GridSearchCV library, as shown below:

import numpy as np
from sklearn.datasets import load_wine
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV, StratifiedKFold

dataset = load_wine()
total_data_num = dataset.data.shape[0]
print('Data shapes:', dataset.data.shape, dataset.target.shape)

idx_tr = np.where(np.arange(total_data_num) % 5 != 0)[0]
x_train = dataset.data[idx_tr]
y_train = dataset.target[idx_tr]
print('Training data shapes:', x_train.shape, y_train.shape)

N_FOLD = 3
model = KNeighborsClassifier()

# 定義要搜尋的超參數範圍
param_grid = {
    'n_neighbors': [1, 3, 5, 7, 9]
}

grid_search = GridSearchCV(
    model, 
    param_grid, 
    cv=StratifiedKFold(n_splits=N_FOLD), 
    scoring='accuracy'
)

grid_search.fit(x_train, y_train)

print('Best parameters:', grid_search.best_params_)
print('Best average acc:', grid_search.best_score_ * 100)

總之，透過交叉驗證，找出能取得最好的準確度（或者其他你需要的評估結果）的超參數後，就可以用這組超參數和所有的訓練資料重新訓練一個模型，並在測試集上看看效果。

In short, after using cross validation to find the hyperparameters that give the best accuracy (or any other evaluation result you need), you can then use this set of hyperparameters together with all the training data to train a model again, and check its performance on the test set.

事實上，在交叉驗證當中，能改變的不只有超參數，甚至連選用的模型本身也可以更動。此外，除了使用找出的最佳超參數和所有的訓練資料重新訓練一個模型以外，你也可以將過程中訓練的模型全部直接保留，並將其全部用來對測試集做預測，再將結果取平均或中位數等統計；這樣在每個 fold 的超參數有明顯差異時，有機會取得比較穩定的預測結果。一個簡單的範例如下：

In fact, in cross validation, you can change not only the hyperparameters, but also the model itself. In addition, instead of using the best hyperparameters found and all the training data to train a new model, you can also directly keep all the models trained during the process, use all of them to predict the test set, and then take the average or median of the results. This way, when the hyperparameters differ noticeably between folds, you may get a more stable prediction result. A simple example is shown below:

import numpy as np
from sklearn.datasets import load_wine
from sklearn.neighbors import KNeighborsClassifier

dataset = load_wine()
total_data_num = dataset.data.shape[0]
print('Data shapes:', dataset.data.shape, dataset.target.shape)

idx_tr = np.where(np.arange(total_data_num) % 5 != 0)[0]
x_train = dataset.data[idx_tr]
y_train = dataset.target[idx_tr]
tr_data_num = x_train.shape[0]
print('Training data shapes:', x_train.shape, y_train.shape)

idx_te = np.where(np.arange(total_data_num) % 5 == 0)[0]
x_test = dataset.data[idx_te]
y_test = dataset.target[idx_te]
print('Test data shapes:', x_test.shape, y_test.shape)

N_FOLD = 3
models = []
accuracies = []
for i in range(N_FOLD):
	idx_fold_tr = np.where(np.arange(tr_data_num) % N_FOLD != i)[0]
	idx_fold_va = np.where(np.arange(tr_data_num) % N_FOLD == i)[0]
	model = KNeighborsClassifier(n_neighbors=1)
	model.fit(x_train[idx_fold_tr], y_train[idx_fold_tr])
	pred = model.predict(x_train[idx_fold_va])
	acc = 100 * np.mean(pred == y_train[idx_fold_va])
	models.append(model)
	accuracies.append(acc)
print('Average acc:', np.mean(accuracies))

pred_all = []
for m in models:
	pred = m.predict(x_test)
	pred_all.append(pred)

pred_all = np.array(pred_all) # Shape: (N_FOLD, data_Num)
pred_all = np.median(pred_all, axis=0) # Shape: (data_Num, )
print('Test acc:', 100 * np.mean(pred_all == y_test))

若將上述範例套用到競賽或研究的實務上，你可能會想要將交叉驗證完畢的模型先儲存起來，再用另外一個檔案載入驗證效果比較好的模型並對測試集做預測，以避免將效果不佳的模型拿去做預測而浪費時間。若有需要儲存與載入模型，可以參考 Python 內建的 pickle 函式庫。

If you apply the example above to a real competition or research task, you may want to first save the models trained during cross validation, then load the model with better validation performance from another file and use it to predict the test set. This avoids wasting time using a poorly performing model to make predictions. If you need to save and load models, you can refer to Python's built-in pickle library.