在實務上,資料很有可能不是完美的,特徵缺失就是其中一類問題。比方說,有某個預測健康狀況的模型,需要測量四肢的血壓,但是可能有人剛好骨折打石膏,因此其中幾肢的血壓無法測量。而依據資料中缺值的欄位或比率分布等狀況,則各自有不同的方法適合來處理,包含但不限於:
If you apply the example above to a real competition or research task, you may want to first save the models trained during cross validation, then load the model with better validation performance from another file and use it to predict the test set. This avoids wasting time using a poorly performing model to make predictions. If you need to save and load models, you can refer to Python's built-in pickle library.
- 如果含有缺值的樣本比率很低,可以嘗試直接刪除那些樣本。
- 如果某個特徵幾乎都是缺值,可以嘗試直接刪除該特徵。
- 如果「缺失」本身可能是一種資訊(例如問卷填答者,因隱私等原因拒答部分題目),則
- 對於數值特徵,例如年齡等,可以嘗試填寫明顯的異常值,例如 -1;也可以進一步新增一維特徵,來代表原本的那一維是缺失。
- 對於類別特徵,例如居住縣市等,可以嘗試新增一類代表缺失。
- 如果資料是時間序列,例如股價等,可以嘗試填寫前一筆或後一筆有效資料。
- 如果不是前述的特定情況,或嘗試過以後效果不好,可以嘗試以非缺失資料的平均數、中位數或眾數等統計值,來填補缺失處。
- 如果效果仍然不好,或者資源允許,可以嘗試其他進階方法,例如基於相近樣本的 KNN Imputer,或者基於模型預測的 MICE (Multivariate Imputation by Chained Equations)。
- If the proportion of samples with missing values is very low, you can try directly removing those samples.
- If a certain feature is almost entirely missing, you can try directly removing that feature.
- If "missingness" itself may carry information (for example, a survey respondent refuses to answer certain questions for privacy reasons), then
- For numerical features, such as age, you can try filling in an obviously abnormal value, such as -1; you can also add an extra dimension to indicate that the original dimension was missing.
- For categorical features, such as county or city of residence, you can try adding a new category to represent missingness.
- If the data is a time series, such as stock prices, you can try filling in the previous or next valid data point.
- If none of the specific cases above apply, or if the results are not good after trying them, you can try filling in the missing values using statistics such as the mean, median, or mode of the non-missing data.
- If the results are still not good, or if resources allow, you can try other more advanced methods, such as KNN Imputer based on similar samples, or MICE (Multivariate Imputation by Chained Equations) based on model prediction.
上述的「很低」、「幾乎」等形容詞,可能需要根據資料狀況等方面,來決定較具體的標準。
The adjectives above, such as "very low" and "almost," may need more specific criteria depending on the actual data situation.
以下範例,會示範當隨機缺值比率從 1~10% 時,補特定值與使用 KNN Imputer 的模型辨識效果差異:
The example below demonstrates the difference in model recognition performance between filling in a fixed value and using KNN Imputer, when the random missing rate ranges from 1% to 10%:
import matplotlib.pyplot as plt import numpy as np from sklearn.datasets import load_wine from sklearn.impute import KNNImputer from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split np.random.seed(0) dataset = load_wine() X_train, X_test, y_train, y_test = train_test_split( dataset.data, dataset.target, test_size=0.2, random_state=42 ) print('Training data shapes:', X_train.shape, y_train.shape) print('Test data shapes:', X_test.shape, y_test.shape) acc_fill = [] acc_knn = [] missing_rates = np.arange(0.01, 0.11, 0.01) for rate in missing_rates: X_train_miss = X_train.copy() X_test_miss = X_test.copy() mask_train = np.random.rand(*X_train_miss.shape) < rate mask_test = np.random.rand(*X_test_miss.shape) < rate X_train_miss[mask_train] = np.nan X_test_miss[mask_test] = np.nan # Fill by fix val X_train_fill = np.nan_to_num(X_train_miss, nan=0) X_test_fill = np.nan_to_num(X_test_miss, nan=0) # Fill by mean # col_mean = np.nanmean(X_train_miss, axis=0) # X_train_fill = np.where(np.isnan(X_train_miss), col_mean, X_train_miss) # X_test_fill = np.where(np.isnan(X_test_miss), col_mean, X_test_miss) model = LogisticRegression(max_iter=500) model.fit(X_train_fill, y_train) pred = model.predict(X_test_fill) acc_fill.append(accuracy_score(y_test, pred) * 100) # Impute by KNN imputer = KNNImputer(n_neighbors=5) X_train_knn = imputer.fit_transform(X_train_miss) X_test_knn = imputer.transform(X_test_miss) model = LogisticRegression(max_iter=500) model.fit(X_train_knn, y_train) pred = model.predict(X_test_knn) acc_knn.append(accuracy_score(y_test, pred) * 100) plt.plot(missing_rates * 100, acc_fill, '.-', label='Impute by fix val') plt.plot(missing_rates * 100, acc_knn, '.-', label='Impute by KNN') plt.xlabel("Missing rate (%)") plt.ylabel("Accuracy (%)") plt.legend() plt.show()在上述範例中:
In the example above:
- 也同時把補平均值的範例程式碼,以註解的形式提供給各位自行嘗試。
- 示範的缺值模式是完全隨機缺失,若你的資料中的缺值模式不同(例如問卷中,某些欄位容易因為隱私問題讓填卷者拒答;或者不同版本的問卷有題目更動,但你需要把兩種版本合併在一起使用時),則可能適合不同的補值方式。
- 對驗證集(範例中未展示)和測試集補值時,只能使用訓練集計算出來的資訊。
- Example code for filling in the mean value is also provided as a comment, for you to try on your own.
- The missing pattern demonstrated here is completely random missingness. If the missing pattern in your data is different (for example, in a survey, some fields are more likely to be left unanswered due to privacy concerns, or different versions of a survey have different questions but you need to combine both versions together), a different imputation method may be more suitable.
- When imputing values for the validation set (not shown in the example) and the test set, you can only use information calculated from the training set.