Privacy-Preserving Filter-based Feature Selection with Secure Multiparty Computation

Abstract

The data pre-processing stage with steps of data cleaning (handling of missing/noisy data, dealing with outliers), data transformation (normalization, discretization, and rebalancing), and data reduction (feature extraction/selection) is crucial for the machine learning work- flow. Existing work on privacy-preserving machine learning (PPML) with Secure Multiparty Computation (MPC) is almost exclusively focused on model training and on inference with trained models, thereby overlooking the important data pre-processing stage. In this work, we propose an MPC based protocol π_FILTER−FS for private feature selection based on the filter method. It is independent of model training, and can be used in combination with any MPC protocol to rank features. For ranking of features in a privacy-preserving manner, we propose a feature scoring protocol π_MS−GINI based on Gini impurity. The computation of a Gini score for continuous valued features traditionally requires sorting of the feature values to determine candidate split points in the feature value range. As sorting is an expensive operation to perform in a privacy-preserving way, we instead propose a “mean-split Gini score” (MS-GINI) that avoids the need for sorting by selecting the mean of the feature values as the split point. Feature selection with MS-GINI leads to accuracy improvements that are on par with those obtained with the traditional Gini score in the data sets used in our experiments.To demonstrate the feasibility of our approach for practical data science, we propose a protocol π_GINI−FS, which combines π_FILTER−FS with π_MS−GINI, and perform experiments with the proposed MPC protocols for feature selection in a commonly used machine-learning- as-a-service configuration where computations are outsourced to three servers (3PC), with semi-honest and malicious adversaries. Regarding effectiveness, we show that secure feature selection with the proposed protocols improves the accuracy of classifiers on a variety of real-world data sets, without leaking information about the feature values or even which features were selected. Regarding efficiency, we document runtimes between ≈48 sec and ≈25 hours for our protocols to finish, depending on the size of the data set, the security settings, and the hardware configuration.