Last updated on 2019年4月13日

## 在Python中实现机器学习功能的4种方法

1. 单变量特征选择

1. 递归特征消除(RFE)

1. 主成分分析（PCA）

1. 特征选择 (feature importance)

#### 单变量特征选择

scikit-learn库提供SelectKBest类，可以与一组不同的统计测试一起使用，以选择特定数量的功能。

```#Feature Extraction with Univariate Statistical Tests (Chi-squared for classification)
#Import the required packages
#Import pandas to read csv import pandas
#Import numpy for array related operations import numpy
#Import sklearn's feature selection algorithm
from sklearn.feature_selection import SelectKBest
#Import chi2 for performing chi square test from sklearn.feature_selection import chi2
url ="https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians diabetes/pima-indians-diabetes.data"
#Define the attribute names
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
#Create array from data values
array = dataframe.values
#Split the data into input and target
X = array[:,0:8]
Y = array[:,8]
#We will select the features using chi square
test = SelectKBest(score_func=chi2, k=4)
#Fit the function for ranking the features by score
fit = test.fit(X, Y)
#Summarize scores numpy.set_printoptions(precision=3) print(fit.scores_)
#Apply the transformation on to dataset
features = fit.transform(X)
#Summarize selected features print(features[0:5,:])```

```[111.52   1411.887 17.605 53.108  2175.565   127.669 5.393
181.304]```

```[[148. 0. 33.6 50. ]
[85. 0. 26.6 31. ]
[183. 0. 23.3 32. ]
[89. 94. 28.1 21. ]
[137. 168. 43.1 33. ]]```

#### 递归特征消除(RFE)

RFE通过递归删除属性并在剩余的属性上构建模型来工作。它使用模型精度来识别哪些属性（和属性组合）对预测目标属性的贡献最大。以下示例使用RFE和逻辑回归算法来选择前三个特征。算法的选择并不重要，只要它技巧性和一致性：

```#Import the required packages
#Import pandas to read csv import pandas
#Import numpy for array related operations import numpy
#Import sklearn's feature selection algorithm from sklearn.feature_selection import RFE
#Import LogisticRegression for performing chi square test from sklearn.linear_model import LogisticRegression
url =
"https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians-dia betes/pima-indians-diabetes.data"
#Define the attribute names
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
#Create array from data values
array = dataframe.values
#Split the data into input and target
X = array[:,0:8]
Y = array[:,8]
#Feature extraction
model = LogisticRegression() rfe = RFE(model, 3)
fit = rfe.fit(X, Y)
print("Num Features: %d"% fit.n_features_) print("Selected Features: %s"% fit.support_) print("Feature Ranking: %s"% fit.ranking_)```

```Num Features: 3
Selected Features: [ True False False False False   True  True False]
Feature Ranking: [1 2 3 5 6 1 1 4]```

#### 主成分分析（PCA）

PCA使用线性代数将数据集转换为压缩形式。通常，它被认为是数据简化技术。PCA的一个属性是您可以选择转换结果中的维数或主成分数。

```#Import the required packages
#Import pandas to read csv import pandas
#Import numpy for array related operations import numpy
#Import sklearn's PCA algorithm
from sklearn.decomposition import PCA
url =
"https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians diabetes/pima-indians-diabetes.data"
#Define the attribute names
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
#Create array from data values
array = dataframe.values
#Split the data into input and target
X = array[:,0:8]
Y = array[:,8]
#Feature extraction
pca = PCA(n_components=3) fit = pca.fit(X)
#Summarize components
print("Explained Variance: %s") % fit.explained_variance_ratio_
print(fit.components_)```

```Explained Variance: [ 0.88854663   0.06159078  0.02579012]
[[ -2.02176587e-03    9.78115765e-02 1.60930503e-02    6.07566861e-02
9.93110844e-01          1.40108085e-02 5.37167919e-04   -3.56474430e-03]
[ -2.26488861e-02   -9.72210040e-01              -1.41909330e-01  5.78614699e-02 9.46266913e-02   -4.69729766e-02               -8.16804621e-04  -1.40168181e-01
[ -2.24649003e-02 1.43428710e-01                 -9.22467192e-01  -3.07013055e-01 2.09773019e-02   -1.32444542e-01                -6.39983017e-04  -1.25454310e-01]]```

#### 特征选择 (feature importance)

```#Import the supporting libraries
#Import pandas to load the dataset from csv file
#Import numpy for array based operations and calculations
import numpy as np
#Import Random Forest classifier class from sklearn
from sklearn.ensemble import RandomForestClassifier
#Import feature selector class select model of sklearn
from sklearn.feature_selection
import SelectFromModel
np.random.seed(1)```

```#Function to create Train and Test set from the original dataset def getTrainTestData(dataset,split):
np.random.seed(0) training = [] testing = []
np.random.shuffle(dataset) shape = np.shape(dataset)
trainlength = np.uint16(np.floor(split*shape[0]))
for i in range(trainlength): training.append(dataset[i])
for i in range(trainlength,shape[0]): testing.append(dataset[i])
training = np.array(training) testing = np.array(testing)
return training,testing```

```#Function to evaluate model performance
def getAccuracy(pre,ytest): count = 0
for i in range(len(ytest)):
if ytest[i]==pre[i]: count+=1
acc = float(count)/len(ytest)
return acc```

```#Load dataset as pandas data frame
#Extract attribute names from the data frame
feat = data.keys()
feat_labels = feat.get_values()
#Extract data values from the data frame
dataset = data.values
#Shuffle the dataset
np.random.shuffle(dataset)
#We will select 50000 instances to train the classifier
inst = 50000
#Extract 50000 instances from the dataset
dataset = dataset[0:inst,:]
#Create Training and Testing data for performance evaluation
train,test = getTrainTestData(dataset, 0.7)
#Split data into input and output variable with selected features
Xtrain = train[:,0:94] ytrain = train[:,94] shape = np.shape(Xtrain)
print("Shape of the dataset ",shape)
#Print the size of Data in MBs
print("Size of Data set before feature selection: %.2f MB"%(Xtrain.nbytes/1e6))```

```Shape of the dataset (35000, 94)
Size of Data set before feature selection: 26.32 MB```

```#Lets select the test data for model evaluation purpose
Xtest = test[:,0:94] ytest = test[:,94]
#Create a random forest classifier with the following Parameters
trees            = 250
max_feat     = 7
max_depth = 30
min_sample = 2
clf = RandomForestClassifier(n_estimators=trees,
max_features=max_feat,
max_depth=max_depth,
min_samples_split= min_sample, random_state=0,
n_jobs=-1)
#Train the classifier and calculate the training time
import time
start = time.time() clf.fit(Xtrain, ytrain) end = time.time()
#Lets Note down the model training time
print("Execution time for building the Tree is: %f"%(float(end)- float(start)))
pre = clf.predict(Xtest)
Let's see how much time is required to train the model on the training dataset:
Execution time for building the Tree is: 2.913641
#Evaluate the model performance for the test data
acc = getAccuracy(pre, ytest)
print("Accuracy of model before feature selection is %.2f"%(100*acc))```

#### 特征选择前的模型精度为98.82

#Once我们培养的模型中，我们的排名将所有功能的功能 在 拉链（feat_labels，clf.feature_importances_）：

```print(feature)
('id', 0.33346650420175183)
('feat_1', 0.0036186958628801214)
('feat_2', 0.0037243050888530957)
('feat_3', 0.011579217472062748)
('feat_4', 0.010297382675187445)
('feat_5', 0.0010359139416194116)
('feat_6', 0.00038171336038056165)
('feat_7', 0.0024867672489765021)
('feat_8', 0.0096689721610546085)
('feat_9', 0.007906150362995093)
('feat_10', 0.0022342480802130366)```

```#Select features which have higher contribution in the final prediction
sfm = SelectFromModel(clf, threshold=0.01) sfm.fit(Xtrain,ytrain)```

```#Transform input dataset
Xtrain_1 = sfm.transform(Xtrain) Xtest_1      = sfm.transform(Xtest)
#Let's see the size and shape of new dataset print("Size of Data set before feature selection: %.2f MB"%(Xtrain_1.nbytes/1e6))
shape = np.shape(Xtrain_1)
print("Shape of the dataset ",shape)
Size of Data set before feature selection: 5.60 MB Shape of the dataset (35000, 20)```

```#Model training time
start = time.time() clf.fit(Xtrain_1, ytrain) end = time.time()
print("Execution time for building the Tree is: %f"%(float(end)- float(start)))
#Let's evaluate the model on test data
pre = clf.predict(Xtest_1) count = 0
acc2 = getAccuracy(pre, ytest)
print("Accuracy after feature selection %.2f"%(100*acc2))
Execution time for building the Tree is: 1.711518 Accuracy after feature selection 99.97```