• 如何使用Keras API创建向量范数约束；
• 如何使用Keras API为MLP、CNN和RNN层添加权重约束；
• 如何通过向现有模型添加权重约束来减少过度拟合；

• Keras中的权重约束；
• 图层上的权重约束；
• 权重约束案例研究；

## Keras中权重约束

API支持权重约束，且约束可以按每层指定。

• 最大范数（max_norm）：强制权重等于或低于给定限制;
• 非负规范（non_neg）：强制权重为正数；
• 单位范数（unit_norm）：强制权重为1.0；
• Min-Max范数（min_max_norm）：强制权重在一个范围之间；

``````# import norm
from keras.constraints import max_norm
# instantiate norm
norm = max_norm(3.0)``````

## 图层上的权重约束

### MLP权重约束

``````# example of max norm on a dense layer
from keras.layers import Dense
from keras.constraints import max_norm
...
...``````

### CNN权重约束

``````# example of max norm on a cnn layer
from keras.layers import Conv2D
from keras.constraints import max_norm
...
...``````

### RNN权重约束

``````# example of max norm on an lstm layer
from keras.layers import LSTM
from keras.constraints import max_norm
...
...``````

## 权重约束案例研究

### 二分类问题

``````# generate 2d classification dataset
X, y = make_moons(n_samples=100, noise=0.2, random_state=1)``````

``````# generate two moons dataset
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
# generate 2d classification dataset
X, y = make_moons(n_samples=100, noise=0.2, random_state=1)
# scatter plot, dots colored by class value
df = DataFrame(dict(x=X[:,0], y=X[:,1], label=y))
colors = {0:'red', 1:'blue'}
fig, ax = pyplot.subplots()
grouped = df.groupby('label')
for key, group in grouped:
group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
pyplot.show()``````

## 多层感知器过拟合

MLP模型只具有一个隐藏层，但具有比解决该问题所需的节点更多的节点，从而提供过拟合的可能。

``````# generate 2d classification dataset
X, y = make_moons(n_samples=100, noise=0.2, random_state=1)
# split into train and test
n_train = 30
trainX, testX = X[:n_train, :], X[n_train:, :]
trainy, testy = y[:n_train], y[n_train:]``````

``````# define model
model = Sequential()

``````# fit model
history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=0)``````

``````# evaluate the model
_, train_acc = model.evaluate(trainX, trainy, verbose=0)
_, test_acc = model.evaluate(testX, testy, verbose=0)
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))``````

``````# plot history
pyplot.plot(history.history['acc'], label='train')
pyplot.plot(history.history['val_acc'], label='test')
pyplot.legend()
pyplot.show()``````

``````# mlp overfit on the moons dataset
from sklearn.datasets import make_moons
from keras.layers import Dense
from keras.models import Sequential
from matplotlib import pyplot
# generate 2d classification dataset
X, y = make_moons(n_samples=100, noise=0.2, random_state=1)
# split into train and test
n_train = 30
trainX, testX = X[:n_train, :], X[n_train:, :]
trainy, testy = y[:n_train], y[n_train:]
# define model
model = Sequential()
# fit model
history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=0)
# evaluate the model
_, train_acc = model.evaluate(trainX, trainy, verbose=0)
_, test_acc = model.evaluate(testX, testy, verbose=0)
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
# plot history
pyplot.plot(history.history['acc'], label='train')
pyplot.plot(history.history['val_acc'], label='test')
pyplot.legend()
pyplot.show()``````

``Train: 1.000, Test: 0.914``

## 具有权重约束的多层感知器过拟合

``model.add(Dense(500, input_dim=2, activation='relu', kernel_constraint=unit_norm()))``

``model.add(Dense(500, input_dim=2, activation='relu', kernel_constraint=min_max_norm(min_value=1.0, max_value=1.0)))``

``model.add(Dense(500, input_dim=2, activation='relu', kernel_constraint=max_norm(1.0)))``

``````# mlp overfit on the moons dataset with a unit norm constraint
from sklearn.datasets import make_moons
from keras.layers import Dense
from keras.models import Sequential
from keras.constraints import unit_norm
from matplotlib import pyplot
# generate 2d classification dataset
X, y = make_moons(n_samples=100, noise=0.2, random_state=1)
# split into train and test
n_train = 30
trainX, testX = X[:n_train, :], X[n_train:, :]
trainy, testy = y[:n_train], y[n_train:]
# define model
model = Sequential()
# fit model
history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=0)
# evaluate the model
_, train_acc = model.evaluate(trainX, trainy, verbose=0)
_, test_acc = model.evaluate(testX, testy, verbose=0)
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
# plot history
pyplot.plot(history.history['acc'], label='train')
pyplot.plot(history.history['val_acc'], label='test')
pyplot.legend()
pyplot.show()``````

``Train: 1.000, Test: 0.943``

## 扩展

• ：更新示例以计算网络权重的大小，并证明使用约束后，确实使得幅度更小；
• ：更新示例以将约束添加到模型的输出层并比较结果；
• ：更新示例以向偏差权重添加约束并比较结果；
• ：更新示例以多次拟合和评估模型，并报告模型性能的均值和标准差；

## 作者信息

Jason Brownlee，机器学习专家