Yann LeCunn

RGB图像

## 卷积层|Convolution Layer：

• 滤波器必须具有与输入图像相同数量的通道；
• 网络越深，使用的滤波器就越多；拥有的滤波器越多，获得的边缘和特征检测就越多；

• W ：输入图像的宽度
• H ：输入图像的高度
• Fw  ：滤波器或内核的宽度
• Fh ：滤波器的高度
• P ：填充
• S ：移动步幅

ReLU激活函数

## 反向传播|Backpropagation：

### 全连接层：

• θi+ 1 ：优化的权重
• θi：初始权重
• α ：学习率
• ∇J（θi）：损失函数的梯度

• ∂hij：损失函数的导数

## TensorFlow实现卷积神经网络：

TensorFlow

### 什么是计算图？

• w3 = cos（x），余弦三角函数操作
• w4 = sin（x），正弦三角函数操作
• w5 = w3∙w4，乘法操作
• w6 = w1 / w2，除法操作
• w7 = w5 + w6，加法操作

## 代码：

```# Import the deep learning library
import tensorflow as tf
# Define our compuational graph
W1 = tf.constant(5.0, name = "x")
W2 = tf.constant(3.0, name = "y")
W3 = tf.cos(W1, name = "cos")
W4 = tf.sin(W2, name = "sin")
W5 = tf.multiply(W3, W4, name = "mult")
W6 = tf.divide(W1, W2, name = "div")
# Open the session
with tf.Session() as sess:
cos = sess.run(W3)
sin = sess.run(W4)
mult = sess.run(W5)
div = sess.run(W6)
# Before running TensorBoard, make sure you have generated summary data in a log directory by creating a summary writer
writer = tf.summary.FileWriter("./Desktop/ComputationGraph", sess.graph)
# Once you have event files, run TensorBoard and provide the log directory
# Command: tensorboard --logdir="path/to/logs"```

## 使用Tensorboard进行可视化：

MNIST数据集

### 代码：

```# Import the deep learning library
import tensorflow as tf
import time
# Import the MNIST dataset
from tensorflow.examples.tutorials.mnist import input_data
# Network inputs and outputs
# The network's input is a 28×28 dimensional input
n = 28
m = 28
num_input = n * m # MNIST data input
num_classes = 10 # MNIST total classes (0-9 digits)
# tf Graph input
X = tf.placeholder(tf.float32, [None, num_input])
Y = tf.placeholder(tf.float32, [None, num_classes])
# Storing the parameters of our LeNET-5 inspired Convolutional Neural Network
weights = {
"W_ij": tf.Variable(tf.random_normal([5, 5, 1, 32])),
"W_jk": tf.Variable(tf.random_normal([5, 5, 32, 64])),
"W_kl": tf.Variable(tf.random_normal([7 * 7 * 64, 1024])),
"W_lm": tf.Variable(tf.random_normal([1024, num_classes]))
}
biases = {
"b_ij": tf.Variable(tf.random_normal([32])),
"b_jk": tf.Variable(tf.random_normal([64])),
"b_kl": tf.Variable(tf.random_normal([1024])),
"b_lm": tf.Variable(tf.random_normal([num_classes]))
}
# The hyper-parameters of our Convolutional Neural Network
learning_rate = 1e-3
num_steps = 500
batch_size = 128
display_step = 10
def ConvolutionLayer(x, W, b, strides=1):
# Convolution Layer
x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
return x
def ReLU(x):
# ReLU activation function
return tf.nn.relu(x)
def PoolingLayer(x, k=2, strides=2):
# Max Pooling layer
return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, strides, strides, 1],
def Softmax(x):
# Softmax activation function for the CNN's final output
return tf.nn.softmax(x)
# Create model
def ConvolutionalNeuralNetwork(x, weights, biases):
# MNIST data input is a 1-D row vector of 784 features (28×28 pixels)
# Reshape to match picture format [Height x Width x Channel]
# Tensor input become 4-D: [Batch Size, Height, Width, Channel]
x = tf.reshape(x, shape=[-1, 28, 28, 1])
# Convolution Layer
Conv1 = ConvolutionLayer(x, weights["W_ij"], biases["b_ij"])
# Non-Linearity
ReLU1 = ReLU(Conv1)
# Max Pooling (down-sampling)
Pool1 = PoolingLayer(ReLU1, k=2)
# Convolution Layer
Conv2 = ConvolutionLayer(Pool1, weights["W_jk"], biases["b_jk"])
# Non-Linearity
ReLU2 = ReLU(Conv2)
# Max Pooling (down-sampling)
Pool2 = PoolingLayer(ReLU2, k=2)
# Fully connected layer
# Reshape conv2 output to fit fully connected layer input
FC = tf.reshape(Pool2, [-1, weights["W_kl"].get_shape().as_list()[0]])
FC = ReLU(FC)
# Output, class prediction
return output
# Construct model
logits = ConvolutionalNeuralNetwork(X, weights, biases)
prediction = Softmax(logits)
# Softamx cross entropy loss function
loss_function = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
logits=logits, labels=Y))
training_process = optimizer.minimize(loss_function)
# Evaluate model
correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
# recording how the loss functio varies over time during training
cost = tf.summary.scalar("cost", loss_function)
training_accuracy = tf.summary.scalar("accuracy", accuracy)
train_summary_op = tf.summary.merge([cost,training_accuracy])
train_writer = tf.summary.FileWriter("./Desktop/logs",
graph=tf.get_default_graph())
# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()
# Start training
with tf.Session() as sess:
# Run the initializer
sess.run(init)
start_time = time.time()
for step in range(1, num_steps+1):
batch_x, batch_y = mnist.train.next_batch(batch_size)
# Run optimization op (backprop)
sess.run(training_process, feed_dict={X: batch_x, Y: batch_y})
if step % display_step == 0 or step == 1:
# Calculate batch loss and accuracy
loss, acc, summary = sess.run([loss_function, accuracy, train_summary_op], feed_dict={X: batch_x,
Y: batch_y})
print("Step " + str(step) + ", Minibatch Loss= " + \
"{:.4f}".format(loss) + ", Training Accuracy= " + \
"{:.3f}".format(acc))
end_time = time.time()
print("Time duration: " + str(int(end_time-start_time)) + " seconds")
print("Optimization Finished!")
# Calculate accuracy for 256 MNIST test images
print("Testing Accuracy:", \
sess.run(accuracy, feed_dict={X: mnist.test.images[:256],
Y: mnist.test.labels[:256]}))```

```Step 1, Minibatch Loss= 74470.4844, Training Accuracy= 0.117
Step 10, Minibatch Loss= 20529.4141, Training Accuracy= 0.250
Step 20, Minibatch Loss= 14074.7539, Training Accuracy= 0.531
Step 30, Minibatch Loss= 7168.9839, Training Accuracy= 0.586
Step 40, Minibatch Loss= 4781.1060, Training Accuracy= 0.703
Step 50, Minibatch Loss= 3281.0979, Training Accuracy= 0.766
Step 60, Minibatch Loss= 2701.2451, Training Accuracy= 0.781
Step 70, Minibatch Loss= 2478.7153, Training Accuracy= 0.773
Step 80, Minibatch Loss= 2312.8320, Training Accuracy= 0.820
Step 90, Minibatch Loss= 2143.0774, Training Accuracy= 0.852
Step 100, Minibatch Loss= 1373.9169, Training Accuracy= 0.852
Step 110, Minibatch Loss= 1852.9535, Training Accuracy= 0.852
Step 120, Minibatch Loss= 1845.3500, Training Accuracy= 0.891
Step 130, Minibatch Loss= 1677.2566, Training Accuracy= 0.844
Step 140, Minibatch Loss= 1683.3661, Training Accuracy= 0.875
Step 150, Minibatch Loss= 1859.3821, Training Accuracy= 0.836
Step 160, Minibatch Loss= 1495.4796, Training Accuracy= 0.859
Step 170, Minibatch Loss= 609.3800, Training Accuracy= 0.914
Step 180, Minibatch Loss= 1376.5054, Training Accuracy= 0.891
Step 190, Minibatch Loss= 1085.0363, Training Accuracy= 0.891
Step 200, Minibatch Loss= 1129.7145, Training Accuracy= 0.914
Step 210, Minibatch Loss= 1488.5452, Training Accuracy= 0.906
Step 220, Minibatch Loss= 584.5027, Training Accuracy= 0.930
Step 230, Minibatch Loss= 619.9744, Training Accuracy= 0.914
Step 240, Minibatch Loss= 1575.8933, Training Accuracy= 0.891
Step 250, Minibatch Loss= 1558.5853, Training Accuracy= 0.891
Step 260, Minibatch Loss= 375.0371, Training Accuracy= 0.922
Step 270, Minibatch Loss= 1568.0758, Training Accuracy= 0.859
Step 280, Minibatch Loss= 1172.9205, Training Accuracy= 0.914
Step 290, Minibatch Loss= 1023.5415, Training Accuracy= 0.914
Step 300, Minibatch Loss= 475.9756, Training Accuracy= 0.945
Step 310, Minibatch Loss= 488.8930, Training Accuracy= 0.961
Step 320, Minibatch Loss= 1105.7720, Training Accuracy= 0.914
Step 330, Minibatch Loss= 1111.8589, Training Accuracy= 0.906
Step 340, Minibatch Loss= 842.7805, Training Accuracy= 0.930
Step 350, Minibatch Loss= 1514.0153, Training Accuracy= 0.914
Step 360, Minibatch Loss= 1722.1812, Training Accuracy= 0.875
Step 370, Minibatch Loss= 681.6041, Training Accuracy= 0.891
Step 380, Minibatch Loss= 902.8599, Training Accuracy= 0.930
Step 390, Minibatch Loss= 714.1541, Training Accuracy= 0.930
Step 400, Minibatch Loss= 1654.8883, Training Accuracy= 0.914
Step 410, Minibatch Loss= 696.6915, Training Accuracy= 0.906
Step 420, Minibatch Loss= 536.7183, Training Accuracy= 0.914
Step 430, Minibatch Loss= 1405.9148, Training Accuracy= 0.891
Step 440, Minibatch Loss= 199.4781, Training Accuracy= 0.953
Step 450, Minibatch Loss= 438.3784, Training Accuracy= 0.938
Step 460, Minibatch Loss= 409.6419, Training Accuracy= 0.969
Step 470, Minibatch Loss= 503.1216, Training Accuracy= 0.930
Step 480, Minibatch Loss= 482.6476, Training Accuracy= 0.922
Step 490, Minibatch Loss= 767.3893, Training Accuracy= 0.922
Step 500, Minibatch Loss= 626.8249, Training Accuracy= 0.930
Time duration: 657 seconds
Optimization Finished!
Testing Accuracy: 0.9453125```