## 矩阵乘法

`y = w[0]*x[0] + w[1]*x[1] + w[2]*x[2] + ... + w[n-1]*x[n-1]`

w和x都是向量，w是权重，x是输入。最终的结果是y是一个标量(scalar)。这个运算称作 multipy-accumulate operations 。而其中的一个操作 `w[0]*x[0] + ..` （先乘后加）称为一个MACC(multipy-accumulate operation)。而上面的计算一共包含n个MACCs。

## 全连接层

`y = matmul(x,W) + b`

## 激活函数

`y = max(x, 0)`

`x` 为输入，这里的输入就是其他层的输出，假如其它层传递给 `RELU层` n个向量，那幺 `RELU层` 对这n个向量进行计算，也就是n个FLOPs。

`y = 1 / (1 + exp(-x))`

## 卷积层

`K  x  K  x  Cin  x  Hout  x  Wout  x  Cout`

```Hout x Wout
K x K
Cin
Cout```

`256 x 256 x 3 x 3 x 3 x 128 = 226,492,416`

## 深度可分离卷积结构

`K x K x C X Hout X Wout`

`3 x 3 x 64 x 112 x 112 = 7,225,344`

`Cin X Hout X Wout X Cout`

`64 x 112 x 112 x 128 = 102,760,448`

```3×3 depthwise          : 7,225,344
1×1 pointwise          : 102,760,448
depthwise separable    : 109,985,792 MACCs
regular 3×3 convolution: 924,844,032 MACCs```

`(K x K x Cin X Hout X Wout) + (Cin x Hout x Wout x Cout)`

`Cin x Hout x Wout X (K x K + Cout)`

```Cexp = (Cin × expansion_factor)
expansion_layer = Cin × Hin × Win × Cexp
depthwise_layer = K × K × Cexp × Hout × Wout
projection_layer = Cexp × Hout × Wout × Cout```

`Cin x Hin X Win X Cexp + (K x K + Cout) x Cexp x Hout x Wout`

`(3 × 3 + 128 + 64) × (64 × 6) × 112 × 112 = 968,196,096`

## 批标准化-BatchNorm

`z = gamma * (y - mean) / sqrt(variance + epsilon) + beta`