CNN 编码深度形状先验知识

3D 数据的不同表征

几何形式：不能直接应用 CNN

```# point_cloud1 and point_cloud2 represent the same 3D structure
# even though they are represented differently in memory
point_cloud1 = [(x1, y1, z1), (x2, y2, z2),..., (xn, yn, zn)]
point_cloud2 = [(x2, y2, z2), (x1, y1, z1),..., (xn, yn, zn)]```

2D 结构生成器

```#--------- Pytorch pseudo-code for Structure Generator ---------#
class Structure_Generator(nn.Module):
# contains two module in sequence, an encoder and a decoder
def __init__(self):
self.encoder = Encoder()
self.decoder = Decoder()
def forward(self, RGB_image):
# Encoder takes in one RGB image and
# output an encoded deep shape-embedding
shape_embedding = self.encoder(RGB_image)
# Decoder takes the encoded values and output
# multiples 2D projection (XYZ + mask)

训练动态

```# --------- Pytorch pseudo-code for training loop ----------#
# Create 2D Conv Structure generator
model = Structure_Generator()
# only need to learn the 2D structure optimizer
optimizer = optim.SGD(model.parameters())
# 2D projections from predetermined viewpoints
# fused point cloud
#fuseTrans is predetermined viewpoints info
XYZid, ML = fuse3D(XYZ, maskLogit, fuseTrans)
# Render new depth images at novel viewpoints
# renderTrans is novel viewpoints info
newDepth, newMaskLogit, collision = render2D(XYZid, ML, renderTrans)
# Compute loss between novel view and ground truth
loss_depth = L1Loss()(newDepth, GTDepth)
# Back-propagation to update Structure Generator
loss_total.backward()
optimizer.step()```

参考

Pytorch 代码：https://github.com/lkhphuc/pytorch-3d-point-cloud-generation

Tensorflow 代码：https://github.com/chenhsuanlin/3D-point-cloud-generation