#### 3.1 文本生成之数据处理

3.1.1 读取数据

```# https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt
input_filepath = "./shakespeare.txt"
text = open(input_filepath, 'r').read()
print(len(text))
print(text[0:100])```

```1115394
First Citizen:
Before we proceed any further, hear me speak.
All:
Speak, speak.
First Citizen:
You```

3.1.2 对数据进行处理

```# 1. generate vocab
# 2. build mapping char->id
# 3. data -> id_data
# 4. abcd -> bcd<eos>
vocab = sorted(set(text))
print(len(vocab))
print(vocab)```

```65
['
', ' ', '!', '\$', '&', "'", ',', '-', '.', '3', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']```

3.1.3 对词表进行映射

```char2idx = {char:idx for idx, char in enumerate(vocab)}
print(char2idx)```

enumerate枚举，枚举之后会给vocal中的每一个元素都有一个idx，将idx作为id就可以了

```{'
': 0, ' ': 1, '!': 2, '\$': 3, '&': 4, "'": 5, ',': 6, '-': 7, '.': 8, '3': 9, ':': 10, ';': 11, '?': 12, 'A': 13, 'B': 14, 'C': 15, 'D': 16, 'E': 17, 'F': 18, 'G': 19, 'H': 20, 'I': 21, 'J': 22, 'K': 23, 'L': 24, 'M': 25, 'N': 26, 'O': 27, 'P': 28, 'Q': 29, 'R': 30, 'S': 31, 'T': 32, 'U': 33, 'V': 34, 'W': 35, 'X': 36, 'Y': 37, 'Z': 38, 'a': 39, 'b': 40, 'c': 41, 'd': 42, 'e': 43, 'f': 44, 'g': 45, 'h': 46, 'i': 47, 'j': 48, 'k': 49, 'l': 50, 'm': 51, 'n': 52, 'o': 53, 'p': 54, 'q': 55, 'r': 56, 's': 57, 't': 58, 'u': 59, 'v': 60, 'w': 61, 'x': 62, 'y': 63, 'z': 64}```

3.1.4 把列表变成numpy.array

```idx2char = np.array(vocab)
print(idx2char)```

```['
' ' ' '!' '\$' '&' "'" ',' '-' '.' '3' ':' ';' '?' 'A' 'B' 'C' 'D' 'E' 'F' 'G' 'H' 'I' 'J' 'K' 'L' 'M' 'N' 'O' 'P' 'Q' 'R' 'S' 'T' 'U' 'V' 'W' 'X' 'Y' 'Z' 'a' 'b' 'c' 'd' 'e' 'f' 'g' 'h' 'i' 'j' 'k' 'l' 'm' 'n' 'o' 'p' 'q' 'r' 's' 't' 'u' 'v' 'w' 'x' 'y' 'z']```

3.1.5 把字符的id转化为id

```text_as_int = np.array([char2idx[c] for c in text])
print(text_as_int[0:10])
print(text[0:10])```

```[18 47 56 57 58  1 15 47 58 47]
First Citi```

3.1.6 生成输出

from_tensor_slices生成dataset

```def split_input_target(id_text):
"""
abcde -> abcd, bcde
"""
return id_text[0:-1], id_text[1:]
char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
seq_length = 100
# 这个batch是未了将字符变成句子
seq_dataset = char_dataset.batch(seq_length + 1,
drop_remainder = True)
for ch_id in char_dataset.take(2):
print(ch_id, idx2char[ch_id.numpy()])
for seq_id in seq_dataset.take(2):
print(seq_id)
print(repr(''.join(idx2char[seq_id.numpy()])))```

```tf.Tensor(18, shape=(), dtype=int64) F
tf.Tensor(47, shape=(), dtype=int64) i
tf.Tensor(
[18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 14 43 44 53 56 43  1 61 43
1 54 56 53 41 43 43 42  1 39 52 63  1 44 59 56 58 46 43 56  6  1 46 43
39 56  1 51 43  1 57 54 43 39 49  8  0  0 13 50 50 10  0 31 54 43 39 49
6  1 57 54 43 39 49  8  0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10
0 37 53 59  1], shape=(101,), dtype=int64)
'First Citizen:
Before we proceed any further, hear me speak.
All:
Speak, speak.
First Citizen:
You '
tf.Tensor(
[39 56 43  1 39 50 50  1 56 43 57 53 50 60 43 42  1 56 39 58 46 43 56  1
58 53  1 42 47 43  1 58 46 39 52  1 58 53  1 44 39 51 47 57 46 12  0  0
13 50 50 10  0 30 43 57 53 50 60 43 42  8  1 56 43 57 53 50 60 43 42  8
0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 18 47 56 57 58  6  1
63 53 59  1 49], shape=(101,), dtype=int64)
'are all resolved rather to die than to famish?
All:
Resolved. resolved.
First Citizen:
First, you k'```

3.1.7 调用split_input_target来获得输入和输出

```seq_dataset = seq_dataset.map(split_input_target)
for item_input, item_output in seq_dataset.take(2):
print(item_input.numpy())
print(item_output.numpy())```

```[18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 14 43 44 53 56 43  1 61 43
1 54 56 53 41 43 43 42  1 39 52 63  1 44 59 56 58 46 43 56  6  1 46 43
39 56  1 51 43  1 57 54 43 39 49  8  0  0 13 50 50 10  0 31 54 43 39 49
6  1 57 54 43 39 49  8  0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10
0 37 53 59]
[47 56 57 58  1 15 47 58 47 64 43 52 10  0 14 43 44 53 56 43  1 61 43  1
54 56 53 41 43 43 42  1 39 52 63  1 44 59 56 58 46 43 56  6  1 46 43 39
56  1 51 43  1 57 54 43 39 49  8  0  0 13 50 50 10  0 31 54 43 39 49  6
1 57 54 43 39 49  8  0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10  0
37 53 59  1]
[39 56 43  1 39 50 50  1 56 43 57 53 50 60 43 42  1 56 39 58 46 43 56  1
58 53  1 42 47 43  1 58 46 39 52  1 58 53  1 44 39 51 47 57 46 12  0  0
13 50 50 10  0 30 43 57 53 50 60 43 42  8  1 56 43 57 53 50 60 43 42  8
0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 18 47 56 57 58  6  1
63 53 59  1]
[56 43  1 39 50 50  1 56 43 57 53 50 60 43 42  1 56 39 58 46 43 56  1 58
53  1 42 47 43  1 58 46 39 52  1 58 53  1 44 39 51 47 57 46 12  0  0 13
50 50 10  0 30 43 57 53 50 60 43 42  8  1 56 43 57 53 50 60 43 42  8  0
0 18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 18 47 56 57 58  6  1 63
53 59  1 49]```

3.1.8 形成batch

```batch_size = 64
buffer_size = 10000
# 这个batch就是未了形成一个batch
# drop_remainder=True：最后一组要是不够一个batch，则丢弃掉
seq_dataset = seq_dataset.shuffle(buffer_size).batch(
batch_size, drop_remainder=True)```

#### 3.2 构建模型

3.2.1定义一个函数来构建模型

```vocab_size = len(vocab)
embedding_dim = 256
rnn_units = 1024
def build_model(vocab_size, embedding_dim, rnn_units, batch_size):
model = keras.models.Sequential([
keras.layers.Embedding(vocab_size, embedding_dim,
batch_input_shape = [batch_size, None]),
keras.layers.SimpleRNN(units = rnn_units,
stateful = True,
recurrent_initializer = 'glorot_uniform',
# 输入和输出都是序列
return_sequences = True),
keras.layers.Dense(vocab_size),
])
return model
model = build_model(
vocab_size = vocab_size,
embedding_dim = embedding_dim,
rnn_units = rnn_units,
batch_size = batch_size)
model.summary()```

```Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding (Embedding)        (64, None, 256)           16640
_________________________________________________________________
simple_rnn (SimpleRNN)       (64, None, 1024)          1311744
_________________________________________________________________
dense (Dense)                (64, None, 65)            66625
=================================================================
Total params: 1,395,009
Trainable params: 1,395,009
Non-trainable params: 0
_________________________________________________________________```

3.2.2 用模型进行预测

```for input_example_batch, target_example_batch in seq_dataset.take(1):
example_batch_predictions = model(input_example_batch)
print(example_batch_predictions.shape)```

`(64, 100, 65)`

64：batch_size

100: 句子的长度

65： vocab_size，相当于一个类别预测

3.2.3 随机采样

tf.random.categorical：用来随机sample
tf.squeeze：用来操作维度，减少一个维度

```# random sampling.
# greedy（贪心策略）, random（随机策略）.
sample_indices = tf.random.categorical(
logits = example_batch_predictions[0], num_samples = 1)
print(sample_indices)
# (100, 65) -> (100, 1)
sample_indices = tf.squeeze(sample_indices, axis = -1)
print(sample_indices)```

```tf.Tensor(
[[52]
[25]
[ 3]
[21]
[43]
[45]
[ 3]
[51]
[56]
[26]
[25]
[27]
[26]
[58]
[25]
[34]
[42]
[48]
[27]
[19]
[24]
[45]
[12]
[44]
[39]
[59]
[26]
[ 5]
[48]
[48]
[23]
[12]
[28]
[58]
[30]
[33]
[15]
[22]
[24]
[ 1]
[19]
[ 7]
[58]
[32]
[ 1]
[62]
[ 5]
[26]
[23]
[51]
[58]
[30]
[16]
[61]
[31]
[17]
[16]
[13]
[26]
[11]
[31]
[63]
[ 0]
[ 0]
[45]
[29]
[54]
[47]
[ 8]
[ 2]
[35]
[ 2]
[39]
[32]
[54]
[41]
[47]
[33]
[56]
[27]
[27]
[28]
[13]
[55]
[62]
[51]
[25]
[64]
[53]
[15]
[51]
[ 3]
[21]
[62]
[37]
[17]
[64]
[ 9]
[30]
[17]], shape=(100, 1), dtype=int64)
tf.Tensor(
[52 25  3 21 43 45  3 51 56 26 25 27 26 58 25 34 42 48 27 19 24 45 12 44
39 59 26  5 48 48 23 12 28 58 30 33 15 22 24  1 19  7 58 32  1 62  5 26
23 51 58 30 16 61 31 17 16 13 26 11 31 63  0  0 45 29 54 47  8  2 35  2
39 32 54 41 47 33 56 27 27 28 13 55 62 51 25 64 53 15 51  3 21 62 37 17
64  9 30 17], shape=(100,), dtype=int64)```

3.2.4 打印输入输出

```print("Input: ", repr("".join(idx2char[input_example_batch[0]])))
print()
print("Output: ", repr("".join(idx2char[target_example_batch[0]])))
print()
print("Predictions: ", repr("".join(idx2char[sample_indices])))```

```Input:  'ture
To mingle faith with him! Undone! undone!
If I might die within this hour, I have lived
To die '
Output:  'ure
To mingle faith with him! Undone! undone!
If I might die within this hour, I have lived
To die w'
Predictions:  "nM\$Ieg\$mrNMONtMVdjOGLg?fauN'jjK?PtRUCJL G-tT x'NKmtRDwSEDAN;Sy
gQpi.!W!aTpciUrOOPAqxmMzoCm\$IxYEz3RE"```

prediction是乱码是因为model还没有进行训练

3.2.5 定义损失函数，对model进行compile

```def loss(labels, logits):
return keras.losses.sparse_categorical_crossentropy(
labels, logits, from_logits=True)
model.compile(optimizer = 'adam', loss = loss)
example_loss = loss(target_example_batch, example_batch_predictions)
print(example_loss.shape)
print(example_loss.numpy().mean())```

```(64, 100)
4.1839275```

3.2.6 训练模型

```output_dir = "./text_generation_checkpoints"
if not os.path.exists(output_dir):
os.mkdir(output_dir)
checkpoint_prefix = os.path.join(output_dir, 'ckpt_{epoch}')
checkpoint_callback = keras.callbacks.ModelCheckpoint(
filepath = checkpoint_prefix,
save_weights_only = True)
epochs = 100
history = model.fit(seq_dataset, epochs = epochs,
callbacks = [checkpoint_callback])```

3.2.7 查看最后保存的模型

`tf.train.latest_checkpoint(output_dir)`

#### 3.3 采样生成文本

3.3.1 载入模型

```model2 = build_model(vocab_size,
embedding_dim,
rnn_units,
batch_size = 1)
model2.build(tf.TensorShape([1, None]))
# 文本生成的流程
# start ch sequence A,
# A -> model -> b
# A.append(b) -> B
# B(Ab) -> model -> c
# B.append(c) -> C
# C(Abc) -> model -> ...
model2.summary()```

```Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_2 (Embedding)      (1, None, 256)            16640
_________________________________________________________________
simple_rnn_2 (SimpleRNN)     (1, None, 1024)           1311744
_________________________________________________________________
dense_2 (Dense)              (1, None, 65)             66625
=================================================================
Total params: 1,395,009
Trainable params: 1,395,009
Non-trainable params: 0
_________________________________________________________________```

3.3.2文本生成：

tf.expand_dims： 增加维度

```def generate_text(model, start_string, num_generate = 1000):
input_eval = [char2idx[ch] for ch in start_string]
input_eval = tf.expand_dims(input_eval, 0)

text_generated = []
model.reset_states()

for _ in range(num_generate):
# 1. model inference -> predictions
# 2. sample -> ch -> text_generated.
# 3. update input_eval

# predictions : [batch_size, input_eval_len, vocab_size]
predictions = model(input_eval)
# predictions : [input_eval_len, vocab_size]
predictions = tf.squeeze(predictions, 0)
# predicted_ids: [input_eval_len, 1]
# a b c -> b c d
predicted_id = tf.random.categorical(
predictions, num_samples = 1)[-1, 0].numpy()
text_generated.append(idx2char[predicted_id])
# s, x -> rnn -> s', y
input_eval = tf.expand_dims([predicted_id], 0)
return start_string + ''.join(text_generated)
new_text = generate_text(model2, "All: ")
print(new_text)```

```All: gentle: And it is, to water of will'd-pullens in the enemy?
On if thou be gone to the cropk.
Clown:
I will.
MARIANA:
Give me thyself was shumber;
To invuinate, as then I recking
To revout then unknown reberish weeds,
As you have true lips as believe us, good sir, witnession great unnot
God'I are murderer, for I will unselt the wallou preventest my bones
Of whom I know, or hatisage of thee?
GRESO:
Ar, but your stuffic ound thou exerving, braved or now? what overme of wrath; to their inferring thousand's tender, that I should private in Romes but of Ingles
That you 'darished recompensy in perfucuted-word:
To neity, give me home:
The true so weary Grevant:
Might there better they, when I lighted mornce, tell me, hither towards here but dead, this Aumerave be married.
ROMEO:
Upon the body, look, thy wisdom, I must beseech your ears, above:
In such arging from Oxford sho!
GLOUCESTER:
Nurse:
Good marvengrannith that borness, by set must arrays, far you would sit by: thou but he should```