译者 | 崔皓
审校 | 孙淑娟
开篇
Transformer模型通过在语言翻译、文本分类和序列建模中提供卓越的性能，彻底改变了自然语言处理（NLP）任务。
Transformer的架构是基于一种自我关注机制，它允许序列中的每个元素关注其他元素并处理输入序列的堆叠编码器。
本文将演示如何建立一个Transformer模型来生成新的鸡尾酒配方。文中将使用Cocktail DB数据集，该数据集包含了成千上万种鸡尾酒的信息，包括它们的成分以及配方。
下载Cocktail DB数据集
首先，我们需要下载并预处理Cocktail DB数据集。我们将使用Pandas库来完成这一工作。
import pandas as pd
url = 'https://www.thecocktaildb.com/api/json/v1/1/search.php?s=' cocktail_df = pd.DataFrame() for i in range(1, 26): response = pd.read_json(tr(i)) cocktail_df = pd.concat([cocktail_df, response['drinks']], ignore_index=True)
预处理数据集
cocktail_df = cocktail_df.dropna(subset=['strInstructions']) cocktail_df = cocktail_df[['strDrink', 'strInstructions', 'strIngredient1', 'strIngredient2', 'strIngredient3', 'strIngredient4', 'strIngredient5', 'strIngredient6']] cocktail_df = cocktail_df.fillna('')接下来，我们需要使用标记器对鸡尾酒菜谱进行标记和编码。
将TensorFlow数据集导入为tfds。
定义标记器和词汇量大小
tokenizer = tfds.features.text.SubwordTextEncoder.build_from_corpus( (text for text in cocktail_df['strInstructions']), target_vocab_size=2**13)
定义编码函数
def encode(text): encoded_text = tokenizer.encode(text) return encoded_text
将编码函数应用到数据集上
cocktail_df['encoded_recipe'] = cocktail_df['strInstructions'].apply(encode)
定义菜谱的最大长度
MAX_LEN = max([len(recipe) for recipe in cocktail_df['encoded_recipe']] )
有了标记化的鸡尾酒菜谱，就可以定义转化器解码层了。Transformer解码器层由两个子层组成：masked multi-head self-attention 层和point-wise feed-forward 层。
import tensorflow as tf from tensorflow.keras.layers import LayerNormalization, MultiHeadAttention, Dense
class TransformerDecoderLayer(tf.keras.layers.Layer): def init(self, num_heads, d_model, dff, rate=0.1): super(TransformerDecoderLayer, self).init()
self.mha1 = MultiHeadAttention(num_heads, d_model)
self.mha2 = MultiHeadAttention(num_heads, d_model)
self.ffn = tf.keras.Sequential([
Dense(dff, activation='relu'),
Dense(d_model)
])
self.layernorm1 = LayerNormalization(epsilon=1e-6)
self.layernorm2 = LayerNormalization(epsilon=1e-6)
self.layernorm3 = LayerNormalization(epsilon=1e-6)
self.dropout1 = tf.keras.layers.Dropout(rate)
self.dropout2 = tf.keras.layers.Dropout(rate)
self.dropout3 = tf.keras.layers.Dropout(rate)
def call(self, x, enc_output, training, look_ahead_mask):
attn1 = self.mha1(x, x, x, look_ahead_mask)
attn1 = self.dropout1(attn1, training=training)
out1 = self.layernorm1(x + attn1)
attn2 = self.mha2(enc_output, enc_output, out1, out1, out1)
attn2 = self.dropout2(attn2, training=training)
out2 = self.layernorm2(out1 + attn2)
ffn_output = self.ffn(out2)
ffn_output = self.dropout3(ffn_output, training=training)
out3 = self.layernorm3(out2 + ffn_output)
return out3
在上面的代码中，TransformerDecoderLayer类需要四个参数：masked multi-head self-attention层的个数、模型的维度、point-wise feed-forward层的单元数和dropout率。
调用方法定义了解码器层的前向传递，其中x是输入序列；enc_output是编码器的输出；training是一个布尔标志，表示模型是处于训练还是推理模式；look_ahead_mask是一个掩码，防止解码器关注未来的令牌。
我们现在可以定义转化器模型，它由多个堆叠的转化器解码器层和一个密集层组成，密集层将解码器的输出映射到词汇量上。
从tensorflow.keras.layer导入Input
input_layer = Input(shape=(MAX_LEN,))
定义Transformer解码器层
NUM_LAYERS = 4 NUM_HEADS = 8 D_MODEL = 256 DFF = 1024 DROPOUT_RATE = 0.1
decoder_layers = [TransformerDecoderLayer(NUM_HEADS, D_MODEL, DFF, DROPOUT_RATE) for _ in range(NUM_LAYERS)]
定义输出层
output_layer = Dense(tokenizer.vocab_size)
连接层
x = input_layer look_ahead_mask = tf.linalg.band_part(tf.ones((MAX_LEN, MAX_LEN)), -1, 0) for decoder_layer in decoder_layers: x = decoder_layer(x, x, True, look_ahead_mask) output = output_layer(x)
定义模型
model = tf.keras.models.Model(inputs=input_layer, outputs=output)
在上面的代码中，我们定义了输入层，接受长度为MAX_LEN的填充序列。然后，通过创建一个堆叠在一起的TransformerDecoderLayer对象的列表来定义转化器解码层，从而处理输入序列。
最后一个Transformer解码层的输出通过一个密集层，其词汇量与标记器中的子词数量相对应。我们可以使用Adam优化器来训练模型，并在一定数量的epochs后评估其性能。
定义损失函数
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')
def loss_function(real, pred): mask = tf.math.logical_not(tf.math.equal(real, 0)) loss_ = loss_object(real, pred)
mask = tf.cast(mask, dtype=loss_.dtype)
loss_ *= mask
return tf.reduce_mean(loss_)
定义学习率时间表
class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule): def init(self, d_model, warmup_steps=4000): super(CustomSchedule, self).init()
self.d_model = tf.cast(d_model, tf.float32)
self.warmup_steps = warmup_steps
def __call__(self, step):
arg1 = tf.math.rsqrt(step)
arg2 = step * (self.warmup_steps**-1)
return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)
定义优化器
LR = CustomSchedule(D_MODEL) optimizer = tf.keras.optimizers.Adam(LR, beta_1=0.9, beta_2=0.98, epsilon=1e-9)
定义准确度指标
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')
定义训练步骤函数
@tf.function def train_step(inp, tar): tar_inp = tar[:, :-1] tar_real = tar[:, 1:]
look_ahead_mask = tf.linalg.band_part(tf.ones((tar.shape[1], tar.shape[1])), -1, 0) look_ahead_mask = 1 - look_ahead_mask
with tf.GradientTape() as tape: predictions = model(inp, True, look_ahead_mask) loss = loss_function(tar_real, predictions)
gradients = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(gradients, model.trainable_variables))
train_accuracy.update_state(tar_real, predictions)
return loss Train the model EPOCHS = 50 BATCH_SIZE = 64 NUM_EXAMPLES = len(cocktail_df)
for epoch in range(EPOCHS): print('Epoch', epoch + 1) total_loss = 0
for i in range(0, NUM_EXAMPLES, BATCH_SIZE): batch = cocktail_df[i:i+BATCH_SIZE] input_batch = tf.keras.preprocessing.sequence.pad_sequences(batch['encoded_recipe'], padding='post', maxlen=MAX_LEN) target_batch = input_batch
loss = train_step(input_batch, target_batch)
total_loss += loss
print('Loss:', total_loss) print('Accuracy:', train_accuracy.result().numpy()) train_accuracy.reset_states
一旦模型被训练出来，我们就可以通过给模型提供一个种子序列，并通过反复预测来生成新的鸡尾酒配方。Shapetokentar：产生Shapee序列末尾的标记。
def generate_recipe(seed, max_len):
encoded_seed = encode(seed) for i in range(max_len):
input_sequence = tf.keras.preprocessing.sequence.pad_sequences([encoded_seed],
padding='post', maxlen=MAX_LEN) predictions = model(input_sequence, False, None)
predicted_id = tf.random.categorical(predictions[:, -1, :], num_samples=1)
if predicted_id == tokenizer.vocab_size:
break encoded_seed = tf.squeeze(predicted_id).numpy().tolist()
recipe = tokenizer.decode(encoded_seed)
return recipe
总结
总之，Transformer是一个强大的序列建模工具，可以用于NLP以外的广泛应用。
通过遵循本文所述的步骤，可以建立一个Transformer模型来生成新的鸡尾酒配方，展示了Transformer架构的灵活性和通用性。
译者介绍
崔皓，51CTO社区编辑，资深架构师，拥有18年的软件开发和架构经验，10年分布式架构经验。
原文标题：Cocktail Alchemy: Creating New Recipes With Transformers，作者：Pavan madduru
