Having seen how to implement the scaled dot-product attention<\/a>, and integrate it within the multi-head attention<\/a> of the Transformer model, we may progress one step further towards implementing a complete Transformer model by implementing its encoder. Our end goal remains the application of the complete model to Natural Language Processing (NLP).<\/p>\n In this tutorial, you will discover how to implement the Transformer encoder from scratch in TensorFlow and Keras.\u00a0<\/span><\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.\u00a0<\/span><\/p>\n Implementing the Transformer Encoder From Scratch in TensorFlow and Keras This tutorial is divided into three parts; they are:<\/p>\n For this tutorial, we assume that you are already familiar with:<\/p>\n Recall<\/a> having seen that the Transformer architecture follows an encoder-decoder structure: the encoder, on the left-hand side, is tasked with mapping an input sequence to a sequence of continuous representations; the decoder, on the right-hand side, receives the output of the encoder together with the decoder output at the previous time step, to generate an output sequence.<\/p>\n The Encoder-Decoder Structure of the Transformer Architecture In generating an output sequence, the Transformer does not rely on recurrence and convolutions.<\/p>\n We had seen that the decoder part of the Transformer shares many similarities in its architecture with the encoder.\u00a0In this tutorial, we will be focusing on the components that form part of the Transformer encoder. \u00a0<\/span><\/p>\n The Transformer encoder consists of a stack of $N$ identical layers, where each layer further consists of two main sub-layers:<\/p>\n The Encoder Block of the Transformer Architecture Following each of these two sub-layers is layer normalisation, into which the sub-layer input (through a residual connection) and output are fed. The output of each layer normalization step is the following:<\/p>\n LayerNorm(Sublayer Input + Sublayer Output)<\/p>\n In order to facilitate such an operation, which involves an addition between the sublayer input and output, Vaswani et al. designed all sub-layers and embedding layers in the model to produce outputs of dimension, $d_{text{model}}$ = 512.<\/p>\n Recall<\/a> as well the queries, keys and values as the inputs to the Transformer encoder.<\/p>\n Here, the queries, keys and values carry the same input sequence after this has been embedded and augmented by positional information, where the queries and keys are of dimensionality, $d_k$, whereas the dimensionality of the values is $d_v$.<\/p>\n Furthermore, Vaswani et al. also introduce regularization into the model by applying dropout to the output of each sub-layer (before the layer normalization step), as well as to the positional encodings before these are fed into the encoder.\u00a0<\/span><\/p>\n Let\u2019s now see how to implement the Transformer encoder from scratch in TensorFlow and Keras.<\/p>\n We shall begin by creating classes for the Feed Forward<\/i> and Add & Norm<\/i> layers that are shown in the diagram above.<\/p>\n Vaswani et al. tell us that the fully connected feed-forward network consists of two linear transformations with a ReLU activation in between. The first linear transformation produces an output of dimensionality, $d_{ff}$ = 2048, while the second linear transformation produces an output of dimensionality, $d_{text{model}}$ = 512.<\/p>\n For this purpose, let\u2019s first create the class, We will add to it the class method, The next step is to create another class, In it, we will include the following class method that sums its sub-layer\u2019s input and output, which it receives as inputs, and applies layer normalization to the result:<\/p>\n Next, we will implement the encoder layer, which the Transformer encoder will replicate identically $N$ times.\u00a0<\/span><\/p>\n For this purpose, let\u2019s create the class, Here you may notice that we have initialized instances of the Let\u2019s now proceed to create the class method, In addition to the input data, the The same class method can receive a The last step is to create a class for the Transformer encoder, which we shall be naming The Transformer encoder receives an input sequence after this would have undergone a process of word embedding and positional encoding. In order to compute the positional encoding, we will make use of the As we have similarly done in the previous sections, here we will also create a class method, The code listing for the full Transformer encoder is the following:<\/p>\n We will be working with the parameter values specified in the paper, Attention Is All You Need<\/a>, by Vaswani et al. (2017):<\/p>\n As for the input sequence we will be working with dummy data for the time being until we arrive to the stage of\u00a0training the complete Transformer model<\/a>\u00a0in a separate tutorial, at which point we will be using actual sentences:<\/p>\n Next, we will create a new instance of the Tying everything together produces the following code listing:<\/p>\n Running this code produces an output of shape, (batch size<\/i>, sequence length<\/i>, model dimensionality<\/i>). Note that you will likely see a different output due to the random initialization of the input sequence, and the parameter values of the Dense layers.\u00a0<\/span><\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to implement the Transformer encoder from scratch in TensorFlow and Keras.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions? The post Implementing the Transformer Encoder From Scratch in TensorFlow and Keras<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"\"](\"http:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/03\/encoder_cover-1024x683.jpg\")
<\/a><\/p>\n
Photo by ian dooley<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/b><\/h2>\n
\n
\n
\n
Prerequisites<\/b><\/h2>\n
\n
Recap of the Transformer Architecture<\/b><\/h2>\n
![\"\"](\"http:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/08\/attention_research_1-727x1024.png\")
<\/a><\/p>\n
Taken from \u201cAttention Is All You Need<\/a>\u201c<\/p>\n<\/div>\nThe Transformer Encoder<\/b><\/h3>\n
\n
![\"\"](\"http:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/10\/transformer_1-727x1024.png\"){kind=spacer}
<\/a><\/p>\n
Taken from \u201cAttention Is All You Need<\/a>\u201c<\/p>\n<\/div>\nImplementing the Transformer Encoder From Scratch<\/b><\/h2>\n
The Fully Connected Feed-Forward Neural Network and Layer Normalization<\/b><\/h3>\n
FeedForward<\/code> that inherits form the Layer<\/code> base class in Keras, and initialize the dense layers and the ReLU activation:<\/p>\nclass FeedForward(Layer):\r\n def __init__(self, d_ff, d_model, **kwargs):\r\n super(FeedForward, self).__init__(**kwargs)\r\n self.fully_connected1 = Dense(d_ff) # First fully connected layer\r\n self.fully_connected2 = Dense(d_model) # Second fully connected layer\r\n self.activation = ReLU() # ReLU activation layer\r\n ...<\/pre>\n
call()<\/code>, that receives an input and passes it through the two fully connected layers with ReLU activation, returning an output of dimensionality equal to 512:<\/p>\n...\r\ndef call(self, x):\r\n # The input is passed into the two fully-connected layers, with a ReLU in between\r\n x_fc1 = self.fully_connected1(x)\r\n\r\n return self.fully_connected2(self.activation(x_fc1))<\/pre>\n
AddNormalization<\/code>, that also inherits form the Layer<\/code> base class in Keras, and initialize a Layer normalization layer:<\/p>\nclass AddNormalization(Layer):\r\n def __init__(self, **kwargs):\r\n super(AddNormalization, self).__init__(**kwargs)\r\n self.layer_norm = LayerNormalization() # Layer normalization layer\r\n ...<\/pre>\n
...\r\ndef call(self, x, sublayer_x):\r\n # The sublayer input and output need to be of the same shape to be summed\r\n add = x + sublayer_x\r\n\r\n # Apply layer normalization to the sum\r\n return self.layer_norm(add)<\/pre>\n<\/p>\n
The Encoder Layer<\/b><\/h3>\n
EncoderLayer<\/code>, and initialize all of the sub-layers that it consists of:<\/p>\nclass EncoderLayer(Layer):\r\n def __init__(self, h, d_k, d_v, d_model, d_ff, rate, **kwargs):\r\n super(EncoderLayer, self).__init__(**kwargs)\r\n self.multihead_attention = MultiHeadAttention(h, d_k, d_v, d_model)\r\n self.dropout1 = Dropout(rate)\r\n self.add_norm1 = AddNormalization()\r\n self.feed_forward = FeedForward(d_ff, d_model)\r\n self.dropout2 = Dropout(rate)\r\n self.add_norm2 = AddNormalization()\r\n ...<\/pre>\n
FeedForward<\/code> and AddNormalization<\/code> classes, which we have just created in the previous section, and assigned their output to the respective variables, feed_forward<\/code> and add_norm<\/code> (1 and 2). The Dropout<\/code> layer is self-explanatory, where rate<\/code> defines the frequency at which the input units are set to 0. We had created the MultiHeadAttention<\/code> class in a previous tutorial<\/a>, and if you had saved the code into a separate Python script, then do not forget to import<\/code> it. I saved mine in a Python script named, multihead_attention.py<\/i>, and for this reason I need to include the line of code, from multihead_attention import MultiHeadAttention.<\/i><\/p>\ncall()<\/code>, that implements all of the encoder sub-layers:<\/p>\n...\r\ndef call(self, x, padding_mask, training):\r\n # Multi-head attention layer\r\n multihead_output = self.multihead_attention(x, x, x, padding_mask)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Add in a dropout layer\r\n multihead_output = self.dropout1(multihead_output, training=training)\r\n\r\n # Followed by an Add & Norm layer\r\n addnorm_output = self.add_norm1(x, multihead_output)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Followed by a fully connected layer\r\n feedforward_output = self.feed_forward(addnorm_output)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Add in another dropout layer\r\n feedforward_output = self.dropout2(feedforward_output, training=training)\r\n\r\n # Followed by another Add & Norm layer\r\n return self.add_norm2(addnorm_output, feedforward_output)<\/pre>\n
call()<\/code> method can also receive a padding mask. As a brief reminder of what we had said in a previous tutorial<\/a>, the padding<\/i> mask is necessary to suppress the zero padding in the input sequence from being processed along with the actual input values.\u00a0<\/span><\/p>\ntraining<\/code> flag which, when set to True<\/code>, will only apply the Dropout layers during training.<\/p>\nThe Transformer Encoder<\/b><\/h3>\n
Encoder<\/code>:<\/p>\nclass Encoder(Layer):\r\n def __init__(self, vocab_size, sequence_length, h, d_k, d_v, d_model, d_ff, n, rate, **kwargs):\r\n super(Encoder, self).__init__(**kwargs)\r\n self.pos_encoding = PositionEmbeddingFixedWeights(sequence_length, vocab_size, d_model)\r\n self.dropout = Dropout(rate)\r\n self.encoder_layer = [EncoderLayer(h, d_k, d_v, d_model, d_ff, rate) for _ in range(n)]\r\n ...<\/pre>\n
PositionEmbeddingFixedWeights<\/code> class described by Mehreen Saeed in this tutorial<\/a>.\u00a0<\/span><\/p>\ncall()<\/code>, that applies word embedding and positional encoding to the input sequence, and feeds the result to $N$ encoder layers:<\/p>\n...\r\ndef call(self, input_sentence, padding_mask, training):\r\n # Generate the positional encoding\r\n pos_encoding_output = self.pos_encoding(input_sentence)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Add in a dropout layer\r\n x = self.dropout(pos_encoding_output, training=training)\r\n\r\n # Pass on the positional encoded values to each encoder layer\r\n for i, layer in enumerate(self.encoder_layer):\r\n x = layer(x, padding_mask, training)\r\n\r\n return x<\/pre>\n
from tensorflow.keras.layers import LayerNormalization, Layer, Dense, ReLU, Dropout\r\nfrom multihead_attention import MultiHeadAttention\r\nfrom positional_encoding import PositionEmbeddingFixedWeights\r\n\r\n# Implementing the Add & Norm Layer\r\nclass AddNormalization(Layer):\r\n def __init__(self, **kwargs):\r\n super(AddNormalization, self).__init__(**kwargs)\r\n self.layer_norm = LayerNormalization() # Layer normalization layer\r\n\r\n def call(self, x, sublayer_x):\r\n # The sublayer input and output need to be of the same shape to be summed\r\n add = x + sublayer_x\r\n\r\n # Apply layer normalization to the sum\r\n return self.layer_norm(add)\r\n\r\n# Implementing the Feed-Forward Layer\r\nclass FeedForward(Layer):\r\n def __init__(self, d_ff, d_model, **kwargs):\r\n super(FeedForward, self).__init__(**kwargs)\r\n self.fully_connected1 = Dense(d_ff) # First fully connected layer\r\n self.fully_connected2 = Dense(d_model) # Second fully connected layer\r\n self.activation = ReLU() # ReLU activation layer\r\n\r\n def call(self, x):\r\n # The input is passed into the two fully-connected layers, with a ReLU in between\r\n x_fc1 = self.fully_connected1(x)\r\n\r\n return self.fully_connected2(self.activation(x_fc1))\r\n\r\n# Implementing the Encoder Layer\r\nclass EncoderLayer(Layer):\r\n def __init__(self, h, d_k, d_v, d_model, d_ff, rate, **kwargs):\r\n super(EncoderLayer, self).__init__(**kwargs)\r\n self.multihead_attention = MultiHeadAttention(h, d_k, d_v, d_model)\r\n self.dropout1 = Dropout(rate)\r\n self.add_norm1 = AddNormalization()\r\n self.feed_forward = FeedForward(d_ff, d_model)\r\n self.dropout2 = Dropout(rate)\r\n self.add_norm2 = AddNormalization()\r\n\r\n def call(self, x, padding_mask, training):\r\n # Multi-head attention layer\r\n multihead_output = self.multihead_attention(x, x, x, padding_mask)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Add in a dropout layer\r\n multihead_output = self.dropout1(multihead_output, training=training)\r\n\r\n # Followed by an Add & Norm layer\r\n addnorm_output = self.add_norm1(x, multihead_output)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Followed by a fully connected layer\r\n feedforward_output = self.feed_forward(addnorm_output)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Add in another dropout layer\r\n feedforward_output = self.dropout2(feedforward_output, training=training)\r\n\r\n # Followed by another Add & Norm layer\r\n return self.add_norm2(addnorm_output, feedforward_output)\r\n\r\n# Implementing the Encoder\r\nclass Encoder(Layer):\r\n def __init__(self, vocab_size, sequence_length, h, d_k, d_v, d_model, d_ff, n, rate, **kwargs):\r\n super(Encoder, self).__init__(**kwargs)\r\n self.pos_encoding = PositionEmbeddingFixedWeights(sequence_length, vocab_size, d_model)\r\n self.dropout = Dropout(rate)\r\n self.encoder_layer = [EncoderLayer(h, d_k, d_v, d_model, d_ff, rate) for _ in range(n)]\r\n\r\n def call(self, input_sentence, padding_mask, training):\r\n # Generate the positional encoding\r\n pos_encoding_output = self.pos_encoding(input_sentence)\r\n # Expected output shape = (batch_size, sequence_length, d_model)\r\n\r\n # Add in a dropout layer\r\n x = self.dropout(pos_encoding_output, training=training)\r\n\r\n # Pass on the positional encoded values to each encoder layer\r\n for i, layer in enumerate(self.encoder_layer):\r\n x = layer(x, padding_mask, training)\r\n\r\n return x<\/pre>\n<\/p>\n
Testing Out the Code<\/b><\/h2>\n
h = 8 # Number of self-attention heads\r\nd_k = 64 # Dimensionality of the linearly projected queries and keys\r\nd_v = 64 # Dimensionality of the linearly projected values\r\nd_ff = 2048 # Dimensionality of the inner fully connected layer\r\nd_model = 512 # Dimensionality of the model sub-layers' outputs\r\nn = 6 # Number of layers in the encoder stack\r\n\r\nbatch_size = 64 # Batch size from the training process\r\ndropout_rate = 0.1 # Frequency of dropping the input units in the dropout layers\r\n...<\/pre>\n
...\r\nenc_vocab_size = 20 # Vocabulary size for the encoder\r\ninput_seq_length = 5 # Maximum length of the input sequence\r\n\r\ninput_seq = random.random((batch_size, input_seq_length))\r\n...<\/pre>\n
Encoder<\/code> class, assigning its output to the encoder<\/code> variable, and subsequently feeding in the input arguments and printing the result. We will be setting the padding mask argument to None<\/code> for the time being, but we shall return to this when we implement the complete Transformer model:<\/p>\n...\r\nencoder = Encoder(enc_vocab_size, input_seq_length, h, d_k, d_v, d_model, d_ff, n, dropout_rate)\r\nprint(encoder(input_seq, None, True))<\/pre>\n
from numpy import random\r\n\r\nenc_vocab_size = 20 # Vocabulary size for the encoder\r\ninput_seq_length = 5 # Maximum length of the input sequence\r\nh = 8 # Number of self-attention heads\r\nd_k = 64 # Dimensionality of the linearly projected queries and keys\r\nd_v = 64 # Dimensionality of the linearly projected values\r\nd_ff = 2048 # Dimensionality of the inner fully connected layer\r\nd_model = 512 # Dimensionality of the model sub-layers' outputs\r\nn = 6 # Number of layers in the encoder stack\r\n\r\nbatch_size = 64 # Batch size from the training process\r\ndropout_rate = 0.1 # Frequency of dropping the input units in the dropout layers\r\n\r\ninput_seq = random.random((batch_size, input_seq_length))\r\n\r\nencoder = Encoder(enc_vocab_size, input_seq_length, h, d_k, d_v, d_model, d_ff, n, dropout_rate)\r\nprint(encoder(input_seq, None, True))<\/pre>\n
tf.Tensor(\r\n[[[-0.4214715 -1.1246173 -0.8444572 ... 1.6388322 -0.1890367\r\n 1.0173352 ]\r\n [ 0.21662089 -0.61147404 -1.0946581 ... 1.4627445 -0.6000164\r\n -0.64127874]\r\n [ 0.46674493 -1.4155326 -0.5686513 ... 1.1790234 -0.94788337\r\n 0.1331717 ]\r\n [-0.30638126 -1.9047263 -1.8556844 ... 0.9130118 -0.47863355\r\n 0.00976158]\r\n [-0.22600567 -0.9702025 -0.91090447 ... 1.7457147 -0.139926\r\n -0.07021569]]\r\n...\r\n\r\n [[-0.48047638 -1.1034104 -0.16164204 ... 1.5588069 0.08743562\r\n -0.08847156]\r\n [-0.61683714 -0.8403657 -1.0450369 ... 2.3587787 -0.76091915\r\n -0.02891812]\r\n [-0.34268388 -0.65042275 -0.6715749 ... 2.8530657 -0.33631966\r\n 0.5215888 ]\r\n [-0.6288677 -1.0030932 -0.9749813 ... 2.1386387 0.0640307\r\n -0.69504136]\r\n [-1.33254 -1.2524267 -0.230098 ... 2.515467 -0.04207756\r\n -0.3395423 ]]], shape=(64, 5, 512), dtype=float32)<\/pre>\n<\/p>\n
Further Reading<\/b><\/h2>\n
Books<\/b><\/h3>\n
\n
Papers<\/b><\/h3>\n
\n
Summary<\/b><\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n