deepof.model_utils.TransformerEncoder

class deepof.model_utils.TransformerEncoder(*args, **kwargs)

Transformer encoder.

Based on https://www.tensorflow.org/text/tutorials/transformer. Adapted according to https://academic.oup.com/gigascience/article/8/11/giz134/5626377?login=true and https://arxiv.org/abs/1711.03905.

__init__(num_layers, seq_dim, key_dim, num_heads, dff, maximum_position_encoding, rate=0.1)

Construct the transformer encoder.

Parameters:

  • num_layers – number of transformer layers to include.

  • seq_dim – dimensionality of the sequence embeddings

  • key_dim – dimensionality of the time series

  • num_heads – number of heads of the multi-head-attention layers used on the transformer encoder

  • dff – dimensionality of the token embeddings

  • maximum_position_encoding – maximum time series length

  • rate – dropout rate

Methods

__init__(num_layers, seq_dim, key_dim, ...)

Construct the transformer encoder.

add_loss(losses, **kwargs)

Add loss tensor(s), potentially dependent on layer inputs.

add_metric(value[, name])

Adds metric tensor to the layer.

add_update(updates)

Add update op(s), potentially dependent on layer inputs.

add_variable(*args, **kwargs)

Deprecated, do NOT use! Alias for add_weight.

add_weight([name, shape, dtype, ...])

Adds a new variable to the layer.

build(input_shape)

Creates the variables of the layer (for subclass implementers).

build_from_config(config)

Builds the layer's states with the supplied config dict.

call(x, training)

Call the transformer encoder.

compute_mask(inputs[, mask])

Computes an output mask tensor.

compute_output_shape(input_shape)

Computes the output shape of the layer.

compute_output_signature(input_signature)

Compute the output tensor signature of the layer based on the inputs.

count_params()

Count the total number of scalars composing the weights.

finalize_state()

Finalizes the layers state after updating layer weights.

from_config(config)

Creates a layer from its config.

get_build_config()

Returns a dictionary with the layer's input shape.

get_config()

Returns the config of the layer.

get_input_at(node_index)

Retrieves the input tensor(s) of a layer at a given node.

get_input_mask_at(node_index)

Retrieves the input mask tensor(s) of a layer at a given node.

get_input_shape_at(node_index)

Retrieves the input shape(s) of a layer at a given node.

get_output_at(node_index)

Retrieves the output tensor(s) of a layer at a given node.

get_output_mask_at(node_index)

Retrieves the output mask tensor(s) of a layer at a given node.

get_output_shape_at(node_index)

Retrieves the output shape(s) of a layer at a given node.

get_weights()

Returns the current weights of the layer, as NumPy arrays.

load_own_variables(store)

Loads the state of the layer.

save_own_variables(store)

Saves the state of the layer.

set_weights(weights)

Sets the weights of the layer, from NumPy arrays.

with_name_scope(method)

Decorator to automatically enter the module name scope.

Attributes

activity_regularizer

Optional regularizer function for the output of this layer.

compute_dtype

The dtype of the layer's computations.

dtype

The dtype of the layer weights.

dtype_policy

The dtype policy associated with this layer.

dynamic

Whether the layer is dynamic (eager-only); set in the constructor.

inbound_nodes

Return Functional API nodes upstream of this layer.

input

Retrieves the input tensor(s) of a layer.

input_mask

Retrieves the input mask tensor(s) of a layer.

input_shape

Retrieves the input shape(s) of a layer.

input_spec

InputSpec instance(s) describing the input format for this layer.

losses

List of losses added using the add_loss() API.

metrics

List of metrics attached to the layer.

name

Name of the layer (string), set in the constructor.

name_scope

Returns a tf.name_scope instance for this class.

non_trainable_variables

Sequence of non-trainable variables owned by this module and its submodules.

non_trainable_weights

List of all non-trainable weights tracked by this layer.

outbound_nodes

Return Functional API nodes downstream of this layer.

output

Retrieves the output tensor(s) of a layer.

output_mask

Retrieves the output mask tensor(s) of a layer.

output_shape

Retrieves the output shape(s) of a layer.

stateful

submodules

Sequence of all sub-modules.

supports_masking

Whether this layer supports computing a mask using compute_mask.

trainable

trainable_variables

Sequence of trainable variables owned by this module and its submodules.

trainable_weights

List of all trainable weights tracked by this layer.

updates

variable_dtype

Alias of Layer.dtype, the dtype of the weights.

variables

Returns the list of all layer variables/weights.

weights

Returns the list of all layer variables/weights.
__init__(num_layers, seq_dim, key_dim, num_heads, dff, maximum_position_encoding, rate=0.1)

Construct the transformer encoder.

Parameters:

  • num_layers – number of transformer layers to include.

  • seq_dim – dimensionality of the sequence embeddings

  • key_dim – dimensionality of the time series

  • num_heads – number of heads of the multi-head-attention layers used on the transformer encoder

  • dff – dimensionality of the token embeddings

  • maximum_position_encoding – maximum time series length

  • rate – dropout rate