def DecoderFeedForward( c_in, # the number of input channels predict_every_n_patches, # for a given sequence of length m with frequency f, number of predictions num_layers, d_ff, attn_dropout, res_attention, pre_norm, store_attn, n_heads, shared_embedding, affine, n_classes, # the number of classes to predict (for sleep stage - there are 6) d_model, # the dimension of the transformer model norm:str='BatchNorm', # batchnorm or layernorm between linear and convolutional layers act:str='gelu', # activation function to use between layers, 'gelu' or 'relu' dropout:float=0.0, # dropout in between linear layers):
transformer decoder with attention for feedforward predictions. This is really just another encoder layer followed by a linear layer + 1d convolution + softmax. However, if used in linear probing, could be useful.
def TimeDistributedConvolutionalFeedForward( c_in, # the number of input channels frequency, # the frequency of the original channels predict_every_seconds, # for a given sequence of length m with frequency f, number of predictions n_classes, # the number of classes to predict (for sleep stage - there are 6) win_length, # the convolved patch length, the first step in this is to do a linear layer to this dimension d_model, # the dimension of the transformer model affine:bool=False, shared_embedding:bool=True):
Convolutional feed forward head that first uses a linear feed forward network to project features into the original convolutional dimension. Then, a convolutional transpose is used to extrapolate the data to its original form. Finally, a final convolution is used to predict the classes.
def LinearProbingHead( c_in, # the number of input channels in the original input predict_every_n_patches, # for a given sequence of length m with frequency f, number of predictions n_classes, # the number of classes to predict (for sleep stage - there are 6) input_size, # the dimension of the transformer model n_layers, # the number of linear layers to use in the prediciton head, with RELU activation and dropout num_patch, shared_embedding:bool=True, # whether or not to have a dense layer per channel or one layer per channel affine:bool=True, # include learnable parameters to weight predictions norm:str='BatchNorm', # batchnorm or layernorm between linear and convolutional layers act:str='gelu', # activation function to use between layers, 'gelu' or 'relu' dropout:float=0.0, # dropout in between linear layers):
A linear probing head (with optional MLP), assumes that the d_model corresponds to a particular segment of time and will make a prediction per patch per channel, and average the results
def TimeDistributedFeedForward( c_in, # the number of input channels n_classes, # the number of classes to predict (for sleep stage - there are 6) n_patches, # the number of stft or time patches d_model, # the dimension of the transformer model pred_len_seconds, # the sequence multiclass prediction length in seconds n_linear_layers, # the number of linear layers to use in the prediciton head, with RELU activation and dropout conv_kernel_stride_size, # the 1d convolution kernel size and stride, in seconds. If you want every 30 second predicitons, put 30 here. dropout:float=0.0, # dropout in between linear layers):
Feed forward head that uses a convolutional layer to reduce channel dimensionality Followed by a feedforward network to make
def ConvGRU1D( input_size, hidden_sizes, # if integer, the same hidden size is used for all cells. kernel_sizes, # if integer, the same kernel size is used for all cells. n_layers):
Base class for all neural network modules.
Your models should also subclass this class.
Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::
import torch.nn as nn
import torch.nn.functional as F
class Model(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 20, 5)
self.conv2 = nn.Conv2d(20, 20, 5)
def forward(self, x):
x = F.relu(self.conv1(x))
return F.relu(self.conv2(x))
Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.
.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.
:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool