Layers

Potentially helpful layers for your models

Linear Layers for Patches

---

source

PatchEncoder

 PatchEncoder (c_in, patch_len, d_model, shared_embedding)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

	Details
c_in	the number of input channels
patch_len	the length of the patches (either stft or interval length)
d_model	the dimension of the initial linear layers for inputting patches into transformer
shared_embedding	indicator of whether to project each channel individually or together

Positional Encoding Layers

---

source

PositionalEncoding

 PositionalEncoding (num_patch, d_model)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

	Details
num_patch	number of patches of time series or stft in input
d_model	dimension of patch embeddings

---

source

tAPE

 tAPE (d_model:int, seq_len:int, scale_factor=1.0)

time Absolute Position Encoding Adapted from tsai

	Type	Default	Details
d_model	int		the embedding dimension
seq_len	int		the max. length of the incoming sequence or num patches
scale_factor	float	1.0	dropout:float=0., # dropout value

Mask and Augmentation Layers

---

source

Mask

 Mask (mask_type, mask_ratio, return_mask=True)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

torch.manual_seed(125)
m = Mask(mask_type='jitter_zero', mask_ratio=0.5)
x = torch.randn((9))

m(x), m(x)

((tensor([ 0.0000,  0.0000,  0.0000,  1.0975,  0.0000,  1.4248,  0.0000, -0.1104,
           1.1865]),
  tensor(8)),
 (tensor([ 1.1462, -1.8379, -0.2368,  2.0488,  0.0000,  0.0000, -0.3927, -1.0523,
           0.1294]),
  tensor(4)))

x = torch.randn((4))
x_aug = x.clone()
mask = torch.tensor([True,True,False,False])

x_aug[mask] = 1
x_aug, x

(tensor([ 1.0000,  1.0000,  1.9390, -0.0338]),
 tensor([-1.7385, -0.5780,  1.9390, -0.0338]))

---

source

PatchAugmentations

 PatchAugmentations (augmentations=['patch_mask', 'jitter_zero_mask',
                     'reverse_sequence', 'shuffle_channels'],
                     patch_mask_ratio=0.0, jitter_zero_mask_ratio=0.0)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

x = torch.randn(4,3600,7,750)

s=PatchAugmentations(patch_mask_ratio=0.1, jitter_zero_mask_ratio=0.1)
s(x).shape

torch.Size([4, 3600, 7, 750])

---

source

EmbeddingAugmentations

 EmbeddingAugmentations (augmentations=['shuffle_dims',
                         'jitter_zero_mask', 'patch_mask'],
                         dims_to_shuffle=[1, 2, 3], patch_mask_ratio=0.0,
                         jitter_zero_mask_ratio=0.0)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

x = torch.randn(4,7,512,3600)

s = EmbeddingAugmentations(augmentations=['jitter_zero_mask'], dims_to_shuffle=[1], patch_mask_ratio=0.1, jitter_zero_mask_ratio=0.1)
s(x).shape

torch.Size([4, 7, 512, 3600])

Patch and Fourier Layers

---

source

Patch

 Patch (patch_len, stride, max_seq_len=None)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

---

source

STFT

 STFT (n_fft, win_length, hop_length, stft_norm, decibel_scale,
       channel_stft_means=None, channel_stft_stds=None,
       pad_win_length_to_nfft=True, pad_mode='reflect', center=False,
       return_complex=True)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

---

source

FFT

 FFT (dim=-1, norm='backward')

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

	Type	Default	Details
dim	int	-1	dimension to calculate fft over
norm	str	backward	“forward” - normalize by 1/n, “backward” - no normalization, “ortho” - normalize by 1/sqrt(n) (making the FFT orthonormal)

Reversible Instance Normalization

---

source

RevIN

 RevIN (num_features:int, eps=1e-05, dim_to_reduce=-1, affine=True)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

	Type	Default	Details
num_features	int		the number of channels or features in the input
eps	float	1e-05	added to avoid division by zero errors
dim_to_reduce	int	-1	the dimension to reduce,
affine	bool	True	learning affine parameters bias and weight per channel

x = torch.randn(4,7,1000)

revin = RevIN(x.shape[1], dim_to_reduce=-1, affine=True)
x_norm = revin(x, mode=True)
x_denorm = revin(x_norm, mode=False)

x.shape, x_norm.shape, x_denorm.shape, revin.mean.shape, revin.stdev.shape

(torch.Size([4, 7, 1000]),
 torch.Size([4, 7, 1000]),
 torch.Size([4, 7, 1000]),
 torch.Size([4, 7, 1]),
 torch.Size([4, 7, 1]))

Attention

---

source

MultiheadFlashAttention

 MultiheadFlashAttention (d_model:int, n_heads:int, qkv_bias:bool=True,
                          is_causal:bool=False, attn_dropout:float=0.0,
                          proj_dropout:float=0.0)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

mha = MultiheadFlashAttention(d_model=512, n_heads=8, attn_dropout=0., proj_dropout=0.)
x = torch.randn(2*7,100,512) # [bs * n_vars x n_patches x d_model]
key_padding_mask = torch.zeros(2*7, 100, dtype=torch.bool)
key_padding_mask[:, -2:] = True  # mask last 2 positions
output = mha(x, key_padding_mask=key_padding_mask)
output.shape

torch.Size([14, 100, 512])

---

source

ScaledDotProductAttention

 ScaledDotProductAttention (d_model, n_heads, attn_dropout=0.0,
                            res_attention=False, lsa=False)

Scaled Dot-Product Attention module (Attention is all you need by Vaswani et al., 2017) with optional residual attention from previous layer (Realformer: Transformer likes residual attention by He et al, 2020) and locality self sttention (Vision Transformer for Small-Size Datasets by Lee et al, 2021)

---

source

MultiheadAttentionCustom

 MultiheadAttentionCustom (d_model, n_heads, d_k=None, d_v=None,
                           res_attention=False, attn_dropout=0.0,
                           proj_dropout=0.0, qkv_bias=True, lsa=False)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

mha_attn = MultiheadAttentionCustom(d_model=512, n_heads=8, attn_dropout=0., proj_dropout=0., res_attention=False)
mha_attn

MultiheadAttentionCustom(
  (W_Q): Linear(in_features=512, out_features=512, bias=True)
  (W_K): Linear(in_features=512, out_features=512, bias=True)
  (W_V): Linear(in_features=512, out_features=512, bias=True)
  (sdp_attn): ScaledDotProductAttention(
    (attn_dropout): Dropout(p=0.0, inplace=False)
  )
  (to_out): Sequential(
    (0): Linear(in_features=512, out_features=512, bias=True)
    (1): Dropout(p=0.0, inplace=False)
  )
)

def test_attention_equivalence():
    # Set random seed for reproducibility
    torch.manual_seed(42)
    
    # Test parameters
    batch_size = 2
    seq_len = 10
    d_model = 64
    n_heads = 4
    
    # Create input tensor (only need one since we're using self-attention)
    x = torch.randn(batch_size, seq_len, d_model)
    
    # Create key padding mask
    key_padding_mask = torch.zeros(batch_size, seq_len, dtype=torch.bool)
    key_padding_mask[:, -2:] = True  # mask last 2 positions

    # Initialize both implementations
    custom_mha = MultiheadAttentionCustom(d_model=d_model, n_heads=n_heads)
    flash_mha = MultiheadFlashAttention(d_model=d_model, n_heads=n_heads)
    
    # Set both models to eval mode to disable dropout
    custom_mha.eval()
    flash_mha.eval()
    
    # Copy weights to ensure identical parameters
    # Combine QKV weights from custom implementation into single matrix for flash attention
    combined_weight = torch.cat([
        custom_mha.W_Q.weight,
        custom_mha.W_K.weight,
        custom_mha.W_V.weight
    ], dim=0)
    combined_bias = torch.cat([
        custom_mha.W_Q.bias,
        custom_mha.W_K.bias,
        custom_mha.W_V.bias
    ], dim=0)
    
    # Copy combined weights to flash attention
    flash_mha.c_attn.weight.data = combined_weight
    flash_mha.c_attn.bias.data = combined_bias
    
    # Output projection weights
    flash_mha.c_proj.weight.data = custom_mha.to_out[0].weight.data.clone()
    flash_mha.c_proj.bias.data = custom_mha.to_out[0].bias.data.clone()
    
    # Forward pass
    with torch.no_grad():
        custom_output, custom_attn = custom_mha(x, key_padding_mask=key_padding_mask)
        
        flash_output = flash_mha(x, attn_mask=key_padding_mask)
    
    # Compare outputs
    print(f"Custom output shape: {custom_output.shape}")
    print(f"Flash output shape: {flash_output.shape}")
    
    output_close = torch.allclose(custom_output, flash_output, rtol=0, atol=0)
    print(f"Outputs match: {output_close}")
    
    if not output_close:
        print("\nOutput differences:")
        print(f"Max difference: {(custom_output - flash_output).abs().max().item()}")
        print(f"Mean difference: {(custom_output - flash_output).abs().mean().item()}")
    
    return custom_output, flash_output

custom_output, flash_output = test_attention_equivalence()
#: 8.940696716308594e-08
#Mean difference: 1.0550138540565968e-08

Custom output shape: torch.Size([2, 10, 64])
Flash output shape: torch.Size([2, 10, 64])
Outputs match: True

d_model=512
n_heads=8
d_k = d_v = d_model // n_heads
attn = ScaledDotProductAttention(d_model=d_model, n_heads=n_heads)
mha_attn = MultiheadAttentionCustom(d_model, n_heads)

W_Q = nn.Linear(d_model, d_k * n_heads)
W_K = nn.Linear(d_model, d_k * n_heads)
W_V = nn.Linear(d_model, d_v * n_heads)
X,_,_ = ds[0]

X = create_patch(X, patch_len=(10*50), stride=(5*50), constant_pad=True)

patch_len = X.shape[-1]

X = X[None, ...].permute(0,2,1,3)  # simulate batch size of 1 [bs x n_vars x num_patch x patch_len]

print(f'X input shape: {X.shape}')
W_P = nn.Linear(patch_len, d_model)

X = W_P(X) # project to d_model
print(f"Projected X shape to d_model: {X.shape}")

X = torch.reshape(X, (X.shape[0]*X.shape[1],X.shape[2],X.shape[3]))
print(f"Reshape for attention: {X.shape}")

# test multihead attention
print("\nTesting MHA and SDA attention, with just 50 elements.")
mha_output, mha_attn_weights = mha_attn(Q=X[:,:50,:])
print(f"MHA attention output shape: {mha_output.shape}, mha attn weight shape: {mha_attn_weights.shape}")

# test scaled dot product attn
K = Q = V = X

# # Linear (+ split in multiple heads)
bs = 1 # 1 * 16
q_s = W_Q(Q).reshape(bs, -1, n_heads, d_k).transpose(1, 2)
k_s = W_K(K).reshape(bs, -1, n_heads, d_k).permute(0, 2, 3, 1)
v_s = W_V(V).reshape(bs, -1, n_heads, d_v).transpose(1, 2)
print(f"Q shape: {q_s.shape}, K shape: {k_s.shape}, V shape: {v_s.shape}")

to_out = nn.Linear(n_heads * d_v, d_model)
output, attn_weights = attn(q_s[:,:,:50,:],k_s[:,:,:,:50], v_s[:,:,:50,:])
output = output.transpose(1, 2).contiguous().view(bs, -1, n_heads * d_v)
print(f"Attn output shape {output.shape}, attn weight shape: {attn_weights.shape}")

X input shape: torch.Size([1, 7, 10799, 500])
Projected X shape to d_model: torch.Size([1, 7, 10799, 512])
Reshape for attention: torch.Size([7, 10799, 512])

Testing MHA and SDA attention, with just 50 elements.
MHA attention output shape: torch.Size([7, 50, 512]), mha attn weight shape: torch.Size([7, 8, 50, 50])
Q shape: torch.Size([1, 8, 75593, 64]), K shape: torch.Size([1, 8, 64, 75593]), V shape: torch.Size([1, 8, 75593, 64])
Attn output shape torch.Size([1, 50, 512]), attn weight shape: torch.Size([1, 8, 50, 50])

---

source

Attention_Rel_Scl

 Attention_Rel_Scl (d_model:int, n_heads:int, seq_len:int, d_k:int=None,
                    d_v:int=None, res_attention:bool=False,
                    attn_dropout:float=0.0, lsa:bool=False,
                    proj_dropout:float=0.0, qkv_bias:bool=True)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

	Type	Default	Details
d_model	int		Embedding dimension
n_heads	int		number of attention heads
seq_len	int		sequence length or num patches
d_k	int	None	key dimension
d_v	int	None	value dimension
res_attention	bool	False	whether to use residual attention
attn_dropout	float	0.0	dropout for attention
lsa	bool	False	whether to use LSA, trainable paramater for scaling
proj_dropout	float	0.0	dropout for projection
qkv_bias	bool	True	bias for q, k, v

d_model=16
c_in = 2
seq_len = 1*360*10
x = torch.randn(4,c_in,seq_len)
embed_layer = nn.Sequential(nn.Conv2d(1, d_model*4, kernel_size=[1, 7], padding='same'), nn.BatchNorm2d(d_model*4), nn.GELU())
embed_layer2 = nn.Sequential(nn.Conv2d(d_model*4, d_model, kernel_size=[c_in, 1], padding='valid'), nn.BatchNorm2d(d_model), nn.GELU())
abs_position = tAPE(d_model, seq_len=seq_len)
x_emb = embed_layer2(embed_layer(x.unsqueeze(1))).squeeze(2)
x_emb = x_emb.permute(0,2,1)
x_emb_pos = abs_position(x_emb)

model = Attention_Rel_Scl(d_model=d_model,
        n_heads=2, # number of attention heads
        seq_len=seq_len, # sequence length or num patches
        )

out, attn_weights = model(x_emb)

## test w patches [bs *c_in x num_patches x d_model]
d_model=512
c_in = 2
num_patches = 10
x_emb = torch.randn(4*c_in,num_patches, d_model)
abs_position = tAPE(d_model, seq_len=num_patches)
x_emb_pos = abs_position(x_emb)

model = Attention_Rel_Scl(d_model=d_model,
        n_heads=2, # number of attention heads
        seq_len=num_patches, # sequence length or num patches
        )

out, attn_weights = model(x_emb)

Pretraining Heads

---

source

MaskedAutogressionFeedForward2

 MaskedAutogressionFeedForward2 (d_model, patch_len, n_layers, dropout)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

---

source

MaskedAutogressionFeedForward

 MaskedAutogressionFeedForward (c_in, patch_len, d_model,
                                shared_recreation=True)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

	Type	Default	Details
c_in			the number of input channels
patch_len			the length of the patches (either stft or interval length)
d_model			the dimension of the initial linear layers for inputting patches into transformer
shared_recreation	bool	True	indicator of whether to project each channel individually or together

Miscellaneous

---

source

Identity

 Identity ()

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

---

source

get_activation_fn

 get_activation_fn (activation)

---

source

Transpose

 Transpose (*dims, contiguous=False)

*Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*