Residual Connections#

class vision_architectures.utils.residuals.Residual(*args, **kwargs)[source]#

Bases: Module

A simple residual connection.

This has been saved as an nn.Module so that it can always be converted to a stochastic version if required.

forward(new_value, old_value)[source]#

Simply adds the new value to the old value.

Parameters:

  • new_value (Tensor) – New value to be added.

  • old_value (Tensor) – Old value to be added to.

Returns:

Value after adding the new value to the old value.

class vision_architectures.utils.residuals.StochasticDepthResidual(survival_prob=1.0, dropout_type='layer')[source]#

Bases: Residual

This class can be wrapped around a list of modules to randomly drop some of them during training.

__init__(survival_prob=1.0, dropout_type='layer')[source]#

Initializes the StochasticDepthResidual module.

Use dropout_type="layer" for most occasions as that implements true stochastic depth dropout as per SOTA papers. Use dropout_type=”neuron” only if you want to drop individual neurons.

Parameters:

  • survival_prob (float) – Prbability that every layer / neuron will survive the residual connection. Defaults to 1.0.

  • dropout_type (Literal['layer', 'neuron']) – Defaults to “layer”.

forward(new_value, old_value)[source]#

Drops the new value with a certain probability and scales the remaining value before adding it to the old value. See Residual for more details.

Parameters:

  • new_value (Tensor) – New value to be added.

  • old_value (Tensor) – Old value to be added to.

Returns:

Value after performing stochastic depth and adding the new value to the old value.

extra_repr()[source]#

Return the extra representation of the module.

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

Return type:

str

vision_architectures.utils.residuals.add_stochastic_depth_dropout(module, *args, **kwargs)[source]#

Converts all instances of Residual in a module to StochasticDepthResidual.

vision_architectures.utils.residuals.remove_stochastic_depth_dropout(module, *args, **kwargs)[source]#

Converts all instances of StochasticDepthResidual in a module to Residual.