![]() |
NeuZephyr
Simple DL Framework
|
Represents a Leaky Rectified Linear Unit (LeakyReLU) activation function node in a computational graph. More...
Public Member Functions | |
LeakyReLUNode (Node *input, Tensor::value_type alpha=0.01f) | |
Constructor to initialize a LeakyReLUNode for applying the Leaky ReLU activation function. | |
void | forward () override |
Forward pass for the LeakyReLUNode to apply the Leaky ReLU activation function. | |
void | backward () override |
Backward pass for the LeakyReLUNode to compute gradients. | |
![]() | |
virtual void | print (std::ostream &os) const |
Prints the type, data, and gradient of the node. | |
void | dataInject (Tensor::value_type *data, bool grad=false) const |
Injects data into a relevant tensor object, optionally setting its gradient requirement. | |
template<typename Iterator > | |
void | dataInject (Iterator begin, Iterator end, const bool grad=false) const |
Injects data from an iterator range into the output tensor of the InputNode, optionally setting its gradient requirement. | |
void | dataInject (const std::initializer_list< Tensor::value_type > &data, bool grad=false) const |
Injects data from a std::initializer_list into the output tensor of the Node, optionally setting its gradient requirement. | |
Represents a Leaky Rectified Linear Unit (LeakyReLU) activation function node in a computational graph.
The LeakyReLUNode
class applies the Leaky ReLU activation function to the input tensor. Unlike the standard ReLU, Leaky ReLU allows a small, non-zero gradient for negative input values. The function is defined as:
where alpha
is a small constant that determines the slope for negative values.
Key features:
alpha
.alpha
.This class is part of the nz::nodes
namespace and is commonly used in deep learning to mitigate the "dying ReLU" problem by allowing small gradients for negative inputs.
alpha
parameter defaults to 0.01
, but can be customized during construction.
|
explicit |
Constructor to initialize a LeakyReLUNode
for applying the Leaky ReLU activation function.
The constructor initializes a LeakyReLUNode
, which applies the Leaky ReLU activation function to an input tensor. It establishes a connection to the input node, initializes the output tensor, and sets the alpha
parameter and node type.
input | A pointer to the input node. Its output tensor will have the Leaky ReLU activation applied. |
alpha | The slope for negative input values, determining how much they are scaled. Defaults to 0.01 . |
inputs
vector to establish the connection in the computational graph.output
tensor is initialized with the same shape as the input tensor, and its gradient tracking is determined based on the input tensor's requirements.alpha
parameter is stored to control the slope for negative input values during both the forward and backward passes.
|
overridevirtual |
Backward pass for the LeakyReLUNode
to compute gradients.
The backward()
method computes the gradient of the loss with respect to the input tensor by applying the derivative of the Leaky ReLU activation function. The gradient computation is defined as:
LeakyReLUBackward
) is launched to compute the gradients in parallel on the GPU.output
tensor to compute the input gradient.alpha
parameter, provided during construction, controls the gradient scale for negative input values.requiresGrad
property is true.Implements nz::nodes::Node.
Definition at line 422 of file Nodes.cu.
|
overridevirtual |
Forward pass for the LeakyReLUNode
to apply the Leaky ReLU activation function.
The forward()
method applies the Leaky ReLU activation function element-wise to the input tensor. Positive input values remain unchanged, while negative input values are scaled by the alpha
parameter. The result is stored in the output
tensor. The function is defined as:
LeakyReLU
) is launched to compute the activation function in parallel on the GPU.output
tensor to ensure efficient GPU utilization.alpha
parameter, provided during construction, determines the slope for negative input values.Implements nz::nodes::Node.
Definition at line 416 of file Nodes.cu.