nz::nodes::calc::ReLUNode Class Reference

Represents a Rectified Linear Unit (ReLU) operation node in a computational graph. More...

Inheritance diagram for nz::nodes::calc::ReLUNode:

Collaboration diagram for nz::nodes::calc::ReLUNode:

Public Member Functions
	ReLUNode (Node *input)
	Constructor to initialize a ReLUNode for applying the ReLU activation function.
void	forward () override
	Forward pass for the ReLUNode to apply the ReLU activation function.
void	backward () override
	Backward pass for the ReLUNode to compute gradients.
Public Member Functions inherited from nz::nodes::Node
virtual void	print (std::ostream &os) const
	Prints the type, data, and gradient of the node.
void	dataInject (Tensor::value_type *data, bool grad=false) const
	Injects data into a relevant tensor object, optionally setting its gradient requirement.
template<typename Iterator >
void	dataInject (Iterator begin, Iterator end, const bool grad=false) const
	Injects data from an iterator range into the output tensor of the InputNode, optionally setting its gradient requirement.
void	dataInject (const std::initializer_list< Tensor::value_type > &data, bool grad=false) const
	Injects data from a std::initializer_list into the output tensor of the Node, optionally setting its gradient requirement.

Detailed Description

Represents a Rectified Linear Unit (ReLU) operation node in a computational graph.

The ReLUNode class applies the ReLU activation function to the input tensor. ReLU is a commonly used non-linear activation function in neural networks, defined as ReLU(x) = max(0, x). It introduces non-linearity and sparsity into the network.

Key features:

• Forward Pass: Applies the ReLU activation function element-wise to the input tensor. Values less than zero are set to zero, while non-negative values remain unchanged.
• Backward Pass: Computes the gradient of the loss with respect to the input tensor. Gradients are passed through unchanged for positive input values and set to zero for negative input values.
• Shape Preservation: The output tensor has the same shape as the input tensor.
• Gradient Management: Automatically tracks gradients if required by the input tensor.

This class is part of the nz::nodes namespace and is typically used in constructing neural network models to introduce non-linearity between layers.

Note

• The input tensor's shape is preserved in the output tensor.
• Gradients are efficiently computed using CUDA kernels.

Usage Example:

// Example: Using ReLUNode in a computational graph
InputNode input({3, 3}, true);  // Create an input node with shape {3, 3}
 
float data[] = {-1.0f, 0.0f, 1.0f, 2.0f, -2.0f, 3.0f, -3.0f, 4.0f, -4.0f};  // Sample input values
input.output->dataInject(data);  // Copy data to the input tensor
 
ReLUNode relu_node(&input);  // Apply ReLU activation
relu_node.forward();  // Perform the forward pass
relu_node.backward();  // Propagate gradients in the backward pass
 
std::cout << "Output: " << *relu_node.output << std::endl;  // Print the result

See also

forward() for the ReLU activation computation in the forward pass.

backward() for gradient computation in the backward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Definition at line 1929 of file Nodes.cuh.

Constructor & Destructor Documentation

ReLUNode()

nz::nodes::calc::ReLUNode::ReLUNode ( Node * input )

explicit

Constructor to initialize a ReLUNode for applying the ReLU activation function.

The constructor initializes a ReLUNode, which applies the Rectified Linear Unit (ReLU) activation function to an input tensor. It establishes a connection to the input node, initializes the output tensor, and sets the type of the node to "ReLU".

Parameters

input

A pointer to the input node. Its output tensor will have the ReLU activation applied.

• The input node is added to the inputs vector to establish the connection in the computational graph.
• The output tensor is initialized with the same shape as the input tensor, and its gradient tracking is determined based on the input tensor's requirements.
• The node's type is set to "ReLU" to reflect its operation.

Note

• The ReLU activation is defined as ReLU(x) = max(0, x). This will be applied during the forward pass.
• This node supports automatic gradient tracking if the input tensor requires gradients.

See also

forward() for the forward pass implementation.

backward() for gradient propagation in the backward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Definition at line 344 of file Nodes.cu.

Member Function Documentation

backward()

void nz::nodes::calc::ReLUNode::backward ( )

overridevirtual

Backward pass for the ReLUNode to compute gradients.

The backward() method computes the gradient of the loss with respect to the input tensor by applying the derivative of the ReLU activation function. Gradients are propagated only for elements where the input tensor values are positive; otherwise, the gradients are set to zero.

• A CUDA kernel (ReLUBackward) is launched to compute the gradients in parallel on the GPU.
• The grid and block dimensions are calculated dynamically based on the size of the output tensor.
• The derivative of ReLU is defined as:

  ReLU'(x) = 1, if x > 0
            0, if x <= 0

  This derivative is applied element-wise to propagate gradients through the ReLU operation.
• Gradients are accumulated in the gradient tensor of the input node.

Note

• Gradients are only propagated if the input tensor's requiresGrad property is true.
• The shape of the gradient tensor matches the shape of the input tensor.

See also

forward() for the ReLU activation computation in the forward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Implements nz::nodes::Node.

Definition at line 357 of file Nodes.cu.

Here is the call graph for this function:

forward()

void nz::nodes::calc::ReLUNode::forward ( )

overridevirtual

Forward pass for the ReLUNode to apply the ReLU activation function.

The forward() method applies the Rectified Linear Unit (ReLU) activation function element-wise to the input tensor. Values less than zero are set to zero, while non-negative values remain unchanged. The results are stored in the output tensor.

• A CUDA kernel (RectifiedLinearUnit) is launched to compute the ReLU activation in parallel on the GPU.
• The grid and block dimensions are calculated dynamically based on the size of the output tensor to optimize GPU performance.
• The output tensor stores the result of applying ReLU(x) = max(0, x) for each element of the input tensor.

Note

• The shape of the output tensor matches that of the input tensor.
• Ensure the input tensor is properly initialized before calling this method.

See also

backward() for gradient computation in the backward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Implements nz::nodes::Node.

Definition at line 351 of file Nodes.cu.

Here is the call graph for this function:

---

The documentation for this class was generated from the following files:

• D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/include/NeuZephyr/Nodes.cuh
• D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/src/Nodes.cu