nz::nodes::calc::HardSigmoidNode Class Reference

Represents a Hard Sigmoid activation function node in a computational graph. More...

Inheritance diagram for nz::nodes::calc::HardSigmoidNode:

Collaboration diagram for nz::nodes::calc::HardSigmoidNode:

Public Member Functions
	HardSigmoidNode (Node *input, Tensor::value_type alpha=0.2f, Tensor::value_type beta=0.5f)
	Constructor to initialize a HardSigmoidNode for applying the Hard Sigmoid activation function.
void	forward () override
	Forward pass for the HardSigmoidNode to apply the Hard Sigmoid activation function.
void	backward () override
	Backward pass for the HardSigmoidNode to compute gradients.
Public Member Functions inherited from nz::nodes::Node
virtual void	print (std::ostream &os) const
	Prints the type, data, and gradient of the node.
void	dataInject (Tensor::value_type *data, bool grad=false) const
	Injects data into a relevant tensor object, optionally setting its gradient requirement.
template<typename Iterator >
void	dataInject (Iterator begin, Iterator end, const bool grad=false) const
	Injects data from an iterator range into the output tensor of the InputNode, optionally setting its gradient requirement.
void	dataInject (const std::initializer_list< Tensor::value_type > &data, bool grad=false) const
	Injects data from a std::initializer_list into the output tensor of the Node, optionally setting its gradient requirement.

Detailed Description

Represents a Hard Sigmoid activation function node in a computational graph.

The HardSigmoidNode class applies the Hard Sigmoid activation function to the input tensor. The Hard Sigmoid function is a computationally efficient approximation of the sigmoid function and is defined as:

HardSigmoid(x) = max(0, min(1, alpha * x + beta))

where alpha and beta control the slope and offset, respectively.

Key features:

• Forward Pass: Applies the Hard Sigmoid activation function element-wise to the input tensor, mapping values to the range [0, 1] with linear interpolation.
• Backward Pass: Computes the gradient of the loss with respect to the input tensor. Gradients are propagated only for input values within the linear range (0 <= alpha * x + beta <= 1):

  HardSigmoid'(x) = alpha, if 0 <= alpha * x + beta <= 1
                    0, otherwise

• Shape Preservation: The output tensor has the same shape as the input tensor.
• Gradient Management: Automatically tracks gradients if required by the input tensor.

This class is part of the nz::nodes namespace and is used in models where efficiency is prioritized over precise non-linearity.

Note

• The alpha and beta parameters default to 0.2 and 0.5, respectively, but can be customized during construction.
• Efficient GPU computations are performed for both forward and backward passes.

Usage Example:

// Example: Using HardSigmoidNode in a computational graph
InputNode input({3, 3}, true);  // Create an input node with shape {3, 3}
 
float data[] = {-1.0f, 0.0f, 1.0f, 2.0f, -2.0f, 3.0f, -3.0f, 4.0f, -4.0f};  // Sample input values
input.output->dataInject(data);  // Copy data to the input tensor
 
HardSigmoidNode hard_sigmoid_node(&input, 0.2f, 0.5f);  // Apply Hard Sigmoid activation
hard_sigmoid_node.forward();  // Perform the forward pass
hard_sigmoid_node.backward();  // Propagate gradients in the backward pass
 
std::cout << "Output: " << *hard_sigmoid_node.output << std::endl;  // Print the result

See also

forward() for the Hard Sigmoid computation in the forward pass.

backward() for gradient computation in the backward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Definition at line 2803 of file Nodes.cuh.

Constructor & Destructor Documentation

HardSigmoidNode()

nz::nodes::calc::HardSigmoidNode::HardSigmoidNode ( Node * input, Tensor::value_type alpha = 0.2f, Tensor::value_type beta = 0.5f )

explicit

Constructor to initialize a HardSigmoidNode for applying the Hard Sigmoid activation function.

The constructor initializes a HardSigmoidNode, which applies the Hard Sigmoid activation function to an input tensor. It establishes a connection to the input node, initializes the output tensor, and sets the alpha and beta parameters as well as the node type.

Parameters

input	A pointer to the input node. Its output tensor will have the Hard Sigmoid activation applied.
alpha	The slope parameter for the linear part of the Hard Sigmoid function. Defaults to 0.2.
beta	The offset parameter for the Hard Sigmoid function. Defaults to 0.5.

• The input node is added to the inputs vector to establish the connection in the computational graph.
• The output tensor is initialized with the same shape as the input tensor, and its gradient tracking is determined based on the input tensor's requirements.
• The alpha and beta parameters control the slope and offset of the Hard Sigmoid activation function, influencing the gradient flow and the range mapping.
• The node's type is set to "HardSigmoid" to reflect its operation.

Note

• The Hard Sigmoid activation function is defined as:

  HardSigmoid(x) = max(0, min(1, alpha * x + beta))

• This node supports automatic gradient tracking if the input tensor requires gradients.

See also

forward() for the forward pass implementation.

backward() for gradient computation in the backward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Definition at line 476 of file Nodes.cu.

Member Function Documentation

backward()

void nz::nodes::calc::HardSigmoidNode::backward ( )

overridevirtual

Backward pass for the HardSigmoidNode to compute gradients.

The backward() method computes the gradient of the loss with respect to the input tensor by applying the derivative of the Hard Sigmoid activation function. The gradient computation is defined as:

HardSigmoid'(x) = alpha, if 0 <= alpha * x + beta <= 1
                  0, otherwise

where alpha and beta control the slope and offset of the Hard Sigmoid function.

• A CUDA kernel (HardSigmoidBackward) is launched to compute the gradients in parallel on the GPU.
• The derivative of the Hard Sigmoid function is applied element-wise to the input tensor's data and combined with the gradient of the output tensor to compute the input gradient.
• The computed gradient is stored in the gradient tensor of the input node.

Note

• Gradients are only computed and propagated if the input tensor's requiresGrad property is true.
• The shape of the gradient tensor matches that of the input tensor.

See also

forward() for the Hard Sigmoid computation in the forward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Implements nz::nodes::Node.

Definition at line 491 of file Nodes.cu.

Here is the call graph for this function:

forward()

void nz::nodes::calc::HardSigmoidNode::forward ( )

overridevirtual

Forward pass for the HardSigmoidNode to apply the Hard Sigmoid activation function.

The forward() method applies the Hard Sigmoid activation function element-wise to the input tensor. The result is stored in the output tensor. The Hard Sigmoid function is defined as:

HardSigmoid(x) = max(0, min(1, alpha * x + beta))

• A CUDA kernel (HardSigmoid) is launched to compute the activation function in parallel on the GPU.
• The grid and block dimensions are dynamically calculated based on the size of the output tensor to ensure efficient GPU utilization.
• The alpha and beta parameters, provided during construction, control the slope and offset of the linear part of the activation function.

Note

• The shape of the output tensor matches that of the input tensor.

See also

backward() for the computation of gradients in the backward pass.

Author

Mgepahmge (https://github.com/Mgepahmge)

Date

2024/12/05

Implements nz::nodes::Node.

Definition at line 485 of file Nodes.cu.

Here is the call graph for this function:

---

The documentation for this class was generated from the following files:

• D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/include/NeuZephyr/Nodes.cuh
• D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/src/Nodes.cu