Base class for nodes in a neural network or computational graph. More...

Inheritance diagram for nz::nodes::Node:

[legend]

Public Member Functions
virtual void	forward ()=0
	Abstract method for the forward pass computation.

virtual void	backward ()=0
	Abstract method for the backward pass (gradient computation).

virtual void	print (std::ostream &os) const
	Prints the type, data, and gradient of the node.

void	dataInject (Tensor::value_type *data, bool grad=false) const
	Injects data into a relevant tensor object, optionally setting its gradient requirement.

template<typename Iterator >
void	dataInject (Iterator begin, Iterator end, const bool grad=false) const
	Injects data from an iterator range into the output tensor of the InputNode, optionally setting its gradient requirement.

void	dataInject (const std::initializer_list< Tensor::value_type > &data, bool grad=false) const
	Injects data from a std::initializer_list into the output tensor of the Node, optionally setting its gradient requirement.

Detailed Description

Base class for nodes in a neural network or computational graph.

The Node class serves as an abstract base class for all types of nodes in a computational graph, commonly used in neural networks. Each node represents an operation or a layer in the graph, with input and output connections that allow data to flow through the network. The forward() and backward() methods define the computations to be performed during the forward and backward passes of the network, respectively.

This class is designed to be subclassed and extended for specific layers or operations. Derived classes are required to implement the forward() and backward() methods to define the specific computations for each node.

Key features:

Inputs: A vector of pointers to other nodes that provide input data to this node.
Output: A shared pointer to a Tensor object that stores the result of this node's computation.
Type: A string indicating the type of the node (e.g., "Basic", "Input", "MatMul").
Forward and Backward Passes: The pure virtual functions forward() and backward() that must be implemented by derived classes to perform the forward and backward propagation steps of the neural network.

This class is part of the nz::nodes namespace, and is intended to be used as a base class for defining custom layers or operations in a neural network.

Note

Derived classes must implement the forward() and backward() functions to define the specific computations for the node.
This class is designed to be used within a larger computational graph, where nodes are connected to form a complete neural network.

Author: Mgepahmge (https://github.com/Mgepahmge)

Date: 2024/11/29

Definition at line 114 of file Nodes.cuh.

Member Function Documentation

◆ backward()

virtual void nz::nodes::Node::backward ( )

pure virtual

Abstract method for the backward pass (gradient computation).

The backward() method is a pure virtual function in the Node class, which must be implemented by derived classes. It is responsible for computing the gradients during the backward pass of the neural network or computational graph, which is used for backpropagation in training.

During the backward pass, the error gradients are propagated backward through the network, from the output nodes to the input nodes. Each node computes the gradient of its output with respect to its input, using the chain rule of calculus, to update the weights or parameters of the network.

Derived classes that represent specific layers or operations must implement this method to define how gradients are calculated for that particular layer or operation.

Note

The backward() method must be implemented by any class derived from Node. It should compute the gradient of the output with respect to the node's input and store it in the node's grad tensor.
This method is essential for the backpropagation process during training, allowing the model to adjust its parameters based on the computed gradients.

See also: forward() for the forward propagation (computation) method.

Author: Mgepahmge (https://github.com/Mgepahmge)

Date: 2024/11/29

◆ dataInject() [1/3]

void nz::nodes::Node::dataInject	(	const std::initializer_list< Tensor::value_type > &	data,
		bool	grad = false ) const

Injects data from a std::initializer_list into the output tensor of the Node, optionally setting its gradient requirement.

Parameters

data	A std::initializer_list containing the data to be injected into the output tensor (host-to-device).
grad	A boolean indicating whether the output tensor should require gradient computation after data injection.

Returns: None.

This function is responsible for injecting data from a std::initializer_list into the output tensor of the Node. Memory management is handled by the underlying dataInject method of the Tensor class. The output tensor is assumed to have already allocated enough memory to accommodate the data in the std::initializer_list.

Regarding exception handling, this function does not explicitly catch any exceptions. Exceptions that might occur during data injection, such as memory allocation errors in the Tensor class, will propagate to the caller.

This function acts as a bridge between the Node and its output tensor, allowing data to be easily provided using a std::initializer_list.

Exceptions

None	explicitly, but the `dataInject` method of the `Tensor` class may throw exceptions, such as `std::bad_alloc` if memory allocation fails during the injection process.

Note

Ensure that the std::initializer_list contains enough elements to fill the output tensor according to its shape.
The CUDA runtime environment should be properly initialized before calling this function if the tensor is using CUDA memory.
The time complexity of this function is O(n), where n is the number of elements in the std::initializer_list, as it involves copying data from the list into the tensor.

```cpp
 
InputNode node({2, 2}, true);
node.dataInject({1.0f, 2.0f, 3.0f, 4.0f});
```

Definition at line 19 of file Nodes.cu.

◆ dataInject() [2/3]

template<typename Iterator >

void nz::nodes::Node::dataInject	(	Iterator	begin,
		Iterator	end,
		const bool	grad = false ) const

inline

Injects data from an iterator range into the output tensor of the InputNode, optionally setting its gradient requirement.

Template Parameters

Iterator The type of the iterators used to define the data range. It should support the standard iterator operations like dereferencing and incrementing.

Parameters

begin	An iterator pointing to the beginning of the data range (host-to-device). The data in this range will be injected into the output tensor.
end	An iterator pointing to the end of the data range (host-to-device).
grad	A boolean indicating whether the output tensor should require gradient computation after data injection. Defaults to false.

Returns: None.

This template function is used to inject data from an iterator range into the output tensor of the InputNode. Memory management is handled by the underlying dataInject method of the Tensor class. It is assumed that the output tensor has already allocated sufficient memory to hold the data from the iterator range.

Regarding exception handling, this function does not explicitly catch any exceptions. Exceptions that might occur during data injection, such as iterator invalidation or memory allocation errors in the Tensor class, will propagate to the caller.

This function serves as a wrapper around the dataInject method of the output tensor, facilitating the use of iterators to provide data for injection.

Exceptions

None	explicitly, but the `dataInject` method of the `Tensor` class may throw exceptions, such as `std::bad_alloc` if memory allocation fails during the injection process.

Note

Ensure that the iterator range [begin, end) is valid and that the data type pointed to by the iterators is compatible with the Tensor::value_type.
The CUDA runtime environment should be properly initialized before calling this function if the tensor is using CUDA memory.
The time complexity of this function is O(n), where n is the number of elements in the iterator range, as it involves copying data from the range into the tensor.

```cpp
#include <vector>
 
std::vector<value_type> data = {1.0f, 2.0f, 3.0f, 4.0f};
InputNode inputNode({2, 2}, true);
inputNode.dataInject(data.begin(), data.end());
```

Definition at line 278 of file Nodes.cuh.

◆ dataInject() [3/3]

void nz::nodes::Node::dataInject	(	Tensor::value_type *	data,
		bool	grad = false ) const

Injects data into a relevant tensor object, optionally setting its gradient requirement.

Parameters

data	A pointer to the data to be injected into the tensor (host-to-device). This data will be used to populate the tensor.
grad	A boolean indicating whether the tensor should require gradient computation after data injection. Defaults to false.

Returns: None.

This function is designed to inject data into a tensor object. Memory management within this function is handled by the underlying tensor operations. It is assumed that the tensor object has already allocated the necessary memory to hold the data pointed to by data.

Regarding exception handling, this function does not explicitly catch any exceptions. Exceptions that might occur during data injection, such as memory access errors or CUDA errors (if applicable), will propagate to the caller.

This function likely interacts with other components related to the tensor, such as the computation graph or the gradient computation system, depending on the value of grad.

Exceptions

None	explicitly, but underlying tensor operations may throw exceptions, such as std::bad_alloc if memory allocation fails during the injection process.

Note

Ensure that the data pointer is valid and points to enough data to fill the target tensor.
The CUDA runtime environment should be properly initialized before calling this function if the tensor is using CUDA memory.
The time complexity of this function is O(n), where n is the number of elements in the tensor, as it involves copying data into the tensor.

```cpp
 
value_type data[] = {1.0f, 2.0f, 3.0f, 4.0f};
// Assume there is an object that has the dataInject method
InputNode input({2, 2}), true);
obj.dataInject(data);
```

Definition at line 15 of file Nodes.cu.

◆ forward()

virtual void nz::nodes::Node::forward ( )

pure virtual

Abstract method for the forward pass computation.

The forward() method is a pure virtual function in the Node class, which must be implemented by derived classes. It is responsible for performing the computation during the forward pass of the neural network or computational graph.

In the forward pass, data flows through the network from input nodes to output nodes, and each node performs its specific computation (e.g., activation, matrix multiplication, etc.) based on the data it receives as input.

Derived classes that represent specific layers or operations (such as activation functions, convolution layers, etc.) must implement this method to define the exact computation to be performed for that layer.

Note

The forward() method must be implemented by any class derived from Node. It should modify the output of the node based on its inputs and computation.
This method does not return any value, as it updates the node's output tensor directly.

See also: backward() for the reverse propagation (gradient calculation) method.

Author: Mgepahmge (https://github.com/Mgepahmge)

Date: 2024/11/29

◆ print()

void nz::nodes::Node::print ( std::ostream & os ) const

virtual

Prints the type, data, and gradient of the node.

The print() method outputs the information about the node, including its type, the tensor data stored in the node's output, and the corresponding gradient. This is useful for debugging and inspecting the state of nodes in a computational graph or during training, allowing for easy visualization of the node's content and gradients.

The method outputs the following details:

Type: The type of the node (e.g., the operation it represents, such as "MatrixMul", "ReLU", etc.).
Data: The tensor data stored in the node's output tensor.
Gradient: If the node has a computed gradient, it is also displayed, providing insights into the gradient values that are being backpropagated through the network during training.

This method is primarily used for debugging and monitoring the state of tensors and gradients, making it easier to inspect how the data and gradients flow through the network.

Note

The output tensor should contain both the data and the gradient information, and both are printed when this method is called.
This method is typically used during development or debugging phases and should not be used in performance-critical code as it involves printing potentially large amounts of data.

Parameters

os	The output stream (e.g., `std::cout`) to which the node's information will be printed.

Author: Mgepahmge (https://github.com/Mgepahmge)

Date: 2024/11/29

Reimplemented in nz::nodes::io::OutputNode.

Definition at line 10 of file Nodes.cu.

The documentation for this class was generated from the following files:

D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/include/NeuZephyr/Nodes.cuh
D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/src/Nodes.cu

Public Member Functions

Detailed Description

Member Function Documentation

◆ backward()

◆ dataInject() [1/3]

◆ dataInject() [2/3]

◆ dataInject() [3/3]

◆ forward()

◆ print()