Coding a Neural Network from Scratch in C: No Libraries Required

Introduction

In this tutorial, we will learn how to build a neural network from scratch using the C programming language without any external libraries. We will focus on training the neural network to predict the output of the XOR logical function, a classic problem in neural network training. By the end of this tutorial, you will understand the basic concepts of neural networks, including layers, weights, biases, and activation functions, and how to implement them in C.

Step 1: Understanding the XOR Problem

• The XOR function takes two binary inputs and produces one binary output based on the following rules:
  • Input: (0, 0) → Output: 0
  • Input: (0, 1) → Output: 1
  • Input: (1, 0) → Output: 1
  • Input: (1, 1) → Output: 0
• This problem cannot be solved with a simple linear classifier; thus, a neural network with at least one hidden layer is required.

Step 2: Setting Up the C Environment

• Ensure you have a C compiler installed (e.g., GCC).
• Use an IDE or text editor, such as Visual Studio Code, for coding and debugging.

Step 3: Initialize Project Files

• Create a new C file (e.g., xor_neural_network.c).
• Include necessary standard libraries at the top of your code:

  #include <stdio.h>
  #include <stdlib.h>
  #include <math.h>
  

Step 4: Define Constants and Variables

• Define constants for inputs, hidden nodes, and outputs:

  #define NUM_INPUTS 2
  #define NUM_HIDDEN_NODES 2
  #define NUM_OUTPUTS 1
  #define NUM_TRAINING_SETS 4
  

• Declare arrays for input data, output data, weights, biases, and activations.

Step 5: Initialize Weights and Biases

• Create a function to initialize weights randomly:

  double init_weight() {
      return (double)rand() / RAND_MAX; // Random values between 0 and 1
  }
  

• Initialize weights and biases using nested loops.

Step 6: Implement the Sigmoid Activation Function

• Define the sigmoid function and its derivative for use in the neural network:

  double sigmoid(double x) {
      return 1.0 / (1.0 + exp(-x));
  }
  
  double sigmoid_derivative(double x) {
      return x * (1.0 - x); // x is the output of the sigmoid function
  }
  

Step 7: Create Forward Pass Function

• Implement the forward pass to calculate activations for the hidden and output layers:

  void forward_pass(double input[NUM_INPUTS], double hidden_layer[NUM_HIDDEN_NODES], double output_layer[NUM_OUTPUTS]) {
      // Calculate hidden layer activations
      for (int j = 0; j < NUM_HIDDEN_NODES; j++) {
          hidden_layer[j] = 0; // Initialize
          for (int i = 0; i < NUM_INPUTS; i++) {
              hidden_layer[j] += input[i] * hidden_weights[i][j];
          }
          hidden_layer[j] = sigmoid(hidden_layer[j]); // Apply activation
      }
      
      // Calculate output layer activation
      for (int k = 0; k < NUM_OUTPUTS; k++) {
          output_layer[k] = 0; // Initialize
          for (int j = 0; j < NUM_HIDDEN_NODES; j++) {
              output_layer[k] += hidden_layer[j] * output_weights[j][k];
          }
          output_layer[k] = sigmoid(output_layer[k]); // Apply activation
      }
  }
  

Step 8: Implement Backpropagation

• Create a function to update weights based on the error between predicted and expected outputs:

  void backpropagation(double input[NUM_INPUTS], double hidden_layer[NUM_HIDDEN_NODES], double output_layer[NUM_OUTPUTS], double expected_output[NUM_OUTPUTS]) {
      // Calculate error
      double output_error[NUM_OUTPUTS];
      for (int k = 0; k < NUM_OUTPUTS; k++) {
          output_error[k] = expected_output[k] - output_layer[k];
      }
      
      // Update output weights
      for (int k = 0; k < NUM_OUTPUTS; k++) {
          for (int j = 0; j < NUM_HIDDEN_NODES; j++) {
              output_weights[j][k] += learning_rate * output_error[k] * sigmoid_derivative(output_layer[k]) * hidden_layer[j];
          }
      }
  
      // Calculate hidden layer error and update hidden weights
      double hidden_error[NUM_HIDDEN_NODES];
      for (int j = 0; j < NUM_HIDDEN_NODES; j++) {
          hidden_error[j] = 0;
          for (int k = 0; k < NUM_OUTPUTS; k++) {
              hidden_error[j] += output_error[k] * output_weights[j][k];
          }
          for (int i = 0; i < NUM_INPUTS; i++) {
              hidden_weights[i][j] += learning_rate * hidden_error[j] * sigmoid_derivative(hidden_layer[j]) * input[i];
          }
      }
  }
  

Step 9: Train the Neural Network

• Create a training loop to iterate through epochs and update weights:

  for (int epoch = 0; epoch < EPOCHS; epoch++) {
      for (int i = 0; i < NUM_TRAINING_SETS; i++) {
          forward_pass(training_inputs[i], hidden_layer, output_layer);
          backpropagation(training_inputs[i], hidden_layer, output_layer, training_outputs[i]);
      }
  }
  

Conclusion

By following this tutorial, you have successfully built a neural network from scratch in C to solve the XOR problem. You learned how to initialize the network, implement forward and backward passes, and train the network using a simple dataset. As a next step, consider experimenting with more complex datasets or modifying the architecture of your neural network to see how it affects performance.