Joel Grus - Livecoding Madness - Let's Build a Deep Learning Library

Title: Building a Deep Learning Library - Step-by-Step Tutorial

Step 1: Introduction

• Who is Joel Grus?
  • Joel Grus is a research engineer at the Allen Institute for Artificial Intelligence in Seattle.
  • He is the author of the book "Data Science from Scratch" and co-host of the podcast "Adversarial Learning."

Step 2: Setting up the Environment

• Using Python 3.6
  • Make sure you have Python 3.6 installed on your system.
  • Joel uses type hints extensively in his code.

Step 3: Creating the Library Structure

• Create a Directory
  • Create a directory named "gillnet" to house the deep learning library.
  • Add an __init__.py file to the folder.

Step 4: Implementing Tensors

• Define the Tensor Class
  • Implement a tensor class with methods for loss calculation and gradients.
  • Use NumPy for numerical computations.

Step 5: Implementing Layers

• Create Layer Classes
  • Implement a base layer class with forward and backward propagation methods.
  • Create specific layers like linear layers and activation layers (e.g., tanh).

Step 6: Defining Neural Networks

• Build Neural Networks
  • Define a neural network class that consists of multiple layers.
  • Implement methods for forward and backward propagation through the network.

Step 7: Implementing Optimizers

• Optimizing Parameters
  • Create an optimizer class (e.g., Stochastic Gradient Descent) to adjust network parameters based on gradients.

Step 8: Data Handling

• Data Iteration
  • Implement a data iterator to handle input-output data in batches.
  • Use binary encoding for data representation.

Step 9: Training the Neural Network

• Training Process
  • Define a training function that iterates over epochs and batches to train the neural network.
  • Use a loss function (e.g., mean squared error) to compute errors and gradients.

Step 10: Testing with FizzBuzz Problem

• Solving FizzBuzz with Neural Network
  • Apply the trained neural network to solve the FizzBuzz problem.
  • Encode input numbers and predict corresponding FizzBuzz outputs.

Step 11: Conclusion and Resources

• Wrap-Up and Resources
  • Review the steps taken to build the deep learning library.
  • Check out Joel Grus' GitHub repository for the code.
  • Explore more about artificial intelligence at Allen Institute for AI.

By following these steps, you can create your own deep learning library inspired by Joel Grus' live coding session.