Categorical Deep Learning

Introduction

This tutorial explores the concepts of Categorical Deep Learning as presented by Petar Veličković. It aims to provide a step-by-step understanding of how Categorical Deep Learning generalizes Geometric Deep Learning using category theory. The focus will be on group actions, monads, algebras, and their implications for neural network architectures.

Step 1: Understand Group Actions and Algebras

• Group actions are operations that transform data based on symmetry groups.
• In category theory, an algebra is a structure that allows execution of computations on objects.
• To execute a group action, define:
  • A carrier object (the data you want to transform).
  • A structure map (the operation to apply the transformation).
• Example: For a translation symmetry group acting on images:
  • Carrier object: A grid of pixels (real numbers of width times height).
  • Structure map: The shift operation that translates pixel positions.

Step 2: Define Monad and Algebra

• A monad is a construct that encapsulates computations and transformations.
• Define a monad using:
  • An endofunctor (a mapping from a category to itself).
  • A structure map that handles the transformation.
• The relationship between monads and algebras is crucial in ensuring that transformations are consistent and valid.
• Commutative diagrams must hold, ensuring that different paths through the transformation yield the same results.

Step 3: Connect Neural Networks with Monad Algebra Homomorphisms

• To create equivariant neural networks, use:
  • Monad algebra homomorphisms to map between different carrier objects.
  • Ensure that the transformation rules respect the structure of the symmetry group.
• For example, a neural network mapping features from one space to another must satisfy:
  • The output for a transformed input equals the transformation of the output.

Step 4: Explore Endofunctor Algebras

• Generalize concepts beyond monads with endofunctor algebras.
• Define an endofunctor as:
  • A mapping from a category to itself with a carrier object and a structure map.
• Example: Lists can be described using endofunctor algebras:
  • Carrier object: A set of list elements.
  • Structure maps: Functions to construct lists (e.g., nil for an empty list and cons for adding elements).

Step 5: Understand Coalgebras and Their Applications

• Coalgebras reverse the direction of morphisms and are useful for representing computations that produce new outputs.
• They can represent infinite processes, such as automata.
• Example: Mealy machines can be expressed with coalgebras:
  • The state changes based on inputs while producing outputs.

Step 6: Implement Parametric Morphisms in Neural Networks

• Transition to two categories to handle parameters more effectively.
• Define parametric morphisms, which include parameters as extra inputs:
  • This allows the representation of weight sharing across layers.
• Use block diagrams to visualize the relationships and parameter flow in neural networks.

Conclusion

Categorical Deep Learning leverages the principles of category theory to enhance our understanding of deep learning architectures. By exploring group actions, monads, algebras, and their applications, practitioners can design neural networks that respect underlying symmetries and constraints. For further exploration, consider diving into resources like nLab for deeper insights into category theory concepts.