Run ALL Your AI Locally in Minutes (LLMs, RAG, and more)

Introduction

This tutorial will guide you through setting up a local AI environment using Ollama, Qdrant, Postgres, and n8n. You will learn how to install and configure these tools to create a RAG (Retrieval-Augmented Generation) AI Agent that functions entirely on your local machine. This setup is ideal for developers looking to leverage AI capabilities without relying on cloud services.

Step 1: Prepare Your Hardware

Ensure that your hardware meets the requirements for running AI models effectively. Recommended specifications include:

• GPU with at least 8GB of VRAM for smaller models (e.g., Llama 3.1 8b).
• At least 16GB of VRAM for larger models (e.g., Llama 3.1 70b).
• A CPU with multiple cores for optimal performance (e.g., i7-12700H with 14 cores and 20 threads is ideal).

Step 2: Install Required Software

1. Install GitHub Desktop

   • Download from GitHub Desktop.
   • Follow the installation instructions for your operating system.

2. Install Docker Desktop

   • Download from Docker.
   • Install Docker and ensure it is running before proceeding.

3. Clone the Local AI Starter Kit Repository

   • Open GitHub Desktop and clone the Local AI Starter Kit from the following link:
     • n8n Self-hosted AI Starter Kit.

Step 3: Set Up the Local AI Package

1. Navigate to the Cloned Directory

   • Open a terminal and navigate to the folder where you cloned the Local AI Starter Kit.

2. Build Docker Containers

   • Run the following command to build the necessary Docker containers:

     docker-compose build
     

3. Start the Docker Containers

   • Use the command below to start the containers:

     docker-compose up
     

   • Allow a few minutes for the containers to initialize properly.

Step 4: Create a Local RAG AI Agent

1. Access n8n Workflow Automation

   • Open your web browser and go to http://localhost:5678 to access the n8n interface.

2. Create a New Workflow

   • Click on “New Workflow” to start designing your RAG AI Agent.

3. Add Nodes to Your Workflow

   • Utilize the available nodes to connect your AI models, databases, and other components.
   • Common nodes include:
     • HTTP Request node for API interactions.
     • Database node for Postgres to store and retrieve data.
     • Function node for custom scripting.

4. Configure Each Node

   • Click on each node to set it up according to your requirements.
   • Make sure to define the inputs and outputs for each node accurately.

Step 5: Test Your Local RAG AI Agent

1. Run Your Workflow

   • After configuring the workflow, click the “Execute Workflow” button to test the entire setup.

2. Review Outputs

   • Check the outputs from each node to ensure that your RAG AI Agent is functioning as expected.
   • Troubleshoot any issues by revisiting your node configurations.

Conclusion

You have successfully set up a local AI environment using Ollama, Qdrant, Postgres, and n8n, and created a RAG AI Agent. This setup allows for both no-code solutions and custom-coded applications. Explore further by modifying your workflow or integrating additional AI models. For future developments, consider deploying this setup in the cloud for enhanced scalability.