Run ALL Your AI Locally in Minutes (LLMs, RAG, and more)
Table of Contents
Introduction
This tutorial will guide you through setting up a local AI environment using Ollama, Qdrant, Postgres, and n8n. You will learn how to install and configure these tools to create a RAG (Retrieval-Augmented Generation) AI Agent that functions entirely on your local machine. This setup is ideal for developers looking to leverage AI capabilities without relying on cloud services.
Step 1: Prepare Your Hardware
Ensure that your hardware meets the requirements for running AI models effectively. Recommended specifications include:
- GPU with at least 8GB of VRAM for smaller models (e.g., Llama 3.1 8b).
- At least 16GB of VRAM for larger models (e.g., Llama 3.1 70b).
- A CPU with multiple cores for optimal performance (e.g., i7-12700H with 14 cores and 20 threads is ideal).
Step 2: Install Required Software
-
Install GitHub Desktop
- Download from GitHub Desktop.
- Follow the installation instructions for your operating system.
-
Install Docker Desktop
- Download from Docker.
- Install Docker and ensure it is running before proceeding.
-
Clone the Local AI Starter Kit Repository
- Open GitHub Desktop and clone the Local AI Starter Kit from the following link:
Step 3: Set Up the Local AI Package
-
Navigate to the Cloned Directory
- Open a terminal and navigate to the folder where you cloned the Local AI Starter Kit.
-
Build Docker Containers
- Run the following command to build the necessary Docker containers:
docker-compose build
- Run the following command to build the necessary Docker containers:
-
Start the Docker Containers
- Use the command below to start the containers:
docker-compose up
- Allow a few minutes for the containers to initialize properly.
- Use the command below to start the containers:
Step 4: Create a Local RAG AI Agent
-
Access n8n Workflow Automation
- Open your web browser and go to
http://localhost:5678
to access the n8n interface.
- Open your web browser and go to
-
Create a New Workflow
- Click on “New Workflow” to start designing your RAG AI Agent.
-
Add Nodes to Your Workflow
- Utilize the available nodes to connect your AI models, databases, and other components.
- Common nodes include:
- HTTP Request node for API interactions.
- Database node for Postgres to store and retrieve data.
- Function node for custom scripting.
-
Configure Each Node
- Click on each node to set it up according to your requirements.
- Make sure to define the inputs and outputs for each node accurately.
Step 5: Test Your Local RAG AI Agent
-
Run Your Workflow
- After configuring the workflow, click the “Execute Workflow” button to test the entire setup.
-
Review Outputs
- Check the outputs from each node to ensure that your RAG AI Agent is functioning as expected.
- Troubleshoot any issues by revisiting your node configurations.
Conclusion
You have successfully set up a local AI environment using Ollama, Qdrant, Postgres, and n8n, and created a RAG AI Agent. This setup allows for both no-code solutions and custom-coded applications. Explore further by modifying your workflow or integrating additional AI models. For future developments, consider deploying this setup in the cloud for enhanced scalability.