Easy videos with FLOW and VACE Wan2.1 Video ComfyUI - IP Adapter, Depth, openpose.

Introduction

This tutorial provides a step-by-step guide on using the VACE framework with ComfyUI for creating and editing synthetic videos and animations. With this guide, you will learn how to set up your environment, utilize essential nodes, and download necessary models to enhance your AI video generation capabilities.

Step 1: Set Up Your Environment

To begin using VACE with ComfyUI, follow these steps:

1. RunPod Setup

   • Visit the RunPod link: RunPod for Koala Nation.
   • Create an account if you don't have one, and set up your environment.

2. Download Required Nodes

   Install the following custom nodes via the manager

   • VideoHelperSuite
   • KJNodes
   • Controlnet_aux
   • DepthAnythingV2
   • Florence2
   • Note: Depth Anything and Florence models will automatically download during the first run of the corresponding nodes.

Step 2: Download Necessary Models

For VACE Wan 2.1 to function properly, you need to download specific models. Here’s how:

1. Text Encoder

   • Download the Text Encoder from this link: Text Encoder Model.
   • Place it in the directory: /workspace/ComfyUI/models/text_encoders/.

2. Variational Autoencoder (VAE)

   • Obtain the VAE model here: VAE Model.
   • Save it in the directory: /workspace/ComfyUI/models/vae/.

3. Diffusion Models

   Choose and download one of the following diffusion models

   • 1.3 Preview: Model 1.3 Preview
   • 14B: Model 14B
   • Place the downloaded model in the directory: /workspace/ComfyUI/models/diffusion_models/.

4. CausVid Model

   • Download CausVid from here: CausVid Model.
   • Save it in the directory: /workspace/ComfyUI/models/loras/.

Step 3: Explore Additional Resources

To further enhance your experience and understanding, consider reviewing the following resources:

• VACE Github Repository

  • Access the VACE project page here: VACE Github.

• Tutorials and Workflows

  • Review the tutorial on setting up RunPod for ComfyUI: ComfyUI Setup Tutorial.
  • Explore workflows using ComfyUI: Amazing ComfyUI Workflows.

Conclusion

By following these steps, you can successfully set up the VACE framework with ComfyUI for generating and editing synthetic videos. Ensure you have all necessary nodes and models installed for optimal performance. For further learning, explore additional resources provided, and start experimenting with your video creations!