I Built Make.com AI Marketplace Scraper

Introduction

In this tutorial, we will explore how to build an AI-powered web scraper using Make.com, Apify, and Airtable. This automation is designed to extract information from the Autotrader website and generate a report based on the collected data. Whether you're looking to automate data collection for your business or enhance your web scraping skills, this step-by-step guide will walk you through the entire process.

Step 1: Setting Up Your Apify Account

• Create an Apify account: Visit the Apify website and sign up for a free account.
• Access the Apify console: Once logged in, navigate to the Apify console where you can manage your scrapers.
• Create a new actor:
  • Click on "Actors" in the sidebar.
  • Select "Create new actor."
  • Choose an appropriate template or start from scratch to build your web scraper for Autotrader.

Step 2: Configuring the Web Scraper

• Define the target URL: Set the URL of the Autotrader page you want to scrape.

• Select the data to extract:

  • Identify the specific elements (e.g., car make, model, price, mileage) you want to scrape from the page.
  • Use CSS selectors or XPath to target these elements accurately.

• Write the scraping code: If needed, customize the default code provided by Apify to suit your requirements. Here’s a simple example:

  const { Apify } = require('apify');
  
  Apify.main(async () => {
      const requestList = await Apify.openRequestList('my-list', [
          { url: 'https://www.autotrader.com/' },
      ]);
      const crawler = new Apify.PuppeteerCrawler({
          requestList,
          handleRequestFunction: async ({ request }) => {
              const page = await Apify.utils.puppeteer.getPage();
              await page.goto(request.url);
              // Extract data here
          },
      });
      await crawler.run();
  });
  

Step 3: Integrating Make.com

• Create an account on Make.com: Sign up for an account if you don’t have one.
• Set up a new scenario:
  • Click on “Create a new scenario.”
  • Search for Apify to connect your web scraper.
• Configure Apify module:
  • Select the action to run your Apify actor.
  • Choose the output format you want to receive the scraped data in.

Step 4: Sending Data to Airtable

• Set up an Airtable account: Go to the Airtable website and create an account.
• Create a new base: Set up a new base where you will store the scraped data.
• Connect Airtable to Make.com:
  • In your Make.com scenario, add an Airtable module.
  • Authenticate and connect your Airtable account.
• Map the fields: Ensure that the fields in your Airtable base correspond to the data you have scraped.

Step 5: Testing and Running the Automation

• Run the scenario: Execute the Make.com scenario to test if the data flows correctly from Apify to Airtable.
• Check for errors: Monitor the output for any errors or missing data points. Adjust the scraping code or field mappings as necessary.
• Schedule the automation: Once confirmed, set up a schedule to run the automation at desired intervals.

Conclusion

You have now built a functional web scraper using Apify, Make.com, and Airtable to automate data collection from the Autotrader website. This setup not only streamlines data gathering but also enhances reporting capabilities for your projects. Explore further by integrating additional features or expanding your scraper to other websites. If you have any questions or need assistance, feel free to reach out to the community or consult the respective documentation for each tool used. Happy scraping!