Remove duplicates from scraped data in ZeroWork

Introduction

In this tutorial, you will learn how to remove duplicate entries from your scraped data using ZeroWork. This process is essential for maintaining clean and accurate datasets, especially when collecting profiles from platforms like LinkedIn. By following these steps, you will ensure that your data table remains free of duplicates, enhancing the quality of your analysis and reporting.

Step 1: Add the Remove Duplicates Building Block

• Open your workflow in ZeroWork.
• Locate the building block section where you can add new actions.
• Find and select the Remove Duplicates block.
• Choose the data table that contains the profiles you want to clean up.
• Select the column that will be used to identify duplicates. It is recommended to use the Profile Link column, as it is unique for each profile.

Step 2: Configure the Building Block

• Ensure you only select one column (e.g., Profile Link) to improve processing speed. If no column is selected, ZeroWork will compare all columns across rows, which can slow down the operation unnecessarily.
• Add the Remove Duplicates block after the Repeat block in your workflow. This ensures that the action runs only once after the entire loop has completed, rather than during each iteration, which could significantly slow down your task.

Step 3: Adjust Loop Settings for Testing

• Before testing your workflow, modify the loop settings:
  • Set the number of repetitions to 0. This will skip the loop and prevent the workflow from paginating through results, allowing you to focus solely on the Remove Duplicates action.
  • Deactivate any blocks that involve opening links (like LinkedIn) if they are not necessary for this demonstration.

Step 4: Run the Taskboard

• With your adjustments made, run the taskboard.
• Monitor the execution to ensure that the Remove Duplicates action processes correctly.

Step 5: Verify the Results

• After the taskboard has completed running, check the data table.
• Scroll through the results to confirm that duplicate profiles have been successfully removed.

Conclusion

By following these steps, you can efficiently remove duplicates from your scraped data in ZeroWork. Remember to always check which column you are using to identify duplicates and make adjustments to your workflow settings for optimal performance. This practice will help maintain the integrity of your data when running similar tasks in the future. For further improvements, consider exploring additional features of ZeroWork that can enhance your data collection and management processes.