{
  "version": 1,
  "slug": "deduplicate-lists",
  "title": "3 Ways To Deduplicate Your Lists in 2026 (AI-Powered)",
  "excerpt": "Deduplication is more than just deleting. Learn how to handle conflicting data by merging and updating records with our comprehensive guide.",
  "cover": {
    "src": "/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-cover.png",
    "optimized": "https://www.datablist.com/_next/image?url=%2Fhowto_images%2Fdeduplication-strategies%2Fthree-ways-to-deduplicate-speadsheet-files-cover.png&w=1200&q=75"
  },
  "url": "https://www.datablist.com/how-to/deduplicate-lists",
  "contentMarkdown": "\nCleaning duplicates involves more than hitting \"delete\"!\n\n**Some records are exact matches. Others contain conflicting values. Many are complementary and need merging instead of removal.**\n\n**Depending on your workflow, you may need to merge records, update a master record, or simply flag duplicates for review.**\n\nBasic tools delete rows without understanding field priority or business rules. That approach destroys useful data.\n\nSolid deduplication requires clear logic. Define how to select the master item, how to resolve conflicts, and what to do with secondary records.\n\nThis article explains the practical methods to merge, update, and remove duplicates in CSV files, Excel sheets, and CRMs.\n\nLet’s go!\n\n> 📌 **Summary For Those In a Rush**\n> \n> This article covers everything you need to know about deduplicating your spreadsheets, including how to merge, update, and remove duplicates the right way.\n> \n> **Problem:** Without understanding prioritization patterns and bulk actions, you'll either lose important data or keep the wrong records when dealing with duplicates.\n> \n> **Solution:** Datablist offers three dedupe methods: simple merging and removal, AI-powered editing for complex rules, and multi-file deduplication.\n> \n> **The Deduplication Methods We Cover:**\n> \n> 1. [Simple duplicates merging and removal on a single file](#simple-duplicates-merging-removal-on-a-single-file)\n> 2. [AI editing for complex prioritization rules before removing](#editing-duplicates-before-removing-them)\n> 3. [Removing duplicates across multiple files](#removing-duplicates-across-two-sheets-or-more)\n\n## In The Next 10 Minutes You’ll Learn {#in-the-next-10-minutes-youll-learn}\n\n- [What Datablist is and what qualifies us to talk about duplicates](#why-you-should-listen-to-us)\n- [What you need to know about duplicates before scrubbing / deduplicating your list](#understanding-deduplication-fundamentals)\n- [The 3 most effective ways of removing your duplicates (step-by-step instructions)](#deduplication-cleaning-duplicate-records-from-your-data)\n\n## Why You Should Listen To Us {#why-you-should-listen-to-us}\n\nDatablist is a platform for building [lead generation workflows](/use-cases/lead-list-building) that allows currently 26000 users to **find, enrich, and clean data** using over [60 different tools](/enrichments) from [AI Agents](/enrichments/ai-agent) to [Email Finders](/enrichments/email-finder), [AI processors](/enrichments/run-anthropic-claudeai-bulk), [Technology enrichments](/how-to/what-is-a-technology-finder), and more. \n\nAdditionally, **Datablist features an extensive deduplication suite** that allows you to merge, update, remove, or flag duplicates with just a few clicks, without coding.\n\n![Three Ways To Deduplicate Spreadsheet Files - Why You Can Trust Datablist](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-why-you-can-trust-datablist.png)\n\n## Understanding Deduplication Fundamentals {#understanding-deduplication-fundamentals}\n\nBefore moving with how to deduplicate your list, here are the principles behind the different deduplication techniques.\n\n**This section will cover:**\n\n- [A short explanation of duplicate types](#what-you-need-to-understand-deduplication-fundamentals)\n- Fundamentals of deduplicating conflicting records\n    - [Prioritization patterns for Master Items](#first-determining-a-prioritization-pattern)\n    - [Bulk actions to save time](#second-choosing-a-bulk-action)\n- [Questions to determine your goal faster](#questions-to-ask-yourself-before-deduplicating-a-list)\n\n### What You Need To Understand: Deduplication Fundamentals {#what-you-need-to-understand-deduplication-fundamentals}\n\n**The following points are only relevant for single-file deduplication.** [For multi-file deduplication](#removing-duplicates-across-two-sheets-or-more), you can only delete your copies from certain files, and not merge or update, making understanding these principles helpful rather than mandatory\n\nBy default, Datablist tries to merge duplicate records automatically. In practice, this doesn’t always work since **most users have conflicting duplicates.** \n\n![Three Ways To Deduplicate Spreadsheet Files - Conflicting Duplicates](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-conflicting-duplicates.png)\n\nWhen conflicts exist, the process relies on two concepts:\n- Prioritization patterns to choose the master record in a duplicate group\n- Bulk actions to handle the secondary records in this duplicate group\n\n#### Understanding Duplicate Types {#understanding-duplicate-types}\n\nWe classify duplicates by how similar their fields are.\n\n1. **Exact duplicates**: all columns contain identical values. These usually come from double imports or accidental copy-paste.\n2. **Conflicting duplicates**: records represent the same entity but conflict on some fields like phone, job title, or revenue.\n3. **Complementing duplicates**: each record holds different useful data that should be combined. One record might have an [email address](/how-to/finding-emails-at-scale) while its duplicate has a phone number, making them complementary.\n\n![Three Ways To Deduplicate Spreadsheet Files - Duplicate Types](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-duplicate-types.png)\n\n#### First: Determining a Prioritization Pattern {#first-determining-a-prioritization-pattern}\n\nYou must decide which record becomes the reference record. We call this the **Master Item Rule**. Remember this term, you’ll need it later.\n\n**Example Patterns/Master Item Rules:**\n\n- **Most complete:** Keeps the record with the most filled-in fields\n- **Last updated:** Keeps the most recently modified record\n- **First created:** Keeps the oldest record\n- **Lowest value:** Keeps the record with the smallest number in a specific column\n- **Highest value:** Keeps the record with the largest number in a specific column\n- **Matching value:** Keeps the record that matches a specific value on a property you define\n\n> 📘 **Master Item Rules**\n> \n> **Important:** \"Last updated\" and \"First created\" are only relevant for data that's been actively managed in Datablist over time. **If you've just uploaded your file, these options won't work** because imported spreadsheets don't include this metadata.\n\n> We recommend choosing “Most complete” or using the technique explained in the second part of this section if you’re not sure which master item rule to choose.\n\nFor complex cases, **Datablist allows you to use AI to create custom prioritization patterns**, for example: If column A contains “Hello people”, and column B contains “of Germany”.\n\n**More on this in the second part of the step-by-step section.**\n\n![Three Ways To Deduplicate Spreadsheet Files - Master Item Rules](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-master-item-rules.png)\n\n#### Second: Choosing a Bulk Action {#second-choosing-a-bulk-action}\n\nWhen you’ve chosen your prioritization pattern, the next thing you need to do is figure out what you want to do with the records that don’t match that pattern.\n\n**Example Bulk Actions To Process Duplicates:**\n\n- Delete secondary items\n- Merge the Master Item and the secondary item into one record\n- Merge selected properties of the secondary item with the Master Item, and delete the rest\n- Update selected properties of the Master Item with the values of the secondary item\n- Flagging duplicates without deleting them. This is especially valuable if you work in a large organization, and the secondary items are needed for compliance purposes\n- …. and everything else you can think of\n\n> 📘 **Understanding Merging Duplicates vs. Updating Duplicates**\n> \n> **Merging is to combine the values of both records.** This is especially good for duplicated CRM contacts where you have notes in both records\n> \n> **Updating means replacing specific values with better data from another source.** Use it when each duplicate has some correct information, like keeping contact A, but fixing its job title using the accurate one from contact B.\n\n![Three Ways To Deduplicate Spreadsheet Files - Deduplication Strategy](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-deduplication-strategy.png)\n\n### Questions To Ask Yourself Before Deduplicating a List {#questions-to-ask-yourself-before-deduplicating-a-list}\n\nNow that you understand patterns and bulk actions, use the following questions to quickly determine your prioritization pattern and what to do with the rest.\n\n#### Which Record Should Be Your Master Item? {#which-record-should-be-your-master-item}\n\nThis question helps you determine your **prioritization pattern**. Think about what makes one duplicate \"better\" than the other.\n\n**Ask yourself:**\n\n- Is there one record that's more complete than the others?\n- Did one record come from a more reliable source?\n- Is one record more recent or freshly updated?\n- Does one record have a specific value that makes it the \"correct\" version?\n\n**Your answer determines your Master Item rule:**\n\n- If completeness matters most → use \"Most complete\"\n- If recency matters most → use \"Last updated\" or \"First created\"\n- If a specific value determines the winner → use \"Matching value\"\n- If the logic is more complex → use AI Editing (Method 2)\n\n#### What Should Happen to the Non-Master Records? {#what-should-happen-to-the-non-master-records}\n\nThis question helps you determine your **bulk action**. Once you've picked a winner, what do you want to do with the losers?\n\n**Ask yourself:**\n\n- Do the other records have any valuable data I want to keep?\n- Should I combine information from multiple records into one?\n- Do I just need to delete the extras and move on?\n- Do I need to flag duplicates for review instead of deleting them?\n\n**Your answer determines your bulk action:**\n\n- If other records have no value → simply **drop all conflicting values/ delete** them\n- If other records have useful data → **combine the conflicting values** or **update** the master item\n- If you need compliance records → **flag** duplicates without deleting\n- If you need to cherry-pick specific values → use AI Editing (Method 2)\n\n![Three Ways To Deduplicate Spreadsheet Files - It’s Really Simple](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-its-really-simple.png)\n\n## Deduplication: Cleaning Duplicate Records From Your Data {#deduplication-cleaning-duplicate-records-from-your-data}\n\n**Datablist has a deduplication suite that handles everything** from simple duplicate removal to multi-file deduplication. Therefore, this section will feature 3 different workflows: \n\n1. [Merging and removing duplicates on a single file based on simple rules](#simple-duplicates-merging-removal-on-a-single-file)\n2. [Updating and removing duplicates on a single file with complex rules](#editing-duplicates-before-removing-them)\n3. [Removing duplicates across multiple files; no merging possible](#removing-duplicates-across-two-sheets-or-more)\n\nLet’s get started!\n\n#### How Datablist Handles Duplicates - Quick Revisit {#how-datablist-handles-duplicates---quick-revisit}\n\n**If you have read the last section, you can skip this;** if you haven’t, use this simple summary so you understand exactly what you will be doing.\n\n1. Datablist scans your data and finds rows that have matching information in the columns you specify. \n2. When it finds duplicates, it lets you auto-merge them for exact matches\n3. If you have conflicting duplicates, it asks you to choose a pattern by which to prioritize one record over the other (we call it the \"Master Item Rule\").\n4. When you’ve defined your Master Item Rule, it allows you to merge, update, flag, or delete the second duplicate record from the pair.\n\n### Simple Duplicates Merging & Removal On A Single File {#simple-duplicates-merging-removal-on-a-single-file}\n\nThis is the simplest way to remove duplicates. You have a list with some entries appearing more than once, and you want to keep only one copy of each record.\n\n#### When it's useful: {#when-its-useful}\n\n- You imported the same CSV file twice by accident\n- Your CRM export contains duplicate contacts\n- Scraped data has repeated entries from pagination errors\n\n#### Step 1: Sign Up And Upload Your Data {#step-1-sign-up-and-upload-your-data}\n\n1. Sign up for Datablist\n    \n![Three Ways To Deduplicate Spreadsheet Files - Datablist Homepage](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-datablist-homepage.png)\n    \n2. ***Upload*** your CSV or Excel\n    \n![Three Ways To Deduplicate Spreadsheet Files - Datablist Start Page](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-datablist-start-page.png)\n    \n\n#### Step 2: Navigate to the Duplicates Finder {#step-2-navigate-to-the-duplicates-finder}\n\nClick on ***Clean*** in the top menu of the app and select ***Duplicates Finder***\n\n![Three Ways To Deduplicate Spreadsheet Files - Duplicated Contacts](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-duplicated-contacts.png)\n\n#### Step 3: Choose Your Unique Identifier {#step-3-choose-your-unique-identifier}\n\nIn this step, you’ll have two options:\n\n**Option 1:** Choose one or a few columns as a unique identifier - RECOMMENDED\n\nThink of a unique identifier as the piece of information that makes each record special. For example: \n\n- **Using one column:** If you choose \"Email\" as your unique identifier, then [john@example.com](mailto:john@example.com) will be considered unique even if everything else matches\n- **Using multiple columns:** If you choose \"First Name\" + \"Company\" together, then \"John\" at \"Microsoft\" is different from \"John\" at \"Google\"\n\nThe more columns you select, the stricter the matching becomes. We recommend starting with just one or two columns that truly identify unique records in your data.\n\n![Three Ways To Deduplicate Spreadsheet Files - Choosing a Unique Identifier](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-unique-identifier.png)\n\n**Option 2:** Deduplicate based on all properties - NOT RECOMMENDED\n\nThis option will check if **every single column** in a row matches exactly with another row. This means that two rows are only considered duplicates if **all their data is identical**.\n\n**Why we don't recommend this:** In real-world data, duplicates rarely match perfectly across all columns. For example, the same person might have slightly different job titles, or the same company might have different employee counts from different sources. If you use this option, you'll miss most duplicates.\n\n![Three Ways To Deduplicate Spreadsheet Files - Exact Duplicates Are Rare](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-all-properties.png)\n\n**When the second option might be useful:** Use this only if you're looking for exact duplicate rows that were imported twice by mistake, where literally every field is identical.\n\nOnce you selected the properties you want to deduplicate on, scroll down and click on ***Next***\n\n#### Step 4: Select Comparison Algorithm {#step-4-select-comparison-algorithm}\n\nIn this step, you have to select a comparison algorithm and processor for each property you want to deduplicate on. **We recommend keeping the default settings except for company names.**\n\n![Three Ways To Deduplicate Spreadsheet Files - Comparison Algorithm And Processors](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-comparison-algorithm-and-processors.png)\n\n**If you’re deduplicating based on company names:** If you’re deduplicating based on company names, then choose the company names processor since it’s the only one that Datablist can not automatically detect.\n\n#### Step 5: Select Master Item, Review, and Resolve Conflicts {#step-5-select-master-item-review-and-resolve-conflicts}\n\n1. Choose master item rule: As explained in the first section, Datablist always asks you to specify a Master Item rule. The default rule is “Most Complete”, but you can also choose another one.\n    \n![Three Ways To Deduplicate Spreadsheet Files - Master Item Rule Selection](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-master-item-rule-selection.png)\n    \n\n1. Review and resolve conflicts if necessary: Many times, when you have duplicates, they are not identical on all properties. That’s also the reason we ask you to specify a master item\n    \n    To resolve conflicts, you can choose either combine or drop the conflicting values. However, combining values works only for text-based properties, so if you have numbers, date time, etc., you’ll need to combine both rules; combining and dropping.\n    \n![Three Ways To Deduplicate Spreadsheet Files - Merge Settings](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-merge-settings.png)\n    \n\n1. Click on ***Refresh Merging Preview*** to see the changes that will be made\n    \n![Three Ways To Deduplicate Spreadsheet Files - Dropping Conflicting Values](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-dropping-conflicting-values.png)\n    \n\n#### Step 6: Running and Reviewing {#step-6-running-and-reviewing}\n\nNow, the only thing you’ll need to do is to click ***Auto-merge when possible.***\n\n![Three Ways To Deduplicate Spreadsheet Files - Merging Preview](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-merging-preview.png)\n\nOnce you’ve merged your duplicates, Datablist will let you download the changes that have been made as a CSV. The file will include: \n\n- All duplicates you had in your file\n- The records where those duplicates have been merged to\n- The changes that have been made\n- The Datablist record ID\n\nDownloading that file is optional\n\n![Three Ways To Deduplicate Spreadsheet Files - Deduping Successful](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-dedupe-successful.png)\n\n> 💡 **If You’ve Made Any Mistakes**\n> \n> You can also revert the changes you made by clicking the history button and undoing the actions when going back to your spreadsheet view.\n\n### Editing Duplicates Before Removing Them {#editing-duplicates-before-removing-them}\n\nSometimes the simple master item rules aren't enough. What if you want to keep the phone number from one record but the job title from another? This is where AI Editing comes in.\n\n**How it works:** Instead of choosing a preset rule, you describe exactly what you want in plain English. Datablist's AI reads your instructions, generates a script, and applies your custom logic to every duplicate group.\n\n#### When It's Useful: {#when-its-useful}\n\n- You have contacts from multiple sources (CRM, LinkedIn, phone lists) and want to combine the best data from each\n- Your duplicates have different fields filled in, and you want to cherry-pick specific values\n- You need custom logic that doesn't fit the standard master item rules\n- You want to update records before deleting them, not just pick a winner\n- You want to flag the duplicates instead of deleting them for compliance reasons\n\n#### Step 1: Sign Up And Upload Your Data {#step-1-sign-up-and-upload-your-data}\n\n1. Sign up for Datablist\n    \n![Three Ways To Deduplicate Spreadsheet Files - Datablist Homepage](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-datablist-homepage-1.png)\n    \n2. ***Upload*** your CSV or Excel\n    \n![Three Ways To Deduplicate Spreadsheet Files - Datablist Start Page](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-datablist-start-page-1.png)\n    \n\n#### Step 2: Navigate to the Duplicates Finder {#step-2-navigate-to-the-duplicates-finder}\n\nClick on ***Clean*** in the top menu of the app and select ***Duplicates Finder***\n\n![Three Ways To Deduplicate Spreadsheet Files - Duplicated Contacts](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-duplicated-contacts-1.png)\n\n#### Step 3: Choose Your Unique Identifier {#step-3-choose-your-unique-identifier}\n\nSelect the column(s) you want to use for matching duplicates. Once selected, scroll down and click on ***Next***\n\n![Three Ways To Deduplicate Spreadsheet Files - Choosing a Unique Identifier](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-unique-identifier-1.png)\n\n#### Step 4: Select Comparison Algorithm {#step-4-select-comparison-algorithm}\n\nSelect a comparison algorithm and processor for each property you want to deduplicate on. **We recommend keeping the default settings except for company names.**\n\n![Three Ways To Deduplicate Spreadsheet Files - Comparison Algorithm And Processors](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-comparison-algorithm-and-processors-1.png)\n\n#### Step 5: Open AI Editing {#step-5-open-ai-editing}\n\nInstead of selecting a master item rule, click on ***AI Editing*** in the deduplication panel.\n\n![Three Ways To Deduplicate Spreadsheet Files - AI Deduplication](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-ai-deduplication.png)\n\n#### Step 6: Write Your Prompt {#step-6-write-your-prompt}\n\nDescribe what you want in plain English. Here's a practical example:\n\nLet's say you have contact data from two sources: phone verification and LinkedIn scraping. The phone records have verified numbers, but LinkedIn has updated job titles and company names. You want to keep the phone record as the master but update it with LinkedIn data.\n\n**Here's the prompt I used:**\n\n```\nSelect the records with \"Phone\" as source as master item and update them with the job title and company name from the record with the \"LinkedIn\" as source. \n\nThe source: /source\nThe job title: /job title\nThe company name: /company\n\nDelete the second item when finished\n```\n\n**Note:** Don’t forget to map your properties to the prompt using ***/*** \n\nClick ***Generate and preview changes*** when ready\n\n![Three Ways To Deduplicate Spreadsheet Files - AI Prompt](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-ai-prompt.png)\n\n#### Step 7: Review and Apply the Changes {#step-7-review-and-apply-the-changes}\n\nDatablist will show you exactly what changes the AI will make before applying them. Review the preview to make sure it matches your expectations.\n\nOnce you're happy with the preview, click ***Run AI Script*** to apply the changes to all duplicate groups. Then export your cleaned data.\n\n![Three Ways To Deduplicate Spreadsheet Files - AI Deduplication Preview](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-ai-deduplication-preview.png)\n\n> 💡 **Prompt Tips for Better Results**\n> \n> Be very specific about your expectations. The more precisely you can describe what you want it to do, the better your results will be.\n\n#### With This You Can Also: {#with-this-you-can-also}\n\n- **Flag duplicates instead of deleting them:** Write a prompt like \"Add 'DUPLICATE' to the status column for all non-master items instead of deleting them\"\n- **Combine text fields:** \"Merge all notes from duplicate records into the master item's notes field, separated by line breaks\"\n- **Prioritize by source quality:** \"Use Salesforce records as master when available, otherwise use HubSpot, then spreadsheet imports\"\n- …. or anything else you can think of.\n\n### Removing Duplicates Across Two Sheets or More {#removing-duplicates-across-two-sheets-or-more}\n\nIf you have two different CSV files and you want to find records that appear in both or deduplicate a new [lead list](/how-to/create-lead-list-with-emails) against your existing CRM export, **Datablist makes it simple.**\n\n**How it works:** Unlike single-file deduplication, this workflow compares records across multiple files and removes duplicates that span different data sources. You can select two files or more with no limit.\n\n#### When It's Useful: {#when-its-useful}\n\n- You're importing new leads and want to avoid duplicates with existing contacts\n- You're merging data from multiple vendors or sources\n- You need to find an overlap between two customer lists\n- You want to prevent contacting the same prospect twice\n- You need to consolidate customer data from various departments or branches\n- … and for many more [data cleaning](/use-cases/data-cleaning) workflows\n\n> 📘 **Important Difference From Single-File Deduplication**\n> \n> When deduplicating across multiple files, Datablist removes duplicates entirely rather than merging them.\n\n#### Step 1: Sign Up And Upload Your Files {#step-1-sign-up-and-upload-your-files}\n\n1. Sign up for Datablist\n    \n![Three Ways To Deduplicate Spreadsheet Files - Datablist Homepage](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-datablist-homepage-2.png)\n    \n2. ***Import*** your first CSV or Excel file\n    \n![Three Ways To Deduplicate Spreadsheet Files - Datablist Start Page](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-datablist-start-page-2.png)\n    \n3. ***Import*** your second file into another collection (and any additional files you want to deduplicate across)\n    \n![Three Ways To Deduplicate Spreadsheet Files - Import Second File](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-upload-second-file.png)\n    \n\n1. **Make Sure You Have a Unique Identifier**\n\nBefore proceeding, confirm that all your files share at least one common column that can be used as a unique identifier. This could be:\n\n- Email address\n- LinkedIn URL\n- Company domain\n- Phone number\n- Any other field that uniquely identifies a record\n\n#### Step 2: Navigate to the Duplicates Finder {#step-2-navigate-to-the-duplicates-finder}\n\nClick on ***Clean*** in the top menu of the app and select ***Duplicates Finder***\n\n![Three Ways To Deduplicate Spreadsheet Files - Duplicated Contacts](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-duplicated-contacts-2.png)\n\n#### Step 3: Enable Multi-Collection Deduplication {#step-3-enable-multi-collection-deduplication}\n\n1. Check ***Check Duplicate Items Across Several Collections***\n2. Select the collection(s) **/** file(s) you just imported\n    \n![Three Ways To Deduplicate Spreadsheet Files - Multi File Selection](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-multi-file-selection-1.png)\n    \n\n#### Step 4: Choose Your Unique Identifier Property {#step-4-choose-your-unique-identifier-property}\n\nSelect the property you want to use for matching duplicates across files. You can select multiple properties, but make sure all files contain these properties to keep your deduplication process accurate.\n\n![Three Ways To Deduplicate Spreadsheet Files - Choosing a Unique Identifier](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-multi-file-deduplication.png)\n\n#### Step 5: Select Comparison Algorithm {#step-5-select-comparison-algorithm}\n\nChoose the comparison mechanism that fits your data:\n\n- **Exact:** Best for URLs, domains, or IDs where you need exact matches\n- **Smart:** Best for text-based properties where slight variations might exist\n    \n![Three Ways To Deduplicate Spreadsheet Files - Comparison Algorithm And Processors](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-comparison-algorithm-and-processors-2.png)\n    \n\nClick on ***Run duplicates check*** once you've chosen the comparison method.\n\n#### Step 6: Set Up Cleaning Rules {#step-6-set-up-cleaning-rules}\n\nChoose how you want to handle the duplicates:\n\n- **Remove duplicate items from collection X:** Removes duplicates from your selected file\n- **Keep duplicate items only in collection X:** Only available when deduplicating across 3 or more collections\n\nClick on ***Process duplicate items*** to continue.\n\n![Three Ways To Deduplicate Spreadsheet Files - Auto Cleaning Rule](/howto_images/deduplication-strategies/three-ways-to-deduplicate-speadsheet-files-auto-cleaning-rule.png)\n\n**That’s it!**\n\n## Conclusion {#conclusion}\n\nCongrats, you reached the end, and know now more about deduplication than most people will ever learn. **Here’s a quick recap of today’s most important lessons:** \n\n1. **Duplicates aren't all the same,** and knowing which type you’re dealing with makes a big difference\n2. Picking the right Master Item and bulk action can **save you hours of manual cleanup**\n3. Unlike other tools that lock you into their way of doing things, **Datablist lets you handle duplicates exactly how you need to**\n\nSo whether you're merging contacts from a messy CRM, applying custom logic with AI, or cleaning new leads against your existing database, you've got the tools and the knowledge to do it right. Happy deduplicating!\n\n## Frequently Asked Question {#frequently-asked-question}\n\n### How Does Datablist Decide Which Duplicate Record To Keep? {#how-does-datablist-decide-which-duplicate-record-to-keep}\n\nDatablist doesn't decide, you do. You choose a **Master Item Rule** (like \"Most complete\" or \"Last updated\") that tells Datablist which record to prioritize. If your logic is more complex, you can use AI Editing to define custom rules in plain English (our AI assistant will handle the rest).\n\n### What Differentiates Datablist's Deduplication And Matching Suite From Other Products? {#what-differentiates-datablists-deduplication-and-matching-suite-from-other-products}\n\nThree things: **flexibility**, **AI-powered customization**, and **price**. Most tools only let you delete duplicates. Datablist lets you merge, update, flag, or delete based on rules you define. The AI Editing feature handles complex logic that other tools simply can't. And the next comparable product costs multiple thousands of dollars per year (enterprise software).\n\n### What If I Don't Want To Delete My Duplicates? {#what-if-i-dont-want-to-delete-my-duplicates}\n\nYou can flag them instead. Use AI Editing and write a prompt like: \"Add 'DUPLICATE' to the status column for all non-master items instead of deleting them.\" This is especially useful for compliance purposes or when you need to review duplicates before removing them.\n\n### What If The Master Item Rules Don't Fit My Use Case? {#what-if-the-master-item-rules-dont-fit-my-use-case}\n\nUse AI Editing. Instead of choosing a preset rule, you describe your logic in plain English, and Datablist's AI creates a custom script for you. For example: \"Keep the record from Salesforce, but use the job title from LinkedIn.\"\n\n### Can I Create Custom Master Item Rules? {#can-i-create-custom-master-item-rules}\n\nYes. Datablist's AI Editing feature lets you write any prioritization rule you can describe. Want to keep records where column A contains a specific value? Or prioritize based on multiple conditions? Just type what you need, and the AI handles the rest.\n\n### What Is A Unique Identifier In Deduplication? {#what-is-a-unique-identifier-in-deduplication}\n\nA unique identifier is the column (or combination of columns) that makes each record distinct. For example, if you use \"Email\" as your unique identifier, two rows with the same email are considered duplicates, even if other fields differ. You can also combine columns like \"First Name\" + \"Company\" for stricter matching.\n\n### How Can I Deduplicate A List With Conflicting Values? {#how-can-i-deduplicate-a-list-with-conflicting-values}\n\nConflicting duplicates happen when two records represent the same entity but have different values in some fields. To handle them: (1) Choose a Master Item Rule to pick which record wins, (2) Decide whether to combine, drop, or update the conflicting values, (3) **Use Datablist's deduplication suite to apply your choices in bulk.** For complex cases, AI Editing lets you cherry-pick specific values from different records.\n\n### How Can I Flag Duplicates Without Deleting Them? {#how-can-i-flag-duplicates-without-deleting-them}\n\nYou can use Datablist’s AI Editing feature inside their Deduplication and Matching Suite. Simply write a prompt like: \"Add 'DUPLICATE' to the status column for all non-master items instead of deleting them.\" This marks your duplicates for review while keeping all your data intact, perfect for compliance or when you need manual approval before removal.\n\n### How To Update Duplicated Records Without Deleting? {#how-to-update-duplicated-records-without-deleting}\n\nUpdating duplicates means replacing specific values in your master record with better data from another source. For this, **you can use Datablist’s AI Editing** feature inside their Deduplication and Matching Suite. **The only thing you need to do is describe what you want, for example:** \"Keep records from Source A, but update the job title and company name using values from Source B.\" The AI applies your logic to all duplicate groups, then you can delete the extras or keep them flagged."
}