Find, remove or merge duplicates

Datablist's Duplicate Finder detects duplicate records in one collection or across several collections. It can compare complete records, match selected properties with exact or fuzzy algorithms, standardize business data before comparison, and then remove, merge, or custom-process the duplicate groups.

The duplicate check is read-only. Datablist does not edit or delete records until you review the results and start a processing action.

For a shorter workflow, see how to dedupe a list online. For a product overview, visit the Duplicates Remover.

Open a collection and choose one of the Duplicate Finder actions from Clean in the collection header:

• Remove duplicates opens the tool with the deletion workflow selected.
• Merge duplicates opens it with data-preserving merge selected.
• Match across collections opens the multi-collection setup. This workflow requires a Premium plan.

The three actions use the same detection engine and results workspace. They start with a different goal so the first choices and primary action match the job you want to complete.

This page is the complete reference for the Duplicate Finder:

• Choose All Properties, Selected Properties, or several collections
• Configure comparison algorithms
• Standardize business data with processors
• Control empty values and multi-value cells
• Understand similarity scores and the review and processing thresholds
• Remove duplicates, merge data, or create a custom AI workflow
• Resolve conflicts and choose the record to keep
• Review, edit, merge, or skip individual groups
• Search and sort the duplicate preview
• Download duplicate groups, match results, and change logs
• See what is included in Free and Premium plans

Before you start: Import the records into a collection. See Importing Data for the available import methods.

Free and Premium feature availability

The core single-collection Duplicate Finder is available on the Free plan. Premium adds cross-collection matching, advanced fuzzy algorithms, advanced business-data processing, configurable survivorship, AI processing, and duplicate-group IDs and change-log downloads. In this page, Premium means a paid Starter or Growth plan.

Visitors using Datablist without an account can run an Exact duplicate check. Creating a free account unlocks Smart comparison and the other Free-plan features.

Choose the duplicate check method

The first screen offers two methods.

All Properties

All Properties compares every editable property in the current collection with the Exact algorithm. Two records become duplicates only when all their property values match.

Use this method to find complete duplicate rows, such as records created by importing the same file twice. It does not provide per-property processors, algorithms, or fuzzy settings.

With the 1,000-row deduplication sample, this configuration finds five two-record groups whose values match across every property.

Selected Properties & Multi Collections

Selected Properties & Multi Collections lets you choose the properties that identify a duplicate. It also unlocks per-property algorithms, processors, and comparison settings.

Examples:

• Select Email to find contacts that share an email address.
• Select First Name and Last Name to find people with similar names.
• Select Website to find companies that use the same domain.
• Select SKU and Supplier to find duplicate products.

For checks that use several properties, Datablist calculates a score for each property and uses their arithmetic mean as the similarity between two records.

Most business-data checks should use Selected Properties because identifiers often need normalization or fuzzy matching.

Select properties that help identify the entity, rather than descriptive properties that commonly change:

• For people, start with Email, Phone, or First Name and Last Name.
• For companies, start with Website or Company Name.
• For products, use a stable SKU, supplier identifier, or another business key.

Adding many non-identifying properties usually makes true duplicates harder to find. Use All Properties only when you specifically need identical rows.

Match data across several collections

Enable Check duplicate items across several collections? under Selected Properties to compare the current collection with one or more other collections in the same workspace.

Multi-collection matching is available on paid plans. Datablist combines the selected collections for detection while preserving each record's source collection.

You can start this workflow from Clean > Match across collections.

Map properties from the other collections

Each selected property in the current collection must be mapped to one property in every other collection. For example:

Datablist suggests a mapping when it finds a similar property, but you can change it. Every mapping is required before the check can run.

The review screen combines the mapped property names into one comparison configuration. Select the algorithm and processor that fit the mapped values.

Multi-collection detection returns matches across collections. Groups that contain records from only one collection are skipped. Run Duplicate Finder on that individual collection to process its internal duplicates.

The multi-collection results screen lets you download or process the matches:

• Open Result files to export matched, needs-review, unmatched, crosswalk, or detailed group CSV files. Select the source collection before generating a source-based file. These files describe the current matching results and do not modify records.
• Use Remove overlaps automatically to remove matches from a chosen collection or, with more than two collections, keep matches only from a chosen collection.
• Use Manual Merging assistant to review and merge a group across collections.

Automatic multi-collection cleaning removes records. It does not consolidate their values, so the screen warns when unique information could be lost. After automatic cleaning, Datablist offers a separate changes file for every collection from which records were deleted. Each file uses that collection's own columns and includes the deleted records' previous values.

The CSV downloads do not modify any collection. They reflect the active confidence thresholds and the current group membership. See Download duplicate groups and match results for the file schemas and one-to-many matching behavior.

Configure the selected properties

After choosing Selected Properties, select one or more properties. On a direct page load, the screen displays Loading... until the collection properties are available.

On the review screen, Datablist shows the source collections, record counts, selected properties, and one comparison configuration for each property.

Select the comparison algorithm and run the deduplication check

Each selected property has an Algorithm and a Processor. Some combinations also expose settings for case, multiple values, URLs, empty values, or fuzzy distance.

Algorithm availability

For authenticated users, text properties default to Smart. Anonymous users start with Exact. Non-text properties use Exact. Exact and Smart are available on the Free plan; Metaphone and Distance require a Premium plan.

Click Run duplicates check after reviewing the settings. The browser worker fetches the records, creates candidate pairs, calculates similarities, and builds duplicate groups.

Data processors

A processor tells Datablist how to interpret a property before comparison. It changes only the duplicate check. It does not rewrite the values stored in the collection.

Datablist initially selects a processor from the property type. You can select a different processor when a property stores another kind of data. For example, a Text property containing company websites can use the URL processor.

Text processor

The Text processor applies the selected comparison algorithm without data-specific rules.

• Exact trims leading and trailing spaces and is case insensitive by default.
• Smart ignores accents, spaces, punctuation, symbols, case, and word order.

Use Text for names, labels, addresses, and other text that does not need a more specific processor.

Email processor

Use the Email processor with Smart to normalize common representations of the same mailbox.

For mailto: links, Datablist removes the prefix and query parameters, then decodes the address. Plus aliases are removed from the part before @ for every provider.

Dot removal applies only to gmail.com and googlemail.com. Datablist keeps dots on Google Workspace and other business domains because jane.doe@example.com and janedoe@example.com may be different mailboxes.

The processor does not correct a misspelled domain or assume that two business domains are aliases. Exact comparison does not apply the Smart email rules.

Company Name processor

The Company Name processor removes common legal suffixes, business terms, and geographical terms before Smart, Metaphone, or Distance comparison.

Select the company property, choose Smart, then select Company Name as the processor.

Examples include:

• Acme Inc. and Acme
• Northwind Group LLC and Northwind
• Example Partners Europe and Example

In the 1,000-row deduplication sample, this check returns 200 two-row groups. Fifty groups contain designed company-name variants. The other 150 designed duplicate pairs already share an identical company value. The exported result contains all 200 expected synthetic entities without combining different entities.

Groups can show a 100% match score while remaining in Needs review. This means other properties need conflict-resolution rules; it does not lower the company match score.

See the company-name deduplication guide for the complete workflow and downloadable test files.

This processor is available on paid plans. Exact keeps the original company name and follows the Exact rules.

Checkbox processor

The Checkbox processor compares checked and unchecked states. An empty checkbox is treated as unchecked during comparison. Checkbox values use Exact and have no processor-specific settings.

URL processor

With Smart, the URL processor ignores protocol differences and a trailing slash. For example, https://www.datablist.com/ matches http://www.datablist.com.

By default, subdomains, paths, and query parameters remain part of the value. Enable the following settings when those URL components should not identify a different website.

Ignore Subdomain

Ignore Subdomain removes the part before the registered domain. https://www.datablist.com and https://app.datablist.com therefore match.

The domain, public suffix, port, path, and query remain unless another setting removes them.

Ignore Path

Ignore Path removes the URL path. It makes www.datablist.com match www.datablist.com/fr and www.datablist.com/fr/about.

Query parameters remain. Enable Ignore Query Params when they should also be removed.

Ignore Query Params

Ignore Query Params removes the query string beginning with ?. For example, https://www.datablist.com?utm_source=crm matches https://www.datablist.com?utm_source=newsletter.

The URL path remains. Enable Ignore Path when different pages on the same website should match.

Measured URL example

The 1,000-row deduplication sample produces 200 two-record groups with Smart URL matching and all three optional controls enabled. The exported groups contain 150 pairs with an identical Website value and 50 pairs with designed protocol, subdomain, path, or query variations. Every pair resolves to the same normalized host in this sample.

Why matched lists the property, algorithm, processor, enabled URL settings, and score used for each group.

Download the sample CSV and the 200 exported Website duplicate groups to reproduce and inspect this result.

Number processor

The Number processor compares numeric values directly. Equivalent integer and decimal values match, such as 10 and 10.0. It has no processor-specific settings.

DateTime processor

The DateTime processor compares the complete stored date and time. Values match when they represent the same timestamp. It has no processor-specific settings.

Phone processor

The Phone processor handles basic formatting only. With Smart, it ignores spaces, punctuation, and symbols, so (415) 555-0123 matches 4155550123.

It does not:

• infer a missing country code;
• use another Country property to interpret a local number;
• convert local and international versions to one format;
• validate whether the number is possible for its country.

For example, 06 12 34 50 08 and +33 6 12 34 50 08 do not match through the Phone processor alone.

Recommended phone-number deduplication workflow

Normalize and validate phone numbers before running the duplicate check:

1. Keep the original Phone and Country properties.
2. Run the Phone Number Cleaner.
3. Map Phone as the input. For one-country data, select that country under Local Phone Countries. For mixed-country data, open Advanced Settings, enable Define country per item, and map the Country property.
4. Write the result to a new Clean Phone Number property. Invalid or unrecognized inputs do not receive a cleaned number, so review the run statuses before matching.
5. Open Clean > Merge duplicates and select Clean Phone Number. Use Exact because the enrichment has already converted valid numbers to one international format.

Do not overwrite the raw Phone property. Keeping it beside the cleaned value makes failed parsing and merge decisions easier to review.

Download the 200-row phone deduplication sample. It contains 50 local/international pairs across five countries and 100 non-matching controls. The sample is designed to produce 50 two-row groups after Phone Number Cleaner processing and Exact matching on the cleaned output. See phone number normalization for the difference between formatting cleanup, validation, and international normalization.

Comparison settings

The settings shown for a property depend on its algorithm, processor, and whether the check uses several properties.

Multiple Values

Enable Multiple Values when one cell contains several semicolon-separated values. It is available on paid plans for text-based properties using Exact or Smart.

For email lists, select the property, choose Smart with the Email processor, then enable Multiple Values.

Datablist:

1. Splits each cell at every semicolon (;).
2. Trims spaces around each part.
3. Applies the selected processor and its settings to every part.
4. Compares every part from one record with every part from the other record.

The property matches when at least one pair matches. It does not require the complete lists to be equal.

Record A: florian@datablist.com;contact@datablist.com
Record B: florian@datablist.com
Result: Match

Processor settings still apply. For example:

• Email Smart makes florian+newsletter@datablist.com;sales@datablist.com match florian@datablist.com.
• URL Smart with Ignore Path makes www.datablist.com/fr;www.example.com match www.datablist.com.
• Exact applies the Case Sensitive setting to each part.

Only semicolons are separators. Commas and line breaks are not split. The setting does not change the stored cell.

On the 200-row test file, this setup returned 60 two-row groups. Validation found all 60 designed pairs and no records from the 80 non-matching controls.

Open Why matched to verify that the explanation includes the selected property, algorithm, processor, and Multiple values flag.

Download the 60 exported groups to inspect the result. Multiple Values controls comparison. During merging, the matching property is handled like any other field. If its cell values conflict, choose whether to combine them, keep the selected record's value, or apply a field rule.

Empty Value Rule

The Empty Value Rule controls the score for one selected property when a check uses several properties. It appears for Smart, Metaphone, and Distance, not Exact.

Value required (default)

With Value required, a property does not match when either record has no value. This includes the case where both values are empty.

Use the default for identifiers such as emails, company names, and websites. It prevents incomplete records from matching because they are missing the same data.

Record A: First Name = John, Last Name = Doe
Record B: First Name = John, Last Name = (empty)
Last Name result: No match

Match on empty value

With Match on empty value:

• Both values empty score 100% for that property.
• Only one value empty scores 75%.
• Records with every compared property empty never form a duplicate group.

For example, compare First Name and Last Name:

Record A: First Name = John, Last Name = Doe
Record B: First Name = John, Last Name = (empty)

First Name scores 100% and Last Name scores 75%. The overall score is (100% + 75%) / 2 = 87.5%, so the records match at an 80% threshold.

Use this rule when another property is a strong identifier and a missing secondary value should not prevent a match.

Case sensitive

Exact is case insensitive by default. Enable Case sensitive when uppercase and lowercase text must be different.

Case sensitive off: ABC-123 matches abc-123
Case sensitive on:  ABC-123 does not match abc-123

Exact also trims spaces before and after text, but keeps spaces inside it.

Distance threshold

Distance accepts a threshold from 50 to 100. The threshold is the minimum similarity needed for the property to match. A value of 100 requires equal strings, while a lower value accepts more variation.

Start around 90 to catch small typos. Lower the value in small steps if valid duplicates are missing. Lower values can produce more false positives.

Exact algorithm

Exact compares values directly after the processor's basic normalization.

For text, it:

• Trims leading and trailing spaces.
• Ignores case unless Case sensitive is enabled.
• Keeps punctuation, word order, and spaces inside the value.

Use Exact for stable IDs, SKUs, external record identifiers, and any field where small differences are meaningful.

Smart algorithm

Smart normalizes text before comparison. It ignores:

• Uppercase and lowercase differences.
• Accents.
• Spaces and punctuation.
• Symbols and emoji.
• Word order.

With the URL or Email processor, Smart also applies the processor-specific rules described above.

For example, John-Doe, doe john, and JOHN DOE match with Text Smart.

Metaphone algorithm

Metaphone is a phonetic algorithm. Datablist uses Double Metaphone on top of Smart normalization to match words that sound alike.

Examples include Filip and Philippe, or Dupon and Dupont.

Use it for names and text where pronunciation is more important than spelling. Review the results because unrelated words can have similar phonetic codes.

For example, select Full Name, choose Metaphone, and keep the Text processor.

This configuration displayed 69 groups on the 1,000-row sample: 5 Ready and 64 Needs review. The export contains 188 rows in groups of two to seven. Against the sample's known entities, 19 groups contain records from one person and 50 groups mix different people.

Open Why matched to see the property, algorithm, processor, and score range used for the group.

Download the 69 exported Metaphone groups to inspect the result. One 80% group contains several abbreviated Rossi names. Their shared phonetic codes create a candidate group, but the names alone do not establish that the rows represent one person.

Metaphone is available on paid plans.

Distance algorithms

Distance calculates a numeric string similarity. Choose:

• Levenshtein distance, which measures the edits needed to transform one string into another.
• Jaro-Winkler distance, which gives more weight to strings with matching prefixes.

Datablist converts Levenshtein distance to a similarity ratio:

1 - LevenshteinDistance / ((StringALength + StringBLength) / 2)

For example, select Full Name, choose Distance with Levenshtein, keep the Text processor, and set the threshold to 90%.

This configuration returned 77 two-record groups on the 1,000-row sample. Ground-truth validation found 21 same-entity groups and 56 cross-entity groups. The false positives mostly contain identical abbreviated names that cannot be resolved from Full Name alone.

Download the 77 exported groups for the complete measured result. Add another identifying property or review the groups manually before processing ambiguous names.

Distance is available on paid plans.

Similarity across several properties

For a multi-property check, Datablist calculates one score per property and uses their arithmetic mean.

Overall similarity =
    (First Name similarity + Last Name similarity + Email similarity) / 3

Distance properties contribute their raw similarity score. Exact, Smart, and Metaphone contribute the score produced by their comparison and empty-value rules.

How duplicate groups are built

Comparing every possible record pair becomes impractical on large collections. Datablist first creates likely candidate pairs, then calculates similarities only for those candidates.

Pairs that reach the active threshold are connected into groups. This connection is transitive: if A matches B and B matches C, A, B, and C can belong to one duplicate group even when A and C are not a direct match.

Processing runs in the browser and is protected by time, pair, group-count, and group-size limits. If a check reaches a limit, the result summary marks the analysis as incomplete. You can process the groups already found, then run Duplicate Finder again on the remaining records.

The comparison stage can take up to three minutes on a large dataset.

Duplicate results

The result page is a workspace for reviewing and processing the groups. It contains:

• Records checked and duplicate groups found.
• One dual-handle confidence control for the result and automatic-processing thresholds on scored checks.
• Download groups for a single-collection check, or Result files for a cross-collection check. After processing changes, paid plans also show the change-log downloads.
• Three processing workflows for single-collection checks.
• A preview split into Ready and Needs review.
• A floating Next step action showing how many groups are ready.

The preview lists the records in each group side by side. For an Exact All-Properties check, identical rows have a 100% score and can be processed as Ready groups.

Download duplicate groups and match results

The available download depends on the matching workflow.

Single-collection duplicate groups

Click Download groups to export the current duplicate records before processing them. The file contains the collection properties and a DuplicateGroupId value that associates records from the same group.

Group IDs are available on Premium plans. Free-plan users can download the duplicate records, but DuplicateGroupId contains a plan notice instead of the group identifier.

The action opens Datablist's exporter, where you can select the output format.

The downloaded file repeats the group ID for records that belong to the same duplicate group. This makes the groups easy to filter or review outside Datablist.

Cross-collection match result files

Click Result files after a cross-collection check. The download workspace shows a row count for each file before generation. Select the source collection for the matched, needs-review, unmatched, and crosswalk files. The selected source does not restrict the Detailed match groups file, which contains members from every collection in the current groups.

The matched and needs-review files preserve the source properties and add:

• Datablist Record ID
• Match Status
• Match Group ID
• Group Similarity (%)
• Best Match Score (%)
• Match Reason
• Matched Collection and Matched Collection ID
• Matched Record ID
• Match Count

The unmatched file contains the source collection's original property columns. It does not add matching metadata because these records do not belong to any included cross-collection group.

The crosswalk uses a neutral schema so it works when collections have different properties. It contains:

• Match Group ID, Match Status, and Group Similarity (%)
• Pair Match Score (%) and Match Reason
• Source collection name and ID, plus Source Record ID
• Matched collection name and ID, plus Matched Record ID

The Detailed match groups file contains one row per group member. It starts with the group ID, status, similarity, collection name and ID, and record ID. It then adds a wide set of collection-specific property columns. Columns belonging to other collections are empty on that row, which allows collections with different schemas to coexist in one CSV.

Group Similarity is the weakest retained connection holding the complete group together. Best Match Score and Pair Match Score are direct source-to-candidate comparisons recalculated with the configured algorithms, processors, and property mappings. The values can differ in groups with three or more records.

All five files reflect the active Include matches from and Process automatically from thresholds plus any manual group-membership changes. A record in a Needs-review group is not also exported as unmatched. CSV generation runs in the background and streams large files without loading every matching record into the results page.

Change the threshold in the results

When a check uses at least one Smart, Metaphone, or Distance algorithm, the result summary shows one confidence scale with two handles:

• Include matches from is the results threshold. Groups below it are hidden.
• Process automatically from is the higher automation threshold. Groups between the two thresholds remain visible in Needs review, but Datablist never includes them in a batch Remove, Merge, or AI action.

Both handles use the same fixed scale, from the retained worker floor to 100%. Moving one handle does not rescale or move the other. The handles cannot cross. The colored track shows three ranges:

• Excluded: below Include matches from.
• Needs review: between the two handles.
• Automatic: at or above Process automatically from.

For example, use 70% for Include matches and 90% for Process automatically to inspect broad candidates while allowing only high-confidence groups into a batch action. Apply either value without running duplicate detection again.

• A single Distance property starts at its configured Distance threshold.
• Other scored checks start at 80%.
• The automatic-processing threshold starts at 90%, or at the results threshold when that value is higher.
• Pure Exact checks stay at 100% and do not show the control.
• The threshold locks after processing or manual changes begin.

Raising the threshold removes weak matches and can split a group. Lowering it can reconnect groups from the retained candidate data.

After applying Include matches from, Datablist rebuilds the groups and refreshes the merge analysis. The Resolve conflicting values panel is then computed from the unresolved conflicts in the new result set. Existing Combine, Keep selected value, and field-survivorship settings are preserved and applied again; the conflict count shown on each card includes only properties that remain unresolved.

Datablist normally retains scored matches down to a 50% floor. On a large or dense dataset, it can raise the floor in five-point steps to remain within browser memory and processing limits. The expanded control displays the effective available range and explains when the floor was raised.

The worker keeps a compact maximum-spanning tree instead of every possible record pair. This makes interactive threshold changes practical for large collections.

Understand a duplicate group's similarity

The link icon and percentage in a group header show its group similarity. This is the weakest retained link required to connect every record in that group, not an average of every possible pair.

Suppose A matches B at 95% and B matches C at 85%. The group similarity is 85%. Raising the threshold above 85% separates C from the stronger A-B group.

Hover over the percentage to see the minimum and maximum connecting-link scores and an explanation of the compact grouping model.

Click Why matched to audit the comparison. Datablist shows every matching property, its algorithm and processor options, and the average and range of the raw property scores across the records displayed in the group. These scores are calculated on demand so large checks do not retain every possible record pair in memory.

If you manually edit a group's membership, the original percentage is hidden because it no longer describes that group. Processing Ready groups does not remove the scores from untouched Needs-review groups.

Preview warnings

The Needs review tab replaces a general fuzzy-matching warning with an action on the groups that need attention:

• A low-confidence group displays Verify match and Confirm duplicate.
• An unresolved merge group displays the number of conflicting properties.
• A Remove group that would lose information displays Some data will be discarded.

Hover over a status for more context. In Remove mode, values that will be lost are bold in the removed rows.

Refresh the preview after changing the context

The preview is a simulation of the current threshold, workflow, group membership, conflict rules, and Record to keep setting. When one of those inputs changes, Datablist can display Preview needs refresh next to Refresh preview.

Refresh before processing so Ready, Needs review, conflict counts, retained records, and proposed changes all describe the current configuration. Refreshing does not clear your conflict-resolution settings. Datablist applies them again to every group in the refreshed result set.

Choose what to do with duplicates

Single-collection results offer three workflows. Remove duplicates is selected by default. Changing the workflow refreshes the preview before processing.

Remove duplicates

Remove duplicates selects one record to keep unchanged and deletes the other records in each Ready group.

It never fills, combines, or rewrites values on the retained record. If a removed record contains extra or conflicting data, that data is lost.

Datablist makes this risk visible in four places:

1. An amber warning above the preview counts affected groups.
2. Each affected group has a Some data will be discarded badge.
3. Values that will be lost are bold in the removed rows.
4. Clicking Remove ready duplicates opens a confirmation dialog before the destructive action.

Both the warning and dialog link to Merge and preserve data, which can copy or combine the values instead.

Groups are Ready whenever the selected Record to keep rule finds a record. They do not need review merely because values conflict, since Remove does not merge those values.

Merge and preserve data

Merge and preserve data selects one record as the destination, fills its empty properties from the other records, then deletes the secondary records.

Groups with equal or complementary data are Ready. A group with different non-empty values needs review until every conflicting property has a rule.

Use Merge ready groups to process all Ready groups and leave unresolved groups untouched.

Custom with AI

Custom with AI generates a group-processing workflow from a prompt. It is available to authenticated users on paid plans.

Use /Property or {{Property}} to reference collection properties. The AI workflow can:

• Add properties, such as a Duplicate checkbox.
• Edit records without deleting them.
• Select a record with a custom rule.
• Normalize a property to the most common group value.
• Sum values or apply another group calculation.
• Delete secondary records.
• Skip groups that meet a condition.

Click Generate and preview changes first. Review the generated explanation and preview, then use Run AI rules. Choose Start Again to replace the prompt and generated workflow.

Ready and Needs review

The preview separates groups into two tabs:

• Ready contains groups for which the selected workflow produced concrete changes with no unresolved requirement.
• Needs review contains unresolved merge conflicts, a Record to keep rule that found no record, groups skipped by AI, or groups below the automatic-processing confidence threshold. Low-confidence cards display Verify match and remain untouched until you review the group.

For a low-confidence group that you recognize as a real duplicate, click Confirm duplicate. Datablist buffers the confirmation and keeps the card in place so you can continue reviewing the tab. Click Refresh preview when you are ready to reclassify the confirmed groups. Refresh removes only the confidence hold: a group moves to Ready when no other requirement remains, but unresolved value conflicts keep it in Needs review. The final batch action also honors the confirmation.

The floating action processes every eligible Ready group, regardless of which tab is visible. Needs-review groups stay unchanged.

For very large results, Datablist keeps a bounded preview in memory. A + after a tab count and a preview-sample note indicate that more groups exist than the cards retained for display. Processing still checks all eligible groups.

Merge settings

Merge and preserve data has two expandable settings: Resolve conflicting values and Record to keep.

Merge non-conflicting and complementary records

Datablist automatically handles values that do not conflict:

• If every record has the same value, it keeps one copy.
• If the selected record has an empty property and another record has a value, it fills the empty property.
• After the destination contains the merged data, it deletes the secondary records.

Example:

Before
james@gmail.com | James | (empty)
james@gmail.com | (empty) | Bond

After
james@gmail.com | James | Bond

Resolve conflicting values

A conflict exists when records contain different non-empty values for the same property. The Resolve conflicting values panel lists all conflicting properties across the current groups. It shows how many properties are resolved and how many remain. A group stays in Needs review until each of its conflicting properties has a rule.

Choose one of the following rules for each conflicting property.

The panel updates when the results threshold changes. Existing rules remain selected and continue to resolve their properties; only newly introduced or still-unresolved properties are added to the remaining count.

Combine conflicting properties

Combine preserves distinct text values in the destination property. Select the property and one of four delimiters:

• Semicolon (;)
• Line break
• Comma (,)
• Space

Before
Phone: +33 1 34 65 23
Phone: 06 13 42 78 23

After with semicolon
Phone: +33 1 34 65 23;06 13 42 78 23

Only text-based properties can be combined. Number, DateTime, and Checkbox properties must use the selected record's value.

Keep the selected record's value

This rule keeps the value from the Record to keep and discards the other non-empty values.

Use it for fields that must contain one value, such as:

• A CRM record ID.
• Last activity date.
• Status checkbox.
• Numeric total that should not be concatenated.

The preview bolds the discarded values so you can verify the result before processing.

Field survivorship rules

Choose values with rules selects a winning value independently for each property while preserving the ID of the Record to keep. This is useful when no single record contains the best value for every field.

This setting is available on paid plans.

Available rules are:

• Most recently updated record.
• Oldest record.
• Highest or lowest value.
• Longest or shortest value.
• Most frequent value in the group.

Empty values are ignored. Ties are resolved deterministically so the preview and final operation select the same value. A property can use only one conflict strategy: combine values, keep the selected record's value, or a field survivorship rule.

Use Use newest values for all remaining to assign Most recently updated to every unresolved compatible property, then adjust individual rules when needed.

Record to keep

The Record to keep becomes the destination in Merge mode. In Remove mode, it stays byte-for-byte unchanged while the other records are deleted.

Most Complete remains the default on all plans. Opening the Record to keep setting and selecting another rule are Premium features. Free-plan checks still use Most Complete automatically for Remove and Merge.

Most Complete

Selects the record with the most populated values. If records tie, Datablist uses their data completeness and text content, then favors the most recently added record for a remaining tie.

Last updated

Selects the record with the newest internal UpdatedAt timestamp.

First created

Selects the record with the oldest internal CreatedAt timestamp.

Highest value on a property

Selects a property, then keeps the record with its highest value. DateTime, Number, Text, and Checkbox properties are supported. If values tie, Datablist favors the most recent record.

Lowest value on a property

Selects a property, then keeps the record with its lowest value. If values tie, Datablist favors the most recent record.

Matching value on a property

Selects a property and required value, then keeps the record that contains that value. String matching is case insensitive.

If no record in a group has the value, Datablist does not process that group. It appears in Needs review.

Review groups manually

Every automatic preview row has a remove-from-group action. Use it to exclude one record from duplicate membership without deleting the record. Datablist invalidates the old group similarity and asks you to refresh the preview. If only one record remains, the duplicate group disappears.

Click Review group on any card to open the complete manual workflow for that group.

Manual review provides these actions:

• Manual Merging assistant: Open the full merging interface, select the destination record, and choose the value to keep for every property.
• Open Item: Edit a record in the item drawer.
• Remove from duplicate group: Exclude a record from this group without deleting it.
• Delete Item: Delete an individual record immediately after confirmation.
• Undo delete: Recreate a record deleted during the manual review.
• Merge Automatically: Return the group to the automatic preview.

Manual edits and deletes take effect immediately. After group membership changes, refresh the preview before processing the remaining automatic groups.

The Merging Assistant also works with multi-collection groups.

See Merging Assistant for its complete value-selection workflow.

Skip a group

Click Skip group when a group should not be processed. Datablist removes it from the current result workflow and leaves its records unchanged.

Skipping is useful for false positives, groups that need external research, or records that you deliberately want to keep separate.

Search and sort results

Use the toolbar above the preview cards to search all records in the remaining duplicate groups, including groups that are not part of the initial preview sample. Enter at least two characters and click Search all groups.

Search checks:

• Duplicate-group numbers and IDs.
• Record IDs.
• Every stored property value on the records.

The search runs in a separate web worker and does not run duplicate detection again. It scans the current groups progressively in bounded batches of up to 1,000 groups or 10,000 records. Matches appear after every completed batch, and the progress panel shows records scanned and groups found.

During a long search:

• Stop and keep results finishes the current batch, then keeps the matches found so far.
• Cancel search stops immediately, discards the search results, and returns to the normal preview.
• After a completed or stopped search, Back to preview restores the normal Ready and Needs-review listing.

Datablist retains at most the first 1,000 matching groups to keep browser memory bounded. If that limit is reached, refine the query to narrow the results. Editing, skipping, or changing a group exits search because the previous search result may no longer describe the current duplicate groups.

Sort the active Ready or Needs-review tab by:

• Highest risk: low-confidence matches first, followed by groups with more unresolved conflicts or potential data loss.
• Lowest confidence: weakest group similarity first.
• Most records affected: largest groups first.

While search results are displayed, the Ready and Needs-review counts describe the matches found so far. Outside search, they describe the complete analyzed preview. Search only changes what is listed: the floating batch action still processes every eligible Ready group, not only the matching cards currently shown.

Process the Ready groups

The floating action bar stays visible while you review the list. It shows the number of Ready groups and the next action:

• Continue: Remove ready duplicates
• Continue: Merge ready groups
• Continue: Run AI rules

Processing applies to all Ready groups, not only the cards on screen. When conflicts remain, Datablist processes the safe groups first and leaves the Needs-review groups for later.

Downloads, History, and undo

Download the change log

Change-log downloads are available on Premium plans.

After a single-collection processing action changes records, the result header shows Change log (N) and provides one file for the current collection.

The file contains:

• DatablistId
• The change type
• Every previous property value
• Every destination property value

Use it as an audit trail or to synchronize deletions and updates with a CRM or another application.

After automatic cross-collection cleaning, the completion screen provides one download button for each collection from which records were removed, for example Download Event Registrations changes and Download CRM Export changes. Datablist does not combine collections with different schemas into one change file.

Each collection-specific file follows the same structure as a single-collection change log:

• DatablistId
• Change, with deleted for records removed by cross-collection cleaning
• One Previous … column for every editable property in that collection
• One Destination … column for every editable property; these cells are empty for deleted records

The Datablist ID is included for audit purposes, while the previous property values—including an external ID when the collection has one—let you identify and reconcile the deleted record in its source system. Datablist snapshots these values before deleting the records.

Undo changes with History

Automatic removal and merge actions are recorded in Datablist History. Return to the collection table and open History to undo the operation.

Manual review can also undo an individual delete while the group is open. For a completed operation, use the collection History.

FAQ

Can Datablist miss duplicates?

Yes. Large datasets can contain billions of theoretical record pairs, so Datablist uses blocking to discard pairs that are unlikely to match before calculating full similarities.

For example, 200,000 records have 200,000 × 199,999 / 2 = 19,999,900,000 possible pairs.

The candidate strategy is effective for most business data, but an unusually dense dataset can reach a processing safeguard. The result summary indicates when the full dataset was not processed.

Process the groups already found, then run Duplicate Finder again to find additional groups among the remaining records.

Why can two records in one group have a low direct similarity?

Groups are connected transitively. A can match B, and B can match C, without A matching C directly. The group-similarity badge shows the weakest retained link needed to connect the group.

Raise the results threshold to split weakly connected groups.

Why is the similarity threshold unavailable?

The control is hidden for Exact-only checks because their retained matches are 100%. It becomes locked after edits or processing begin so a threshold change cannot reintroduce records already changed or deleted.

When should I remove instead of merge?

Use Remove when one record is already authoritative and all other values can be discarded. Use Merge when secondary records contain information you want to copy or combine.

The Remove preview warns when data would be lost and offers a direct switch to Merge and preserve data.

To learn more

• Dedupe a list online
• Dedupe email addresses and aliases
• Merge duplicate rows in an Excel file
• Find and merge CSV duplicates
• Merge duplicate leads without losing data
• Online data matching tool