Changelog

Import files up to 500k items

Another milestone in Datablist handling large collections. After moving the limit from 10k to 50k in April, I've been able to increase it ten-fold in May to 500k. From 10k to 500k in 4 months is a big step forward. When importing a data file (like a CSV file), the data is parsed, and stored in a local database. Datablist uses a database to filter, sort the data, and save edits. In my test, importing a 500k file takes less than 2 minutes. My goal is to import and edit files up to 1 million items.

Join big CSV files

In the process of increasing the number of items in a collection, I've rewrote some features. The algorithm to join several CSV files on a unique key has been improved to handle bigger collections and edge cases.

Joining two CSV files with hundred of thousand of items is now possible.

Collection Filters

Filtering data is finally available on Datablist. Select one or several filter conditions to show a subset of your collection items. Filtering conditions depend on your data types. Number properties can be filtered on numerical operators, DateTime values are filtered related to timestamps. To export your filtered view, select all items and click on "Export selected items".

Auto create properties during first import

Datablist is both used by regular users and users coming from Google who wants to perform a single task (like deduplicating a CSV file). The import process must be as straightforward as possible. On an empty collection, when importing a file, properties are auto-created using column names and detected data types. If the collection already has properties, the mapping process is shown.

Clone collection

Instead of exporting a collection to a CSV file to re-import it in another collection, you can use the shortcut "Clone collection". All the properties and the collection items are duplicated in a new collection.

Select CSV Export separator

When exporting your data to a CSV file, a new option to select the separator character is available. Choices are "comma" (default) and "semicolon".

Import files up to 50k items

In April, I released a first step toward managing a higher number of items with Datablist. From 10k items per collection, the limit has been increased to 50k items.

A lot has changed just to multiply the limit by five. Instead of loading all the data in memory (like all spreadsheet tools), Datablist now loads it on demand from a local database. The user interface is still responsive and easy to use. And this stack will scale well to millions of items (at least it does in my head 🤞).

Performance Improvements

Faster file import

Importing files (CSV/Excel) into a collection for registered users has been improved. Previously, items were saved in the cloud during the file import process. For big files with thousands of items, this leads to frustrating seconds/minutes waiting for cloud sync before accessing the data.

Cloud sync is now asynchronous. Access and manage your data instantly after importing a file while the data is being synced to the cloud in the background.

Faster duplicates finder

Duplicates finder algorithm has been improved. It is now faster with big collections.

Also:

• Duplicates comparison is now case insensitive.
• DateTime values are now compared
• Duplicates on empty values are skipped

Action Runner for big collection

Running actions (verify email addresses, find a LinkedIn profile for an email address, etc.) on thousands of items was challenging. The action runner now split the collection into small parts (chunks) and sends them in sequence. A stop button is available to stop the action before processing all the items.

Better errors handling

A lot can happen in a web application. Internet can be lost, servers might have intermittent issues, etc. Shit happens 🤷‍♂️

I continue to improve how Datablist web client deals with any errors. Retries, showing feedback, etc.

New features

Datablist Help Center

Learn to use Datablist and discover how to get the most of it with our new Help Center: https://www.datablist.com/docs

New action: LinkedIn Profile Finder

This action takes a name and keyword properties and returns a LinkedIn Profile URL when found. Read our new guide: How to scrape Facebook group members and find their LinkedIn Profile.

Notifications

Long running tasks are being moved to background jobs to improve UI reactivity. For example, when a collection is deleted, the task takes several seconds to complete but it does't prevent the user to navigate on the Datablist App.

Notifications have been implemented on: Collection Delete, Item edit from the drawer, undo and redo operations.

Network lost, API errors while editing items now return visible error notifications.

Improvements

Improve selected items export

Select export format between CSV and Excel files.

Improve history (Undo/Redo)

History actions (Undo and Redo) are new shown directly in the collection header.

Also, it's now possible to undo collection name and icon changes. And, after creating a new item, calling "undo" will delete it.

Keep CSV rows order on import

During import, file rows are split in several chunks and saved using parallel calls. Before, this could lead to a reordering of items order depending of what call was saved first. File import has been improved to keep file rows order.

Create a collection fast with keyboard shortcut

Press "n" to create a new Datablist collection. See Keyboard Shortcuts documentation https://www.datablist.com/docs/keyboard-shortcuts

New features

Automatic duplicates merging

Last month, we released Datablist Duplicates Finder to find and list collection items with duplicate values. Yet, processing those duplicates was a manual task.

Not anymore! In November, Duplicates Finder got an upgrade to automatically merge non-conflicting items.

What are non-conflicting items?

• Items with similar values for all their properties
• Items with complementary values

Merging Assistant

For items with conflicting values found with Datablist Duplicates Finder or to merge items directly from a collection, we introduce a Merging Assistant.

It shows two or more items, automatically select the item with the most information as Primary Item and allow merging data from other items into this Primary Item.

Import/Export Excel files

In addition to CSV, Microsoft Excel files have been added as import and export format.

To import a Microsoft Excel file, the data must be on the first worksheet and display a Table like structure.

Improvements

Properties data types

In November, built-in data types received improvements. Current built-in types are:

• Text
• Long Text
• Checkbox
• Email
• Url
• Datetime
• Date
• Number

During data import (CSV or Excel), Datablist scans the first 100 lines to automatically detect data types.

Collection search and sort actions are compatible with all those built-in data types.

In the detail drawer, data validation is performed to forbid non valid data inputs.

Empty cell en backspace/delete key press

Last month, bulk delete was added to empty several cells with the "Delete"/"Backspace" key. Now it also works on a single cell.

Fix freeze on big CSV file import

On very large files (more than 50mo), Datablist CSV Importer could freeze as browser memory became full.

Now, CSV files are read chunk by chunk to handle large files.

Currently, Datablist only import the first 10k lines but this is an important step toward higher volume of data.

New features

Use Datablist without signing up

Datablist is a great tool for simple data manipulations like removing duplicates, filtering CSV, consolidating lists, etc.

It is now possible to upload CSV files and play with them without being registered.

Go to https://app.datablist.com/ and create a collection to get started.

The data is kept in your browser storage. Sign in to sync it to the cloud and share it with your team.

Duplicates Finder

Until now, you could merge duplicate items by using the "unique values" setting on a property. It worked but it was not possible to preview duplicates before merging them and it worked only during imports.

With our new Duplicates Finder feature, finding duplicates is easier than ever. Select the property(ies) to compare and run the analysis!

"Long Text" property

Writing notes (or any long text) on Datablist is now possible with the new Long Text property type. It opens a write zone on the table and shows as a Textarea on the drawer.

Undo/Redo Manager

Try, iterate, cancel. Undo/Redo has been implemented on:

• Item edit
• Column size change
• Cells bulk delete (see improvements)

Use Cmd+Z (undo) and Cmd+Shift+Z (redo) on your collections.

Improvements

Cells bulk delete

Empty all the selected cells with the "Delete"/"Backspace" key as you would do on a spreadsheet.

Quick collection creation

Creating a collection is now a one step process. Click the "+" button and you will be redirected to an empty collection page.

Export selected items

Sometimes you want to export selected items instead of the whole listing. Now you can 🤟