Every business wants to grow. That's why marketing exists. But, inside "marketing", growth hacking is a new field with a simple goal: getting new customers. While traditional marketing is about branding, "growth hacking" focuses on building a system to win new customers, fast! 🚀
Growth hacking is at the crossroad between marketing and tech. Startups set up multi competencies growth hacking teams with developers, marketers, and product managers. And growth hacking, once limited to early-stage startups, is becoming mainstream.
During the last months, growth hacking teams (or simply "growth teams") have contacted me to discuss how their company does prospecting and to see how to use Datablist to store prospects. In their organization, the growth team is responsible for prospecting. The sales team is used later on in the workflow once a prospect shows a sign of interest.
Growth teams build large prospect lists using scraping and companies' databases. Then, they score prospects to engage only with relevant ones. Interested prospects are then passed to the sales CRM and the conversation continues with a salesperson.
Their results are stunning, with multi digits growth rate! I wanted to share with you what I've learned from those interviews and explain how modern startups use data and engineering to fuel their growth.
Below is a summary of how growth teams set up prospecting.
<h2 id="user-content-step-1-prospect-list-building"><a href="#step-1-prospect-list-building" aria-hidden="true" tabindex="-1"></a>Step 1: Prospect List building</h2>
It always starts with building prospect lists. Prospects are companies (or people) who may be interested in the product you are offering.
B2B company databases (e.g. Crunchbase or AngelList) let you export large datasets with basic business data: the company industry, funding rounds, employee headcount, etc.
Another way to build a list of prospects is "scraping". LinkedIn offers premium members to search for profiles using job titles, or industries. Third-party tools extract the LinkedIn profiles into spreadsheets or CSV.
Scraping works also with Twitter and Facebook Groups. I've talked with people scanning for new Facebook B2B group members and engaging with them on LinkedIn. See our <a href="/how-to/scrape-facebook-group-members-linkedin">guide on how to scrape Facebook group members</a>.
Growth teams are really creative to find new prospects. I've seen teams tracking job offers or employees changes. For example, a Fintech startup has coded an "algorithm" to detect businesses that have just recruited a CFO (Chief Financial Officer).
Another software company selling an e-commerce plugin parses website HTML code. HTML tags indicate which e-commerce framework is used. All websites built with Magento or Shopify (e-commerce frameworks) are added to their prospects list.
Other growth teams screen new startups on ProductHunt or IndieHackers.
<h3 id="user-content-storing-b2b-prospect-lists"><a href="#storing-b2b-prospect-lists" aria-hidden="true" tabindex="-1"></a>Storing B2B prospect lists</h3>
I asked for the tool they use to store their prospects:
<ul>
<li>Small teams use spreadsheet tools like Google Sheets and Airtable. Airtable comes with basic data types, some collaboration features and an API which makes it great to store prospects.</li>
<li>In bigger startups, teams have coded their workflow with a custom database. <a href="https://www.postgresql.org/" target="_blank" rel="nofollow">PostgreSQL</a>, an open-source database, is often cited; along with <a href="https://cloud.google.com/firestore" target="_blank" rel="nofollow">Google Firestore</a>, a cloud database. With the database, some have added a data-warehouse (Google BigQuery, Snowflake) to log all events generated from their prospects. When a data-warehouse is added, the system becomes a "Customer Data Platform". They log events from prospects, users with a free plan, paying customers, churned customers, etc. A Customer Data Platform offers a full view of the funnel acquisition from the prospect to the paying customer.</li>
</ul>
Both spreadsheets and custom developments bring challenges:
<ul>
<li>Airtable reaches its limits with 10k prospects and Google Sheets doesn't have data structures.</li>
<li>With a custom database, developers have to code an interface, manage user permissions, deal with integration to their sales CRM and code data management feature such as deduplication, and leads merging. Instead of focusing on algorithms to source prospects, they waste time reinventing the wheel.</li>
</ul>
Prospecting relies on a large volume of data and <a href="/">Datablist</a> is perfect to store prospects. With an easy-to-use interface, <a href="/csv-editor">CSV imports</a>, <a href="/how-to/remove-csv-duplicates">deduplication</a>, <a href="/features/duplicates-remover">items merging</a>, users management, and a <a href="https://developers.datablist.com/" target="_blank" rel="nofollow">developer-friendly API</a>.
<h2 id="user-content-step-2-prospect-enrichment"><a href="#step-2-prospect-enrichment" aria-hidden="true" tabindex="-1"></a>Step 2: Prospect Enrichment</h2>
The next step is prospects enrichment. Enrichment's purpose is to have a complete identity for every prospect you have, whether they come from a business database, from scraping, or for manually added prospects.
With identifiers such as a website URL or an email address, it's easy to fill in the missing information.
The growth teams I've interviewed perform manual and automatic enrichment. Teams with prospect lists on spreadsheets use manual enrichment. When a new prospect is added, they use LinkedIn, Crunchbase, etc. sources to fill important attributes.
Bigger teams, with larger prospect lists, have automated this process by plug-in external APIs into their workflows. New prospects trigger enrichment functions. A wide variety of SaaS products exist to search for any kind of data whether you want to enrich a contact or a company.
For contacts, enrichment services include:
<ul>
<li>Email Finder</li>
<li>Email verifier</li>
<li>Job Title</li>
</ul>
For companies, enrichments include:
<ul>
<li>Company Tech/Sales/Finance Stack</li>
<li>Website traffic estimation</li>
<li>Funding rounds</li>
<li>Employees Headcount</li>
<li>Social Network profiles</li>
<li>Country</li>
<li>Industry</li>
</ul>
<h2 id="user-content-step-3--prospect-scoring"><a href="#step-3--prospect-scoring" aria-hidden="true" tabindex="-1"></a>Step 3 : Prospect Scoring</h2>
Then, every prospect is scored based on its properties and the prospects list is ranked. Defining a scoring formula is tricky but you need to define your "Ideal Customer Profile" and the score must reflect how close to this ideal customer your prospect is.
A prospect score may change. For example, if a company raises money, it means the company is accelerating and your product might fit with this change.
Scoring in spreadsheets or with custom databases is similar. But when a data warehouse is connected to the system, event logs from the data-warehouse are added to the score formula. Events from your website analytic software can feed the data-warehouse.
Scoring is meant to narrow your prospects list and avoid spamming everyone on your list.
Have you ever received an invitation from a stranger on LinkedIn to connect? Most of the time, they come from an automated workflow. And have you noticed you receive fewer invitations since the beginning of 2021? That's because LinkedIn restricted the number of invitations a member can send from 100 per day to 100 per week, i.e. 300 invitations per month (almost 90% less). When you have 300 prospects per month to contact, you want to cherry-pick them.
Same with cold emailing. Spam filters detect accounts sending mass mailing and analyze how people react to the emails. Do they open them, do they click, reply, etc. Sending hundreds of emails per day will get your domain blacklisted in no time.
That is why scoring and segmenting are so important. And because nobody likes spam, you aim to contact fewer prospects but increase your conversion rate with relevant ones.
<h2 id="user-content-step-4-engage-with-prospects"><a href="#step-4-engage-with-prospects" aria-hidden="true" tabindex="-1"></a>Step 4: Engage with prospects</h2>
Next, select prospects with the highest scores, and engage with them. "Engaging" means you want them to at least read what you have to offer. You can engage by contacting them or with targeted advertising.
Here is a list of solutions growth teams use to engage with prospects:
<ul>
<li>Direct contact
<ul>
<li>Cold emailing: Lemlist, Reply.io, etc.</li>
<li>LinkedIn Automation to connect and send LinkedIn messages</li>
<li>SMS, Whatsapp, Facebook, etc.</li>
</ul>
</li>
<li>Events</li>
<li>Targeted Advertising</li>
</ul>
Engaging with prospects generates events that can fuel your growth system. Email clicks, user visits, and other events can be logged to a data warehouse to update the score of the prospect.
Once a prospect replies to a message (email, LinkedIn), it ignites a sales approach. Steps 1 to 4 are dealing with thousands of prospects. In the fifth and last step, the prospect becomes a lead and is now the responsibility of the sales team.
<h2 id="user-content-step-5-import-leads-to-sales-crm"><a href="#step-5-import-leads-to-sales-crm" aria-hidden="true" tabindex="-1"></a>Step 5: Import leads to Sales CRM</h2>
In the companies I've interviewed, a reply (or any sign of interest) converts a prospect into a lead. Leads are sent to sales CRM (for example Salesforce) and are assigned to salespersons.
Prospects and Leads belong to different teams and tools. Prospection is for the growth team and prospect lists can have hundreds of thousands of contacts.
And leads are managed by the sales team in a sales CRM. CRMs should be only for warn leads that know your product and show a minimum of interest. You must avoid polluting the CRM with thousands of useless prospects.
<h2 id="user-content-conclusion"><a href="#conclusion" aria-hidden="true" tabindex="-1"></a>Conclusion</h2>
Growth teams are reinventing prospecting. From a manual task performed by salespersons, it becomes an "automated system" built like <a href="/blog/internal-software-is-key-to-your-growth">internal software</a>. Datablist is a perfect tool to store your prospect, <a href="/how-to/remove-csv-duplicates">merge duplicate entries</a>, and run enrichment services. <a href="https://docs.google.com/forms/d/e/1FAIpQLSeAZfqCjOg-ok7yYAwz4hmKxhPOeLMIwZrLozsro5HTQhrsxw/viewform?usp=sf_link" target="_blank" rel="nofollow">Feel free to contact us!</a>


Every business wants to grow. That's why marketing exists. But, inside "marketing", growth hacking is a new field with a simple goal: getting new customers. **While traditional marketing is about branding, "growth hacking" focuses on building a system to win new customers, fast! 🚀**

Growth hacking is at the crossroad between marketing and tech. Startups set up multi competencies growth hacking teams with developers, marketers, and product managers. **And growth hacking, once limited to early-stage startups, is becoming mainstream.**

During the last months, growth hacking teams (or simply "growth teams") have contacted me to discuss how their company does prospecting and to see how to use Datablist to store prospects. **In their organization, the growth team is responsible for prospecting. The sales team is used later on in the workflow once a prospect shows a sign of interest.**

Growth teams build large prospect lists using scraping and companies' databases. Then, they score prospects to engage only with relevant ones. Interested prospects are then passed to the sales CRM and the conversation continues with a salesperson.

Their results are stunning, with multi digits growth rate! I wanted to share with you what I've learned from those interviews and explain how modern startups use data and engineering to fuel their growth.

Below is a summary of how growth teams set up prospecting.


## Step 1: Prospect List building

It always starts with building prospect lists. **Prospects are companies (or people) who may be interested in the product you are offering.**

B2B company databases (e.g. Crunchbase or AngelList) let you export large datasets with basic business data: the company industry, funding rounds, employee headcount, etc.

**Another way to build a list of prospects is "scraping".** LinkedIn offers premium members to search for profiles using job titles, or industries. Third-party tools extract the LinkedIn profiles into spreadsheets or CSV.

**Scraping works also with Twitter and Facebook Groups.** I've talked with people scanning for new Facebook B2B group members and engaging with them on LinkedIn. See our [guide on how to scrape Facebook group members](/how-to/scrape-facebook-group-members-linkedin).

Growth teams are really creative to find new prospects. **I've seen teams tracking job offers or employees changes.** For example, a Fintech startup has coded an "algorithm" to detect businesses that have just recruited a CFO (Chief Financial Officer).

Another software company selling an e-commerce plugin parses website HTML code. HTML tags indicate which e-commerce framework is used. All websites built with Magento or Shopify (e-commerce frameworks) are added to their prospects list.

Other growth teams screen new startups on ProductHunt or IndieHackers.

### Storing B2B prospect lists

I asked for the tool they use to store their prospects:

- **Small teams use spreadsheet tools like Google Sheets and Airtable.** Airtable comes with basic data types, some collaboration features and an API which makes it great to store prospects.
- **In bigger startups, teams have coded their workflow with a custom database.** [PostgreSQL](https://www.postgresql.org/), an open-source database, is often cited; along with [Google Firestore](https://cloud.google.com/firestore), a cloud database. With the database, **some have added a data-warehouse (Google BigQuery, Snowflake) to log all events generated from their prospects.**  When a data-warehouse is added, the system becomes a "Customer Data Platform". They log events from prospects, users with a free plan, paying customers, churned customers, etc. A Customer Data Platform offers a full view of the funnel acquisition from the prospect to the paying customer.

Both spreadsheets and custom developments bring challenges:

- **Airtable reaches its limits with 10k prospects and Google Sheets doesn't have data structures.**
- With a custom database, developers have to code an interface, manage user permissions, deal with integration to their sales CRM and code data management feature such as deduplication, and leads merging. **Instead of focusing on algorithms to source  prospects, they waste time reinventing the wheel.**

Prospecting relies on a large volume of data and [Datablist](/) is perfect to store prospects. With an easy-to-use interface, [CSV imports](/csv-editor), [deduplication](/how-to/remove-csv-duplicates), [items merging](/features/duplicates-remover), users management, and a [developer-friendly API](https://developers.datablist.com/).

## Step 2: Prospect Enrichment

The next step is prospects enrichment. **Enrichment's purpose is to have a complete identity for every prospect you have, whether they come from a business database, from scraping, or for manually added prospects.**

With identifiers such as a website URL or an email address, it's easy to fill in the missing information.

The growth teams I've interviewed perform manual and automatic enrichment. Teams with prospect lists on spreadsheets use manual enrichment. When a new prospect is added, they use LinkedIn, Crunchbase, etc. sources to fill important attributes.

**Bigger teams, with larger prospect lists, have automated this process by plug-in external APIs into their workflows.** New prospects trigger enrichment functions. A wide variety of SaaS products exist to search for any kind of data whether you want to enrich a contact or a company.

For contacts, enrichment services include:

- Email Finder
- Email verifier
- Job Title


For companies, enrichments include:

- Company Tech/Sales/Finance Stack
- Website traffic estimation
- Funding rounds
- Employees Headcount
- Social Network profiles
- Country
- Industry


## Step 3 : Prospect Scoring

**Then, every prospect is scored based on its properties and the prospects list is ranked.** Defining a scoring formula is tricky but you need to define your "Ideal Customer Profile" and the score must reflect how close to this ideal customer your prospect is.

A prospect score may change. For example, if a company raises money, it means the company is accelerating and your product might fit with this change.

Scoring in spreadsheets or with custom databases is similar. But when a data warehouse is connected to the system, event logs from the data-warehouse are added to the score formula. Events from your website analytic software can feed the data-warehouse.

**Scoring is meant to narrow your prospects list and avoid spamming everyone on your list.**

Have you ever received an invitation from a stranger on LinkedIn to connect? Most of the time, they come from an automated workflow. And have you noticed you receive fewer invitations since the beginning of 2021? **That's because LinkedIn restricted the number of invitations a member can send from 100 per day to 100 per week, i.e. 300 invitations per month (almost 90% less).** When you have 300 prospects per month to contact, you want to cherry-pick them.

Same with cold emailing. Spam filters detect accounts sending mass mailing and analyze how people react to the emails. Do they open them, do they click, reply, etc. Sending hundreds of emails per day will get your domain blacklisted in no time.

**That is why scoring and segmenting are so important. And because nobody likes spam, you aim to contact fewer prospects but increase your conversion rate with relevant ones.**

## Step 4: Engage with prospects

Next, select prospects with the highest scores, and engage with them. **"Engaging" means you want them to at least read what you have to offer.** You can engage by contacting them or with targeted advertising.

Here is a list of solutions growth teams use to engage with prospects:

- Direct contact
  - Cold emailing: Lemlist, Reply.io, etc.
  - LinkedIn Automation to connect and send LinkedIn messages
  - SMS, Whatsapp, Facebook, etc.
- Events
- Targeted Advertising

**Engaging with prospects generates events that can fuel your growth system. Email clicks, user visits, and other events can be logged to a data warehouse to update the score of the prospect.**

Once a prospect replies to a message (email, LinkedIn), it ignites a sales approach. Steps 1 to 4 are dealing with thousands of prospects. **In the fifth and last step, the prospect becomes a lead and is now the responsibility of the sales team.**

## Step 5: Import leads to Sales CRM

In the companies I've interviewed, **a reply (or any sign of interest) converts a prospect into a lead. Leads are sent to sales CRM (for example Salesforce) and are assigned to salespersons.**

Prospects and Leads belong to different teams and tools. Prospection is for the growth team and prospect lists can have hundreds of thousands of contacts.

And leads are managed by the sales team in a sales CRM. **CRMs should be only for warn leads that know your product and show a minimum of interest.** You must avoid polluting the CRM with thousands of useless prospects.

## Conclusion

Growth teams are reinventing prospecting. From a manual task performed by salespersons, it becomes an "automated system" built like [internal software](/blog/internal-software-is-key-to-your-growth). Datablist is a perfect tool to store your prospect, [merge duplicate entries](/how-to/remove-csv-duplicates), and run enrichment services. [Feel free to contact us!](https://docs.google.com/forms/d/e/1FAIpQLSeAZfqCjOg-ok7yYAwz4hmKxhPOeLMIwZrLozsro5HTQhrsxw/viewform?usp=sf_link)


Prospect List building: a growth hacking guide

Happy new year! 🎉,
First, thanks to all of you for following my entrepreneurship journey with <a href="/">Datablist</a>.
2021 was a building year with the launch during the summer of a first MVP and a focus on some basic data use cases:
<ul>
<li><a href="/how-to/excel-merge-duplicates">Find and merge duplicates in Excel</a> and <a href="/how-to/remove-csv-duplicates">CSV files</a></li>
<li><a href="/how-to/join-csv-files-unique-identifier-or-column">Combine several CSV files into a consolidated list</a></li>
<li><a href="/how-to/email-list-cleaning">Clean mailing lists to remove invalid and disposable email addresses</a></li>
<li>And a Chrome Extension to import LinkedIn profiles to Airtable or Google Sheets (now removed following LinkedIn complaint)</li>
</ul>
Those articles have generated a few thousand visits each month from Google (not bad for just 4 articles!). Which leads to interesting feedbacks and talks with users.
=> A special thanks to all beta users for using Datablist and sharing your use cases 🙏
Most of the time, users are dealing with lists of people (subscribers, customers, employees), or companies (prospects, leads) and are stuck in a no man's land between CRM software and spreadsheet tools.
<ul>
<li>For growth hackers, how to build prospects lists with cleaning, enrichment, and scoring without polluting their CRM.</li>
<li>For mailing lists, how to clean and enrich subscribers to find business opportunities.</li>
<li>For Venture Capital funds, how to manage a list of startups and score them to find the next unicorns.</li>
<li>For head hunters, how to source candidates, then segment and score them to find perfect matches.</li>
</ul>
On one side, CRMs focus on sales professionals and are not suited for any of those use cases. And on the other side, many people still choose to use spreadsheets (or Airtable/Notion) or develop custom tools.
Airtable is a great alternative to spreadsheet tools and is getting momentum. With typed properties and relationships, it brings improvements over spreadsheet tools. Yet it comes with its own limitations. Airtable is not built for "large" collections with tens of thousands of items, doesn't have enrichments APIs, and lacks basic data management features (update items from CSV/APIs, consolidate listings, etc.).
The sweet spot I want to target with Datablist is use cases unaddressed by CRMs, where spreadsheet tools are not enough, and when Airtable hits its limitations. Whether Datablist stays a generic tool or focuses on specific use cases remains to be defined.
For 2022, my goal is to keep building Datablist to target one or more of those use cases. 🙏 If you think Datablist can bring value to you in any other way, <a href="https://docs.google.com/forms/d/e/1FAIpQLSeAZfqCjOg-ok7yYAwz4hmKxhPOeLMIwZrLozsro5HTQhrsxw/viewform?usp=sf_link" target="_blank" rel="nofollow">please contact me</a> 🙂.

Happy new year! 🎉,

**First, thanks to all of you for following my entrepreneurship journey with [Datablist](/).**

2021 was a building year with the launch during the summer of a first MVP and a focus on some basic data use cases:

- [Find and merge duplicates in Excel](/how-to/excel-merge-duplicates) and [CSV files](/how-to/remove-csv-duplicates)
- [Combine several CSV files into a consolidated list](/how-to/join-csv-files-unique-identifier-or-column)
- [Clean mailing lists to remove invalid and disposable email addresses](/how-to/email-list-cleaning)
- And a Chrome Extension to import LinkedIn profiles to Airtable or Google Sheets (now removed following LinkedIn complaint)

Those articles have generated a few thousand visits each month from Google (not bad for just 4 articles!). Which leads to interesting feedbacks and talks with users.

**=> A special thanks to all beta users for using Datablist and sharing your use cases 🙏**

Most of the time, users are dealing with lists of people (subscribers, customers, employees), or companies (prospects, leads) and are **stuck in a no man's land between CRM software and spreadsheet tools.**

- For growth hackers, how to build prospects lists with cleaning, enrichment, and scoring without polluting their CRM.
- For mailing lists, how to clean and enrich subscribers to find business opportunities.
- For Venture Capital funds, how to manage a list of startups and score them to find the next unicorns.
- For head hunters, how to source candidates, then segment and score them to find perfect matches.

**On one side, CRMs focus on sales professionals and are not suited for any of those use cases. And on the other side, many people still choose to use spreadsheets (or Airtable/Notion) or develop custom tools.**

Airtable is a great alternative to spreadsheet tools and is getting momentum. With typed properties and relationships, it brings improvements over spreadsheet tools. Yet it comes with its own limitations. Airtable is not built for "large" collections with tens of thousands of items, doesn't have enrichments APIs, and lacks basic data management features (update items from CSV/APIs, consolidate listings, etc.).

**The sweet spot I want to target with Datablist is use cases unaddressed by CRMs, where spreadsheet tools are not enough, and when Airtable hits its limitations.** Whether Datablist stays a generic tool or focuses on specific use cases remains to be defined.

**For 2022, my goal is to keep building Datablist to target one or more of those use cases. 🙏 If you think Datablist can bring value to you in any other way, [please contact me](https://docs.google.com/forms/d/e/1FAIpQLSeAZfqCjOg-ok7yYAwz4hmKxhPOeLMIwZrLozsro5HTQhrsxw/viewform?usp=sf_link) 🙂.**


Datablist 2021 in review


# Introduction
## What is internal software?

Before digging further, let's agree on a definition. When searching "Internal Software" on Google, results are trusted by "internal tool builders" like [Retool](https://retool.com/) or [Internal](https://internal.io/). For them, internal software is built in-house by developers or with low/no-code solutions (especially with their solutions).

My definition is broader:

 > **Internal Software is software that is not customer-facing.**

Whether they are commercial solutions, internally built, or created using spreadsheets. **They are internal digital tools to streamline organization processes.**

## Why is internal software important?

Every business deals with the same concepts: employees, customers, partners, leads, etc. but each business is so unique that operations processes have remained performed by humans. Lately, as we move to digital organizations, we witness a digitalization of the organization processes. Slowly, business operations move to software.

**Moving business operations to software comes with a promise: Increase the organization productivity.**

In the past 30 years, we have seen a phenomenal increase in individual productivity. Personal computers and the Internet have extended human capabilities and drove productivity gains.

**The next step is to increase productivity at the organization level with digital organizations.**

> **In a digital organization, processes are crafted by humans but are embedded in software systems.**

The piece of software developed internally to improve the organization's productivity are internal tools. Investing in internal tools is not about getting rid of employees but a part of the employee workforce must focus on designing and monitoring the software processes.

Businesses must look at their processes with a product mindset so employees understand how the organization works and participate to improve those processes. **It's the beginning of the "organization as a product".**

Companies invest in internal tools not just to replace human repetitive tasks, but also because internal tools allow them to grow faster while remaining Agile. Using internal software enable organizations to be data-driven and analytical. Day-to-day operations produce data that is analyzed, generate insights, and lead to better decisions.

The best examples of digital organizations are the tech giants like Amazon, Netflix or Google. They have invested massively in internal software. Google has built digital workflows for its hiring processes, employee career management. Amazon has digitized its warehouse management and logistic centers like no other.

**While complexity becomes the downfall of traditional organizations as they scale, digital organizations continue to improve despite the size and complexity of their operations.**

## Internal Software examples

There are many examples of internal software, from CRM to invoicing apps. In the examples below, I focus on systems that are organization-wide and on which we will see the most productivity gains as organizations become data-centric and leverage automation.

### Customer Data Platform

Customer Data Platforms are trending! They are used to consolidate all customer data into a single platform to generate insights and create a single view of the customers.

![Customer Data Platform on Google Trends](/blog_articles_images/2021/06/google_trends_cdp_large.png#1600x1252)

Customer Data Platforms are built with a composition of applications and they benefit every team in the organization.

**Customer Data Platforms are built**

In a nutshell, Customer Data Platforms are built on a data warehouse (for example [BigQuery](https://cloud.google.com/bigquery) or [Snowflake](https://www.snowflake.com/)). Tools exist to connect data warehouse to any data sources (web applications, marketing software, etc.). Once you have the data, you can use business intelligence solutions to visualize it and you can build metrics for every customer that can be synchronized back to your operational applications. If you want to learn more about customer data platforms, you can read [this article](https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/).

A Customer Data Platform is a collection of software connected together. There is no one-size-fits-all solution and you need technical skills to set up a Customer Data Platform. **It's a great example of how digital organizations use both commercial software and custom-built internal tools to generate value.**

**Customer Data Platforms are company-wide**

A customer data platform is anti-silos. It's an organizational foundation on which every team finds value:

- Sales get insights on their ideal customer profile and can detect upselling opportunities from the data
- The marketing team use it to build a personalized customer journey
- Product people find insights on how their product is used
- The support team can use interactions history and data to help customers

### Leads Generation and Management

Another interesting trend is how companies start using a data-centric approach to manage their leads. Whether it's venture funds to detect new companies, or B2B businesses that build listings of potential customers, they are looking to automate leads generation as much as possible. CRM (Customer relationship management) is good to manage the process once a lead has responded to an inquiry, but they are not suited for prospects listing and leads discovery.

**Some companies have built internal software to boost their leads generation. This is part of the new growth hacking strategies.**

While being different in each organization, we can summarize the steps as follow:

- **Step 1: Leads discovering.** Internet is a big messy database. When a startup is launched, the information is posted online. Almost any data about a company can be found online.

  Building a prospects listing is key to win customers. Now, companies use data scrapping, APIs, automation, etc. to build listings of prospects.

- **Step 2: Enrichment.** Once detected, leads are enriched using external data APIs or with information scrapped from their website or LinkedIn pages. Enrichment sources include data about the companies, their employees, the technologies they use, the market in which they are, etc.

- **Step 3: Scoring and segmentation.** With all this data, leads are scored to detect the best opportunities. Leads can also be segmented into several listings. Maintaining this listing up to date is important and the scores and data must be kept fresh and updated as news appears or changes occur for the leads.

- **Step 4: Syncing with other tools.** Opportunities are then synced with prospection tools to perform cold emailing, send LinkedIn requests, etc. Or they can just be synced with a CRM.

### Other internal software examples

We've detailed 2 examples of internal software but they can be found in all departments.
For Product Management, DevOps, Accounting, organizations have already moved to internal software for a while.
We see new areas like Finance and Human Resources (for hiring and employee management) where internal software use is growing.

---

# What composes internal software?

Now we have seen how organizations benefit from internal software, let's look at what composes them. You have two options: use existing products or build custom internal tools!

On the products side, we find commercial applications developed by software editors and open-source solutions. They are either hosted in the cloud or require to be installed on-premise on the organization's servers.

When no solution exists on the market or when it doesn't fit the task, organizations rely on custom internal tools. To build them, they can:

- **Use generic tools like Spreadsheets.** Spreadsheets are still used when it comes to building internal tools.
- **Build on top of existing software.** For example, ERP solutions are extensible and new applications can be developed on them. See our article [*Where is enterprise software heading*](/blog/where-is-enterprise-software-heading) to know more.
- **Use frameworks or low-code/no-code solutions.** Their goal is to speed up development time for internal tools.
- **Code tools from scratch.** Usually using web technologies.
- Or use a mix of all those solutions


**Dedicated team**

Some organizations have a dedicated team to set up internal software. In this configuration, the team works best when members are multi-competent with technical people. Thus, they have the skills to build custom internal tools when needed.

Before, internal tools development repelled developers who prefer using their skills on customer-facing products. But this is changing, as the impact of internal software in organizations increases. In digital organizations, internal software is a key factor in the growth of a business.

When the organization doesn't have a dedicated team, technical employees can build internal tools but it will always be in the background of their primary role. As soon as they get busy with their primary tasks, the internal tools they maintain will be secondary. A team must have a focus and must choose between building a product for customers or building products for internal uses.

Of course, every employee should be involved in the brainstorming and can give feedback. But a team must be responsible for internal software.


**Hybrid approach**

There is no "Buying vs Make" debate. It depends on the issue the organization wants to solve, on the existing market solutions, on internal software strategy, and the organization's team size.

Also, organizations must stay open once they have made a decision. Context and priorities change and it's ok to throw a custom-built tool away or start over.

With cloud services and building blocks, an hybrid option with both commercial applications and custom internal tools working together is possible. At [Datablist](/), we believe in this hybrid approach, with data as foundation to connect them.

---

# What's coming next for internal software?

With the growth of digital organizations, internal software will continue to develop. We see a lot of innovations to make building internal software faster and promote the connectivity of every organization's internal applications on a single data view.

Generally speaking, internal tools manage customers, employees, orders, transactions, leads, etc. They store data, provide analytics, and use automation to streamline workflows. And end-users are employees. Every company deals with the same concepts when building internal tools.

**It represents enough similarities to see platforms that will provide those features as a foundation to build on top. Those platforms would be open, API first, and integrated with the ecosystem. They will have to implement bulletproof permission management and data access.**

It would be similar to what the Cloud giants (Amazon Web Services, Google Cloud, Microsoft Azure, etc.) have done in web development: a suite of web services to build digital processes.

> We can ask whether existing Cloud Platforms will win the internal tools market? AWS has launched an internal tools builder with [Honeycode](https://www.honeycode.aws/) and is building business applications targeted to employees and not just developers (see [AWS business applications](https://aws.amazon.com/business-applications/))


## **Flexible platforms**

We see more and more users migrating away from specialized and rigid tools to generic but flexible ones. [Airtable](https://airtable.com/) for leads generation or simple CRM, [Notion](https://www.notion.so/) for project management, etc.

**Specialized software will never disappear. A CRM is still mandatory to manage sales once the team grows. And Notion can not compete with project management software on complex projects. Nevertheless, the ever-growing space of ultra-specialized SaaS tools from employees directory to employees emotions gathering will compete with simple tools built with generic platforms like Notion, Airtable, or [Datablist](/).**

In the past, flexible and generic platforms meant ERP (see our article [*Where is enterprise software heading?*](/blog/where-is-enterprise-software-heading)) or Spreadsheets. As a result, employees had two options:

- Custom internal tools with ugly user interfaces built with ERP or spreadsheets interface.
- Shiny user interfaces with specialized SaaS software

Spreadsheets are still used as internal tools because they are flexible and we all know how to use them. They have flexible data models, data manipulation is easy with copy/paste, etc. which makes them efficient to deal with data. But, they haven't been created to manage processes (no files, images) and they haven't been adapted to the mobile world.

**New flexible platforms offer both flexible data models and a beautiful interface!**


## The rise of building blocks

Building an internal tool is not necessarily coding it from scratch. Using a spreadsheet to manage employee onboarding is an internal tool.

In the world of internal software, we talk about the "citizen developers". They are users who build software applications while not being technical people. We have all seen the **no-code**, **low-code** hype lately. With a no-code or low-code platform, you build custom applications by putting existing building blocks together. Some platforms are for non-technical people (no-code) and/or others for technical people (low-code). They leverage modularity to promise faster app development than coding it.

Building blocks include several parts of an application:

- **Building blocks for UI/interface with reusable components (for table, forms, buttons, etc.)**. [Retool](https://retool.com/) and [Internal](https://internal.io/) are good examples.
- **Building blocks on top of a smart database.** An example is [Datablist](/). In Datablist, [common data models](/blog/common-data-model-vs-cloud-information-model) are promoted and a catalog of data actions is available. Another example is the Airtable platform. Airtable uses a spreadsheet interface with blocks to display data in different ways (Gantt chart, calendar, map).
- **No-code software integration with workflow automation platform.** [Zapier](https://zapier.com/) and [Integromat](https://www.integromat.com/) are well known in this category.
- **Building blocks for developers.** With Cloud provider (Amazon, Google, Azure) providing higher-end services, it becomes easy to just plug those services together and let them manage the infrastructure.

![Datablist action launcher](/blog_articles_images/2021/06/datablist_action_launcher.png#1600x879)

![Airtable Map feature](/blog_articles_images/2021/06/airtable.png#1600x920)


## The new data-centric organizations

Current applications map organizational boundaries. Customer Relationship Software (CRM) for customer-facing teams, Product Management Software for product teams, etc. all seating in their siloed department. While improving the business efficiency, they've had a limited impact on the digitalization of the company operations. Data and processes are still confined to organizational units.

To leverage automation, digitized processes must go beyond the organization's siloed structure. Analytics and AI require to have a unified view of the data across departments.

Data-centric companies invest in data storage to have a global view of their data. **Data is the foundation on which internal tools are built.** While organizations share a unified data storage, teams are autonomous to build processes and internal software on top of it.


> A unified data foundation is a prerequisite for data-centric organizations. **When talking about data, we must distinguish between data analytics and databases.**


### Data analytics is an append-only data storage

The first concept to understand is data analytics. In data analytics, we harvest large sets of data to generate insights, discover information, and support decision-making. And **data warehouse** is the underlying technology in data analytics.

In the [Customer Data Platform](#customer-data-platform) use case, we would store every user action (application click, signup date, email click, support request created, payments, etc.) into a data warehouse. From that data, by running queries on the data warehouse, we analyze funnels, signup conversions, etc. It uses data from the data warehouse to generate information.

**However, data in a data warehouse is immutable!** Once loaded into the data warehouse, data can't be updated. It can be replaced with a new import or new data can be added to the existing data.

Data warehouse use cases are read-only. Once a value has been calculated with a query on a data warehouse, the value needs to be stored back into an application or a database.


### Organization Databases are mandatory for internal software

For editable data, internal software requires databases! Indeed, when building applications with "read and write" features, internal databases are mandatory.

Currently, database options to build internal tools are limited. Organizations can use spreadsheets, new "spreadsheet-like" platforms (like Airtable) or set up real web database engines from the developer's world (well-known open-source databases are PostgreSQL, MySQL, MongoDB, etc.)

Spreadsheets used as database suffer limitations:

- The number of items they can store is limited: 50k items per base for Airtable.
- It's not possible to implement unicity constraints with a property. In a customer listing, a unicity on email values is essential to avoid duplicates.
- Object relationships are limited. No relationship for Excel, Google Sheet. Airtable provides a relationship feature for items inside a base, but not with another base.

And while database engines are powerful, they are too low-level and lack built-in features to be easy to use:

- No built-in interface with permission-based access
- While possible, they are limited when it comes to storing files
- Data revision history and rollback is not built-in
- Data backup has to be implemented on top of it

**That's why new data platforms like Datablist will be necessary to build internal software.**

> Datablist aims to offer an easy-to-use data foundation usable by any organization employee to create internal software without the limitations we find in spreadsheets tools.

---

At Datablist, we work to make this vision a reality 🚀. [Join us](/) or subscribe to the newsletter to follow our progress 👇.


Why internal software is key to your growth

<h1 id="user-content-introduction"><a href="#introduction" aria-hidden="true" tabindex="-1"></a>Introduction</h1>
In a 1937 article, the Nobel Prize-Winning author Ronald Coase wrote the <a href="https://onlinelibrary.wiley.com/doi/full/10.1111/j.1468-0335.1937.tb00002.x" target="_blank" rel="nofollow">"Nature of the Firm"</a>. In this article, he laid down the conditions for how people organize to produce goods and services. For Coase, the two main ways of producing were with individuals in a market ruled by prices, and employees grouped in a firm. Markets are an efficient means of creating wealth but Coase argues the best way to serve the requirement of customers was through the firm. Indeed, according to Coase, the firm structure allows reducing the coordination cost mandatory for any collaborative project. But this theory was only valid until the digital age!
A new form of collaborative work has emerged with the digital age where individuals contribute without the border of a firm. These collaborative productions can be open-source software like Linux, Node.js (a javascript webserver), culture, or information like Wikipedia. Following Coase article and his 2 models of production, <a href="https://en.wikipedia.org/wiki/Yochai_Benkler" target="_blank" rel="nofollow">Yochai Benkler</a> wrote in 2002 an essay called <a href="http://www.benkler.org/CoasesPenguin.PDF" target="_blank" rel="nofollow">"Coase’s Penguin, Or, Linux and ‘The Nature of the Firm.’"</a> to add a third model, now possible with online collaboration, that he called <a href="https://en.wikipedia.org/wiki/Commons-based_peer_production" target="_blank" rel="nofollow">"commons-based peer production (CBPP)"</a>. This new model of production relies on decentralized information gathering, self-organization, and most of the time, no financial incentive. This is an incredible age, where thousands of people can participate in projects beyond the limit of their firm. After the first wave, driven by individuals, companies now embrace this philosophy as it brings a sense of belonging to employees and serves the companies as they benefit from contributions from outside their labor force.
Open-source produces code libraries to be used as building blocks in larger projects but it also produces standalone applications. In the latter, modularisation and the cloud are transforming how we develop web applications and as a result, will change how people collaborate on open source projects.
<h1 id="user-content-web-applications-are-now-distributed"><a href="#web-applications-are-now-distributed" aria-hidden="true" tabindex="-1"></a>Web applications are now distributed</h1>
When I started coding fifteen years ago, a web application was a big program that handled everything. User accounts, email sendings, file storage, etc. were all coded in a single program.
Two limitations emerged with this approach:
<ul>
<li>First, when several people work on the same program, the code base grows quickly, and the complexity increases. Developer changes can conflicts when they edit the same code and it brings overhead in code maintenance. Furthermore, at some point, it becomes impossible for a single mind to keep the whole program modeled in his head.</li>
<li>And it became obvious developers were reinventing the wheel and spending too much time on basic features.</li>
</ul>
The solution was to split the application into several services. Each service is responsible for a subset of features and can be maintained without interfering with other service's code. Services can be developed by the application developers or can be maintained (and sometimes host) by other providers.
Amazon and its <a href="https://aws.amazon.com/" target="_blank" rel="nofollow">Amazon Web Services (AWS)</a> understood this shift before anyone. AWS started with infrastructure services but now provides high-level services to deal with authentication, emails, push notifications, analytics, machine learning, etc. Amazon teams develop the services, update them when security fixes are needed and take care of the availability.
<a href="https://aws.amazon.com/products/" target="_blank" rel="nofollow"><img src="/blog_articles_images/2021/02/aws-toolkit.png#1200x910" alt="Amazon Web Services Infographic"></a>
With this transformation, the infrastructure is now an integral part of the whole application. It is not enough to have the software code idle in Github to be able to run it, the intelligence from the cloud infrastructure is required. How do the services communicate with each other? How do you share settings between services? How errors are handled when a service is down? How do you deploy a new service in this mesh?
This transformation has been going on for some years in closed source applications. For open-source applications, it is harder to define this infrastructure intelligence in the code repository. Today, serverless technologies can change how open-source distributed applications are built and move code collaboration to the cloud.
<h1 id="user-content-serverless-technologies-will-change-how-we-consume-open-source-software"><a href="#serverless-technologies-will-change-how-we-consume-open-source-software" aria-hidden="true" tabindex="-1"></a>Serverless technologies will change how we consume open source software...</h1>
The open-source software distribution model currently consists of choosing between starting from the source code and self-deploying it, or using a SaaS model some companies have built on top of open-source software. In the latter, the software is hosted on instances paid by the editor and the editor generates revenues with licenses. The editor's role is to make a profit between what the instances cost and the revenues the licenses generate.
<blockquote>
For example, the WordPress software code is open source and can be self-deployed or used directly using the SaaS plan provided by <a href="https://wordpress.com/pricing/" target="_blank" rel="nofollow">Wordpress.com</a>.
</blockquote>
Now, imagine what the <a href="https://en.wikipedia.org/wiki/Serverless_computing" target="_blank" rel="nofollow">serverless infrastructure</a> brings.
Take an application built around serverless technologies. The application is hosted on a cloud platform, ready to run. Obviously, it is made of compute serverless services to run code, but also serverless databases, serverless cache instances, serverless queue services, etc.
<blockquote>
Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers.
<a href="https://en.wikipedia.org/wiki/Serverless_computing" target="_blank" rel="nofollow">Check the Serverless definition on Wikipedia.</a>
</blockquote>
When a customer runs the applications, the cloud platform allocates resources on-demand to launch and run the needed services. At the end of the execution, it is possible to count each service invocation and bill the customer.
In this model, the starting point is not a Github repository but a ready to run application hosted in a cloud platform App Store.
<h1 id="user-content--and-bring-collaborative-cloud-platforms"><a href="#-and-bring-collaborative-cloud-platforms" aria-hidden="true" tabindex="-1"></a>... and bring collaborative cloud platforms</h1>
Serverless technology is still new, yet we can already witness the change.
<h2 id="user-content-aws-serverless-application-repository"><a href="#aws-serverless-application-repository" aria-hidden="true" tabindex="-1"></a>AWS Serverless Application Repository</h2>
Amazon Web Services platform's primary role is to distribute AWS services but they <a href="https://aws.amazon.com/blogs/aws/now-available-aws-serverless-application-repository/" target="_blank" rel="nofollow">release in 2018</a> their <a href="https://aws.amazon.com/serverless/serverlessrepo/" target="_blank" rel="nofollow">Serverless Application Reposity</a>. An example is <a href="https://serverlessrepo.aws.amazon.com/applications/us-east-1/526515951862/image-resizer-service" target="_blank" rel="nofollow">this service to handle image manipulation</a>. AWS has a leading role in serverless and it would make sense for the other cloud providers to follow. Google currently has a <a href="https://cloud.google.com/marketplace" target="_blank" rel="nofollow">marketplace</a> to run third-party solutions but the marketplace doesn't provide software with serverless pricing.
<a href="https://serverlessrepo.aws.amazon.com/applications" target="_blank" rel="nofollow"><img src="/blog_articles_images/2021/02/aws-serverless-store.png#1418x1124" alt="AWS Serverless Application Reposity"></a>
<h2 id="user-content-algorithmia"><a href="#algorithmia" aria-hidden="true" tabindex="-1"></a>Algorithmia</h2>
Another example is the <a href="https://algorithmia.com/algorithms" target="_blank" rel="nofollow">Algorithmia services marketplace</a>. Alorithmia is a serverless cloud platform to host machine learning models. Anyone can publish its trained models on the marketplace for others to use. A published model exposes an API. Hosting and scaling of the underlying service are managed by Algorithmia. Publishers can also charge a commission fee for the execution of their services.
<a href="https://algorithmia.com/algorithms" target="_blank" rel="nofollow"><img src="/blog_articles_images/2021/02/algorithmia.png#2106x1436" alt="Algorithmia Algorithms store"></a>
<h2 id="user-content-glitch"><a href="#glitch" aria-hidden="true" tabindex="-1"></a>Glitch</h2>
I love <a href="https://glitch.com/" target="_blank" rel="nofollow">Glitch</a>. I love <a href="https://medium.com/glitch/welcome-to-glitch-fe161d0fc39b" target="_blank" rel="nofollow">the story</a> and the company culture. Glitch is a collaborative coding environment where the applications are always "runnable". Users can create applications from scratch or duplicate open source applications from other users. Once someone accesses the application, Glitch runs it on a serverless platform and the instance is stopped after 5 minutes of inactivity. Glitch blends both code hosting and application hosting in a collaborative cloud platform.
<a href="https://glitch.com/" target="_blank" rel="nofollow"><img src="/blog_articles_images/2021/02/glitch.png#1818x1266" alt="Glitch"></a>
<h2 id="user-content-datablist"><a href="#datablist" aria-hidden="true" tabindex="-1"></a>Datablist</h2>
<a href="https://www.datablist.com" target="_blank" rel="nofollow">Datablist is a database platform</a> to store any kind of items and to trigger actions on them (manually or using automation). For example, you can trigger enrichment actions on people items, web scrapping actions on URLs, OCR (optical character recognition) actions on PDF, etc.
Under the hood, every action on Datablist is a serverless service called whenever a user runs the action. All the actions available on Datablist are listed in an "Actions Directory".
Users can use actions from this Actions Directory or code their own actions that will be hosted and run on Datablist.
An important concept on Datablist is to have <a href="/blog/common-data-model-vs-cloud-information-model">common data types shared by all users</a>. It allows having generic actions on data types and properties: People, PDF, URL, etc. Because they are generic, an action created by a user can be published on the "Actions Directory" and be available to everyone.
By using tens, hundreds, or thousands of actions on your data, and by automating when and how each action is triggered, you end up building applications. An application being a mesh of actions. The goal is to have applications that suit your needs while you benefit from community actions, so you don't have to maintain each service.
If you share our vision, subscribe to our newsletter below and join our <a href="https://www.datablist.com" target="_blank" rel="nofollow">beta program</a>.


# Introduction

In a 1937 article, the Nobel Prize-Winning author Ronald Coase wrote the ["Nature of the Firm"](https://onlinelibrary.wiley.com/doi/full/10.1111/j.1468-0335.1937.tb00002.x). In this article, he laid down the conditions for how people organize to produce goods and services. For Coase, the two main ways of producing were with individuals in a market ruled by prices, and employees grouped in a firm. Markets are an efficient means of creating wealth but Coase argues the best way to serve the requirement of customers was through the firm. **Indeed, according to Coase, the firm structure allows reducing the coordination cost mandatory for any collaborative project. But this theory was only valid until the digital age!**

**A new form of collaborative work has emerged with the digital age where individuals contribute without the border of a firm.** These collaborative productions can be open-source software like Linux, Node.js (a javascript webserver), culture, or information like Wikipedia. Following Coase article and his 2 models of production, [Yochai Benkler](https://en.wikipedia.org/wiki/Yochai_Benkler) wrote in 2002 an essay called ["Coase’s Penguin, Or, Linux and ‘The Nature of the Firm.’"](http://www.benkler.org/CoasesPenguin.PDF) to add a third model, now possible with online collaboration, that he called ["commons-based peer production (CBPP)"](https://en.wikipedia.org/wiki/Commons-based_peer_production). This new model of production relies on decentralized information gathering, self-organization, and most of the time, no financial incentive. This is an incredible age, where thousands of people can participate in projects beyond the limit of their firm. **After the first wave, driven by individuals, companies now embrace this philosophy as it brings a sense of belonging to employees and serves the companies as they benefit from contributions from outside their labor force.**

Open-source produces code libraries to be used as building blocks in larger projects but it also produces standalone applications. In the latter, modularisation and the cloud are transforming how we develop web applications and as a result, will change how people collaborate on open source projects.


# Web applications are now distributed

When I started coding fifteen years ago, a web application was a big program that handled everything. User accounts, email sendings, file storage, etc. were all coded in a single program.

Two limitations emerged with this approach:
- First, when several people work on the same program, the code base grows quickly, and the complexity increases. Developer changes can conflicts when they edit the same code and it brings overhead in code maintenance. Furthermore, at some point, it becomes impossible for a single mind to keep the whole program modeled in his head.
- And it became obvious developers were reinventing the wheel and spending too much time on basic features.

**The solution was to split the application into several services. Each service is responsible for a subset of features and can be maintained without interfering with other service's code. Services can be developed by the application developers or can be maintained (and sometimes host) by other providers.**

Amazon and its [Amazon Web Services (AWS)](https://aws.amazon.com/) understood this shift before anyone. AWS started with infrastructure services but now provides high-level services to deal with authentication, emails, push notifications, analytics, machine learning, etc. Amazon teams develop the services, update them when security fixes are needed and take care of the availability.

[![Amazon Web Services Infographic](/blog_articles_images/2021/02/aws-toolkit.png#1200x910)](https://aws.amazon.com/products/)

**With this transformation, the infrastructure is now an integral part of the whole application. It is not enough to have the software code idle in Github to be able to run it, the intelligence from the cloud infrastructure is required.** How do the services communicate with each other? How do you share settings between services? How errors are handled when a service is down? How do you deploy a new service in this mesh?

This transformation has been going on for some years in closed source applications. For open-source applications, it is harder to define this infrastructure intelligence in the code repository. **Today, serverless technologies can change how open-source distributed applications are built and move code collaboration to the cloud.**


# Serverless technologies will change how we consume open source software...

The open-source software distribution model currently consists of choosing between starting from the source code and self-deploying it, or using a SaaS model some companies have built on top of open-source software. In the latter, the software is hosted on instances paid by the editor and the editor generates revenues with licenses. The editor's role is to make a profit between what the instances cost and the revenues the licenses generate.

> For example, the WordPress software code is open source and can be self-deployed or used directly using the SaaS plan provided by [Wordpress.com](https://wordpress.com/pricing/).

Now, imagine what the [serverless infrastructure](https://en.wikipedia.org/wiki/Serverless_computing) brings.
Take an application built around serverless technologies. The application is hosted on a cloud platform, ready to run. Obviously, it is made of compute serverless services to run code, but also serverless databases, serverless cache instances, serverless queue services, etc.

> Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers.
> [Check the Serverless definition on Wikipedia.](https://en.wikipedia.org/wiki/Serverless_computing)

When a customer runs the applications, the cloud platform allocates resources on-demand to launch and run the needed services. At the end of the execution, it is possible to count each service invocation and bill the customer.

**In this model, the starting point is not a Github repository but a ready to run application hosted in a cloud platform App Store.**

# ... and bring collaborative cloud platforms

Serverless technology is still new, yet we can already witness the change.

## AWS Serverless Application Repository

Amazon Web Services platform's primary role is to distribute AWS services but they [release in 2018](https://aws.amazon.com/blogs/aws/now-available-aws-serverless-application-repository/) their [Serverless Application Reposity](https://aws.amazon.com/serverless/serverlessrepo/). An example is [this service to handle image manipulation](https://serverlessrepo.aws.amazon.com/applications/us-east-1/526515951862/image-resizer-service). AWS has a leading role in serverless and it would make sense for the other cloud providers to follow. Google currently has a [marketplace](https://cloud.google.com/marketplace) to run third-party solutions but the marketplace doesn't provide software with serverless pricing.


[![AWS Serverless Application Reposity](/blog_articles_images/2021/02/aws-serverless-store.png#1418x1124)](https://serverlessrepo.aws.amazon.com/applications)


## Algorithmia
Another example is the [Algorithmia services marketplace](https://algorithmia.com/algorithms). Alorithmia is a serverless cloud platform to host machine learning models. Anyone can publish its trained models on the marketplace for others to use. A published model exposes an API. Hosting and scaling of the underlying service are managed by Algorithmia. **Publishers can also charge a commission fee for the execution of their services.**

[![Algorithmia Algorithms store](/blog_articles_images/2021/02/algorithmia.png#2106x1436)](https://algorithmia.com/algorithms)


## Glitch

I love [Glitch](https://glitch.com/). I love [the story](https://medium.com/glitch/welcome-to-glitch-fe161d0fc39b) and the company culture. Glitch is a collaborative coding environment where the applications are always "runnable". Users can create applications from scratch or duplicate open source applications from other users. Once someone accesses the application, Glitch runs it on a serverless platform and the instance is stopped after 5 minutes of inactivity. Glitch blends both code hosting and application hosting in a collaborative cloud platform.

[![Glitch](/blog_articles_images/2021/02/glitch.png#1818x1266)](https://glitch.com/)


## Datablist

[Datablist is a database platform](https://www.datablist.com) to store any kind of items and to trigger actions on them (manually or using automation). For example, you can trigger enrichment actions on people items, web scrapping actions on URLs, OCR (optical character recognition) actions on PDF, etc.

Under the hood, every action on Datablist is a serverless service called whenever a user runs the action. All the actions available on Datablist are listed in an "Actions Directory".

Users can use actions from this Actions Directory or code their own actions that will be hosted and run on Datablist.

An important concept on Datablist is to have [common data types shared by all users](/blog/common-data-model-vs-cloud-information-model). It allows having generic actions on data types and properties: People, PDF, URL, etc. Because they are generic, an action created by a user can be published on the "Actions Directory" and be available to everyone.

**By using tens, hundreds, or thousands of actions on your data, and by automating when and how each action is triggered, you end up building applications. An application being a mesh of actions. The goal is to have applications that suit your needs while you benefit from community actions, so you don't have to maintain each service.**


If you share our vision, subscribe to our newsletter below and join our [beta program](https://www.datablist.com).


Serverless will move software collaboration to the cloud!

As companies continue digitizing their operations, data is more valuable than ever. But as we wrote in our article <a href="/blog/where-is-enterprise-software-heading">Where enterprise software is heading?</a>, the number of applications used in an organization keeps increasing. It is then not surprising to see businesses invest in data platforms to centralized their data.
Once all the data is in one place, it's possible to have a 360 view of the customer journey, control costs, and have real-time accounting. Data is also the fuel needed to run predictions. In the next years, artificial intelligence models will help companies run accurate finance forecasts and it could detect operational anomalies automatically.
However, building a data platform is complex. First, you need to extract the data from all the applications and store it as a single data model. The customer data coming from the CRM and the one coming from the accounting must have the same identifiers and attributes to be consolidated. This is known as Extract Transform Load (ETL). <a href="https://hbr.org/2018/08/what-data-scientists-really-do-according-to-35-data-scientists" target="_blank" rel="nofollow">It is said that data engineers spend 80% of their time cleaning and organizing data.</a>
Once the data is stored. Whether it's in a data warehouse or a database, the second step is to build analytic views and models on the data to get insights and create value over the raw data. This is heavy work. The tools at play are complex to configure and you need a data team to benefit from your data platform.
Every business deals with the same data models: Customers, Organizations, Invoices, etc. Yet, they need to redefine manually the data schemas to store their data and create analytics views almost from scratch. So, could this process be simplified?
At Datablist, we believe it's time to have common data models that define all those concepts (Customers, Organizations, Finance, etc.) independently of the applications that store them.
In the past months, the big players (Microsoft, Salesforce, AWS, and Google Cloud) have announced working on unified data models to federate an ecosystem of tools compatible with the same data schemas. Are all of them working on a single data model? No, they have decided to go on a separate road!
<h1 id="user-content-what-is-a-data-model-and-why-does-it-matter"><a href="#what-is-a-data-model-and-why-does-it-matter" aria-hidden="true" tabindex="-1"></a>What is a data model and why does it matter?</h1>
First, we need to define what a data model is. It is made of:
<ul>
<li>Entities - Classes that represent business concepts. For example Person, Organization, Product. Entities can be drill-down in nested levels. The entity "Person" can have sub-entities like Contact, Lead, Customer for example. Usually, a data model is composed of few root classes but each one has a nested tree of sub-entities.</li>
<li>Properties - Entities are empty shells. They need Properties (or attributes) to store information. name is an example of property and the same property could be used in several entities: Person, Product, Organization. Properties are also defined in a hierarchical tree. We can imagine firstName and lastName as sub properties of name.</li>
<li>Relationships - With the properties, it is possible to define relations. For example, a Product entity would have a brand property that is represented by an Organization value.</li>
<li>Data validations - On a generic data model, data validation is loose to avoid imposing constraints on the input data. Nevertheless, a data model could define rules to validate mandatory properties or property value types.</li>
</ul>
Together, Entities, Properties and Relationships compose a data model. It is also called a vocabulary.
So, why is it important? Because it's the first step toward standardization!. Standardization of data organization and expected structure. Data cleaning, the most time-consuming task of any data engineer could be automated (at least partially). When you export data from different applications in a single data model, the consolidation will be automatic.
Then, once the data is stored and consolidated, come the analytics and data visualization parts. A unified data model would bring ready-to-use tools. Instead of building analytics views from scratch, generic views would be used to make dashboards that represent common concepts (revenues, customer churn rate, etc.).
Finally, a unified model would empower non-technical people to work with the data instead of relying on data scientists.
<h1 id="user-content-microsoft-and-the-open-data-initiative"><a href="#microsoft-and-the-open-data-initiative" aria-hidden="true" tabindex="-1"></a>Microsoft and the Open Data Initiative</h1>
The first move was made by Microsoft with the "Open Data Initiative" in 2018. Microsoft was joined by SAP and Adobe to agree on a list of data schemas so their respective applications could speak the same language.
The Open Data Initiative has created the Common Data Model (CDM). When exporting customer data from SAP, Adobe, or Microsoft Dynamics 365, among the formats, the Common Data Model is now available.
<a href="https://aka.ms/cdmposter" target="_blank" rel="nofollow"><img src="/blog_articles_images/2021/01/cdm-entities-v1.png#2772x1908" alt="Common Data Model Entities"></a>
The full Data Poster is available <a href="https://aka.ms/cdmposter" target="_blank" rel="nofollow">here</a> and an interactive version <a href="https://microsoft.github.io/CDM/" target="_blank" rel="nofollow">here</a>.
Microsoft Dynamics 365 internal data structure is the main source for this data model. It has the advantage of providing a complete and broad model. Entities for banking, education, medical fields are available. On the other hand, the model is very complex and the inner workings of Dynamics 365 are too visible.
This is a smart move from Microsoft and the Azure platform. All the data tools on the Azure platform will be compatible eventually. Obviously, the Azure Data Lake to store data but also the Artificial Intelligence tools that will benefit from standard data structures.
To read more, visit the Microsoft page on the Open Data Initiative: <a href="https://www.microsoft.com/en-us/open-data-initiative" target="_blank" rel="nofollow">https://www.microsoft.com/en-us/open-data-initiative</a> and the Github repository: <a href="https://github.com/microsoft/CDM" target="_blank" rel="nofollow">https://github.com/microsoft/CDM</a>.
<h1 id="user-content-salesforce-and-the-cloud-information-model"><a href="#salesforce-and-the-cloud-information-model" aria-hidden="true" tabindex="-1"></a>Salesforce and the Cloud Information Model</h1>
Recently, at the end of 2019, Salesforce and Amazon Web Services decided to start their own initiative called the <a href="https://cloudinformationmodel.org/" target="_blank" rel="nofollow">Cloud Information Model</a> (CIM). Google Cloud, Twilio and other companies have joined the consortium since.
Unlike the Common Data Model, the Cloud Information Model is not a replication of an existing software data structure. Thus, it is lighter and the number of entities and properties is nothing to compare with the Open Data Initiative. It sounds like a weak point but it is not, in the long term. The project is at the beginning and it will grow slowly. In the end, the data model will be easier to understand and to implement.
<a href="https://cloudinformationmodel.org/party.html" target="_blank" rel="nofollow"><img src="/blog_articles_images/2021/01/cloudinformationmodel-party.png#2553x1357" alt="Cloud Information Model - Party Entities"></a>
Another difference with the Microsoft data model is the governance of the project. It is open-source and claims to welcome any company that wants to contribute.
On the negative side, the project is less active than the Common Data Model. Salesforce has announced it would be available on Salesforce Customer 360 and Salesforce Mulesoft solution is compatible, but that's all. The ecosystem doesn't exist yet.
With AWS and Google Cloud on board, we will see if they implement it on their big data tools.
To read more, visit the <a href="https://cloudinformationmodel.org/" target="_blank" rel="nofollow">Cloud Information Model website</a> and the Github repository <a href="https://github.com/cloudinformationmodel" target="_blank" rel="nofollow">https://github.com/cloudinformationmodel</a>.
<h1 id="user-content-do-we-need-several-data-models"><a href="#do-we-need-several-data-models" aria-hidden="true" tabindex="-1"></a>Do we need several data models?</h1>
One can ask why Salesforce, AWS, and the other members of the Cloud Information Model haven't joined the Microsoft initiative and decided instead to go alone. Clearly, this has to do with having control over the course of the project and be at the center of the future ecosystem.
But it doesn't matter. Both models share common entities and properties. A property from one will have a matching property on the other. We might see data format converters to translate data from a vocabulary to another one, in the same way, we have PNG, JPEG, etc. to store images and converters to go from one format to the other.
Competition has always been an accelerator for innovation.
<h1 id="user-content-what-about-datablist"><a href="#what-about-datablist" aria-hidden="true" tabindex="-1"></a>What about Datablist?</h1>
At Datablist, we are building our own data model using the W3C standard <a href="https://www.w3.org/TR/json-ld11/" target="_blank" rel="nofollow">JSON-LD</a>. The Cloud Information Model (CIM) also uses JSON-LD to structure its data model. Technologically speaking, our data model is quite similar to CIM. Unfortunately, at the moment, the CIM model has a narrow list of business domains (basically Customer, Payment, Product, Order) and is not stable. To move fast, we will continue building our own data model.
Data interoperability is coming. The 2020 decade will be all about data!


As companies continue digitizing their operations, data is more valuable than ever. But as we wrote in our article [Where enterprise software is heading?](/blog/where-is-enterprise-software-heading), the number of applications used in an organization keeps increasing. **It is then not surprising to see businesses invest in data platforms to centralized their data.**

Once all the data is in one place, it's possible to have a 360 view of the customer journey, control costs, and have real-time accounting. Data is also the fuel needed to run predictions. In the next years, artificial intelligence models will help companies run accurate finance forecasts and it could detect operational anomalies automatically.

**However, building a data platform is complex.** First, you need to extract the data from all the applications and store it as a single data model. The customer data coming from the CRM and the one coming from the accounting must have the same identifiers and attributes to be consolidated. This is known as Extract Transform Load (ETL). **[It is said that data engineers spend 80% of their time cleaning and organizing data.](https://hbr.org/2018/08/what-data-scientists-really-do-according-to-35-data-scientists)**

Once the data is stored. Whether it's in a data warehouse or a database, the second step is to build analytic views and models on the data to get insights and create value over the raw data. This is heavy work. The tools at play are complex to configure and you need a data team to benefit from your data platform.

Every business deals with the same data models: Customers, Organizations, Invoices, etc. **Yet, they need to redefine manually the data schemas to store their data and create analytics views almost from scratch.** So, could this process be simplified?

**At Datablist, we believe it's time to have common data models that define all those concepts (Customers, Organizations, Finance, etc.) independently of the applications that store them.**

**In the past months, the big players (Microsoft, Salesforce, AWS, and Google Cloud) have announced working on unified data models to federate an ecosystem of tools compatible with the same data schemas.** Are all of them working on a single data model? No, they have decided to go on a separate road!



# What is a data model and why does it matter?

First, we need to define what a data model is. It is made of:

  - **Entities** - Classes that represent business concepts. For example Person, Organization, Product. Entities can be drill-down in nested levels. The entity "Person" can have sub-entities like Contact, Lead, Customer for example. Usually, a data model is composed of few root classes but each one has a nested tree of sub-entities.
  - **Properties** - Entities are empty shells. They need Properties (or attributes) to store information. *name* is an example of property and the same property could be used in several entities: Person, Product, Organization. Properties are also defined in a hierarchical tree. We can imagine *firstName* and *lastName* as sub properties of *name*.
  - **Relationships** - With the properties, it is possible to define relations. For example, a *Product* entity would have a *brand* property that is represented by an *Organization* value.
  - **Data validations** - On a generic data model, data validation is loose to avoid imposing constraints on the input data. Nevertheless, a data model could define rules to validate mandatory properties or property value types.

**Together, *Entities*, *Properties* and *Relationships* compose a data model**. It is also called a **vocabulary**.

So, why is it important? **Because it's the first step toward standardization!.** Standardization of data organization and expected structure. **Data cleaning, the most time-consuming task of any data engineer could be automated (at least partially).** When you export data from different applications in a single data model, the consolidation will be automatic.

Then, once the data is stored and consolidated, come the analytics and data visualization parts. **A unified data model would bring ready-to-use tools.** Instead of building analytics views from scratch, generic views would be used to make dashboards that represent common concepts (revenues, customer churn rate, etc.).

**Finally, a unified model would empower non-technical people to work with the data instead of relying on data scientists.**


# Microsoft and the Open Data Initiative

**The first move was made by Microsoft with the "Open Data Initiative" in 2018.** Microsoft was joined by SAP and Adobe to agree on a list of data schemas so their respective applications could speak the same language.

The Open Data Initiative has created the Common Data Model (CDM). When exporting customer data from SAP, Adobe, or Microsoft Dynamics 365, among the formats, the Common Data Model is now available.

[![Common Data Model Entities](/blog_articles_images/2021/01/cdm-entities-v1.png#2772x1908)](https://aka.ms/cdmposter)

The full Data Poster is available [here](https://aka.ms/cdmposter) and an interactive version [here](https://microsoft.github.io/CDM/).

**Microsoft Dynamics 365 internal data structure is the main source for this data model.** It has the advantage of providing a complete and broad model. Entities for banking, education, medical fields are available. **On the other hand, the model is very complex and the inner workings of Dynamics 365 are too visible.**

This is a smart move from Microsoft and the Azure platform. **All the data tools on the Azure platform will be compatible eventually**. Obviously, the Azure Data Lake to store data but also the Artificial Intelligence tools that will benefit from standard data structures.

To read more, visit the Microsoft page on the Open Data Initiative: [https://www.microsoft.com/en-us/open-data-initiative](https://www.microsoft.com/en-us/open-data-initiative) and the Github repository: [https://github.com/microsoft/CDM](https://github.com/microsoft/CDM).


# Salesforce and the Cloud Information Model

Recently, at the end of 2019, Salesforce and Amazon Web Services decided to start their own initiative called the **[Cloud Information Model](https://cloudinformationmodel.org/)** (CIM). Google Cloud, Twilio and other companies have joined the consortium since.

Unlike the Common Data Model, the Cloud Information Model is not a replication of an existing software data structure. Thus, it is lighter and the number of entities and properties is nothing to compare with the Open Data Initiative. It sounds like a weak point but it is not, in the long term. The project is at the beginning and it will grow slowly. In the end, the data model will be easier to understand and to implement.

[![Cloud Information Model - Party Entities](/blog_articles_images/2021/01/cloudinformationmodel-party.png#2553x1357)](https://cloudinformationmodel.org/party.html)

Another difference with the Microsoft data model is the governance of the project. It is open-source and claims to welcome any company that wants to contribute.

On the negative side, the project is less active than the Common Data Model. Salesforce has announced it would be available on Salesforce Customer 360 and Salesforce Mulesoft solution is compatible, but that's all. The ecosystem doesn't exist yet.

**With AWS and Google Cloud on board, we will see if they implement it on their big data tools.**

To read more, visit the [Cloud Information Model website](https://cloudinformationmodel.org/) and the Github repository [https://github.com/cloudinformationmodel](https://github.com/cloudinformationmodel).


# Do we need several data models?

**One can ask why Salesforce, AWS, and the other members of the Cloud Information Model haven't joined the Microsoft initiative and decided instead to go alone.** Clearly, this has to do with having control over the course of the project and be at the center of the future ecosystem.

**But it doesn't matter**. Both models share common entities and properties. A property from one will have a matching property on the other. We might see data format converters to translate data from a vocabulary to another one, in the same way, we have PNG, JPEG, etc. to store images and converters to go from one format to the other.
**Competition has always been an accelerator for innovation.**


# What about Datablist?

**At Datablist, we are building our own data model using the W3C standard [JSON-LD](https://www.w3.org/TR/json-ld11/).** The Cloud Information Model (CIM) also uses JSON-LD to structure its data model. Technologically speaking, our data model is quite similar to CIM. Unfortunately, at the moment, the CIM model has a narrow list of business domains (basically Customer, Payment, Product, Order) and is not stable. To move fast, we will continue building our own data model.


**Data interoperability is coming. The 2020 decade will be all about data!**


Brace yourself, the battle for a unified data model is coming!

<h1 id="user-content-introduction"><a href="#introduction" aria-hidden="true" tabindex="-1"></a>Introduction</h1>
Last month, <a href="https://www.forbes.com/sites/alexkonrad/2020/10/09/twilio-to-acquire-cloud-startup-segment-for-3-billion/" target="_blank" rel="nofollow">Twilio bought Segment.io</a>, a vertical SaaS software that focuses on customer data integration. This move by Twilio comes after buying Sendgrid last year and follows a bundling strategy. Twilio wants to be a 360° platform when it comes to customer value proposition.
<a href="https://hbr.org/2014/06/how-to-succeed-in-business-by-bundling-and-unbundling" target="_blank" rel="nofollow">In a famous quote</a>, Jim Barksdale said the two ways to make money is by bundling products together or unbundling them and creating targeted offers.
We see both in the software industry today. Microsoft, Salesforce, Oracle and other behemoths keep adding to their software suite, making them ubiquitous. In the meanwhile, we can see an increase in niche SaaS products. From image segmentation APIs to content sentiment analysis tools. The number of SaaS products we use at work keeps increasing.
Integrated suites have the advantage of an unified experience while specialized products take advantage of their expertise and can be laser focus on the problem they solve.
In my previous article (<a href="/blog/where-is-enterprise-software-heading">where is enterprise software heading</a>), I discussed the evolutions in enterprise software as we have moved from ERP to a galaxy of SaaS products (also know as best of breed). This is an integrated suite vs best of breed debate we have now.
In a perfect world, we would be able to compose our own suite by making the different softwares work together. Unfortunately, we are not in a perfect world. Few months ago, a Microsoft sales contacted me to make us move from Slack to Microsoft Teams. I told him my experience with Skype Enterprise and Teams, from an external point of view, was a disaster. It was a hell to work with companies using those tools when you were on another suite (aka Google + Slack). He acknowledged it has not been strategic for Microsoft, but he assured the company started to work on more interoperability.
It might come from my desire to stay away from big corporations but I keep dreaming of a world where software collaboration could exist outside of big enterprise suites.
The number of specialized APIs is impressive. In my last count, I found about twenty companies providing an API for invoice PDF parsing, a good hundred of APIs to check the validity of an email address, and hundreds of companies/people enrichment tools.
Each of these APIs focuses on a single problem and addresses it well. But chances are you don't want to implement tens of different APIs to build an internal tool that would automate your accounting or your customer management. The question is: how can we bundle them to create our own suite?
<h1 id="user-content-why-workflow-automation-zapier-integromat-etc-is-not-the-best-solution"><a href="#why-workflow-automation-zapier-integromat-etc-is-not-the-best-solution" aria-hidden="true" tabindex="-1"></a>Why workflow automation (Zapier, Integromat, etc.) is not the best solution?</h1>
The first wave of tools to address this problematic are called "workflow automation tools" and Zapier, Mulesoft, Integromat, etc. are in this category.
<img src="/blog_articles_images/2020/11/integromat-screenshot.png#1024x628" alt="Integromat Workflow Automation">
The idea is simple: APIs are "Apps". They have inputs and outputs. Something goes in, the app does its job, and something else goes out. The workflow automation tool ensures the data moves from one app to the next one. Apps can be chained to make complex workflows.
While this solves a lot of problems, it is not suited for everything. With workflow automation, the starting point is an event from an app (or a scheduled event) and the workflow will move the data from one app to another one. It acts as a plumbing pipe.
It connects APIs together more than creating a bundle of products. Here are some limitations:
<ul>
<li>It is not possible to trigger manually an action for a specific data item. Imagine you have an existing item in your CRM and you want to run it though an automation workflow you have. It is not possible without manually adding a new source event in your workflow.</li>
<li>Data moves as a whole between apps. It is not possible to transform the value for one property and then merge it back to the data object to be send to the next workflow app. You would need a data storage layer in between.</li>
</ul>
<h1 id="user-content-connect-apps-through-a-data-hub"><a href="#connect-apps-through-a-data-hub" aria-hidden="true" tabindex="-1"></a>Connect Apps through a data hub</h1>
So how can we bundle APIs/Products together to create value? The answer is: through data. Instead of starting from app events like in workflow automation, the data must be at the center.
Businesses need a centralized data hub for data related to customers, suppliers, employees, etc. from which partial data can be send to external APIs to be processed. The results of this process then return to the data hub. It is about consolidating several APIs processes into a single data storage.
From this data hub, automations can be configured on data events like with workflow automation. When a new item (customer, invoice, etc.) is added to the data hub, actions run to process the new data and the result is saved for later use. But now, actions can be trigged manually directly from a data item. For example: send a mail to a customer.
Building with a data first approach opens the possibility of data standardisation. With workflow automation, every time you integrate an app (Application A) with another one (Application B), you need to define what properties from the data exposed by Application A will be used as input in Application B. Integration between two apps needs configuration. By starting from the data, it is not how one app integrates with another one, but how each app connects to the data. The data is the source for each integration.
If a data property is of type email, all APIs with an email input type can be automatically listed and shown.
<h1 id="user-content-own-your-platform-and-use-the-power-of-external-apis"><a href="#own-your-platform-and-use-the-power-of-external-apis" aria-hidden="true" tabindex="-1"></a>Own your platform and use the power of external APIs</h1>
The combo Data hub + APIs is a powerful mix. It gives companies the flexibility to build a platform tailored for their needs without re-developing every feature.
Import leads in your data hub, and get access to a catalog of thousands of APIs to enrich them, validate their information, or send them to dedicated tools like a CRM so the Sales team starts a selling process.
At Datablist, we work to make this vision a reality 🚀. <a href="https://www.datablist.com" target="_blank" rel="nofollow">Join the beta</a> (and get 30% off on launch!) or subscribe to the newsletter to know more 🤝.

# Introduction
Last month, [Twilio bought Segment.io](https://www.forbes.com/sites/alexkonrad/2020/10/09/twilio-to-acquire-cloud-startup-segment-for-3-billion/), a vertical SaaS software that focuses on customer data integration. This move by Twilio comes after buying Sendgrid last year and follows a bundling strategy. Twilio wants to be a 360° platform when it comes to customer value proposition.

[In a famous quote](https://hbr.org/2014/06/how-to-succeed-in-business-by-bundling-and-unbundling), Jim Barksdale said the two ways to make money is by bundling products together or unbundling them and creating targeted offers.

We see both in the software industry today. Microsoft, Salesforce, Oracle and other behemoths keep adding to their software suite, making them ubiquitous. **In the meanwhile, we can see an increase in niche SaaS products**. From image segmentation APIs to content sentiment analysis tools. **The number of SaaS products we use at work keeps increasing.**

Integrated suites have the advantage of an unified experience while specialized products take advantage of their expertise and can be laser focus on the problem they solve.

In my previous article ([where is enterprise software heading](/blog/where-is-enterprise-software-heading)), I discussed the evolutions in enterprise software as we have moved from ERP to a galaxy of SaaS products (also know as best of breed). This is an integrated suite vs best of breed debate we have now.

In a perfect world, we would be able to compose our own suite by making the different softwares work together. Unfortunately, we are not in a perfect world. Few months ago, a Microsoft sales contacted me to make us move from Slack to Microsoft Teams. I told him my experience with Skype Enterprise and Teams, from an external point of view, was a disaster. It was a hell to work with companies using those tools when you were on another suite (aka Google + Slack). He acknowledged it has not been strategic for Microsoft, but he assured the company started to work on more interoperability.

It might come from my desire to stay away from big corporations but I keep dreaming of a world where software collaboration could exist outside of big enterprise suites.

The number of specialized APIs is impressive. In my last count, I found about twenty companies providing an API for invoice PDF parsing, a good hundred of APIs to check the validity of an email address, and hundreds of companies/people enrichment tools.

Each of these APIs focuses on a single problem and addresses it well. But chances are you don't want to implement tens of different APIs to build an internal tool that would automate your accounting or your customer management. The question is: **how can we bundle them to create our own suite?**


# Why workflow automation (Zapier, Integromat, etc.) is not the best solution?

The first wave of tools to address this problematic are called "workflow automation tools" and Zapier, Mulesoft, Integromat, etc. are in this category.

![Integromat Workflow Automation](/blog_articles_images/2020/11/integromat-screenshot.png#1024x628)

The idea is simple: APIs are "Apps". They have inputs and outputs. Something goes in, the app does its job, and something else goes out. The workflow automation tool ensures the data moves from one app to the next one. Apps can be chained to make complex workflows.

While this solves a lot of problems, it is not suited for everything. **With workflow automation, the starting point is an event from an app (or a scheduled event)** and the workflow will move the data from one app to another one. **It acts as a plumbing pipe.**

It connects APIs together more than creating a bundle of products. Here are some limitations:

- **It is not possible to trigger manually an action for a specific data item.** Imagine you have an existing item in your CRM and you want to run it though an automation workflow you have. It is not possible without manually adding a new source event in your workflow.
- Data moves as a whole between apps. It is not possible to transform the value for one property and then merge it back to the data object to be send to the next workflow app. You would need a data storage layer in between.


# Connect Apps through a data hub

So how can we bundle APIs/Products together to create value? **The answer is: through data**. Instead of starting from app events like in workflow automation, the data must be at the center.

<div class="content-video">
 <video autoplay muted loop playsinline width="100%" height="auto" class="d-block">
 <source src="https://storage.googleapis.com/public_datablist/website/videos/run-action-datablist.mp4" type="video/mp4" />
 </video>
 <div class="text-center">Datablist screencast</div>
</div>

**Businesses need a centralized data hub for data related to customers, suppliers, employees, etc. from which partial data can be send to external APIs to be processed.** The results of this process then return to the data hub. It is about consolidating several APIs processes into a single data storage.

From this data hub, automations can be configured on data events like with workflow automation. When a new item (customer, invoice, etc.) is added to the data hub, actions run to process the new data and the result is saved for later use. But now, actions can be trigged manually directly from a data item. For example: send a mail to a customer.

**Building with a data first approach opens the possibility of data standardisation.** With workflow automation, every time you integrate an app (Application A) with another one (Application B), you need to define what properties from the data exposed by *Application A* will be used as input in *Application B*. Integration between two apps needs configuration. By starting from the data, it is not how one app integrates with another one, but how each app connects to the data. The data is the source for each integration.

If a data property is of type **email**, all APIs with an **email** input type can be automatically listed and shown.

# Own your platform and use the power of external APIs

The combo Data hub + APIs is a powerful mix. It gives companies the flexibility to build a platform tailored for their needs without re-developing every feature.

Import leads in your data hub, and get access to a catalog of thousands of APIs to enrich them, validate their information, or send them to dedicated tools like a CRM so the Sales team starts a selling process.

At Datablist, we work to make this vision a reality 🚀. [Join the beta](https://www.datablist.com) (and get 30% off on launch!) or subscribe to the newsletter to know more 🤝.

How to use a Data Hub to connect APIs


Since the beginning of the digital transformation in the 1960s, enterprise software has played a key role in increasing companies productivity. They have moved from custom development, to ERP, to the current best-of-breed approach. Automation and AI are around the corner to drive the next revolution. If the different shifts in technologies that happened in the past are any clues to look forward to, chances are the future of enterprise software will look very different than today.

# 1. The ERP age

Before the 1980s, most of enterprise software programs were custom built. The 1980s brought software standardization with applications developed to manage inventory and manufacturing processes in the industries. Applications spread to all the business needs including finance and human resources. In 1990, the advisory firm Gartner coined the term ERP (for "Enterprise Resource Planning") to refer to this category of business management software programs.

The following decade, the 1990s, was the golden age for ERPs with rapid growth in adoption across medium and large companies in a market dominated by firms like SAP, Oracle, and HP with on-premise solutions.

![A screenshot of Odoo, an open-source ERP](/blog_articles_images/2020/10/example-erp-screenshot-odoo.png#1100x668)

## 1.1. Where ERPs shine

### Data centralisation

The major innovation introduced with ERP is a central and real-time view of the company data. Inventory, Finance, HR and other applications share a unified data storage. This data consolidation enables cross departments analytics and simplifies company planning.

### Extensible

Moreover, ERPs are extensible. All the applications living in the ERP share a common underlying architecture and have the same interface components. All the ERP market leaders expose their platform to external developers. This creates opportunities for third parties applications, and internal processes that are too specific to the company can be built as applications on top of the platform.

## 1.2. ERP limitations

### Complexity

**This globally consistent view of company financials and inventory come at the expense of complexity.** A triple complexity! Complexity to implement, to use, and to extend.

ERP systems come in a one-size-fits-all monolith strategy. **To be implemented in a company, the company must comply with how the ERP works.** ERP applications are not flexible enough to map custom processes the company might be using. When you add that the implementation must cover several departments, therefore several applications, to be valuable, you end up with scary stories. In 2017, the German grocery chain Lidl, [after 6 years and $500 million spent in SAP implementation](https://www.handelsblatt.com/today/companies/programmed-for-disaster-lidl-software-disaster-another-example-of-germanys-digital-failure/23582902.html), decided to roll back the entire inventory system to their previous custom made platform. More recently in Australia, [a SAP implementation in Australia](https://www.youngwitness.com.au/story/6938588/qld-health-still-fixing-inventory-system/) cost 33,5 Millions $ (AUD) more than planned [with mixed results](https://www.theregister.com/2020/09/28/queensland_health_sap_s4hana_rollout/).

It is not surprising that small companies, looking for fast implementation and flexibility, are only a few to use ERPs.

When implemented, ERPs are hard to use. Because all applications are built with the same components, they all share the same User Interface patterns. **The interface is more a representation of the underlying data than a targeted interface for a defined application.**

To finish on the complexity issue, unlike the promise, extending an ERP is not easy. First of all, they are monoliths. Meaning all the applications and data live in a single system. Developing over it can impact the stability of other applications. **Motivation to develop an application on the ERP must be validated by the company.** A single team wouldn't take the responsibility to extend the company ERP without asking permission. In addition, if an existing application doesn't suit the organization, customizing the code prevents the organization from automatically updating the application when a new version is available. Resulting in maintenance time and instability.

![Someone, somewhere, trying to implement an ERP](/blog_articles_images/2020/10/erp-complexity.jpg#1000x656)


### Closed world

ERPs live in a closed world. They are only designed to interact with the company employees. **Suppliers, partners, and customers, although modeled in the ERP database, do not interact directly with it.**

The isolation is also true for the applications. Only applications designed for the ERP will play nicely with the data and the processes integrated with it. **Interconnecting an ERP with external HR or Finance software is complex and this is not the way ERP vendors want you to use their product.**

As a result, you need an SAP software developer to build an application on the SAP platform. Some technologies are proprietary and diverged from other programming languages and practices used elsewhere. As a result, salaries for ERP developers are higher than average.

### Lock-in

Long implementation, centralized data, and closed world inevitably introduce vendor lock-in. The average ERP life cycle in a company is 15 years (time between a first implementation and a new implementation). As a result, the market is concentrated in the hands of a few big players. And it is not to displease them.


# 2. The best of breed revolution

Starting in the 2000s, enterprises moved from vertically integrated organizations toward agile and domain specialized entities. **This move was accompanied by companies opening to the outside world, especially to interact with customers.** The one-size-fits-all ERP systems started to lose momentum in favor of specialized cloud-native applications. **From a single system, the organizations were now choosing to have different software programs, often from different vendors, to meet their ever-growing needs.** Over the past few years, an increasing number of companies have switched to this approach, called “best of breed”.

![SaaS Spending Evolution (source: Blissfully)](/blog_articles_images/2020/10/SaaSSpending.png#1100x618)


## 2.1. The benefits of best of breed

### Cloud Native

The rising of best of breed philosophy is concomitant with the rise of SaaS (Software as A Service) that live in the cloud. This new wave of enterprise software embraced the possibilities created by the cloud and offered a totally different experience.

**Compared to on-premise hosting, cloud-native applications are fast to implement.** The setup no longer involves buying servers and sizing the infrastructure but is reduced to configuration and interconnection with the company's other systems, shortening the time to market. Software update responsibility returns to the software vendor, saving the company maintenance cost.

It also changes the business model of enterprise software vendors. They switched from set up fees + product licenses to on-demand pricing. How many employees do you have? How many customers? **The goal is to capture value as the company grows.**

With this fast implementation and linear pricing, a myriad of new software editors was created to target all company size. From small companies to multinationals.

### Flexibility

Single-purpose applications bring flexibility to the company. Because it is hosted by the vendor, **decision to use an application is slowly pushed back to the team responsibility.** With companies embracing agile organization, teams are empowered to test and decide on their own the tools they want to use. Once implemented, if the results are insufficient, the team can switch to another vendor while keeping the other applications it might use. This “test fast, change fast” is another principle of agility and it is meant to accelerate innovation.

Most of the best-of-breed applications expose API to connect them with other applications. The enterprise processes are then scattered across multiple applications.


### Easy to use

Because applications address specific needs, **the user interface can be adapted to the intended user who is using the applications.** Applications meant for salespeople can have a different user interface than the ones for human resources people. With all the existing devices (smartphones, tablets, laptops, smartwatches, etc.), software vendors can offer personalized user experiences.

### Open to the external world

Finally, best-of-breed applications unveil a first step toward opening the enterprise software to the external world. **Where ERPs focuses on company members, applications now interact directly with external parties: customers, suppliers, partners, etc.**

This opening is still limited but the trend is accelerating.


## 2.2. Best of breed downsides

The best of breed approach comes with its own challenges, especially the consequences of having a system made of multiple programs, data storages, and vendors.

### Siloed Data

Ironically, while ERPs main advantage is a central data storage, the biggest issue with best-of-breed patterns is data silos. Every software is independent and has a dedicated database. **On a company scale, data silos reflect the different business units and teams.**

This has consequences. First, generic data types like customer information might be stored in a CRM only accessible for a part of the company, preventing people from having access to important information. And even if the CRM is connected to other programs to expose customer information, it can lead to data duplication and inconsistencies. This is referred to as the "source of truth" problem.

Finally, when the data is kept in several places, the company doesn't have a global view of its activity. Running analytics to find a correlation between data is not possible. Aligning vision, automation, analytics and planning get complicated without a global view of the data.

![Data Silos in an organization](/blog_articles_images/2020/10/data-silos.jpg#1000x480)

### Integration Complexity

In the best-of-breed world, software vendors focus on their application. **Complexity is moved to integrating the applications together.** How to connect data between applications? How to keep it secure? How to synchronize common objects (like customers) across them? These questions are left to the companies to answer.

In a 2019 survey conducted by Mulesoft, a provider of integration solutions, [IT leaders reported that only 29% of their applications are currently integrated together](https://www.mulesoft.com/press-center/technology-trends-2019-connectivity-benchmark).

There are diverse reasons for this lack of integration. Data models are different between applications. Customer attributes in a CRM can be different than the ones needed in a financial application. They are different and the data models can change. The frequency of deployments (and changes) is increasing with the use of agile methodologies adopted by more and more software vendors. Furthermore, some software vendors don't want to expose their data. In the data gold rush, it is better to receive data from other applications than expose its own data for others to create value with it.

**Companies are far from reaching a global view of their operations with this scattered data.**

![Number of applications per company has skyrocketed in the last years. (Source Blissfully 2019)](/blog_articles_images/2020/10/number-app-per-company.png#1100x526)

### Hard to build on top of an applications mesh

From an ERP market with few big players, the best-of-breed approach has enabled new vendors to join the enterprise software dance. This leads to vendors offering specialized applications to many niche markets.** Yet, often, an organization's digital system can't be only made of vendor applications. Custom development is needed to answer specific needs.**

Best of breed makes it difficult to build a custom application on top of this distributed system. Integration with vendor applications is not always possible. **Some applications offer packaged integration with other market leaders but don't open their API to custom development.** Hosting, testing, and maintenance lead to high cost for internal application development.


### Duplication of costs

To conclude on the downsides, the costs of all the applications can add up to become quite expensive. Keeping track of all the applications used in the company is not easy, some vendor licenses keep being paid while not used. Hundred, or thousands of business applications can drain down the profitability of the organization.

![Number of applications paid several times by different teams. (Source Blissfully 2019)](/blog_articles_images/2020/10/duplication-cost.png#1100x629)



# 3. The future of enterprise software

Marc Andreessen famously said in 2011 that ["software is eating the world"](https://a16z.com/2011/08/20/why-software-is-eating-the-world/). Companies that embrace software both in their operations and as a driver of growth will continue to lead the digital transformation. Predicting the evolution of enterprise software is not easy, but we can look at the current trends to predict what will happen in the market.

The following paragraphs are what we, [at Datablist](/blog/announcing-datablist), believe will shape the future of enterprise software. We can see four strong tendencies:

- **Agility and the power of autonomous teams.**
- **AI, automation and analytics in data-driven organizations.**
- **Leveraging collaboration with the external world**
- **Security concerns in a competitive and hostile world**


## 3.1. Agility and the power of autonomous teams

Agility can be defined as an employee's capacity to analyze information in the environment and respond to that information quickly and autonomously. **It has been proven that decentralization is key to maintaining innovation and adaptation in an ever-changing world.**

We believe the enterprise software landscape will continue to be trusted by best-of-breed vendors instead of all-in-one ERP solutions.

- **SaaS solution living in the cloud and pay as you go**. Flexibility and fast implementation is where SaaS software shine. Cloud-native applications bring automatic updates, few maintenance costs, and predictable pricing. Data ownership and security with the cloud are growing concerns for companies but the SaaS vendors are adapting to those needs and we see no reason the cloud would lose it current momentum.
- **Customer-centric experience**. Agility means the buyers are now the users. Developers can choose the tools they want to use, salespeople can decide what CRM they will work with. This need for flexibility can only be achieved with the best-of-breed approach. They choose their products based on online reviews and network feedback. They are informed buyers and want to choose and not be sold to. User experience and design will be more and more important in enterprise software.



## 3.2. AI, automation, and analytics in data-driven organizations

Productivity increase in an organization comes from both employee productivity gain and automation of organization tasks. Automation relies on data. **The businesses that will benefit from automation are the ones that will build a consolidated data view on top of their mesh of best-of-breed applications.**

### A centralized data space

Connecting best-of-breed applications together is not enough. First, you are limited by the connectors the applications provide. Applications have connectors to market leaders. But what about small vendors? Worse, what about your internal custom applications? **Second, it is not in the best-of-breed vendor's DNA to make sense of the other application's data.**

> Your CRM doesn’t care about your employees data

A centralized data space gives the organization a consolidated view over the data, whether it is users generated data or organization-generated data.

**We believe enterprise data warehouses will gain popularity as a foundation for automation and internal custom applications.**


### Platform to build on top of the data

Despite the ever-growing number of enterprise software programs, there will still be times where custom development would make more sense. Those internal applications will sit on top of the centralized data space and use the data to deliver value to the organization.

Modern development tools are shortening the development life cycle. Serverless and container technologies are perfectly suited to embrace this trend.

**Components and APIs that solve a single purpose will work as building blocks for those internal applications. OCR technologies, text analysis, company verification services are generic needs that will be provided off the shelf.**

![A centralized data space for unified analytic views and automation (Datablist)](/blog_articles_images/2020/10/datablist-centralized-data-space.png#790x666)

## 3.3. Leveraging collaboration with the external world

Innovation comes when the best idea wins regardless of whether the idea comes from. This first means organization must aggregate insights and ideas from everywhere. We have seen companies opening up to customers to listen to their feedback. This trend will accelerate. Contributions can come from employees, customers but also vendors or partners. **Organizational boundaries will be porous so all the community can participate in the company mission and be involved in the decision-making process.**

We already see tech companies betting on open source, sharing their working methodology, the design patterns, and overall being transparent on what they learn. It engages employees and creates purpose.

Enterprise applications will foster collaboration, with people inside the company and outside of it.

## 3.4. Security concerns in a competitive and hostile world

The more organizations become digital and open, the more they are vulnerable. Losing trust from customers in this flexible market is not acceptable.

Organizations will use automatic tools to constantly audit their assets and vulnerabilities. Developer's codes, vendor applications, and products will be mapped in a global view of the organization systems.


# Conclusion

As companies move from a single one-size-fits-all ERP world to a galaxy of SaaS applications, new problems arise:

- **Data is siloed** between Sales, R&D, Operations, etc.
- Processes based on multiple data sources **cannot be automated**
- **Companies can’t build** applications over this mesh of SaaS tools

[At Datablist](/blog/announcing-datablist), we believe those problems can be solved with a secure centralized data space coupled with a platform to build custom applications and automation.

If you share our vision, subscribe to our newsletter below and follow us on [Twitter](https://twitter.com/datablist) 🤝.


Where is enterprise software heading?

Datablist idea comes from a personal frustration in my previous company. I was a Co-Founder and the CTO of an e-commerce software company. One of the goals for the product team was to automate everything redundant. We had built a development pipeline to streamline and automate all the actions between a developer code push and the deployment in production of our software. On every deployment, a Slack bot pushed the release note it has built on its own using the Jira backlog. The DevOps philosophy we followed encourages automation to speed deployment and to remove human error from it. My frustration came from the gap between the level of automation we had in our software development processes compared with all the manual tasks that needed to be done to "run" the company. We used tools for sales, human resources, customer management, etc. Those tools accentuate the existing organization silos by keeping control over the data.
I'm a coder and I tried to aggregate the data by building data bridges and workflows using Google Spreadsheet, Zapier, or other workflow automation software. I found myself to be quickly limited by those tools.
I needed to have more control over the data and I had to store it in an external database, with computing instances to read files and images. I finally gave up when I realized I was building a web infrastructure as complex as the one we had for our e-commerce software.
I thought about this problem for a long time. Now that we have digitized everything from supplier invoices to customer contracts, why don't we move toward more automation for the organization processes?
<ul>
<li>Supplier invoices scattered in our emails or their websites must be gathered together</li>
<li>Employees onboarding with welcoming emails, account configurations must be automated</li>
<li>Reconciliation between invoices and bank transactions must be automatic</li>
<li>Customers and business data must be stored in a data warehouse fully accessible to the whole company</li>
</ul>
Same as in a software development team, the organization processes and repetitive tasks must move to a digital operating model. A digital representation of the processes made with software and algorithms.
This vision cannot simply be done with pipes that connect software programs together. Nor with bots that mimic human actions. Complete access to the data and a platform to create tailored algorithms on this data is mandatory.
A digital operating model is infinitely scalable, can be improved at an impressive rate, and is more resilient to human error. The organization data, when aggregated and consolidated, give people information about the situation so they can take initiative and build on this data platform. This is what Agile organizations are about.
It is about moving human activity from performing the processes to designing and monitoring the automation applications that perform the processes.
This is just the beginning. If you share our vision and are interested in influencing the direction of Datablist, subscribe to our newsletter below and follow us on <a href="https://twitter.com/datablist" target="_blank" rel="nofollow">Twitter</a> 🤝.

Datablist idea comes from a personal frustration in my previous company. I was a Co-Founder and the CTO of an e-commerce software company. One of the goals for the product team was to automate everything redundant. We had built a development pipeline to streamline and automate all the actions between a developer code push and the deployment in production of our software. On every deployment, a Slack bot pushed the release note it has built on its own using the Jira backlog. The DevOps philosophy we followed encourages automation to speed deployment and to remove human error from it. **My frustration came from the gap between the level of automation we had in our software development processes compared with all the manual tasks that needed to be done to "run" the company.** We used tools for sales, human resources, customer management, etc. **Those tools accentuate the existing organization silos by keeping control over the data.**

I'm a coder and I tried to aggregate the data by building data bridges and workflows using Google Spreadsheet, Zapier, or other workflow automation software. I found myself to be quickly limited by those tools.
I needed to have more control over the data and I had to store it in an external database, with computing instances to read files and images. I finally gave up when I realized I was building a web infrastructure as complex as the one we had for our e-commerce software.

I thought about this problem for a long time. Now that we have digitized everything from supplier invoices to customer contracts, why don't we move toward more automation for the organization processes?


- Supplier invoices scattered in our emails or their websites must be gathered together
- Employees onboarding with welcoming emails, account configurations must be automated
- Reconciliation between invoices and bank transactions must be automatic
- **Customers and business data must be stored in a data warehouse fully accessible to the whole company**


Same as in a software development team, the organization processes and repetitive tasks must move to a digital operating model. A digital representation of the processes made with software and algorithms.

**This vision cannot simply be done with pipes that connect software programs together. Nor with bots that mimic human actions. Complete access to the data and a platform to create tailored algorithms on this data is mandatory.**

**A digital operating model is infinitely scalable**, can be improved at an impressive rate, and is more resilient to human error. **The organization data, when aggregated and consolidated, give people information about the situation so they can take initiative and build on this data platform.** This is what Agile organizations are about.

It is about moving human activity from performing the processes to designing and monitoring the automation applications that perform the processes.

**This is just the beginning. If you share our vision and are interested in influencing the direction of Datablist, subscribe to our newsletter below and follow us on [Twitter](https://twitter.com/datablist) 🤝.**


Blog