The persistence of database time

Database habits die hard. Just ask Oracle, which continues to rake in billions in database revenue despite being one of developers’ “most dreaded” databases, according to Stack Overflow’s 2021 survey of 72,517 developers. But let’s not focus on the negative. Just like last year (and 2019, 2018, 2017, and 2016), Redis topped the charts as developers’ “most loved” database, followed closely by PostgreSQL and MongoDB. Go back to 2017 and the headliners on this database love-in are largely the same, though SQL Server has fallen down the rankings since then, and Google’s Firebase has climbed up.

Developer preferences in web frameworks may change relatively frequently, but databases are sticky. As Gartner’s Merv Adrian once said, “The greatest force in legacy databases is inertia.” Hence, although it takes a long time to establish a new database, it takes even longer for a once-loved database to finally get dumped. Even when developers move on, their employers don’t.

In short, it’s hard to make accurate technology predictions, but here’s one you can bank on: The databases developers love today will be the ones that permeate enterprises 10 years from now.

The more things change...

Developers have never had more options in databases. According to DB-Engines, there are currently 373 different databases. The most popular are old standards such as Oracle, MySQL, and Microsoft SQL Server, as well as relative newcomers like MongoDB and Redis. (PostgreSQL has been around forever but has only recently grown in popularity.)

But what does it mean for a database to be popular, anyway?

DB-Engines calculates database popularity by tallying up searches, mentions on sites like Stack Overflow, job postings, LinkedIn references, and more. While not perfect, DB-Engines does a reasonably good job of estimating how broadly used a database is. Databases such as Oracle, SQL Server, and IBM’s DB2 are popular in the sense of being broadly used, even though that usage has declined relative to newcomers like MongoDB ever since DB-Engines started tracking databases in 2012.

Here’s what the top 10 rankings looked like in 2012:

1. Oracle
2. Microsoft SQL Server
3. MySQL
4. Microsoft Access
5. IBM DB2
6. PostgreSQL
7. MongoDB
8. SQLite
9. Cassandra
10. Memcached

Here’s what it looks like today:

1. Oracle
2. MySQL
3. Microsoft SQL Server
4. PostgreSQL
5. MondoBD
6. Redis
7. IBM DB2
8. Elasticsearch
9. SQLite
10. Microsoft Access

Same cast of characters, right? Roughly nine years have passed, but we’re still talking about the same databases with few exceptions (Apache Cassandra is just off screen in 2021, at number 11). What’s interesting, however, are the significant changes in relative popularity or usage. Database shifts are measured in decades, not years, but there’s a very clear shift away from legacy databases such as Oracle, DB2, and SQL Server in favor of PostgreSQL, MongoDB, and Redis.

Don’t believe me? OK. Let’s ask developers.

Feel the developer love

I suspect the margin of error on Stack Overflow’s survey of more than 70,000 developers is relatively low, given the sample size. When asked which databases they love and which they dread, look who shows up at the top and bottom of the list:

Stack Overflow

Even though we rightly hear a lot about graph, time series, and other such databases today, those that make the loved or loathed list have been around for a while. Not surprisingly, the ones that developers love are the same ones making popularity/usage gains on DB-Engines for the past nine years. Also not surprisingly, the ones that developers dread are the same ones dropping on DB-Engines.

Stack Overflow provides an additional crystal ball view into the future of databases: the “wanted” list. Back in 2017 (the first year Stack Overflow asked about databases), developers’ most wanted databases were MongoDB, PostgreSQL, and Redis, in that order. In 2021, those positions have hardly changed. Of the 72,517 developers surveyed, here are the percentages of developers who are not developing with a particular database but have expressed interest in it:

Stack Overflow

At one time, Oracle was beloved, and rightly so. Oracle and other database stalwarts such as IBM and Microsoft transformed enterprises with relational databases. Enterprises may not love working with Oracle the company, but they’ve loved its database for decades.

Yet developers are embracing other options now, including open source spins on relational databases (PostgreSQL, MySQL) or dramatically better speed and scale with MongoDB or Redis. As developers look to innovate for their employers, they’re exploring the host of new options, but they’re also settling into just a few new-school databases. When I say new, I don’t really mean recent. PostgreSQL has been around in some form since 1986. MongoDB since 2007. Redis since 2009. It has taken more than a decade for these and others in their camp (like Cassandra) to cement themselves as popular, safe choices for developers.

Will it take another 10 years for these to fully become enterprise defaults? Perhaps. Database changes, as I mentioned, simply move slowly, given that IT views database replatforming of existing workloads as risky. This often means new database alternatives tend to see adoption in greenfield opportunities.

However slowly, it’s definitely happening, fueled by developers. Maybe this means that in 20 years, today’s most loved databases will become the most dreaded. Maybe that’s the natural cycle of IT. That’s a lot of maybes. What we know with some certitude however, is that today’s most loved databases will become tomorrow’s enterprise defaults.