Back to the (SQL) Future, Part 3

In the first post in this series, I discussed the emergence of NoSQL to address the need to make databases compatible with cloud needs. And in my last blog post, I talked about the dichotomy between smart databases and smart applications. I asked whether we can have smart databases that can scale on the cloud. So now let’s talk about why I think the answer is yes.

The Big Data revolution was substantially powered by Hadoop, initially a combination of a distributed store and the new-fangled Map/Reduce query model. But in recent years Hadoop has evolved from being the Map/Reduce alternative to SQL data warehouse technologies into a stack that strongly embraces SQL.  In the Big Data analytics world there has been a substantial return to SQL.

Operational database technologies are following a similar pattern, although the technical challenges are greater in the operational database world of high-frequency concurrent updates.  It is much harder to retrofit SQL to a NoSQL operational store than it is to add a SQL querying capability to a Hadoop cluster or data lake, but we are seeing the trend nonetheless.

NoSQL vendors are seeking to add both SQL and ACID transactions to their offerings. They are aiming to deliver Elastic SQL systems: The elasticity of NoSQL but with the power of SQL.  

Additionally SQL vendors and service providers are adding NoSQL features and cloud operational models to their offerings.  These systems increasingly offer the traditional benefits of NoSQL products.  They can easily handle unstructured data, have flexible or dynamic schemas, have excellent redundancy to support HA/DR, and provide an excellent developer experience.

And we are seeing major cloud vendors investing in SQL-on-cloud operational database services with varying degrees of cloud-native behavior.  These fall into the categories of:

1. Lift-and-shift deployments of standard RDBMS systems, with cloud APIs for provisioning and management (eg Amazon RDS, Microsoft SQL Server Managed Instances)
2. Standard RDBMS systems with limited elastic features (eg Amazon Aurora, Microsoft Azure SQL Database)
3. Cloud-native or cloud-focused SQL engines (eg Google Spanner, Cockroach Labs, NuoDB)

Of course there are many applications for which NoSQL data stores are the right solution, and that will always be the case.  Nonetheless what we are seeing in the market is a market convergence towards database systems that offer the benefits of SQL and ACID transactions as well as the cloud-style characteristics of elasticity, automation, and always-on operation.

Return to smart databases

The starting point for this discussion was about how the CIO should think about their database strategy.  The following is where things have been and are going:

1. We have been through some decades in which it has been a straightforward choice of which RDBMS platform to standardize on (Oracle, Microsoft, IBM).  
2. In recent years there has been an application-specific choice between smart databases and smart applications (ie between SQL and NoSQL).
3. Technology solutions are now in the market that offer elastic scalability whilst delivering all the benefits of simple applications with smart databases.

The most economically attractive model for application development is to simplify applications by deploying database systems with significant server-side processing capability, and with strong guarantees about your data.  The industry standard way of doing that is to use SQL with ACID transactions.  

In a cloud era, with a higher premium on elastic scale-out, these server-side capabilities were seemingly incompatible with many modern applications.  But technology has caught up with the new data center paradigms, and the simple-applications/smart-databases model has become a reality on elastic infrastructure.

We should expect the Simpler Applications/Smarter Databases trend to continue, which is good news for the budget conscious CIO. And additional good news is the ability to leverage application development skills, DBA skills, tool chains, and data governance processes.