On Specializing AKA Databases Are Easy Now

After hanging my shingle in the independent consulting business — trying to pay the bills as a financing for entrepreneurial efforts[1] – one of my original specialties was high-performance database systems.

I pitched my expertise primarily in large-scale high-demand database systems, providing solutions that would dramatically improve operational performance and relieve pain points. Just as a facet of my network of contacts this was usually in the financial industry.

It made for relatively easy work: Many groups don’t understand how to properly leverage the available database products, their data needs, or lever it to their query profile, so I could often promise at least a 50x performance improvement for some particular pain point, on existing hardware, and achieve that and more with ease. Everyone happy, achievement unlocked, etc.

I started backing away from the database world, however, because many database products have dramatically improved and reduced the ability to shoot one’s own feet, but more significantly hardware has exponentially improved.

From massive memory, caching the entirety of all but the most enormous databases, to flash storage (and now Optane storage) that makes the most inefficient of access patterns still effectively instant. Many have gone from a big expensive RAID array that could barely manage 1000 IOPS, on a platform with maybe 32GB of RAM, to monster servers with many many millions of IOPS and sometimes TBs of very high speed memory. Even the bottom feeder single-instance AWS services are usually fairly adequately equipped.

It has gotten hard to build a poorly performance database system, at least at the scale that many solutions operate at. Expert implementations can still yield significant speedups, but to many potential customers a 100ms query is effectively identical to a <1ms query, with low enough loading that hardware saturation isn’t a concern (or the hardware so inexpensive that a cluster of ten huge machines is fine despite being significant overkill if properly implemented). The most trivial of caching and even a modicum of basic knowledge (e.g. rudimentary indexes) is usually enough to achieve workable results.

Egregious inefficiency is operable. Which is a great thing in many cases, and is similar to our move to less efficient runtimes and platforms across the industry. There are still the few edge situations demanding extreme efficiency, but as with assembly programming it just isn’t enough to demand a focus for many.

The possible market for database optimization just isn’t as large or as lucrative. And when things are awry on that millions of IOPS system with 2TB of RAM and 96 cores, it’s usually so catastrophically, fundamentally broken that it isn’t really a fun pursuit, usually necessitating a significant rebuild that no one wants to hear as the answer.

Very large or specialty shops still have high performance database needs, but naturally those orgs have specialized internal staff.

Being an employee in a very narrow niche doesn’t interest me.

For the past half year I’ve moved primarily to cryptocurrencies and their related technologies (e.g. public blockchain implementations and standards, as well as payment systems). Not chasing a “whale” (or having anything to do with the enormous bubble of some currencies), or hoping to ride a bubble to great riches (I’ve traded $0 in fiat currencies to cryptocurrency — any that I have I’ve got through other mechanisms — and see the current market as incredibly dangerous and vulnerable to incredible variations on the slightest push), but rather becoming completely knowledgeable of every emergent standard, and of the currently dominant platforms and their source code, and offering that knowledge to individuals and firms looking to integrate, to leverage, or to bet. It has been rewarding. My specialization has been in high scale, extremely high performance and low latency solutions.

All the while I continue with the entrepreneurial efforts. My latest is with the iOS ARKit, which has been a fun experience. I should release something on that soon. I also revived a video stabilization project after spending some time on the machine learning side of that field.

Otherwise I continue with fun projects. I still have a contour mapping application on the go. It started as a real estate sales project, but that went so quickly to great success that it became just an interesting project to explore Kotlin, undertow, targeting iOS and Android and then learning about pressure propagation, etc. Then WebGL mapping, millisecond time synchronization (I use four current smartphones in concert), etc. I have a mostly complete posting on that, long promised, that I’ll finish soon.

[1] – I hit some financial roadblocks, leading to a crisis, when I put aside all revenue production that try to focus entirely on entrepreneurial efforts that seemed, for so long, on the absolute cusp of success. Lessons learned.

3D XPoint is Pretty Cool

Five years ago I wrote a post about SSDs/flash storage being a necessary ingredient for most modern build outs, and opponents were wasting time and efforts by not adopting them in their stack. While it is profoundly obvious now, at the time there was a surprising amount of resistance from many in the industry who were accustomed to their racks of spinning rust, RAID levels, and so on. People who had banked their profession on a lot of knowledge about optimizing against extremely slow storage systems (a considerable factor in the enthusiasm for NoSQL), so FUD ruled the day.

Racks of spinning rust still have a crucial role in our infrastructure, often treated as almost nearline storage (stuff you seldom touch, and when you do the performance is so out of bounds of normal expectations). But many of our online systems are worlds improved with latency in microseconds instead of milliseconds courtesy of flash. It changed the entire industry.

In a related piece I noted that “Optimizing against slow seek times is an activity that is quickly going to be a negative return activity.” This turned out to be starkly true, and many efforts that were undertaken to engineer around glacially slow magnetic and EBS IOPS ended up being worse than useless.

We’re coming upon a similar change again, and it’s something that every developer / architect should be considering because it’s about to be real in a very big way.

3D XPoint, co-developed by Micron and Intel (the Intel one has some great infographics and explanatory videos), is a close to RAM-speed, flash-density, non-volatile storage/memory technology (with significantly higher write endurance than flash, though marketing claims vary from 3x to as high as 1000x), and it’s just about to start hitting the market. Initially it’s going to be seen in very high performance, non-volatile caches atop slower storage: the 2TB TLC NVMe with 32GB of 3d xpoint non-volatile cache (better devices currently have SLC flash serving the same purpose), offering extraordinary performance, both in throughput and IOPS / latency, while still offering large capacities.

Over a slightly longer period it will be seen in DRAM-style, byte-accessible form (circumventing the overhead of even NVMe). Not as literally the main memory, which still outclasses it in pure performance, but as an engineered storage option where our databases and solutions directly and knowingly leverage it in the technology stack.

2017 will be interesting.

Micro-benchmarks as the Canary in the Coal Mine

I frequent a number of programming social news style sites as a morning ritual: You don’t have to chase every trend, but being aware of happenings in the industry, learning from other people’s discoveries and adventures, is a useful exercise.

A recurring source of content are micro-benchmarks of some easily understood sliver of our problem space, the canonical example being trivial web implementations in one’s platform of choice.

A Hello World for HTTP.

package main

import (
   "fmt"
   "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
   fmt.Fprintf(w, "Hello world!")
}

func main() {
   http.HandleFunc("/", handler)
   http.ListenAndServe(":8080", nil)
}

Incontestable proof of the universal superiority of whatever language is being pushed. Massive numbers of meaningless requests served by a single virtual server.

As an aside that I should probably add as a footnote, I still strongly recommend that static and cached content be served from a dedicated platform like nginx (use lightweight unix sockets to the back end if on the same machine), itself very likely layered by a CDN. This sort of trivial type stuff should never be in your own code, nor should it be a primary focus of optimizations.

Occasionally the discussion will move to a slightly higher level and there’ll be impassioned debates about HTTP routers (differentiating URLs, pulling parameters, etc, then calling the relevant service logic), everyone optimizing the edges. There are thousands of HTTP routers on virtually every platform, most differentiated by tiny performance differences.

People once cut their teeth by making their own compiler or OS, but now everyone seems to start by making an HTTP router. Focus moves elsewhere.

In a recent discussion where a micro-benchmark was being discussed (used to promote a pre-alpha platform), a user said in regards to Go (one of the lesser alternatives compared against)-

it’s just that the std lib is coded with total disregard for performance concerns, the http server is slow, regex implementation is a joke”

total disregard. A jokeSlow.

On a decently capable server, that critiqued Go implementation, if you’re testing it in isolation and don’t care about doing anything actually useful, could serve more requests than seen by the vast majority of sites on these fair tubes of ours. With a magnitude or two to spare.

100s of thousands of requests per second is simply enormous. It wasn’t that long ago that we were amazed at 100 requests per second for completely static content cached in memory. Just a few short years ago most frameworks tapped out at barely double digit requests per second (twas the era of synchronous IO and blocking a threads for every request).

As a fun fact, a recent implementation I spearheaded attained four million fully robust web service financial transactions per second. This was on a seriously high-end server, and used a wide range of optimizations such as a zero-copy network interface and secure memory sharing between service layers, and ultimately was just grossly overbuilt unless conquering new worlds, but it helped a sales pitch.

Things improve. Standards and expectations improve. That really was a poor state of affairs, and not only were users given a slow, poor experience, it often required farms of servers for even modest traffic needs.

Choosing a high performance foundation is good. The common notion that you can just fix the poor performance parts after the fact seldom holds true.

Nonetheless, the whole venture made me curious what sort of correlation trivial micro-benchmarks hold to actual real-world needs. Clearly printing a string to a TCP connection is an absolutely minuscule part of any real-world solution, and once you’ve layered in authentication and authorization and models and abstractions and back-end microservices and ORMs and databases, it becomes a rounding error.

But does it indicate choices behind the scenes, or a fanatical pursuit of performance, that pays off elsewhere?

It’s tough to gauge because there is no universal web platform benchmark. There is no TPC for web applications.

The best we have, really, are the TechEmpower benchmarks. These are a set of relatively simple benchmarks that vary from absurdly trivial to mostly trivial-

  • Return a simple string (plaintext)
  • Serialize an object (containing a single string) into a JSON string and return it (json)
  • Query a value from a database, and serialize it (an id and a string) into a JSON string and return it (single query)
  • Query multiple values from a database and serialize them (multiple queries)
  • Query values from a database, add an additional value, and serialize them (fortunes)
  • Load rows into objects, update the objects, save the changes back to the database, serialize to json (data updates)

It is hardly a real world implementation of the stacks of dependencies and efficiency barriers in an application, but some of the tests are worlds better than the trivial micro-benchmarks that dot the land. It also gives developers a visible performance reward, just as Sunspider led to enormous Javascript performance improvements.

So here’s the performance profile of a variety of frameworks/platforms against the postgres db on their physical test platform, each clustered in a sequence of plaintext (blue), JSON (red), Fortune (yellow), Single Query (green), and Multiple Query (brown) results. The vertical axis has been capped at 1,000,000 requests per second to preserve detail, and only frameworks having results for all of the categories are included.

When I originally decided that I’d author this piece, my intention was to actually show that you shouldn’t trust micro-benchmarks because they seldom have a correlation with more significant tasks that you’ll face in real life. While I’ve long argued that such optimizations often indicate a team that cares about performance holistically, in the web world it has often been the case that products that shine at very specific things are often very weak in more realistic use.

But in this case my core assumption was only partly right. The correlation between the trivial micro-benchmark speed — simply returning a string — and the more significant tasks that I was sure would be drown out by underlying processing (when you’re doing queries at a rate of 1000 per second, an overhead of 0.000001s is hardly relevant), is much higher than I expected.

  • 0.75 – Correlation between JSON and plaintext performance
  • 0.58 – Correlation between Fortune and plaintext performance
  • 0.646 – Correlation between Single query and plaintext performance
  • 0.21371 – Correlation between Multiple query and plaintext performance

As more happens in the background, outside of the control of the framework, invariably the raw performance advantage is lost, but my core assumption was that there would be a much smaller correlation.

So in the end this is simply a “well, that’s interesting” post. It certainly isn’t a recommendation for any framework or the other — developer aptitude and suitability for task reign supreme — but I found it interesting.

 

Everything You Read About Databases Is Obsolete

Six and a half years ago I wrote (in a piece about NoSQL) –

Optimizing against slow seek times is an activity that is quickly going to be a negative return activity.

This time has long since passed, yet much of the dogma of the industry remains the same as it was back when our storage tier was comprised of 100 IO/second magnetic drives. Many of our solutions still have query engines absolutely crippled by this assumption (including pgsql. mssql has improved, but for years using it on fast storage was as exercise in waste as it endlessly made the wrong assumptions around conserving IOs).

There are now TB+ consumer storage solutions with 300,000 IO/second (3000x the classic magnetic drive, while also offering sequential rates above 3.5GB/s…yes, big G) for under $600. There are enterprise solutions serving 10,000,000 IOPS.

That’s if your solution even needs to touch the storage tier. Memory is so inexpensive now, even on shared hosting like AWS, that all but the largest databases sits resident in memory much of the time. My smartphone has 3GB, and could competently host the hot area of 99.5%+ of operational databases in memory.

For self-hosted hardware, TBs of memory is now economical for a small business, while tens of GBs is operationally inexpensive on shared hosting.

I totally made up that 99.5% stat, but it’s amazing how relatively tiny the overwhelming bulk of databases I encounter in my professional life are, yet how much so many fret about them.

Obviously writes still have to write thru, yet when you remove the self-defeating tactic of trying to pre-optimize by minimizing read IO — eliminating denormalization, combined-storage (e.g. document-oriented), materializations and trigger renderings/precomputing, excessive indexes, etc — in most cases writes narrow to a tiny trickle1.

When writes reduce, not only does practical write performance increase (given that you’re writing much less per transaction, the beneficial yield increases), the effectiveness of memory and tier caches increases as the hot area shrinks, and the attainability of very high performance storage options improves (it’s a lot more economical buying a 60GB high performance, reliable and redundant storage tier than a 6TB system. As you scale up data volumes, often performance is sacrificed for raw capacity).

Backups shrink and systems become much more manageable. It’s easy to stream a replica across a low grade connection when the churn of change is minimized. It’s easy to keep backups validated and up to date when they’re GBs instead of TBs.

Normalize. Don’t be afraid of seeks. Avoid the common pre-optimizations that are in the whole destructive to virtually every dimension of a solution on modern hardware (destroying write performance, long-term read performance, economics, maintainability, reliability). Validate assumptions.

Because almost everything written about databases, and from that much of what you read, is perilously outdated. This post was inspired when seeing another database best practices guideline make the rounds, most suggestions circling a very dated notion that every effort should be made to reduce IOs, the net result being an obsolete, overwrought solution out of the gates.

1 – One of the most trying aspects of writing technical blog entries is people who counter with edge cases to justify positions: Yes, Walmart has a lot of writes. So does Amazon, CERN and Google too. The NY Taxi Commission logs loads of data, being in a city area of tens of millions.

There are many extremely large databases with very specialized needs. They don’t legitimize the choices you make, and they shouldn’t drive your technical needs.

Quick Update. Upcoming Post.

I haven’t posted much for a bit due to some heavy commitments with a fantastic client, however I’ve pulled an old post out of the drafts — essentially the long-delayed part III of the high performance SQL series from eight years ago — and will finish it up imminently.

It is tentatively titled “The Absolute Minimum Every Developer Should Know About Indexes“, and includes a demonstration database that you can run in PostgreSQL.

On other matters, occasionally I look at the sorts of traffic that bring people here, and I’m always amazed at some of the Google search results. Not only that Google sends so many people here, but in how incredibly obsolete technologies hang on.

A frequent search hit, for instance, is regarding the internet provider for Visual SourceSafe. I’d authored a piece almost a decade ago, it being mangled in the moves between various blog engines, but somehow it still yields some search juice.

To which I would say: People still use SourceSafe? With all of the incredibly powerful options freely and easily available, they’re still working with that derelict, obsolete, broken source control provider?

Though of course this isn’t entirely surprising. Many teams and shops have an inability to decouple from past decisions, and those things just hang around for perpetuity. How those things usually happen is that a team or company or product gets loaded with more and more baggage, zero effort or expense paid to continuously refine and remove accumulating technical debt, until eventually the solution is so overwrought and broken that the the entire thing is wholesale abandoned. Not as some voluntary choice, but rather the detritus slows it down the point that eventually it hits a breaking point and involuntary actions are taken: The internal team and product replaced with a greenfield product of an outside consultancy, for instance. It is the ballistic arc of so many projects, where necessary choices and actions aren’t taken through the lifecycle, until the problem is so great that abandonment is the only recourse.

Always be re-evaluating the stack that you use. This isn’t a call to blindly follow what is new, but honestly if you’re still using products like Visual Sourcesafe for anything, there is probably an issue.