After hanging my shingle in the independent consulting business — trying to pay the bills as a financing for entrepreneurial efforts – one of my original specialties was high-performance database systems.
I pitched my expertise primarily in large-scale high-demand database systems, providing solutions that would dramatically improve operational performance and relieve pain points. Just as a facet of my network of contacts this was usually in the financial industry.
It made for relatively easy work: Many groups don’t understand how to properly leverage the available database products, their data needs, or lever it to their query profile, so I could often promise at least a 50x performance improvement for some particular pain point, on existing hardware, and achieve that and more with ease. Everyone happy, achievement unlocked, etc.
I started backing away from the database world, however, because many database products have dramatically improved and reduced the ability to shoot one’s own feet, but more significantly hardware has exponentially improved.
From massive memory, caching the entirety of all but the most enormous databases, to flash storage (and now Optane storage) that makes the most inefficient of access patterns still effectively instant. Many have gone from a big expensive RAID array that could barely manage 1000 IOPS, on a platform with maybe 32GB of RAM, to monster servers with many many millions of IOPS and sometimes TBs of very high speed memory. Even the bottom feeder single-instance AWS services are usually fairly adequately equipped.
It has gotten hard to build a poorly performance database system, at least at the scale that many solutions operate at. Expert implementations can still yield significant speedups, but to many potential customers a 100ms query is effectively identical to a <1ms query, with low enough loading that hardware saturation isn’t a concern (or the hardware so inexpensive that a cluster of ten huge machines is fine despite being significant overkill if properly implemented). The most trivial of caching and even a modicum of basic knowledge (e.g. rudimentary indexes) is usually enough to achieve workable results.
Egregious inefficiency is operable. Which is a great thing in many cases, and is similar to our move to less efficient runtimes and platforms across the industry. There are still the few edge situations demanding extreme efficiency, but as with assembly programming it just isn’t enough to demand a focus for many.
The possible market for database optimization just isn’t as large or as lucrative. And when things are awry on that millions of IOPS system with 2TB of RAM and 96 cores, it’s usually so catastrophically, fundamentally broken that it isn’t really a fun pursuit, usually necessitating a significant rebuild that no one wants to hear as the answer.
Very large or specialty shops still have high performance database needs, but naturally those orgs have specialized internal staff.
Being an employee in a very narrow niche doesn’t interest me.
For the past half year I’ve moved primarily to cryptocurrencies and their related technologies (e.g. public blockchain implementations and standards, as well as payment systems). Not chasing a “whale” (or having anything to do with the enormous bubble of some currencies), or hoping to ride a bubble to great riches (I’ve traded $0 in fiat currencies to cryptocurrency — any that I have I’ve got through other mechanisms — and see the current market as incredibly dangerous and vulnerable to incredible variations on the slightest push), but rather becoming completely knowledgeable of every emergent standard, and of the currently dominant platforms and their source code, and offering that knowledge to individuals and firms looking to integrate, to leverage, or to bet. It has been rewarding. My specialization has been in high scale, extremely high performance and low latency solutions.
All the while I continue with the entrepreneurial efforts. My latest is with the iOS ARKit, which has been a fun experience. I should release something on that soon. I also revived a video stabilization project after spending some time on the machine learning side of that field.
Otherwise I continue with fun projects. I still have a contour mapping application on the go. It started as a real estate sales project, but that went so quickly to great success that it became just an interesting project to explore Kotlin, undertow, targeting iOS and Android and then learning about pressure propagation, etc. Then WebGL mapping, millisecond time synchronization (I use four current smartphones in concert), etc. I have a mostly complete posting on that, long promised, that I’ll finish soon.
 – I hit some financial roadblocks, leading to a crisis, when I put aside all revenue production that try to focus entirely on entrepreneurial efforts that seemed, for so long, on the absolute cusp of success. Lessons learned.