Own My Efforts

I’ve put this sort of pitch out there a few times, in a much more dilute form, and I’m going to do it one last time before dramatically shifting gears and going in a very different direction (partly by choice, and partly by having limited other options). I have utterly no shame in doing this, as ultimately the well-being of my family is my driving concern on this planet, and my ego takes a backseat.

A $615,000 CAD short term (~6 months) secured loan, if we could come to an agreement and you can deliver it in the next week (edit – with availability of funds and completed legal work in the next few days. If considering this, which I know multiple parties are, contact me as soon as possible because I have to act immediately), and you can own the entirety of the output of my facilities for that period. I’ll build for you, pitch for you, and make your enterprises a success for the entire term of the loan. We can build a financial engine, a calculation engine, a visual processing system, an audio analysis engine…whatever. A web property or a mobile app, or architect your idea.

I’m at the point of desperation at this point. I’m open to suggestions, and very open to negotiations. I’m a pretty capable developer and architect in many domains, and am willing to prove myself as necessary. I have lots of my own nascent projects that I’ll make yours.

If curious about the potential, the details, or what the situation is behind this, send me an email (dforbes@yafla.com) or give me a call (email for contact info/skype/etc). I understand that a good percentage of my readers are in other nations, and this simply isn’t possible, but if you have a well healed contact in Canada, let’s make an arrangement as soon as possible.

If you want to seriously take advantage of me, now is a quickly passing opportunity.

Programming An Itch Away

My eldest son’s rig consists of a gaming PC as his main device, with one of my old laptops on the side for ancillary use.

The audio output on his motherboard of the desktop failed mysteriously (with no driver or BIOS fix resolving the issue, the hardware of that subsystem seemingly failing), and swapping out the motherboard isn’t something I’m keen to do given the activation issues, despite having one available.

With the holiday season, getting a wireless headset or USB soundcard wouldn’t be a speedy venture. And anyways it was an opportunity for a fun software distraction.

Visual Studio powered up, a C++ solution pursued, and an hour later a solution was built. Capturing low-latency 44100 32-bit floating point stereo master mix audio on the desktop PC, resampling it to 48000 16-bit integer audio, compressing it with Opus (the resampling is courtesy of Opus demanding a multiple of 6000 sample rate, while Opus itself is purely because both machines are on wifi, and often have large network traffic, so keeping packets tiny improved the chances of a speedy delivery), UDP sending it to an argument-driven target, where the process reverses and yields an extraordinarily high fidelity reproduction of the source audio, with UDP packets tiny enough that they’re easily delivered by the network even under high saturation situations.

The audio delay is less than 10ms, which is imperceptible, and is a magnitude or more below the delay of many Bluetooth headsets.

So he games and the audio from his desktop plays on his laptop, which he has a set of high quality wired headphones connected to. It works well for now, until a long term solution is implemented (probably the motherboard swap). I could jimmy up an Android solution in minutes, having already done some Opus codec/UDP transport projects.

Those are some of the most rewarding projects. Even if I undertook the facile imagination of fantasy billing out those hours and declaring that the opportunity cost lost, those fun projects expose us to technologies and avenues, gaining educational value. Doing strange and interesting side projects is the vehicle of my most interesting ideas (as is misinterpreting or making assumptions about descriptions of products, then discovering that my assumptions or guesses are vastly off the mark, but have merit as a novel invention)

It’s a pretty trivial need and solution, but having an itch, slamming out a solution, and having a bulk of code simply work with minimal issue on first run is a glorious feeling. In this case the single defect among the sender and receiver, despite the fact that these were APIs that I’d never used before and it involved a considerable amount of bit mangling and buffer management code — the traditional shoot-yourself-in-the-foot quagmires — was a transposition of loop variables in nested loops.

Such a great feeling of satisfaction doing something like that. It is quite a nice change from the large scale projects with slower rewards that we generally ply, where rewards come slowly, if at all, diluted in the effluence of time.

Of course doing a microproject like this yields the sort of tab hilarity that we often endure when we’re dealing with technologies or APIs we don’t normally use.

And for the curious, there are some products that do what I described (send the mixed master audio from a PC to other devices), but each that we tested yielded quarter second or more latency, even over a direct twisted-pair, which just made it useless for the purpose. And even if a suitable solution existed, I really just wanted to build something, so I would have unfairly discarded it regardless.

 

3D XPoint is Pretty Cool

Five years ago I wrote a post about SSDs/flash storage being a necessary ingredient for most modern build outs, and opponents were wasting time and efforts by not adopting them in their stack. While it is profoundly obvious now, at the time there was a surprising amount of resistance from many in the industry who were accustomed to their racks of spinning rust, RAID levels, and so on. People who had banked their profession on a lot of knowledge about optimizing against extremely slow storage systems (a considerable factor in the enthusiasm for NoSQL), so FUD ruled the day.

Racks of spinning rust still have a crucial role in our infrastructure, often treated as almost nearline storage (stuff you seldom touch, and when you do the performance is so out of bounds of normal expectations). But many of our online systems are worlds improved with latency in microseconds instead of milliseconds courtesy of flash. It changed the entire industry.

In a related piece I noted that “Optimizing against slow seek times is an activity that is quickly going to be a negative return activity.” This turned out to be starkly true, and many efforts that were undertaken to engineer around glacially slow magnetic and EBS IOPS ended up being worse than useless.

We’re coming upon a similar change again, and it’s something that every developer / architect should be considering because it’s about to be real in a very big way.

3D XPoint, co-developed by Micron and Intel (the Intel one has some great infographics and explanatory videos), is a close to RAM-speed, flash-density, non-volatile storage/memory technology (with significantly higher write endurance than flash, though marketing claims vary from 3x to as high as 1000x), and it’s just about to start hitting the market. Initially it’s going to be seen in very high performance, non-volatile caches atop slower storage: the 2TB TLC NVMe with 32GB of 3d xpoint non-volatile cache (better devices currently have SLC flash serving the same purpose), offering extraordinary performance, both in throughput and IOPS / latency, while still offering large capacities.

Over a slightly longer period it will be seen in DRAM-style, byte-accessible form (circumventing the overhead of even NVMe). Not as literally the main memory, which still outclasses it in pure performance, but as an engineered storage option where our databases and solutions directly and knowingly leverage it in the technology stack.

2017 will be interesting.

Android’s “Secure Enclave” / Private Content and Strong Encryption

Recent iterations of the Android OS have exposed more of the ARM Trusted Execution Environment or Secure Element, allowing you to use encryption that can be strongly tied to a piece of hardware. It’s a subsystem where you can create strongly protected keys (symmetric and asymmetric), protected against extraction and rate limited (via user authentication) against brute force attacks, using them against streams of data to encrypt or decrypt securely.

The private elements of these keys can’t be extracted from the device (theoretically, at least), regardless of whether the application or even operating system were compromised or subverted.

A foe can’t do an adb backup or flash copy and find a poorly hidden private key to extract confidential data.

In an idealized world this would defend against even nation state levels of resources, though that isn’t necessarily the case.  Implementations slowly move towards perfection.

Imagine that you’re making an app to capture strongly encrypted video+audio (a sponsored upgrade solicitation I was recently offered for a prior product I built). The reasons could be many and are outside of the technical discussion: field intelligence gathering or secret R&D, catching corruption as a whistle blower, romantic trysts where the parties don’t really want their captures to be accidentally viewed by someone looking at holiday pictures on their device or automatically uploaded to the cloud or shared, etc.

There are nefarious uses for encryption, as there are with all privacy efforts, but there are countless entirely innocent and lawful uses in the era of the “Fappening”. We have credible reasons for wanting to protect data when it’s perilously easy to share it accidentally and unintentionally.

Let’s start with a modern Android device with full disk encryption. As a start you’re in a better place than without, but this still leaves a number of gaps (FDE becomes irrelevant when you happily unlock and hand your device to a family member to play a game, or when the Android media scanner decides to enumerate your media and it wasn’t appropriately protected, or you pocket shared to Facebook, etc).

So you have some codec streams emitting HEVC and AAC stream blocks (or any other source of data, really), and you want to encrypt it in a strong, device coupled fashion, above and beyond FDE. You accept that if the device is lost or broken, that data is gone presuming you aren’t live uploading the original streams, presumably over an encrypted connection, to some persistence location, which obviously brings up a litany of new concerns and considerations and may undermine this whole exercise.

Easy peasie, at least on Android 6.0 or above (which currently entails about 27% of the active Android market, which sounds small until you consider that this would account for hundreds of millions of devices, which by normal measures is a massive market).

final String keyIdentifier = "codecEncrypt";
final KeyStore ks = KeyStore.getInstance("AndroidKeyStore");
ks.load(null);

SecretKey key = (SecretKey) ks.getKey(keyIdentifier, null);
if (key == null) {
   // create the key
   KeyGenerator keyGenerator = KeyGenerator.getInstance(
      KeyProperties.KEY_ALGORITHM_AES, "AndroidKeyStore");
   keyGenerator.init(
      new KeyGenParameterSpec.Builder(keyIdentifier,
         KeyProperties.PURPOSE_ENCRYPT | KeyProperties.PURPOSE_DECRYPT)
            .setBlockModes(
               KeyProperties.BLOCK_MODE_CBC,
               KeyProperties.BLOCK_MODE_CTR, 
               KeyProperties.BLOCK_MODE_GCM)
            .setEncryptionPaddings(
               KeyProperties.ENCRYPTION_PADDING_PKCS7, 
               KeyProperties.ENCRYPTION_PADDING_NONE)
            .setUserAuthenticationRequired(true)
            .setUserAuthenticationValidityDurationSeconds(30)
            .build());
   key = keyGenerator.generateKey();

   // verify that key is in secure hardware.
   SecretKeyFactory factory = SecretKeyFactory.getInstance(key.getAlgorithm(), "AndroidKeyStore");
   KeyInfo keyInfo = (KeyInfo) factory.getKeySpec(key, KeyInfo.class);
   if (!keyInfo.isInsideSecureHardware()) {
      // is this acceptable? Depends on the app
   }
}

The above sample is greatly simplified, and there are a number of possible exceptions and error states that need to be accounted for, as does the decision of whether secure hardware is a necessity or a nicety (in the nicety case the OS still acts with best efforts to protect the key, but has less of a barrier to exploitation if someone compromised the OS itself).

In this case it’s an AES key that will allow for a number of block mode and padding uses. Notably the key demands user authentication 30 seconds before use: For a device with a finger print or passcode or pattern, the key won’t allow for initialization in a cipher unless that requirement has been met, demanding that your app imperatively demand a re-authentication on exceptions. Whether this is a requirement for a given use is up to the developer.

You can’t pull the key materials as the key is protected from extraction, both through software and hardware.

Using the key is largely normal cipher operations.

Cipher cipher = Cipher.getInstance("AES/CBC/PKCS7Padding");
cipher.init(Cipher.ENCRYPT_MODE, key);
// the iv is critically important. save it.
byte[] iv = cipher.getIV();
// encrypt the data.
byte [] encryptedBytes = cipher.update(dataToEncrypt);
... persist and repeat with subsequent blocks.
encryptedBytes = cipher.doFinal();

And to decrypt

Cipher decryptCipher = Cipher.getInstance("AES/CBC/PKCS7Padding");
AlgorithmParameterSpec IVspec = new IvParameterSpec(iv);
decryptCipher.init(Cipher.DECRYPT_MODE, key, IVspec);
byte [] decryptedBytes = decryptCipher.update(encryptedBlock);
... 
byte [] decryptedBytes = decryptCipher.doFinal

Pretty straightforward, and the key is never revealed to the application, nor often even to the OS. If you had demanded that the user be recently authenticated and that requirement isn’t satisfied (e.g. the timeout had elapsed), the Cipher init call would yield a UserNotAuthenticatedException exception, which you could deal with by calling-

Intent targetIntent = keyguardManager.createConfirmDeviceCredentialIntent(null, null);
startActivityForResult(targetIntent, 0);

Try again on a successful authentication callback, the secure “enclave” doing the necessary rate limiting and lockouts as appropriate.

And of course you may have separate keys for different media files, though ultimately nothing would be gained by doing that.

Having the key in secure hardware is a huge benefit, and ensuring that an authentication happened recently is crucial, but if you take out your full disk encryption Android device, unlock it with your fingerprint, and then hand it to your grandmother to play Townships, she might accidentally hit recent apps and clicks to open a video where you’re doing parkour at the tops of city buildings (family controversy!). It would open because all of the requirements have been satisfied and the hardware key would be happily allowed to be used to decrypt the media stream.

It’d be nice to have an additional level of protection above and beyond simple imperative fences. One thing that is missing from the KeyStore implementation is the ability to add an imperative password to each key (which the traditional key vaults have).

Adding a second level password, including per media resource (without requiring additional keys) is trivial. Recall that each Cipher starts with a randomly generated IV (initialization vector which is, as the name states, is the initial state of the cipher) that ultimately is not privileged information, and generally is information that is stored in the clear (the point of an IV is that if you re-encrypt the same content again and again with the same key, an unwanted observer could discern that it’s repeating, so adding a random IV as the starting point makes each encrypted message entirely different).

Without the IV you can’t decrypt the stream. So let’s encrypt the IV with a pass phrase.

SecretKeyFactory factory = SecretKeyFactory.getInstance("PBEwithSHAandTWOFISH-CBC");
SecretKey ivKey = factory.generateSecret(new PBEKeySpec(password.toCharArray(), new byte[] {0}, 32, 256));

Cipher ivCipher = Cipher.getInstance("AES/GCM/NoPadding");
ivCipher.init(Cipher.ENCRYPT_MODE, ivKey);
byte [] ivIv = ivCipher.getIV();
byte [] encryptedIv = ivCipher.doFinal(iv);

In this case I used GCM, which cryptographically tags the encrypted data with an integrity digest and on decryption validates the payload against corruption or modification. A successful GCM decryption is a clear thumbs up that the password was correct. You could of course use GCM for the media streams as well, and if it’s individual frames of audio or video each in distinct sessions that would probably be ideal, but for large messages GCM has the downside that the entirety of output is buffered until completion to allow it to compute and validate the GCM.

Now we have an encrypted IV (the 16-byte IV becoming a 32-byte encrypted output given that it contains the 16-byte GCM tag, plus we need to save the additional 16-byte IV for this new session, so 48-bytes to store our protected IV). Note that I used a salt of a single 0 byte because it doesn’t add value in this case.

You can do time-intensive many-round KDFs, but in this case where it’s acting as a secure hardware augmentation over already strong, rate-limited encryption, it isn’t that critical.

To decrypt the IV-

ivCipher = Cipher.getInstance("AES/GCM/NoPadding");
AlgorithmParameterSpec IVspec = new IvParameterSpec(ivIv);
ivCipher.init(Cipher.DECRYPT_MODE, ivKey, IVspec);
byte [] decryptedIv = ivCipher.doFinal(encryptedIv);

We store the encrypted streams and the encrypted IV (including its IV), and now to access that media stream the user needs to authenticate with the OS rate-and-try limited authentication, the hardware needs the associated trusted environment key, and the user needs the correct passphrase to access the IV to successfully decrypt.

In the end it’s remarkably simple to add powerful, extremely effective encryption to your application. This can be useful simply to protect you from your own misclicks, or even to defend against formidable, well-resourced foes.

Micro-benchmarks as the Canary in the Coal Mine

I frequent a number of programming social news style sites as a morning ritual: You don’t have to chase every trend, but being aware of happenings in the industry, learning from other people’s discoveries and adventures, is a useful exercise.

A recurring source of content are micro-benchmarks of some easily understood sliver of our problem space, the canonical example being trivial web implementations in one’s platform of choice.

A Hello World for HTTP.

package main

import (
   "fmt"
   "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
   fmt.Fprintf(w, "Hello world!")
}

func main() {
   http.HandleFunc("/", handler)
   http.ListenAndServe(":8080", nil)
}

Incontestable proof of the universal superiority of whatever language is being pushed. Massive numbers of meaningless requests served by a single virtual server.

As an aside that I should probably add as a footnote, I still strongly recommend that static and cached content be served from a dedicated platform like nginx (use lightweight unix sockets to the back end if on the same machine), itself very likely layered by a CDN. This sort of trivial type stuff should never be in your own code, nor should it be a primary focus of optimizations.

Occasionally the discussion will move to a slightly higher level and there’ll be impassioned debates about HTTP routers (differentiating URLs, pulling parameters, etc, then calling the relevant service logic), everyone optimizing the edges. There are thousands of HTTP routers on virtually every platform, most differentiated by tiny performance differences.

People once cut their teeth by making their own compiler or OS, but now everyone seems to start by making an HTTP router. Focus moves elsewhere.

In a recent discussion where a micro-benchmark was being discussed (used to promote a pre-alpha platform), a user said in regards to Go (one of the lesser alternatives compared against)-

it’s just that the std lib is coded with total disregard for performance concerns, the http server is slow, regex implementation is a joke”

total disregard. A jokeSlow.

On a decently capable server, that critiqued Go implementation, if you’re testing it in isolation and don’t care about doing anything actually useful, could serve more requests than seen by the vast majority of sites on these fair tubes of ours. With a magnitude or two to spare.

100s of thousands of requests per second is simply enormous. It wasn’t that long ago that we were amazed at 100 requests per second for completely static content cached in memory. Just a few short years ago most frameworks tapped out at barely double digit requests per second (twas the era of synchronous IO and blocking a threads for every request).

As a fun fact, a recent implementation I spearheaded attained four million fully robust web service financial transactions per second. This was on a seriously high-end server, and used a wide range of optimizations such as a zero-copy network interface and secure memory sharing between service layers, and ultimately was just grossly overbuilt unless conquering new worlds, but it helped a sales pitch.

Things improve. Standards and expectations improve. That really was a poor state of affairs, and not only were users given a slow, poor experience, it often required farms of servers for even modest traffic needs.

Choosing a high performance foundation is good. The common notion that you can just fix the poor performance parts after the fact seldom holds true.

Nonetheless, the whole venture made me curious what sort of correlation trivial micro-benchmarks hold to actual real-world needs. Clearly printing a string to a TCP connection is an absolutely minuscule part of any real-world solution, and once you’ve layered in authentication and authorization and models and abstractions and back-end microservices and ORMs and databases, it becomes a rounding error.

But does it indicate choices behind the scenes, or a fanatical pursuit of performance, that pays off elsewhere?

It’s tough to gauge because there is no universal web platform benchmark. There is no TPC for web applications.

The best we have, really, are the TechEmpower benchmarks. These are a set of relatively simple benchmarks that vary from absurdly trivial to mostly trivial-

  • Return a simple string (plaintext)
  • Serialize an object (containing a single string) into a JSON string and return it (json)
  • Query a value from a database, and serialize it (an id and a string) into a JSON string and return it (single query)
  • Query multiple values from a database and serialize them (multiple queries)
  • Query values from a database, add an additional value, and serialize them (fortunes)
  • Load rows into objects, update the objects, save the changes back to the database, serialize to json (data updates)

It is hardly a real world implementation of the stacks of dependencies and efficiency barriers in an application, but some of the tests are worlds better than the trivial micro-benchmarks that dot the land. It also gives developers a visible performance reward, just as Sunspider led to enormous Javascript performance improvements.

So here’s the performance profile of a variety of frameworks/platforms against the postgres db on their physical test platform, each clustered in a sequence of plaintext (blue), JSON (red), Fortune (yellow), Single Query (green), and Multiple Query (brown) results. The vertical axis has been capped at 1,000,000 requests per second to preserve detail, and only frameworks having results for all of the categories are included.

When I originally decided that I’d author this piece, my intention was to actually show that you shouldn’t trust micro-benchmarks because they seldom have a correlation with more significant tasks that you’ll face in real life. While I’ve long argued that such optimizations often indicate a team that cares about performance holistically, in the web world it has often been the case that products that shine at very specific things are often very weak in more realistic use.

But in this case my core assumption was only partly right. The correlation between the trivial micro-benchmark speed — simply returning a string — and the more significant tasks that I was sure would be drown out by underlying processing (when you’re doing queries at a rate of 1000 per second, an overhead of 0.000001s is hardly relevant), is much higher than I expected.

  • 0.75 – Correlation between JSON and plaintext performance
  • 0.58 – Correlation between Fortune and plaintext performance
  • 0.646 – Correlation between Single query and plaintext performance
  • 0.21371 – Correlation between Multiple query and plaintext performance

As more happens in the background, outside of the control of the framework, invariably the raw performance advantage is lost, but my core assumption was that there would be a much smaller correlation.

So in the end this is simply a “well, that’s interesting” post. It certainly isn’t a recommendation for any framework or the other — developer aptitude and suitability for task reign supreme — but I found it interesting.

 

Link Rot Pt 2

Over the years I’ve moved between a number of content management systems, URL schemes, and even whole domain name changes. So when I did a move a while back I put in a large number of URL redirects for all of those ancient URLs in use around the tubes. A year ago I announced that I was removing them, but happenstance had me installing an nginx variant that had perl shortly after, so they lived for a while longer.

I finally moved on, removing the surface area risk of that rewrite subsystem. Those URLs that I have been serving 301 permanent redirects for years are now dead. I have added to the global accumulation of link rot. I see dozens to hundreds of people coming from old HN or other links daily, get redirected to the front page, and click back (interesting to note that no one just searches for whatever it was that they were looking for once on the site. Attention spans have fallen to essentially zero).

I considered just making the 404 page a search of the wrong URL, but..eh. Not worth it. Technically it’s an easy problem to solve, but that doesn’t mean it’s worth solving.

A pretty boring tale about link rot, but really it’s an observation about technology and simple solutions being ignored: Again, for years those old links were being responded to with a courtesy note that the URL has changed and a new URL should be used permanently. Link rot didn’t have to become rot if any of those systems ever did any verification at all on their links, remembering the new location for the future.

While that sort of link verification and upkeep is a slightly more involved task with something like social news or comment links (and has some considerations that would need to be considered like PageRank gaming, where a bunch of old links were all spaghetti fed to some spam site after gaining credibility, though that’s the case with a persistent redirect so simply fixing the new location is no regression), it’s so bad that even feed readers ignored 301 redirects for years. They followed them, but the next request they were back at the original URL, once again awaiting the onset of link rot.

The Dark Clouds of Choice

I enjoy Go (the programming language/tooling). It rapidly facilitates efficient, concise solutions. Whether it’s processing some data, building a simple connector web service, hosting a system monitoring tool (where I might layer it on some C libraries), it just makes it a joy to bang out a solution. The no-dependency binary outputs are a bonus.

It’s a great tool to have in pocket, and is fun to leverage.

I recently pondered why this was. Why I don’t have the same feeling of delight building solutions in any of the other platforms that I regularly use. C# or Java, for instance, are both absolutely spectacular languages with extraordinary ecosystems (I’d talk about C or C++ here, using both in the regular mix, but while there are usually very practical reasons to drop to them they don’t really fit in this conversation, despite the fact that from a build-a-fast-native-executable perspective they’re the closest).

Goroutines? Syntactic sugar over thread pools. Channels? That’s a concurrent queue. Syntactical sugar is nice and lubricates usage, but once you’ve done it enough times it just becomes irrelevant. A static native build? A bonus, but not a critical requirement.

There is nothing I build in Go that I can’t build at a similar pace in the other languages. And those languages have rich IDEs and toolsets, while Go remains remarkably spartan.

The reason I enjoy it, I think, is that Go is young enough that it isn’t overloaded with the paradox of choice: You throw together some Go using the basic idioms and widely acknowledged best practices, compile it, and there’s your solution. Move on. A lot of strongly held opinions are appearing (about dependencies and versioning, etc — things Go should have gotten right in v1), and an evolutionary battle is happening between a lot of supporting libraries, but ultimately you can mostly ignore it. Happily build and enjoy.

Doing the same project in Java or C#, on the other hand, is an endless series of diverging multi-path forks. Choices. For the most trivial of needs you have countless options of implementation approaches and patterns, and supporting libraries and dependencies. With each new language iteration the options multiply as more language elements from newer languages are grafted on.

Choices are empowering when you’re choosing between wrong and right options, where you can objectively evaluate and make informed, confident decisions. Unfortunately our choices are often more a matter of taste, with countless ways to achieve the same goal, with primarily subjective differences (I’ll anger 50% of C# programmers by stating that LINQ is one of the worst things to happen to the language, and is an inelegant hack that is overwhelming used to build terrible, inefficient, opaque code).

We’re perpetually dogged with the sense that you could have gone a different way. Done it a different way. I actually enjoy C++ (I admit it…Stockholm syndrome?), but with each new standard there are more bolted-on ways to achieve existing solutions in slightly different ways.

I of course still build many solutions on those other platforms, and am incredibly thankful they exist, but I never have the completely confident sense that it is optimal in all ways, or that someone couldn’t look at it and ask “Couldn’t you have…” and I could firmly retort. I continue an internal debate about forks not taken.

The Best of the Best of the Best…Maybe?

I’ve talked about the consulting thing on here a lot, and the trouble involved with the pursuit. While I’ve been working with a fantastic long-running client, and have primarily been focused on speculative technology builds, I considered keeping skills diverse and fresh by mixing things up and working occasionally through a freelancing group that purports to have the best of the best. Doing this would theoretically remove the bad parts of chasing engagements, pre-sales, badgering for payments, etcThe parts that are a giant pain when you hang your own shingle.

If it was just challenging engagements with vetted clients, cool. Just the fun and rewarding parts, please and thank-you.

Freelancing groups almost always end up being a race to the bottom, generally becoming dens of mediocrity, so the notion of a very selective group made it more interesting. I like a challenge, and if someone wants to build a collection of Boss Levels for me to battle, the barriers yielding a group that customers would pay a premium for, maybe it’d be interesting.

So I took a look to find that one of their requirements — not mandatory, but strong recommended — is that you post on your blog how much you want to work with them. This is before you know anything about the credibility of their process, rates, the quality of peers, etc. And this isn’t something you can easily find: they demand that you don’t talk about their process or virtually anything involved with working with them, so in the absence of any information about them (beyond some very negative threads I later found on HN, primarily posts by throwaway accounts), you need to tell the world how eager you are to join them.

This is a profound asymmetry of motives.

Who would do that? It is an adverse selection criteria, and it instantly deflated my impression of their selectivity and had me clicking the back button: The sort of desperation where someone would pander like that — while certainly possible among fantastic talents in bad situations — is not a good criteria. To try to harvest cheap link and namespace credibility like that itself makes the service look lame, like a cheap 1990s long distance carrier.

I still want those challenging smaller efforts, however — variety keeps me fresh and in love with this field, and some extra cash and contacts are always nice — so instead I’m going to start offering micro-consulting via yafla, pitching it on here as well: Pay some set amount (e.g. $500), describe your problem and supply supporting data, and I’ll spend from four to eight hours on your problem, offering up appropriate deliverables (utility, shell of a solution, decision input, directions, analysis, etc). I’ll get that online quickly.

Going for the smaller chunk efforts, of the sort that someone with a corporate credit card can easily pay for, should prove much more tenable than seeking significant engagements where the sales process just drags on forever.

It also is a cross-motivation desire to encourage me to spend more time on this blog, and if I’m pitching bite-sized consulting on posts and actually seeing uptake, it’ll keep me authoring content that people want to see.

While I have always enjoyed the cathartic aspect of blogging, it has never been profitable for me. Every engagement has always come via referral and word of mouth, and even when I’ve sought traditional employment I’ve always been amazed that no one ever even does a simple Google search, much less read anything on here. I never expected any return, and adore that anyone reads these things at all, but it does make it so at times it’s tough to justify time spent making content.

Paid For Solutions, Not The Pursuit

A fun read via HN this morn – You Are Not Paid to Write Code.

I’ve touched on this many times here, including an entry a decade ago where I called SLOC the “Most Destructive Metric in Software Development“. A decade of experience has only made me double (neigh, quadruple) my belief in that sentiment: outside of truly novel solutions, SLOC often has a negative correlation with productivity, and high SLOC shops almost universally slow to a crawl until they hit the rock bottom of no progress. Eventually a new guard is brought in, a giant volume of code is trashed, and the same futile pursuit started fresh again.

This time, you see, it’s a giant node.js codebase instead of that silly giant python codebase they pursued the last time, replacing the giant Delphi codebase that replaced the giant Visual Basic codebase that…

This time it’s different.

An entry on here that gets a number of daily visitors is one from 2005 – Internal Code Reuse Considered Dangerous (I’ve always wondered why, and my weak assumption is it’s coworkers trying to convince peers that they need to move on from the internal framework mentality). That piece came from firsthand observations of a number of shops that had enormous volumes of internal frameworks and libraries that were second-rate, half-complete duplications of proven industry options. But it was treated as an asset, as if just a bit more code would give them the swiss army knife that would allow them to annihilate competitors. Every one of those shops eventually failed outright or did a complete technology shift to leave the detritus in the past.

It isn’t hard to figure out how this happens. If someone asks you to process a file from A to B — the B is the part they care about, not particularly how you do it — and you present a solution including some free and appropriate ETL tools and options, there is no glory in that. If, on the other hand, you make a heavily abstracted, versatile, plug-in engine that can (hypothetically and in some future reality where it ever gets finished) process any form of file to any form of output with a pluggable reference engine and calculation derivative, you can pitch the notion of IP. That instead of just providing a solution, you’ve also built an asset.

This is a lie almost all of the time. There is incredibly little code theft in this industry. That giant internal framework, when uncoupled from the internal mythology of a shop, suddenly has negligible or even negative value to outsiders. A part of that is a not invented here syndrome endemic in this industry (I’ve been in the business of trying to sell completed, proven solutions, and even then it’s tough to find customers because everyone imagines that they can build it themselves better), but a larger part is simply that broadly usable, generalized solutions don’t happen by accident.

This is one of those sorts of entries where some might contrive exceptions, and of course there are exceptions. There are cases where the business says “turn A to B” and you discover that you really need to turn A to B-Z, and 0-9, and… There are many situations where novel solutions are necessary. But so many times they simply aren’t.

Some of the most fulfilling consulting engagements I’ve taken on have been fixed deliverable / fixed price contracts. These are a sort that everyone in this industry will tell you never to do, but really if you have a good grip on your abilities, the problem, and you can obtain a clearly understood agreement of scope and capabilities and deficiencies of the proposed build, it is incredibly liberating. Being literally paid for the solution leads to some of the most frictionless gigs.

Things You Probably Don’t Know About Your Digital Camera

Another periphery post about imaging as a recent project brought it into my thoughts. It’s always nice to have broadly interesting, non-confidential material to talk about on here, and while this is well known to many, I’ve discovered in some projects that many in this field aren’t aware of this magic happening in their devices.

To get right to it.

Your Camera’s Imaging Resolution is Lower Than Advertised

The term “pixel” (picture element) is generally holistic — a discretely controlled illumination of any visible shade of color, whatever the mechanism of achieving that result. Your 1080p television or computer monitor has ~1920 by 1080 pixels, or about 2 million “pixels”. If you look closely, however, your monitor has discrete subpixels for the emissive colors (generally Red-Green-Blue, though there is a display technology that adds yellow as well for an RGBY subpixel arrangement). These might be horizontally oriented, vertical, staggered, in a triangle pattern…in any order.

At appropriate combinations of resolution density and viewing distance, your eyes naturally demosaic, blending the individual colors into a discrete full color representation.

The vast majority of digital cameras, however, use the term pixel in a very different way.

To visually explain, here’s a portion of RAW sensor data taken directly from a Nexus 6p. The only processing applied was source color channel gains and scaling from the original 100 by 63 to a more visible 600 by 378.

car_bayer_closer

In the digital camera world, each of these discrete colors is a whole pixel.

If you inspect the pixels you’ll notice that they aren’t full color (though that is obvious just by eyeballing the image). Each unique location is one of either red, green, or blue, at varying intensities. There are no mixes of the colors.

The imaging sensor has pits that can measure photons, but they have no awareness of wavelength. To facilitate color measurements a physical color filter is overlaid over each pit, alternating between the three colors. This is generally a Bayer color filter.

There is another type of sensor that layers wavelength sensitive silicon (much like the layers of classic film), capturing full color at each site, however it is very rarely used and has its own problems.

Green is most prevalent, comprising 50% of the pixels given that it’s the color band where the human eye is most sensitive to intensity changes and detail. Red alternates with green on one line, while Blue alternates with green on the next.

The functional resolution of detailed color information, particularly in the red and blue domains, is much lower than many believe (because of which many devices have physical and processing steps — e.g. anti-aliasing — that further reduce the usable resolution, blurring away the defects).

The Nexus 6P ostensibly has a 4032 x 3024 imaging resolution, but really, courtesy of the Bayer filter, has a 2016 x 3024 green resolution, a 2016 x 1512 blue resolution, and a 2016 x 1512 red resolution. For fine hue details the resolution can be 1/4 expectations, and this is why fully zoomed in pictures are often somewhat disappointing (also courtesy of processing and filtering to try to mask the color channel information deficiencies).

Your Camera’s Imaging Sensor Has Many Defects

Due to defects in silicon, the application of the physical bayer filter, and electrical gain noise, many of the photo sites on your digital sensor are defective.

Some read nothing, while many more see ghosts, reporting some or significant false readings. Readings of a constant brightness target will vary, sometimes significantly, across pixels (yielding a grainy, noisy output image).

falsereadings

This is a random 150 pixel wide reading from the 6p when taking a 1/10s picture of pure darkness. These defective readings cover the entire capture in varying densities, comprising up to hundreds of false data points. Most are permanent, often with new ones appearing as the device ages. Some defects temporarily worsen when the sensor is warm. Most SLRs have a special mode where it will take a full darkness picture and then catalog and remove all hot pixels from the output material. Android also has the notion of remembering hot pixels.

This is the case with every digital sensor, from your smartphone to your high end SLR. I remember being somewhat horrified first looking at a wholly unprocessed RAW image from my SLR, seeing hundreds of fully lit pixels scattered across the image.

Algorithms Saves The Day

The solution to all of these problems is processing, but it does have consequences.

Hot pixels are eliminated both through prior knowledge (a hot pixel database for a given sensor), and through simply eliminating pixels that shine a little too bright relative to her neighbors. They get replaced with an interpolated average of neighbors.

The Bayer pattern source is turned into a full color image via a demosaicing algorithm, and there is considerable academic research into finding the optimal solution. In that case I linked to an army research paper, the military having a significant interest in this field given the broad use of Bayer imaging sensors, and a need to know that the resulting images/data are the highest fidelity possible (especially given that machine vision systems are then analyzing that resulting heavily processed output, and with the wrong choices can be triggering on algorithm detritus and side-effects).

The choice of demosaicing algorithm can have a significant impact on the quality of the resulting image. Do you know what algo your device is using?

After demosaicing, color corrections are applied (both to move between color spaces, and to provide white point corrections), and then the image is de-noised — those fine grainy variations are homogenized (which can yield unique results if the subject itself has a grainy appearance — the algorithm can’t discern whether variations are from the source or from the sensor).

The resulting image is generally close to perceptually perfect, but an enormous amount of human knowledge and guesswork went into turning some very imperfect source data into a good result. The quality of an image from a digital device is as significantly impacted by software as the hardware (many devices have terrible color fringing courtesy of poor demosaicing). Which is why many choose to shoot RAW photos, saving those source single-band pixels as is before destructively applying corrections. This allows for improvements or alterations of algorithms when the magic mix didn’t work quite right for a given photo.

If you look closely at the results, you start to see the minor compromises necessary to yield a workable output.

Optical vs Electronic Image Stabilization

The recently unveiled Google Pixel smartphones features electronic image stabilization in lieu of optical image stabilization, with Google reps offering up some justifications for their choice.

While there is some merit to their arguments, the contention that optical image stabilization is primarily for photos is inaccurate, and is at odds with the many excellent video solutions that feature optical image stabilization, including the competing iPhone 7.

Add that video is nothing more than a series of pictures, a seemingly trite observation that will have apparent relevance later in this piece.

This post is an attempt to explain stabilization techniques and their merits and detriments.

Why should you listen to me? I have some degree of expertise on this topic. Almost two years ago I created an app for Android1 that featured gyroscope-driven image stabilization, with advanced perspective correction and rolling shutter compensation. It offers sensor-driven Electronic Image Stabilization for any Android device (with Android 4.4+ and a gyroscope).

It was long the only app that did this for Android (and to my knowledge remains the only third-party app to do it). Subsequent releases of Android included extremely rudimentary EIS functionality in the system camera. Now with the Google Pixel, Google has purportedly upgraded the hardware, paid attention to the necessity of reliable timing, and offered a limited version for that device.

They explain why they could accomplish it with some hardware upgrades-

“We have a physical wire between the camera module and the hub for the accelerometer, the gyro, and the motion sensors,” Knight explains. “That makes it possible to have very accurate time-sensing and synchronization between the camera and the gyro.”

We’re talking about gyroscope data triggering 200 times per second, and frames of video in the 30 times per second range. The timing sensitivity is in the 5ms range (that the event sources are timestamped within that range of accuracy). This is a trivial timing need and should need no special hardware upgrades to be accomplished. The iPhone has had rock solid gyroscope timing information going back many generations, along with rock solid image frame timing. It simply wasn’t a need for Google, so the poor design of the timing insanity was the foundation of data on Android (and let me be clear that I’m very pro-Android. I’m pro all non-murdery technology, really. This isn’t advocacy or flag-waving for some alternative: it’s just an irritation that something so simple became so troublesome and wasted so much of my time).

Everyone is getting in on the EIS stabilization game now, including Sony, one of the pioneers of OIS, with several of their new smartphones, and even GoPro with their latest device (their demos again under the blazing midday sun, and still they’re unimpressive). EIS lets you use a cheaper, thinner, less complex imaging module, reducing the number of moving parts (so better yields and reliability over time. Speaking of which, I’ve had two SLR camera bodies go bad because the stabilized sensor system broke in some way).

A number of post-processing options have also appeared (e.g. using only frame v frame evaluations to determine movement and perspective), including Microsoft’s stabilization solution, and the optional solution built right into YouTube.

There are some great white papers covering the topic of stabilization .

Let’s get to stabilization techniques and how EIS compares with OIS.

With optical image stabilization, a gyro sensor package is coupled with the imaging sensor. Some solutions couple this with some electromagnets to move the lens, other solutions move the sensor array, while the best option (there are optical consequences of moving the lens or sensor individually, limiting the magnitude before there are negative optical effects) moves the entire lens+sensor assembly (frequently called “module tilt”), as if it were on a limited range gimbal. And there are actual gimbals2 that can hold your imaging device and stabilize it via gyroscope directed motors.

A 2-axis OIS solution corrects for minor movements of tilt or yaw — e.g. tilting the lens down or up, or tilting to the sides — the Nexus 5 came with 2-axis stabilization, although it was never well used by the system software, and later updates seem to have simply disabled it altogether.

More advanced solutions add rotation (roll), which is twisting the camera, upping it to a 3-axis solution. The pinnacle is 5-axis which also incorporate accelerometer readings and compensates for minor movements left or right, up and down.

EIS also comes in software 2-, 3- and 5-axis varieties: Correlate the necessary sensor readings with the captured frames and correct accordingly. My app is 3-axis (adding the lateral movements was too unreliable across devices, not to mention that while rotational movements could be very accurately corrected and perspective adjusted, the perspective change of lateral movements is a non-trivial consideration, and most implementations are naive).

With an OIS solution the module is trying to fix on a static orientation so long as it concludes that any movement is unintended and variations fall within its range of movement. As you’re walking and pointing down the street, the various movements are cancelled out as the lens or sensor or entire module does corrective counter-movements. Built-in modules have a limited degree of correction — one to two degrees in most cases, so you still have to be somewhat steady, but it can make captures look like you’re operating a steadicam.

An OIS solution does not need to crop to apply corrections, and doesn’t need to maintain any sort of boundary buffer area. The downside, however, is that the OIS system is trying to guess, in real time, the intentions of movements, and will often initially cancel out the onset of intentional movements: As you start to pan the OIS system will often counteract the motion, and then rapidly try to “catch up” and move back to the center sweet spot where it has the maximum range of motion for the new orientation.

The imaging sensor in OIS solutions is largely looking at a static scene, mechanically kept aligned.

With an EIS solution, in contrast, the sensor is fixed, and is moving as the user moves. As the user vibrates around and sways back and forth, and so on, that is what the camera is capturing. The stabilization is then applied either in real-time, or as a post-processing step.

A real-time EIS system often maintains a fixed cropping to maintain a buffer area (e.g. only a portion of the frame is recorded, allowing the active capture area to move around within the buffer area without changing digital zoom levels), and as with OIS solution it predictively tries to infer the intentions of movements. From the demo video Google gave, their system is real-time (or with a minimal number of buffer frames), yielding the displeasing shifts as it adjusts from being fixed on one orientation to transitioning to the next fixed orientation (presumably as range of movement started to push against the edge of the buffer area), rather than smoothly panning between.

A sensor-driven post-processing EIS system, which is what Gallus is, captures the original recording as is, correlating the necessary sensor data and using attributes of the device (focal length, sensor size, field of views, etc) in post processing can evaluate the motion with a knowledge of the entire sequence, using low-pass filters and other smoothing techniques to make a movement spline within the set variability allowance.

Let’s start with an illustrative sample. Moments before writing this entry, the sun beginning to set on what was already a dreary, dim day, I took a walk in the back of the yard with my app, shooting a 4K video on my Nexus 6p. Here it is (it was originally recorded in h265 and was transcoded to h264, and then YouTube did its own re-encoding, so some quality was lost) –

This is no “noon in San Francisco” or “ride on the ferry” sort of demo. It’s terrible light, subjects are close to the camera (and thus have a high rate of relative motion in frame during movements) and the motions are erratic and extreme, though I was actually trying to be stable.

Here’s what Gallus — an example of sensor-driven post-processing EIS — yielded when it stabilized the result.

I included some of the Gallus instrumentation for my own edification. Having that sort of informational overlay on a video is an interesting concern because it conflicts with systems that do frame-v-frame stabilization.

Next up is YouTube and their frame-v-frame stabilization.

YouTube did a pretty good job, outside of the random zooming and jello effect that appears in various segments.

But ultimately this is not intending to be a positive example of Gallus. Quite the opposite, I’m demonstrating exactly what is wrong with EIS: Where it fails, and why you should be very wary of demonstrations that always occur under perfect conditions. And this is a problem that is common across all EIS solutions.

A video is a series of photos. While some degree of motion blur in a video is desirable when presented as is, with all of the original movements, as humans we have become accustomed to blur correlating with motion — a subject is moving, or the frame is moving. We filter it out. You likely didn’t notice that a significant percentage of the frames were blurry messes (pause at random frames) in the original video, courtesy of the lower-light induced longer shutter times mixed with device movements.

Stabilize that video, however, and motion blur of a stabilized frame3 is very off-putting. Which is exactly what is happening above: Gallus is stabilizing the frame perfectly, but many of the frames it is stabilizing were captured during rapid motion, the entire frame marred with significant motion blur.

Frames blurred by imaging device movement are fine when presented in their original form, but are terrible when the motion is removed.

This is the significant downside of EIS relative to OIS. Where individual OIS frames are usually ideal under even challenging conditions, such as the fading sun of a dreary fall day, captured with the stability of individual photos, EIS is often working with seriously compromised source material.

Google added some image processing to make up for the lack of OIS for individual photos — taking a sequence of very short shutter time photos in low light, minimizing photographer motion, and then trying to tease out a usable image from the noise — but this isn’t possible when shooting video.

An EIS system could try to avoid this problem by using very short exposure times (which itself yields a displeasing strobe light effect) and wide apertures or higher, noisier ISOs, but ultimately it is simply a compromise. To yield a usable result other elements of image capture had to be sacrificed.

But the Pixel surely does it better than your little app!” you confidently announce (though they’re doing exactly the same process), sitting on your Pixel order and hoping that advocacy will change reality. As someone who has more skin in this game than anyone heralding whatever their favorite device happens to have, I will guarantee you that the EIS stabilization in the Pixel will be mediocre to unusable in challenging conditions (though the camera will naturally be better than the Nexus 6p, each iteration generally improving upon the last, and is most certainly spectacular in good conditions).

Here’s a review of the iPhone 7 (shot with an iPhone 7), and I’ll draw your attention to the ~1:18 mark — as they walk with the iPhone 7, the frame is largely static with little movement, and is clear and very usable courtesy of OIS (Apple combines minimal electronic stabilization with OIS, but ultimately the question is the probability that static elements of the scene are capturing the majority of a frame’s shutter time on a fixed set of image sensors, and OIS vastly improves the odds). As they pan left, pause and view those frames. Naturally, given the low light, with significant relative movement of the scene it’s a blurry mess. On the Pixel every frame will be like this under that sort of situation presuming the absence of inhuman stability or an external stabilizer.

I’m not trying to pick specifically on the Pixel, and it otherwise looks like a fantastic device (and would be my natural next device, having gone through most Nexus devices back to the Nexus One, which replaced my HTC Magic/HTC Dream duo), but in advocating their “an okay compromise in some situations” solution, they went a little too far with the bombast. Claiming that OIS is just for photos is absurd in the way they intended it, though perhaps it is true if you consider a video a series of photos, as I observed at the outset.

A good OIS solution is vastly superior to the best possible EIS solution. There is no debate about this. EIS is the cut-rate, discount, make-the-best-of-a-bad-situation compromise. That the Pixel lacks OIS might be fine on the streets of San Francisco at noon, but it’s going to be a serious impediment during that Halloween walk, in Times Square at night, or virtually anywhere else where the conditions aren’t ideal.

The bar for video capture has been raised. Even for single frame photography any test that uses static positioning is invalid at the outset: It doesn’t matter if the lens and sensor yield perfect contrast and color if it’s only in the artificial scenario where the camera and subject are both mounted and perfectly still, when in the real world the camera will always be swaying around and vibrating in someone’s hands.

Subjects moving in frame of course will yield similar motion blur on both solutions, but that tends to be a much smaller problem in real world video imaging, and tends to occur at much smaller magnitudes. When you’re swaying back and forth with a fixed, non-OIS sensor, the entire frame is moving across differing capture pixels at a high rate of speed, versus a small subject doing a usually small in frame motion. They are a vastly different scale of problem.

The days of shaky cam action are fast fading, and the blurry cam surrogates are the hanger-ons. The best option is a stabilized rig (but seriously). Next up is 5-axis optical image stabilization, and then its 3-axis cousin. Far behind is sensor-driven EIS. In last place are post-processing frame versus frame comparison options (they often falter in complex scenes, but will always be demoed with a far off horizon line in perfect conditions, with gentle, low frequency movements).

Often on-camera OIS will be augmented with minimal EIS — usually during transition periods when OIS predicted the future wrong (to attempt to resolve the rapid catch-up), and also to deal with rolling shutter distortion.

To explain rolling shutter distortion, each line of the CMOS sensor is captured and read individually and sequentially, so during heavy movement the frame can skew because the bottom of the scene was pulled from the sensor as much as 25ms after the beginning line of the scene (as you pan down things compress to be smaller, grow when panning up, and skew left and right during side movements). So during those rapid transition periods the camera may post process to do some gentle de-skewing, with a small amount of overflow capture resolution to provide the correction pixels. Rolling shutter distortion is another interesting effect because it’s a pretty significant problem with every CMOS device, but it didn’t become obvious until people started stabilizing frames.

And to digress for a moment, the hero of an enormous amount of technology progress in the past ten years are the simple, unheralded MEMS gyroscopes. These are incredible devices, driven by a fascinating principle (vibrating structures reacting to the Coriolis effect), and they’re the foundation of enormous technology shifts. They’re a very recent innovation as well. Years ago it was stabilized tank turrets that had this technology (okay, some murdery technology is pretty cool), courtesy of giant, expensive mechanical gyroscopes. Now we have cheap little bike mount gimbals doing the same thing.

For the curious, here’s the unzoomed original corrections as applied by Gallus. It was a fun, technically challenging project, despite the frustrations and sleepless nights. It began with some whiteboard considerations of optics and how they work, then field of view, sensor sizes, offset calculations, and so on.

1 – Developing Gallus presented unnecessary challenges. Android lacks reliable event timing, though that has improved somewhat in recent iterations. A lot of the necessary imaging metadata simply didn’t exist, because Google had no need for it (and this is a problem when sharecropping on a platform). As they got new interests, new functionality would appear that would expose a little more of the basic underlying hardware functionality (versus starting with a logical analysis of what such a system would consist of and designing a decent foundation). The whole camera subsystem is poorly designed and shockingly fragile. The number of Android handsets with terrible hardware is so high that building such an advanced, hardware-coupled application is an exercise in extreme frustration.

And given some cynical feedback, note that this post is not a plea for people to use that app (this is not a subtle ad), though that should be obvious given that I start by telling you that the very foundation of EIS is flawed. I go through occasional spurts of updating the app (occasionally breaking it in the process), and having users bitching because an update notice inconvenienced their day kind of turned me off of the whole “want lots of users” thing, at least for an app as so “edge of the possible” as Gallus.

2 – While this post was ostensibly about the Pixel EIS claims, I was motivated to actually write it after seeing many of the comments on this video. That bike run, shot with a “Z1-ZRider2” actively stabilized gimbal (not a pitch for it — there are many that are largely identical) is beautifully smoothed, so it’s interesting to see all of the speculation about it being smoothed in post (e.g. via some frame-v-frame solution). Not a chance. If it was shot unstabilized or with EIS (which is, for the purpose of discussion, unstabilized) it would have been a disastrous mess of blurred foliage and massive cropping, for the reasons discussed, even under the moderate sun. Already objects moving closer to the lens (and in a frame relative sense faster) are heavily blurred, but the entirety of the scene would have that blur or worse minus mechanical stabilization.

There is an exaggerated sense of faith in what is possible with post-process smoothing. Garbage in = garbage out.

3 – One of my next technical challenge projects relates to this. Have a lot of cash and want to fund some innovation? Contact me.

Everything You Read About Databases Is Obsolete

Six and a half years ago I wrote (in a piece about NoSQL) –

Optimizing against slow seek times is an activity that is quickly going to be a negative return activity.

This time has long since passed, yet much of the dogma of the industry remains the same as it was back when our storage tier was comprised of 100 IO/second magnetic drives. Many of our solutions still have query engines absolutely crippled by this assumption (including pgsql. mssql has improved, but for years using it on fast storage was as exercise in waste as it endlessly made the wrong assumptions around conserving IOs).

There are now TB+ consumer storage solutions with 300,000 IO/second (3000x the classic magnetic drive, while also offering sequential rates above 3.5GB/s…yes, big G) for under $600. There are enterprise solutions serving 10,000,000 IOPS.

That’s if your solution even needs to touch the storage tier. Memory is so inexpensive now, even on shared hosting like AWS, that all but the largest databases sits resident in memory much of the time. My smartphone has 3GB, and could competently host the hot area of 99.5%+ of operational databases in memory.

For self-hosted hardware, TBs of memory is now economical for a small business, while tens of GBs is operationally inexpensive on shared hosting.

I totally made up that 99.5% stat, but it’s amazing how relatively tiny the overwhelming bulk of databases I encounter in my professional life are, yet how much so many fret about them.

Obviously writes still have to write thru, yet when you remove the self-defeating tactic of trying to pre-optimize by minimizing read IO — eliminating denormalization, combined-storage (e.g. document-oriented), materializations and trigger renderings/precomputing, excessive indexes, etc — in most cases writes narrow to a tiny trickle1.

When writes reduce, not only does practical write performance increase (given that you’re writing much less per transaction, the beneficial yield increases), the effectiveness of memory and tier caches increases as the hot area shrinks, and the attainability of very high performance storage options improves (it’s a lot more economical buying a 60GB high performance, reliable and redundant storage tier than a 6TB system. As you scale up data volumes, often performance is sacrificed for raw capacity).

Backups shrink and systems become much more manageable. It’s easy to stream a replica across a low grade connection when the churn of change is minimized. It’s easy to keep backups validated and up to date when they’re GBs instead of TBs.

Normalize. Don’t be afraid of seeks. Avoid the common pre-optimizations that are in the whole destructive to virtually every dimension of a solution on modern hardware (destroying write performance, long-term read performance, economics, maintainability, reliability). Validate assumptions.

Because almost everything written about databases, and from that much of what you read, is perilously outdated. This post was inspired when seeing another database best practices guideline make the rounds, most suggestions circling a very dated notion that every effort should be made to reduce IOs, the net result being an obsolete, overwrought solution out of the gates.

1 – One of the most trying aspects of writing technical blog entries is people who counter with edge cases to justify positions: Yes, Walmart has a lot of writes. So does Amazon, CERN and Google too. The NY Taxi Commission logs loads of data, being in a city area of tens of millions.

There are many extremely large databases with very specialized needs. They don’t legitimize the choices you make, and they shouldn’t drive your technical needs.

The Dead End of The Social Pyramid Scheme

This is a random observational post while I take a break from real work. I’m revisiting a topic that I touched upon before, and ultimately this is really just a lazy rewriting of that piece.


A few days ago I saw a new commercial for Toronto’s SickKids hospital.

The commercial is powerful.

“This is new and fresh and important, so I’ll share it with the people I know on Facebook”, I thought.

It isn’t original content, obviously, but I thought it was something they’d find interesting.

So I shared it. Seconds later I deleted the post.

I don’t post on Facebook (or Google+, or Twitter) outside of the rare photo of the kids limited to family. By deleting I was returning to my norm.

Most of the people among my contacts have trended toward the same behavior, with a small handful of social feeders alone among the whole. Most now use Facebook for discussion groups and as a feed aggregator: If a site (e.g. Anandtech) shares on Facebook, I just rely upon it appearing in my feed rather than visiting their site. It’s also a great feed for game day news as well.

Individual sharing is trending way down on Facebook. Many other sites are showing the same trend. LinkedIn feels like a graveyard of abandoned profiles, and “celebrities” who have assistants post various self-promotional pieces occasionally (I recently deactivated my LinkedIn profile after realizing that I have gotten zero value from it over my career, yet have gotten a lot of negative consequences including just unnecessary exposure of information random people don’t have any need to know).

We have like, share and retweet fatigue. It sits there as a little judgy footer on every post, each reaction carefully meted out and considered. As a social obligation both on our own posts, and on the posts of our friends and family.

So if I post something and it sits un-liked, should I be offended? Should I fish for likes, building a social crew? If my niece posts something interesting, should I like it or is that weird that I’m her uncle liking her post? If a former female coworker posts an interesting recipe, should I like it or is that going to be perceived as an advance?

If I get a pity like from a relative, should I reciprocate?

Some will dismiss this as overthinking, but what I’m describing above is exactly what this service, and every one like it, is designed to demand as your response. It is the gamification of users, used masterfully, and the premise is that if you make the social counts front and center, it obligates users towards building those numbers up. Some shared blog platforms are now plying this tactic to entice users to become essentially door to door pitchmen to draw people to the platform (as they sharecrop on someone else’s land, repeating a foolish mistake we learned not to make well over a decade ago), lest their blog posts get deranked. People aren’t pitching Avon or Amway now, they’re trying to get you to help them make a foundation for their medium blog or pinterest board or Facebook business group or LinkedIn profile or…

Sometimes it works for a while as a sort of social pyramid scheme. Eventually the base starts to stagnant, the “incentives” lose their luster if not rusting and becoming a disincentive for newer or more casual users. If it isn’t carefully managed, the new users will cast the old guard as obsolete and irrelevant.

I made a Soundcloud account purely to access a personal audio recording across multiple devices, so why do I keep getting notifications of spammy followers, all of whom are front and center on my profile that I don’t want? I don’t want followers or hearts or likes or shares.

Let me qualify that statement a bit: I love when readers think that these blog posts are interesting enough to share on various venues, growing the circle of exposure. That happens organically when readers thinks content is worthwhile, and it’s very cool. But that is something that the reader owns, and doesn’t sit as a social signal of relevance on this page: There are no social toolbars or tags on this post trying to act as a social proof that this is worth reading, beyond that most of you have read these missives for a while and I assume find some value in them.

Users should absolutely have these curating inputs (training the system on the things that they like and dislike), and the feed should of course adapt to the things the user actually enjoys seeing: If zero users find anything interesting that I post, zero people should see it. But by making it a public statement it becomes much more than that, losing its purpose and carrying a significant social obligation and stigma that is unwanted.

Virtually every social site follows the same curve as we all dig the social well, and when it runs dry we simply chase the next experience. Facebook has done well by pivoting the use of the service, but other services (Flickr, Twitter, and others) that attempted the same strategy peaked and then hit a period of stark decline: if someone with less than 100 twitter followers are perceived as “angry and disenfranchised”, new users find more benefits simply waiting out this generation, or moving to something new — a sort of ground zero where everyone goes back to the beginning again — than to try to gain some namespace among established users.

Back in the early days of the Internet, MUDs (definition varies) saw the same curve. Each instance would start as fresh greenfield full of opportunity and excitement. As the initial euphoria settled, soon it was a small set of regular users, maxed out in every regard. Now that the pyramid scheme of fresh meat was exhausted — new users knew that there was little fun or benefit to be had, and went to newer, fresher sites, leaving the existing users with their blessed armor and no skulls to smash — malaise set in. Eventually the universe was wiped and begun anew.

There’s no real aha or “what to do” out of this. I don’t know what to make of it. Clearly the tactic is fantastically successful in the ascent part of the curve, and has been leveraged masterfully by a number of sites, but if you don’t pivot at the right time it ends up turning a site into Slashdot or Kuro5hin — a decayed remnants of a yesteryear internet.