Filler Post

All has been quiet on here lately during a busy professional and personal time. Just a quick update that everything is great, I certainly haven’t abandoned this, and hope to have some interesting updates very soon.

Technology and Population Density Trends

A bit of a rambling, conversational piece today.

Amara’s Law states-

We tend to overestimate the effect of a technology in the short run, and underestimate the effect in the long run.

We declare that a technology changes everything, realize that entrenched patterns/behaviors and small hangups limit broad adoption. We discount it as over-hyped and not as significant as expected. It quietly takes over and changes the very foundations of society and our lives.

Recently I was pondering what electric cars and self-driving cars would do to population density. The former — using mechanically simpler vehicles with a much less expensive energy source — will significantly reduce the cost of driving as it achieves the economy of scale, while the latter will reduce the inconvenience of driving, commuting in particular.

Self-driving cars will not only allow us to do other things during the ride, it will significantly increase the capacity of our roads to handle traffic by reducing human error and inefficiencies.

Intuitively, at least to me, these changes will propagate lower density living. That home that previously would have been an expensive, three-hour commute becomes a relaxing period to watch a Netflix series or catch up on emails.

Considering the probable social change of self-driving EVs led me to consider the changes over the past several decades. In Canada, as an example, lower density areas of the country — the Atlantic provinces, rural areas, small towns and villages — are hollowing out. The high density areas, such as the Golden Horseshoe around Toronto, is a magnetic draw for all of Canada and continues growing at a blistering pace.

Even if a home in the Toronto area costs 5x the price for a given set of amenities, and even if a hypothetical person might prefer lower density, many forces still draw them in.

Which is strange, in a way. I grew up in a small city and seemed to be completely isolated from the larger world. Calling a relative 20 minutes down the road necessitated long distance. My town had no computer store, a mediocre BBS, few channels on television, no radio station, etc. There were few resources for learning.

I was wide eyed at the options available in the big city.

Yet today we live in a world where that same small town has inexpensive 100Mbps internet, and can communicate with anyone over the globe in an instant. Where you can order just about anything and have it the next day, or even the same day. Every form of entertainment is available. Every resource and learning tool is a couple of clicks away (aside — education is one area that has yet to see the coming change from the new world). Few of the benefits of the density are missing.

But those same changes led to centralization, and a hollowing out of most of the better jobs, entailing the workforce having to follow.

We centralized government and administration, pulling the school boards and government offices, banks, etc, out of those small towns in the quest for efficiency, moving up the density ladder. Those five small villages amalgamated to a single board, that then got pulled into a larger board in the city an hour up the road, etc. Connectivity means that management for the few remaining auspices of structure can be at a far flung location.

Every medical specialty was moved to larger centers as the ownership of cars became prevalent, and long drives were accepted. Seeing a pediatrician 200km away becomes a simple norm. Service and even retailing gets centralized to some unknown place elsewhere on the globe.

Everything centralizes. Because it can.

Most decent jobs require a move to density. The same forces that gave the convenience of the city in far flung locations also relegated it to being essentially a retirement home.

Reconsidering the probable change of EVs and self-driving cars will likely accelerate that migration.

Updates: Pay Apps / Date-Changing Posts / Random Projects

Recently a couple of readers noticed that some posts seemed to be reposted with contemporary dates. The explanation might be broadly interesting, so here goes.

I host this site on Amazon’s AWS, as that’s where I’ve done a lot of professional work, I trust the platform, etc. It’s just a personal blog so I actually host it on spot instances — instances that are bid upon and can be terminated at any moment — and there was a dramatic spike late in the week on the pricing of c3 instances, exceeding my bid maximum. My instance was terminated with extreme prejudice. I still had the EBS volume, and could easily have replicated the data on the new instance for zero data loss (just a small period of unavailability), however I was just heading out so I just ramped up an AMI image that I’d previously saved, posted a couple of the lost posts from Google cache text, and let it be. Apologies.

Revisiting Gallus

Readers know I worked for a while on a speculative app called Gallus — a gyroscope-stabilized video solution with a long litany of additional features. Gallus came close to being sold as a complete technology twice, and was the source of an unending amount of stress.

Anyways, recently wanted a challenge of frame-v-frame image stabilization and achieved some fantastic results, motivated by my Galaxy S8 that features OIS (which it provides no developer accessible metrics upon), but given the short range of in-camera OIS it can yield an imperfect result. The idea with be a combination of EIS and OIS, and the result of that development benefits everything. I rolled it into Gallus to augment the existing gyroscope feature, coupling both for fantastic results (it gets rid of the odd gyro mistiming issue, but still has the benefit that it fully stabilizes with highly dynamic and complex scenes). Previously I pursued purely a big pop outcome — I only wanted a tech purchase, coming perilously close — but this time it’s much more sedate in my life and my hope is relaxed. Nonetheless it will return as a pay app, with a dramatically simplified and optimized API. I am considering restricting it only to devices I directly test on first hand. If there are zero or a dozen installs that’s fine, as it’s a much different approach and expectation.

Project with my Son

Another project approaching release is novelty app with my son, primarily to acclimate him to “team” working with git. Again expectations are amazingly low and it’s just for fun, but might make for the source of some content.

AMP Isn’t All Bad

AMP (Accelerated Mobile Pages) is generally reviled in the tech community. Highly critical pieces have topped Hacker News a number of times over the past couple of months. That Register piece ends with the declaration “If we reject AMP, AMP dies.“, which you can ironically read in AMP form.

The complaint is that AMP undermines or kills the web. A lesser complaint that it has poor usability (though not all criticism has held up).

Web Developers Can’t Stop Themselves

Facebook has Instant Articles. Apple has News Format. Google has AMP.

Everyone can leverage AMP, whether producer or consumer. Bing already makes use of AMP in some capacity, as can any other indexer or caching tier. AMP, if available, is publicly announced on the source page (via a link rel=”amphtml” tag) and available to all, versus the other formats that are fed directly into a silo. A quick survey of the Hacker News front page found almost half of the entries had AMP available variants, made possible given that exposing AMP is often nothing more than a simple plug-in on your content management system (and would be a trivial programming task even on a greenfield project).

The impetus for these varied formats is the harsh reality that the web has been abused, and is flexible to the point of immolation. This is especially prevalent on mobile where users are less likely to have content blockers or the ability to easily identify and penalize abusive behaviors.

Auto-play videos, redirects (back capture), abusive ads, malicious JavaScript even on reputable sites, model dialogs (subscribe! follow us on Facebook!), content reflowing that happens dozens of times for seconds on end (often due to simple excessive complexity, but other times an intentional effort to solicit accidental ad clicks as content moves). Every site asking to send desktop notifications or access your location. Gigantic video backgrounds filling the above the fold header for no particular reason.

In an ideal world web properties would refrain from such tragedy of the commons behaviors, worried about offending users and on their best behavior. The prevalent usage doesn’t motivate that, however: many simply see whatever tops Hacker News or Reddit or trending on Facebook and jump in and out of content sources, each site having incredibly little stickiness. The individual benefit of good behavior for any particular site declines.

Bad behavior worsens. Users become even less a check on practices. The good emergent sites suffer, everyone sticking to a tiny selection of sites that they visit daily. It parallels the Windows software download market where once we freely adopted whatever was new and interesting, but after pages of toolbars and daemons and malware many just install the basics and take no risks, new entrants finding no market for their efforts.

AMP (and the other options) is the natural outcome of the wild web. It represents padded walls that constrains bad behavior, giving the content priority. It isn’t appropriate for rich web apps, or even marginally interactive pieces like my bit on floating point numbers, but for the vast majority of media it is a suitable compromise, representing an excellent compromise of the power of HTML with the constraint to yield a speedily rendering, low resource utilization solution. Most AMP pages rendering extraordinarily quickly, with absolutely minimal CPU and network usage. Yes, sites could just optimize their content without being forced to, but somehow we’ve been moving in exactly the opposite direction for years. A simple cooperative effort will never be fruitful.

Google thus far has stated that they do not prioritize AMP content in search results, and given how fervently the SEO industry watches their rankings this isn’t as cloaked as one might imagine. They do, however, have a news carousel for trending topics (e.g. “news”), and most if not all of those entries in the carousel are AMP pages on mobile.

The news carousel has merited significant criticism. For instance a given carousel has a selection of items on a trending topic (e.g. Trump), and swiping within one of the articles brings you to the next in the carousel. As a user, this is fantastic. As a publisher, it’s an attack on non-consumption, easily giving users a trustworthy, speed mechanism of consuming their content (and seeing their ads and their branding, etc).

Other criticism is more subtle. For instance all AMP pages load a script at https://cdn.ampproject.org/v0.js, which of course is served up by a surrogate of Google. This script has private caching and is mandated by the spec, and is utilized for metrics/tracking purposes. Ideally this script would be long cached, but currently it is held for just 50 minutes. If a browser were leveraging AMP it could theoretically keep a local copy for all AMP content, disregarding the caching rules.

And most criticisms are just entirely baseless. Claims that it renders content homogeneous and brand-less, for instance, yet each site can drop a header with a link to their site, as always, just as they always could. For instance The Register does in the initially linked piece, with a logo and link to the homepage. And then there’s simple user confusion, like the blogger who claimed that Google was “stealing” his traffic after he enabled AMP and discovered that yes, AMP implies allowing caching.

Be Charitable

The root of most toxicity on online conversation boards is a lack of charity: Assuming that everyone who disagrees or does something different is an idiot, malicious, has ill intent or is a part of a conspiracy, etc. I could broaden that out and say that the root of most toxicity throughout humanity comes from the same source. If people realized that others just made mistakes when they made a dumb move on the roadway — the foibles of humanity — instead of taking it as a personal affront that must be righted, we’d all have less stressful lives.

This applies to what businesses do as well. We can watch moves like AMP and see only the negatives, and only malicious, ad-serving intentions, or we can see possible positive motives that could potentially benefit the web. Google has developed AMP openly and clearly, and has been responsive to criticism, and the current result is something that many users, I suspect, strongly benefit from.

I’d take this even further and say that the model should be carried to a “HTML Lite” that rolls back the enormous flexibility of HTML5 to a content-serving subset, much like AMP but geared for the desktop or other rich clients. If we could browse in HTML Lite on the majority of sites, enabling richness only for those few that make a credible case, it would be fantastic for the web at large.

Small Optimizations

A couple of months ago I posted an entry titled Rust Faster Than C? Not So Fast. It was simple analysis of the k-nucleotide benchmark game, my interest piqued by Rust taking the lead over C.

After a brief period of profiling I proposed some very simple changes to the C entrant, or rather to the hash code that it leverages (klib) that would allow C to retake the lead. It was pursued purely in a lighthearted fashion, and certainly wasn’t a dismissal of Rust, but instead was just a curiosity given how simple that particular benchmark was (versus other benchmarks like the regex benchmark where Rust’s excellent regular expression library annihilates the Boost regex implementation, which itself eviscerates the stdlib regex implementation in most environments).

The changes were simple, and I take absolutely no credit for the combined work of others, but given that a number of people have queried why I didn’t publish it in a normal fashion, here it is.

Original Performance
3.9s clock time. 11.0s user time.

[on the same hardware Rust yields 3.6s clock time, 9.8s of user time, taking the lead]

Remove a single-line pre-check that while ostensibly for speed-ups, actually was a significant net negative.

Updated Performance
3.3s clock time. 9.55s user time.

Such a trivial change, removing what was assumed to be a performance improvement, yielded a 15% performance improvement. This particular change has zero positive negative impact so I have submitted a pull request.

Switch the flags from bit-packing into 32-bit integers to fixed-position byte storage

Updated Performance
2.9s clock time. 7.4s user time.

As mentioned in the prior post, the implementation used a significant amount of bit packing, necessitating significant bitshifting. By moving to a byte storage it remains incredibly efficient for a hash table, but significantly reduces high-usage overhead.

If the first pull request is accepted, C will take the lead again. Does it matter? Not at all. But it’s the fun of the spirit of the game.

Regardless, as I finally start contributing to public projects it was an opportunity to share the notion of profiling and the high impact of incredibly simple changes.