Mindful Software Development

This is a minor distraction piece while I complete the promised article on some fun high performance SIMD financial calculations across several programming languages.

What Is Mindfulness?

Skip this section if you just want to get right to the part speaking directly to software development.

The topic of mindfulness often segues to meditation, which is a rewarding activity that unfortunately falls in Venn diagrams of various superstitions and spiritual beliefs: I’m not pitching any of those. While I’ve long pursued the practice of meditation (finally having some success as of late1), my interest is philosophical and psychological: Gaining a tool of mental relaxation and focus, both critical abilities in the software development/technology field. This piece is more an observation of personal discoveries rather than advice, so consider skeptically, implement appropriately, and adapt according to your own results.

Mindfulness is something we all experience2, and is common when we encounter changing or unexpected conditions.

The first snowfall of the season is often a mindful experience. The crisp air buffeting against your face. The gentle murmur of snowflakes blanketing the ground, absorbing and dulling the ambiance of the normal environmental sounds. Everything feeling remote and less real, and a little more magical. We note the warmth and humidity when returning indoors, with a new awareness of every unique scent.

We enjoy the crackle of the fireplace, and the smell of burning wood. The water gurgling to a boiling over the wood stove, and the ebbs and flows of patterns created when poured into your hot chocolate. The billowing steam dancing atop the cup. The comfort of a warm drink.

We have a heightened sense of mental clarity.

That is mindfulness. It is being completely involved with the present – by being truly present – absorbing and interpreting all of the senses.

My dog, Piper, hanging out while I work

Put in a computer science perspective, mindfulness is like assigning the process responsible for interpreting the now a high priority, preventing it from being pushed to a background role. Mindful periods are often recalled as if time had slowed down (reported especially when we experience emergency mindfulness, like in a car accident or fall), which from a computer science perspective has a rational explanation: Normally our senses, which are the significant arbiter of our perception of time, are a low priority background thread, consuming some hypothetically low percentage of our thought capacity, but in a period of mindfulness it gets far more of the cycles, experiencing much more in a given period of time.

Some experience mindfulness with chemical assistance, where we might become fascinated by things that we normally overlook. A glass of wine in and I find myself captivated by diced onions sizzling in a pan as they dance atop a thin layer of hot oil, overwhelmed with the beauty of the aromas. I become fascinated as slivers of steam wisp off to their escape.

People’s experiences with various illicit substances, often at enormous human cost, is oft pursued as chemically assisted mindfulness. To enjoy and experience the present, people often sacrifice their future.

Meditation can be used to exercise mindfulness, gaining a better handle on safely achieving it on demand. A common exercise is mindful breathing where you sit in a comfortable position (e.g. one where you’re safe and secure and not in distracting discomfort, usually in a quiet location) and focus on the senses involved with breathing: The rush of the air into your nose; The rise of your chest; The fall as you breath out of your mouth. Or you might mindfully focus your attention on parts of your body, “scanning” the various feelings that ordinarily you completely disregard.

Mindfulness is being fully involved. Of being completely invested and aware of your existence at that moment. Not caught in the past or the future, or planning or reconsidering or reliving, or endlessly going through internal narratives and planning about social interactions and obligations, anxiously considering what to say next to the peer sitting with you or in the meeting tomorrow or reliving the meeting yesterday, but being entirely focused on the moment and the beauty and complexity of the world around us.

That’s mindfulness.

It seems obvious and trivial, and seemingly available on demand, but actively engaging in mindfulness is a difficult task, our minds having an evolutionary tendency to try to move anything routine or expected to a background process, clearing capacity like an over-eager paging algorithm for the event that something unexpected comes along. Noisy contemplation about the past and the future fill the vacuum.

From a survival perspective, where historically threats were everywhere and our continued existence was a moment to moment gambit, having a mental priority model and IPC that focused on deviations from the norm would be an advantage: Ignore everything not deemed a threat or critical, and only apprise me consciously of changes while I worry about tigers in the area and contemplate how they almost got me a week ago and ways to avoid making that mistake again, predicting how they might get me this time. Taking in the trivia of a moment isn’t an advantage or a luxury to be enjoyed when threats are everywhere and many activities have a very rapid risk/reward return.

Survival dictates that you only stop and smell the roses when they’re novel and you’re figuring out if you can eat them or if somehow they’ll eat you. From then on they’re irrelevant.

We now live in a world where threats are rare, and our survival (or rather prosperity) usually relies upon long term activities, often with a long payback period. Where increasingly careers depend upon intense, sustained focus and attention to very small details, but we often live incredibly repetitive lives, with the same sensory inputs day after day, performing close to identical tasks. Driving the same roads. Drinking the same coffee. Making the same small talk. Having the same arguments. Doing many of the same tasks in slightly different ways.

A copy of a copy of a copy.

The software development field isn’t immune to repetition, and many if not most developers have spent decades doing minor variations on a theme. Most social news sites see the bulk of their traffic during the audience’s work days as we look for something to try to add some novelty to our days, rapidly jumping into and out of links on social news, rationalizing that it’s somehow making us better at our jobs.

[note: try creating a list of “how social news made me better today” at the end of each day. Most days it will be entirely empty, because the truth is that most content is extremely low value, and the minority of enriching content is often quickly scanned as we look to see the things to find wrong – particularly on programming related venues – or just to confirm our longstanding assumptions]

We pass time making grand plans for the future, and anxiously reflecting on the many plans that drifted into the past uncompleted.

Mindfulness is spending some time in the present. It makes life more enjoyable. It makes it more interesting, and makes us better witnesses of our environment, where we are surrounded by incredible beauty (cue the oft parodied plastic bag scene from American Beauty).

Being mindful adds enjoyment to life, and gives us more control over our monkey brain. Controlled mindfulness is the ability to appreciate the routine: That coffee that was just like every other coffee you’ve drank for the past three years is arguably better than one that the greatest king in the middle ages could have enjoyed. As is almost every food you enjoy.

We are surrounded by incredibly luxuries, and enormous beauty, that the process of acclimation makes us jaded and dulled about.

On Demand Mindful Programming

Software development is a mental exercise. We have problems that we focus our neural powers upon and generate results.

Optimal software development is a balance between planning for the future, incorporating lessons from the past, and implementing in the now. The now is the part where we work in the IDE, fingers to the keyboard, grinding out beautiful code and designs. Or it’s the part where we work on a whiteboard and plan for the future. Or it’s the part where we analyze the defects of the past and plan the changes the improve the situation.

Whatever embarrassment, pride, shame, or arrogance we have about the past, and whatever worries about scaling or security or re-usability we might have for the future, the now is when we make a difference.

So what does mindfulness have to do with programming?

Mindfulness is the key to optimal effectiveness in this field.

The zone — the magical mental state where one becomes intensely focused on their effort — is applied mindfulness. The ability to actively control or at least maximize the likelihood of a mental flow state is very beneficial.

I’m not going to describe how to meditate on here – it is a very well worn subject – but it is worth an investment of some time. Meditation is the longer term approach, but given all that I’ve written above what could we draw as lesson for a non-practitioner today? How could you better achieve mindfulness today?

I have a couple of suggestions to give a try. The hope is to force your monkey brain to focus on the now, which can be used to bridge to a flow state.

Mix up your environment and routine.

Change the background color of your IDE. Change the font color. Change the font (a well known trick in the proof-reading world). Sit at a different desk. Sit in a different building (e.g. a library or a cafe, or the cafeteria, tailoring to your ability to ignore distractions). Sit on a bouncy ball. Turn your monitor to portrait mode. Switch to a laptop. Put your mouse on the left side. Skip breakfast, or skip lunch. Use a standing desk one day and a sitting desk the next. Start at 6am one day and noon the next (obviously contingent on ability to do so). Take different routes to work, or stop at different shops. Take different doors. Experience different scents. Talk to different people. Work in the dark. Work outside.

Change things up daily.

This notion of changing one’s environment goes completely contrary to most advice in this industry, where the dominant advice is one where you find the conceptually perfect setup with the perfect colors and fonts and seating and situation, and then follow a routine regiment. The foundation of that approach is that you focus on only the code, all of the other distractions removed by making them blend into the background.

It’s wrong, or at least wrong for most people. The net result of this consistency is that life becomes a grey and dull monotony with a detachment from the now. We skitter around looking for something to try to draw us in (we look for a funny cat picture to jar us into conscious existence), and lose the ability to focus or solve difficult problems.

This notion of focusing better after mixing things up has been demonstrated countless times, but people usually draw the wrong conclusion.

Switching to a standing desk likely didn’t make you more productive. Change itself made you more productive. The same can be said of the countless blog posts on going out to lunch, staying in for lunch, sitting on a yoga ball, using a new chair, working in cafes, getting up extra early, working in notepad (in digital or paper form) or a different IDE, etc. People often euphorically report on the increased productivity seen by a change, never attributing it change itself. There are seldom follow-up reports on these silver bullets as the new and novel turns into the old and mundane.

People experience change and report magnificent results. We often become more capable when a new member joins our team and there’s that period of the new, and just about everyone reports how fantastic it is when they first start a new job, or move to that cool new platform or language.

There is a well known effect that’s oft referenced called the Hawthorne effect, which in a nutshell is the belief that being monitored makes people (temporarily) more productive (demonstrating the observer effect). In that original experiment the observers were trying to measure if lighting changes would improve productivity, so they changed the lighting and productivity improved, slowly fading back to normal. They changed it again and productivity improved, returning to normal. Then they changed it to the original lighting and productivity improved yet again, seemingly undermining the entire study. There have been many similar experiments, including some where the subjects weren’t aware that they were being studied, and again a change of conditions yielded improved productivity.

The reason seems obvious, doesn’t it? While we all strive for consistency under the notion of removing distractions, for most of us consistency is the greatest distraction of all. Monotony makes the effort of mentally focusing on the now a herculean task (which can be embraced as essentially monotonous meditation, of a sort, where the intent is to deeply force a focus on the mundane, such as the case of focused breathing, or even things like a Japanese tea ceremony — this isn’t really usable when it comes to the environment around your work, however), in absolute conflict with the intended effect.

Switch your world up. See how it impacts your focus on the code. Look into meditation.

[the pictures in this post were taken minutes before hitting publish on this post — my dog Piper who always hangs in here when I’m working in the home office, and a few out the back as spring makes its presence known]

1 – I’ve started meditating while exercising on the elliptical. I love the elliptical because it’s smooth and presents very little abuse on the body, and allows me to be mentally free. I have found it very useful to practice meditation while doing this, killing two birds with one stone as it is. And yes, you can meditate doing almost anything that doesn’t demand your attention, and the rhythm and blood flow of the elliptical makes the meditation extra immersive.

2 – As always, there’s a caveat about terminology: Get in a discussion with a serious Zen aficionado and mindfulness will often demand a more restrictive set of conditions, and some demand that it mean being essentially free of thought. In this piece I am referring to mindfulness as truly being wholly and completely present, which is something that we seldom truly experience.

Rust Faster Than C? Not So Fast…

On Hacker News a few days ago –

That submission links to the Computer Language Benchmark Game leaderboard for k-nucleotide, where the top performer is a Rust entrant at the time of this post.

Which is impressive. Rust is a relatively young language (although it heavily leverages LLVM on the back-end, standing on the shoulder of a giant), and is making waves while bringing a lot of needed concurrency and memory safety benefits.

For some classes of problems, Rust is proving itself a compelling option even in performance critical code.

This is a very simple test of rudimentary functionality, however, so if C is lagging comparatively, some part of its implementation is sandbagging or focusing on another optimization.

Fired up VTune And Profiling the Laggards

A quick profile and the overwhelming outlier were identified as the set of instructions geared to bit mangling to test and set occupied and deleted flags: In an effort to optimize memory usage, khash stores flags — one for empty or used, and another to indicate deletion — for each hash bucket in a bit array, 16 buckets sharing a single 32-bit integer courtesy of bit packing.

While memory efficient, the process of determining the host integer for a bucket, and then the bit offsets for each constituent, adds significant overhead to the process: It’s a tiny cost by itself, but when iterated hundreds of millions of times becomes substantial. This was made worse by the code style (shoehorning generic-type functionality in a C header file) making it difficult for the compiler to optimize to bit test instructions.

This sort of optimization is an interesting one because the overhead of bit mangling could theoretically be balanced out by a better cache hit rate — more data in less memory — though this is unlikely to hold for an open addressing, double-hash implementation, as they’re generally cache unfriendly.

Curious to the impact, I switched the flag type to an unsigned char and eliminated the bit shifting (there is still fixed bit testing for the first and second bits). The trivial modification to khash.h can be found here (edit: I updated that version slightly to add another small optimization and to improve the readability of the flags).

C Takes The Lead Again

This yielded identical calculations/results, and the same max memory usage (130,592 per time -v), but completed some 32% faster than the original khash implementation, the hashing being significantly faster with the lagging points moving to input parsing and other overhead of the test. My test device is obviously not the same as the one they’re running (though the results are very similar), but assuming the same speedup it would push the C implementation to 4.32 seconds —  a sizable lead — courtesy of a trivial, and arguably optimal change. Memory overhead is slightly worse, with 6 wasted bits per bucket, but the change dramatically reduces the overhead of bucket operations.

GCC Yielded Faster Code Than Clang

As an aside, several of the comments on that discussion questioned why clang wasn’t used, the hypothesis being that it’d yield better performance courtesy of more optimization. In my quick tests, including generating profile outputs and using profile-guided optimizations, GCC generated better optimized code, always winning the performance battle for this test. This was between GCC 6.2.0 and clang 3.9.1. And as an interesting aside, the GCC and clang built binaries were significantly faster running in a VirtualBox Linux VM1 than the binary built with either Visual Studio 2015 or 2017, running in the host environment, even when factoring out the slow stdin handling of Windows.

The C/C++ compiler in Visual Studio has gotten very good (and surprises me with its standards compliance, once its biggest weakness), but falls behind in this test.


There is no conclusion: Rust is great, C is great, and profilers are an awesome tool to lean on. It was a fun exercise in leveraging a performance profiler (in this case VTune). It took seconds to identify the “culprit”, which was a memory optimization that comes at the cost of a significant number of CPU cycles. Shifting the focus to performance, and away from optimizing the memory profile, and performance improves significantly.

These game/toy benchmarks don’t mean a tremendous amount, and no nights should go without sleep over them, but they are a fun exercise and an entertaining distraction.

1 – Totally irrelevant for this test (this is not a candidate for vectorization), but it’s notable that you can enable AVX2 availability in VirtualBox VMs via-

VBoxManage setextradata "$vm_name" VBoxInternal/CPUM/IsaExts/AVX2 1

Gallus / Lessons In Developing For Android

I recently unpublished Gallus — the first and only gyroscope stabilizing hyperlapse app on Android, capturing and stabilizing 4K/60FPS video in real time with overlaid instrumentation like route, speed, distance, etc — from the Google Play Store. It had somewhere in the range of 50,000 installs at the time of unpublishing, with an average rating just under 4.

It was an enjoyable project to work on primarily because it was extremely challenging (75% of the challenge being dealing with deficiencies of Android devices), and in the wake of the change a previous negotiation to sell the technology/source code and possible involvement was revived. That old discussion had been put on indefinite hold in an advanced stage when a buyer was acquired by a larger company, which quite reasonably led them to regroup to analyze how their technologies overlapped and what their ongoing strategy would be.

I put far too many eggs into that deal going smoothly, leading to significant personal costs. It taught me a lesson about contingencies. I’m pleased it is coming to fruition now.

I never had revenue plans with the free app by itself (no ads, no data collection, no coupling with anything else…just pure functionality), and ultimately the technology sell was my goal.

Nonetheless, the experience was illuminating, so I’m taking a moment to document my observations for future me to search up later-

  • Organic search is useless as a means of getting installs. Use a PR-style approach. 50,000 is two magnitudes below my expectation.
  • No matter how simple and intuitive your interface is, people will misunderstand it. When a product is widely used we naturally think “I must be doing something wrong” with the worst interfaces, putting in the time and effort to learn its idioms and standards. When a product isn’t widely used we instead think “This product is doing something wrong.” and blame the product. I abhor the blame the user type mentality, but at the same time it was eye opening how little time people would spend with the most rudimentary of investigations before giving up.
  • The higher expectations are about your product — the more a user wants to leverage it and sees it adding value to their life — the more vigorous and motivated their anger/disappointment is if for some reason they can’t. Over the duration on the market some of the feedback I got because an app wouldn’t work on someone’s strange brand device that I’d never heard of, with a chipset that I’d never seen before, was extraordinary. A sort of “you owe me this working perfectly on my device“.
  • The one-star-but-will-five-star-if-you… users are terrible people. There’s simply no other way to put it. Early on I played along, but quickly learned that they’ll be back with a one star in the future with a new demand.
  • I have spoken about the exaggeration of Android fragmentation on here many times before: For most apps it is a complete non-issue. Most apps can reasonably target 4.4 and above with little market impact. Cutting out obsolete devices will often just save you from negative feedback later — it is Pyrrhic to reach back and support people’s abandoned, obsolete devices — but if you do try to reach back for 100% coverage it’s made easier with the compatibility support library.
    At the same time, though, if you touch the camera pipeline it is nothing but a world of hurt. The number of defects, non-conformance, broken codecs, and fragile systems that I encountered on devices left me shell-shocked. Wrong pixel formats, codecs that die in certain combinations with other codecs, the complete and the utter mystery of what a given device can handle (resolution, bitrate, i-frame interval, frame rate, etc).For a free little app I certainly couldn’t have a suite of test devices (for a given Samsung Galaxy S device there are often dozens of variations, each with unique quirks), and could only base my work on the six or so devices I do have, so it ends up being a game of “if someone complains just block the device”, but then I’d notice months earlier that someone on a slight variation of the device gave a five star review and reported great results.

I enjoyed the project, and finally it looks like it will pay off financially, but the experience dramatically changed my future approach to Android development.

Programming An Itch Away

My eldest son’s rig consists of a gaming PC as his main device, with one of my old laptops on the side for ancillary use.

The audio output on his motherboard of the desktop failed mysteriously (with no driver or BIOS fix resolving the issue, the hardware of that subsystem seemingly failing), and swapping out the motherboard isn’t something I’m keen to do given the activation issues, despite having one available.

With the holiday season, getting a wireless headset or USB soundcard wouldn’t be a speedy venture. And anyways it was an opportunity for a fun software distraction.

Visual Studio powered up, a C++ solution pursued, and an hour later a solution was built. Capturing low-latency 44100 32-bit floating point stereo master mix audio on the desktop PC, resampling it to 48000 16-bit integer audio, compressing it with Opus (the resampling is courtesy of Opus demanding a multiple of 6000 sample rate, while Opus itself is purely because both machines are on wifi, and often have large network traffic, so keeping packets tiny improved the chances of a speedy delivery), UDP sending it to an argument-driven target, where the process reverses and yields an extraordinarily high fidelity reproduction of the source audio, with UDP packets tiny enough that they’re easily delivered by the network even under high saturation situations.

The audio delay is less than 10ms, which is imperceptible, and is a magnitude or more below the delay of many Bluetooth headsets.

So he games and the audio from his desktop plays on his laptop, which he has a set of high quality wired headphones connected to. It works well for now, until a long term solution is implemented (probably the motherboard swap). I could jimmy up an Android solution in minutes, having already done some Opus codec/UDP transport projects.

Those are some of the most rewarding projects. Even if I undertook the facile imagination of fantasy billing out those hours and declaring that the opportunity cost lost, those fun projects expose us to technologies and avenues, gaining educational value. Doing strange and interesting side projects is the vehicle of my most interesting ideas (as is misinterpreting or making assumptions about descriptions of products, then discovering that my assumptions or guesses are vastly off the mark, but have merit as a novel invention)

It’s a pretty trivial need and solution, but having an itch, slamming out a solution, and having a bulk of code simply work with minimal issue on first run is a glorious feeling. In this case the single defect among the sender and receiver, despite the fact that these were APIs that I’d never used before and it involved a considerable amount of bit mangling and buffer management code — the traditional shoot-yourself-in-the-foot quagmires — was a transposition of loop variables in nested loops.

Such a great feeling of satisfaction doing something like that. It is quite a nice change from the large scale projects with slower rewards that we generally ply, where rewards come slowly, if at all, diluted in the effluence of time.

Of course doing a microproject like this yields the sort of tab hilarity that we often endure when we’re dealing with technologies or APIs we don’t normally use.

And for the curious, there are some products that do what I described (send the mixed master audio from a PC to other devices), but each that we tested yielded quarter second or more latency, even over a direct twisted-pair, which just made it useless for the purpose. And even if a suitable solution existed, I really just wanted to build something, so I would have unfairly discarded it regardless.


3D XPoint is Pretty Cool

Five years ago I wrote a post about SSDs/flash storage being a necessary ingredient for most modern build outs, and opponents were wasting time and efforts by not adopting them in their stack. While it is profoundly obvious now, at the time there was a surprising amount of resistance from many in the industry who were accustomed to their racks of spinning rust, RAID levels, and so on. People who had banked their profession on a lot of knowledge about optimizing against extremely slow storage systems (a considerable factor in the enthusiasm for NoSQL), so FUD ruled the day.

Racks of spinning rust still have a crucial role in our infrastructure, often treated as almost nearline storage (stuff you seldom touch, and when you do the performance is so out of bounds of normal expectations). But many of our online systems are worlds improved with latency in microseconds instead of milliseconds courtesy of flash. It changed the entire industry.

In a related piece I noted that “Optimizing against slow seek times is an activity that is quickly going to be a negative return activity.” This turned out to be starkly true, and many efforts that were undertaken to engineer around glacially slow magnetic and EBS IOPS ended up being worse than useless.

We’re coming upon a similar change again, and it’s something that every developer / architect should be considering because it’s about to be real in a very big way.

3D XPoint, co-developed by Micron and Intel (the Intel one has some great infographics and explanatory videos), is a close to RAM-speed, flash-density, non-volatile storage/memory technology (with significantly higher write endurance than flash, though marketing claims vary from 3x to as high as 1000x), and it’s just about to start hitting the market. Initially it’s going to be seen in very high performance, non-volatile caches atop slower storage: the 2TB TLC NVMe with 32GB of 3d xpoint non-volatile cache (better devices currently have SLC flash serving the same purpose), offering extraordinary performance, both in throughput and IOPS / latency, while still offering large capacities.

Over a slightly longer period it will be seen in DRAM-style, byte-accessible form (circumventing the overhead of even NVMe). Not as literally the main memory, which still outclasses it in pure performance, but as an engineered storage option where our databases and solutions directly and knowingly leverage it in the technology stack.

2017 will be interesting.