Mindful Software Development

This is a minor distraction piece while I complete the promised article on some fun high performance SIMD financial calculations across several programming languages.

What Is Mindfulness?

Skip this section if you just want to get right to the part speaking directly to software development.

The topic of mindfulness often segues to meditation, which is a rewarding activity that unfortunately falls in Venn diagrams of various superstitions and spiritual beliefs: I’m not pitching any of those. While I’ve long pursued the practice of meditation (finally having some success as of late1), my interest is philosophical and psychological: Gaining a tool of mental relaxation and focus, both critical abilities in the software development/technology field. This piece is more an observation of personal discoveries rather than advice, so consider skeptically, implement appropriately, and adapt according to your own results.

Mindfulness is something we all experience2, and is common when we encounter changing or unexpected conditions.

The first snowfall of the season is often a mindful experience. The crisp air buffeting against your face. The gentle murmur of snowflakes blanketing the ground, absorbing and dulling the ambiance of the normal environmental sounds. Everything feeling remote and less real, and a little more magical. We note the warmth and humidity when returning indoors, with a new awareness of every unique scent.

We enjoy the crackle of the fireplace, and the smell of burning wood. The water gurgling to a boiling over the wood stove, and the ebbs and flows of patterns created when poured into your hot chocolate. The billowing steam dancing atop the cup. The comfort of a warm drink.

We have a heightened sense of mental clarity.

That is mindfulness. It is being completely involved with the present – by being truly present – absorbing and interpreting all of the senses.

My dog, Piper, hanging out while I work

Put in a computer science perspective, mindfulness is like assigning the process responsible for interpreting the now a high priority, preventing it from being pushed to a background role. Mindful periods are often recalled as if time had slowed down (reported especially when we experience emergency mindfulness, like in a car accident or fall), which from a computer science perspective has a rational explanation: Normally our senses, which are the significant arbiter of our perception of time, are a low priority background thread, consuming some hypothetically low percentage of our thought capacity, but in a period of mindfulness it gets far more of the cycles, experiencing much more in a given period of time.

Some experience mindfulness with chemical assistance, where we might become fascinated by things that we normally overlook. A glass of wine in and I find myself captivated by diced onions sizzling in a pan as they dance atop a thin layer of hot oil, overwhelmed with the beauty of the aromas. I become fascinated as slivers of steam wisp off to their escape.

People’s experiences with various illicit substances, often at enormous human cost, is oft pursued as chemically assisted mindfulness. To enjoy and experience the present, people often sacrifice their future.

Meditation can be used to exercise mindfulness, gaining a better handle on safely achieving it on demand. A common exercise is mindful breathing where you sit in a comfortable position (e.g. one where you’re safe and secure and not in distracting discomfort, usually in a quiet location) and focus on the senses involved with breathing: The rush of the air into your nose; The rise of your chest; The fall as you breath out of your mouth. Or you might mindfully focus your attention on parts of your body, “scanning” the various feelings that ordinarily you completely disregard.

Mindfulness is being fully involved. Of being completely invested and aware of your existence at that moment. Not caught in the past or the future, or planning or reconsidering or reliving, or endlessly going through internal narratives and planning about social interactions and obligations, anxiously considering what to say next to the peer sitting with you or in the meeting tomorrow or reliving the meeting yesterday, but being entirely focused on the moment and the beauty and complexity of the world around us.

That’s mindfulness.

It seems obvious and trivial, and seemingly available on demand, but actively engaging in mindfulness is a difficult task, our minds having an evolutionary tendency to try to move anything routine or expected to a background process, clearing capacity like an over-eager paging algorithm for the event that something unexpected comes along. Noisy contemplation about the past and the future fill the vacuum.

From a survival perspective, where historically threats were everywhere and our continued existence was a moment to moment gambit, having a mental priority model and IPC that focused on deviations from the norm would be an advantage: Ignore everything not deemed a threat or critical, and only apprise me consciously of changes while I worry about tigers in the area and contemplate how they almost got me a week ago and ways to avoid making that mistake again, predicting how they might get me this time. Taking in the trivia of a moment isn’t an advantage or a luxury to be enjoyed when threats are everywhere and many activities have a very rapid risk/reward return.

Survival dictates that you only stop and smell the roses when they’re novel and you’re figuring out if you can eat them or if somehow they’ll eat you. From then on they’re irrelevant.

We now live in a world where threats are rare, and our survival (or rather prosperity) usually relies upon long term activities, often with a long payback period. Where increasingly careers depend upon intense, sustained focus and attention to very small details, but we often live incredibly repetitive lives, with the same sensory inputs day after day, performing close to identical tasks. Driving the same roads. Drinking the same coffee. Making the same small talk. Having the same arguments. Doing many of the same tasks in slightly different ways.

A copy of a copy of a copy.

The software development field isn’t immune to repetition, and many if not most developers have spent decades doing minor variations on a theme. Most social news sites see the bulk of their traffic during the audience’s work days as we look for something to try to add some novelty to our days, rapidly jumping into and out of links on social news, rationalizing that it’s somehow making us better at our jobs.

[note: try creating a list of “how social news made me better today” at the end of each day. Most days it will be entirely empty, because the truth is that most content is extremely low value, and the minority of enriching content is often quickly scanned as we look to see the things to find wrong – particularly on programming related venues – or just to confirm our longstanding assumptions]

We pass time making grand plans for the future, and anxiously reflecting on the many plans that drifted into the past uncompleted.

Mindfulness is spending some time in the present. It makes life more enjoyable. It makes it more interesting, and makes us better witnesses of our environment, where we are surrounded by incredible beauty (cue the oft parodied plastic bag scene from American Beauty).

Being mindful adds enjoyment to life, and gives us more control over our monkey brain. Controlled mindfulness is the ability to appreciate the routine: That coffee that was just like every other coffee you’ve drank for the past three years is arguably better than one that the greatest king in the middle ages could have enjoyed. As is almost every food you enjoy.

We are surrounded by incredibly luxuries, and enormous beauty, that the process of acclimation makes us jaded and dulled about.

On Demand Mindful Programming

Software development is a mental exercise. We have problems that we focus our neural powers upon and generate results.

Optimal software development is a balance between planning for the future, incorporating lessons from the past, and implementing in the now. The now is the part where we work in the IDE, fingers to the keyboard, grinding out beautiful code and designs. Or it’s the part where we work on a whiteboard and plan for the future. Or it’s the part where we analyze the defects of the past and plan the changes the improve the situation.

Whatever embarrassment, pride, shame, or arrogance we have about the past, and whatever worries about scaling or security or re-usability we might have for the future, the now is when we make a difference.

So what does mindfulness have to do with programming?

Mindfulness is the key to optimal effectiveness in this field.

The zone — the magical mental state where one becomes intensely focused on their effort — is applied mindfulness. The ability to actively control or at least maximize the likelihood of a mental flow state is very beneficial.

I’m not going to describe how to meditate on here – it is a very well worn subject – but it is worth an investment of some time. Meditation is the longer term approach, but given all that I’ve written above what could we draw as lesson for a non-practitioner today? How could you better achieve mindfulness today?

I have a couple of suggestions to give a try. The hope is to force your monkey brain to focus on the now, which can be used to bridge to a flow state.

Mix up your environment and routine.

Change the background color of your IDE. Change the font color. Change the font (a well known trick in the proof-reading world). Sit at a different desk. Sit in a different building (e.g. a library or a cafe, or the cafeteria, tailoring to your ability to ignore distractions). Sit on a bouncy ball. Turn your monitor to portrait mode. Switch to a laptop. Put your mouse on the left side. Skip breakfast, or skip lunch. Use a standing desk one day and a sitting desk the next. Start at 6am one day and noon the next (obviously contingent on ability to do so). Take different routes to work, or stop at different shops. Take different doors. Experience different scents. Talk to different people. Work in the dark. Work outside.

Change things up daily.

This notion of changing one’s environment goes completely contrary to most advice in this industry, where the dominant advice is one where you find the conceptually perfect setup with the perfect colors and fonts and seating and situation, and then follow a routine regiment. The foundation of that approach is that you focus on only the code, all of the other distractions removed by making them blend into the background.

It’s wrong, or at least wrong for most people. The net result of this consistency is that life becomes a grey and dull monotony with a detachment from the now. We skitter around looking for something to try to draw us in (we look for a funny cat picture to jar us into conscious existence), and lose the ability to focus or solve difficult problems.

This notion of focusing better after mixing things up has been demonstrated countless times, but people usually draw the wrong conclusion.

Switching to a standing desk likely didn’t make you more productive. Change itself made you more productive. The same can be said of the countless blog posts on going out to lunch, staying in for lunch, sitting on a yoga ball, using a new chair, working in cafes, getting up extra early, working in notepad (in digital or paper form) or a different IDE, etc. People often euphorically report on the increased productivity seen by a change, never attributing it change itself. There are seldom follow-up reports on these silver bullets as the new and novel turns into the old and mundane.

People experience change and report magnificent results. We often become more capable when a new member joins our team and there’s that period of the new, and just about everyone reports how fantastic it is when they first start a new job, or move to that cool new platform or language.

There is a well known effect that’s oft referenced called the Hawthorne effect, which in a nutshell is the belief that being monitored makes people (temporarily) more productive (demonstrating the observer effect). In that original experiment the observers were trying to measure if lighting changes would improve productivity, so they changed the lighting and productivity improved, slowly fading back to normal. They changed it again and productivity improved, returning to normal. Then they changed it to the original lighting and productivity improved yet again, seemingly undermining the entire study. There have been many similar experiments, including some where the subjects weren’t aware that they were being studied, and again a change of conditions yielded improved productivity.

The reason seems obvious, doesn’t it? While we all strive for consistency under the notion of removing distractions, for most of us consistency is the greatest distraction of all. Monotony makes the effort of mentally focusing on the now a herculean task (which can be embraced as essentially monotonous meditation, of a sort, where the intent is to deeply force a focus on the mundane, such as the case of focused breathing, or even things like a Japanese tea ceremony — this isn’t really usable when it comes to the environment around your work, however), in absolute conflict with the intended effect.

Switch your world up. See how it impacts your focus on the code. Look into meditation.

[the pictures in this post were taken minutes before hitting publish on this post — my dog Piper who always hangs in here when I’m working in the home office, and a few out the back as spring makes its presence known]

1 – I’ve started meditating while exercising on the elliptical. I love the elliptical because it’s smooth and presents very little abuse on the body, and allows me to be mentally free. I have found it very useful to practice meditation while doing this, killing two birds with one stone as it is. And yes, you can meditate doing almost anything that doesn’t demand your attention, and the rhythm and blood flow of the elliptical makes the meditation extra immersive.

2 – As always, there’s a caveat about terminology: Get in a discussion with a serious Zen aficionado and mindfulness will often demand a more restrictive set of conditions, and some demand that it mean being essentially free of thought. In this piece I am referring to mindfulness as truly being wholly and completely present, which is something that we seldom truly experience.

Rust Faster Than C? Not So Fast…

On Hacker News a few days ago –

That submission links to the Computer Language Benchmark Game leaderboard for k-nucleotide, where the top performer is a Rust entrant at the time of this post.

Which is impressive. Rust is a relatively young language (although it heavily leverages LLVM on the back-end, standing on the shoulder of a giant), and is making waves while bringing a lot of needed concurrency and memory safety benefits.

For some classes of problems, Rust is proving itself a compelling option even in performance critical code.

This is a very simple test of rudimentary functionality, however, so if C is lagging comparatively, some part of its implementation is sandbagging or focusing on another optimization.

Fired up VTune And Profiling the Laggards

A quick profile and the overwhelming outlier were identified as the set of instructions geared to bit mangling to test and set occupied and deleted flags: In an effort to optimize memory usage, khash stores flags — one for empty or used, and another to indicate deletion — for each hash bucket in a bit array, 16 buckets sharing a single 32-bit integer courtesy of bit packing.

While memory efficient, the process of determining the host integer for a bucket, and then the bit offsets for each constituent, adds significant overhead to the process: It’s a tiny cost by itself, but when iterated hundreds of millions of times becomes substantial. This was made worse by the code style (shoehorning generic-type functionality in a C header file) making it difficult for the compiler to optimize to bit test instructions.

This sort of optimization is an interesting one because the overhead of bit mangling could theoretically be balanced out by a better cache hit rate — more data in less memory — though this is unlikely to hold for an open addressing, double-hash implementation, as they’re generally cache unfriendly.

Curious to the impact, I switched the flag type to an unsigned char and eliminated the bit shifting (there is still fixed bit testing for the first and second bits). The trivial modification to khash.h can be found here (edit: I updated that version slightly to add another small optimization and to improve the readability of the flags).

C Takes The Lead Again

This yielded identical calculations/results, and the same max memory usage (130,592 per time -v), but completed some 32% faster than the original khash implementation, the hashing being significantly faster with the lagging points moving to input parsing and other overhead of the test. My test device is obviously not the same as the one they’re running (though the results are very similar), but assuming the same speedup it would push the C implementation to 4.32 seconds —  a sizable lead — courtesy of a trivial, and arguably optimal change. Memory overhead is slightly worse, with 6 wasted bits per bucket, but the change dramatically reduces the overhead of bucket operations.

GCC Yielded Faster Code Than Clang

As an aside, several of the comments on that discussion questioned why clang wasn’t used, the hypothesis being that it’d yield better performance courtesy of more optimization. In my quick tests, including generating profile outputs and using profile-guided optimizations, GCC generated better optimized code, always winning the performance battle for this test. This was between GCC 6.2.0 and clang 3.9.1. And as an interesting aside, the GCC and clang built binaries were significantly faster running in a VirtualBox Linux VM1 than the binary built with either Visual Studio 2015 or 2017, running in the host environment, even when factoring out the slow stdin handling of Windows.

The C/C++ compiler in Visual Studio has gotten very good (and surprises me with its standards compliance, once its biggest weakness), but falls behind in this test.


There is no conclusion: Rust is great, C is great, and profilers are an awesome tool to lean on. It was a fun exercise in leveraging a performance profiler (in this case VTune). It took seconds to identify the “culprit”, which was a memory optimization that comes at the cost of a significant number of CPU cycles. Shifting the focus to performance, and away from optimizing the memory profile, and performance improves significantly.

These game/toy benchmarks don’t mean a tremendous amount, and no nights should go without sleep over them, but they are a fun exercise and an entertaining distraction.

1 – Totally irrelevant for this test (this is not a candidate for vectorization), but it’s notable that you can enable AVX2 availability in VirtualBox VMs via-

VBoxManage setextradata "$vm_name" VBoxInternal/CPUM/IsaExts/AVX2 1

CMOS and Rolling Shutter Artifacts / Double Precision Compute

EDIT (2017-03-01) – It’s been a bit quiet on here, but that will change soon as some hopefully compelling posts are finished. I’ve taken a couple of weeks to get into a daily fitness and relaxation routine (I would call it a meditation if that term wasn’t loaded with so much baggage), organize life better, etc. Then it’s back to 100% again with these new habits and behaviors.

While I finish up a more interesting CS-style post (SIMD in financial calculations across a variety of programming languages), just a couple of interesting news items I thought worth sharing.

In a prior entry on optical versus electronic image stabilization I noted rolling shutter artifacts –an image distortion where moving subjects or the entire frame during motion can be skewed and distorted — and their negative impact on electronic stabilization.

During video capture, especially under less than ideal conditions, it is a significant cause of distortion that often goes unnoticed until you stabilize the frames.

Sony announced a CMOS sensor with a data buffering layer that allows it to have something approximating a global shutter (Canon previously announced something similar). While their press release focuses on moving subjects in stabilized type situations, the same benefit dramatically reduces the rolling shutter skew during motion of video capture. It also offers some high speed capture options which is enticing.

Sony sensors are used by almost every mobile device now, so it’ll likely see very rapid adoption across many vendors.

EDIT: Sony is already showing off a device with a memory-layer equipped CMOS, so it’s going to become prevalent quickly.

Another bit of compelling news for the week is the upcoming release of the GP100, Pascal-based workstation GPU/compute device by nvidia (they previously released the P100 server based devices).

Double-precision calculations of upwards of 5 TeraFLOPS (for comparison a 72-core/AVX-512 Knights Landing Xeon Phi 7290 offers about 3.4 TeraFLOPS of DP performance, while any traditional processor will be some two magnitudes lower even when leveraging full SIMD such as AVX2). Traditionally these workstation cards massively compromised double precision calculations, so this update brings it into much greater utility for a wide array of uses (notably the financial and scientific world where the significant digits of single precision made it unworkable).


I’m okay.

Months of incredible levels of stress, coupled with weeks of little sleep and the recent sudden passing of my brother Darrell, yielded some deeply irrational, illogical thinking that I of course regret, but can’t erase from my timeline. For any sort of intellectual exercise, like software development, no sleep+stress = a recursive loop of ineffectiveness that generates more stress and even less sleep.

I apologize to those I caused distress, from family to remote individuals who I’ve never met but who cared. This is my first opportunity to post anything on this, and the lack of communications wasn’t an intentional act of dramatic exercise.

In any case, a quick set of thanks-

  • thanks to the many people who cared
  • thanks to the Halton Regional Police (the many fantastic officers, and one profoundly talented K9), who literally saved my life
  • thanks to the remarkable staff of Joseph Brant Hospital
  • thanks to the patients/community of 1 West @ JBH, who opened my eyes to the battles that so many are fighting, and offered friendship and support during a rough period

Not to make light of this ridiculous and very serious situation, but this event yielded a few life happenings that I never expected to have in my biography-

  • Chased by a number of officers through a frigid river
  • Taken down/bitten by a police K9 (who was a model of K9 professionalism, and is a beautiful, extraordinarily well-trained canine officer)
  • Committed involuntarily under the mental health act (e.g. escorted by a guard, locked ward, bed checks, restricted to a hospital gown for several days)

It was an interesting experience that I don’t plan on repeating. The week+ in the hospital gave me the quietest, most reflective period that I’ve had since…forever.

It was the first time I’ve ever truly, successfully meditated. The real, lotus-pose, mind-at-ease meditation that went on for tens of minutes.

I learned an enormous amount about myself as a result (and got various health tests that were long overdue, being conveniently located and all), and came out of it a much better person. Finally dealt with some pretty severe social anxiety that has always been a problem for me.

And for those concerned, various other things got resolved to a good outcome at the same time. The stress+overwhelming tiredness clouded my eyes to options that were available, and everything else is in a much better place.


Gallus / Lessons In Developing For Android

I recently unpublished Gallus — the first and only gyroscope stabilizing hyperlapse app on Android, capturing and stabilizing 4K/60FPS video in real time with overlaid instrumentation like route, speed, distance, etc — from the Google Play Store. It had somewhere in the range of 50,000 installs at the time of unpublishing, with an average rating just under 4.

It was an enjoyable project to work on primarily because it was extremely challenging (75% of the challenge being dealing with deficiencies of Android devices), and in the wake of the change a previous negotiation to sell the technology/source code and possible involvement was revived. That old discussion had been put on indefinite hold in an advanced stage when a buyer was acquired by a larger company, which quite reasonably led them to regroup to analyze how their technologies overlapped and what their ongoing strategy would be.

I put far too many eggs into that deal going smoothly, leading to significant personal costs. It taught me a lesson about contingencies. I’m pleased it is coming to fruition now.

I never had revenue plans with the free app by itself (no ads, no data collection, no coupling with anything else…just pure functionality), and ultimately the technology sell was my goal.

Nonetheless, the experience was illuminating, so I’m taking a moment to document my observations for future me to search up later-

  • Organic search is useless as a means of getting installs. Use a PR-style approach. 50,000 is two magnitudes below my expectation.
  • No matter how simple and intuitive your interface is, people will misunderstand it. When a product is widely used we naturally think “I must be doing something wrong” with the worst interfaces, putting in the time and effort to learn its idioms and standards. When a product isn’t widely used we instead think “This product is doing something wrong.” and blame the product. I abhor the blame the user type mentality, but at the same time it was eye opening how little time people would spend with the most rudimentary of investigations before giving up.
  • The higher expectations are about your product — the more a user wants to leverage it and sees it adding value to their life — the more vigorous and motivated their anger/disappointment is if for some reason they can’t. Over the duration on the market some of the feedback I got because an app wouldn’t work on someone’s strange brand device that I’d never heard of, with a chipset that I’d never seen before, was extraordinary. A sort of “you owe me this working perfectly on my device“.
  • The one-star-but-will-five-star-if-you… users are terrible people. There’s simply no other way to put it. Early on I played along, but quickly learned that they’ll be back with a one star in the future with a new demand.
  • I have spoken about the exaggeration of Android fragmentation on here many times before: For most apps it is a complete non-issue. Most apps can reasonably target 4.4 and above with little market impact. Cutting out obsolete devices will often just save you from negative feedback later — it is Pyrrhic to reach back and support people’s abandoned, obsolete devices — but if you do try to reach back for 100% coverage it’s made easier with the compatibility support library.
    At the same time, though, if you touch the camera pipeline it is nothing but a world of hurt. The number of defects, non-conformance, broken codecs, and fragile systems that I encountered on devices left me shell-shocked. Wrong pixel formats, codecs that die in certain combinations with other codecs, the complete and the utter mystery of what a given device can handle (resolution, bitrate, i-frame interval, frame rate, etc).For a free little app I certainly couldn’t have a suite of test devices (for a given Samsung Galaxy S device there are often dozens of variations, each with unique quirks), and could only base my work on the six or so devices I do have, so it ends up being a game of “if someone complains just block the device”, but then I’d notice months earlier that someone on a slight variation of the device gave a five star review and reported great results.

I enjoyed the project, and finally it looks like it will pay off financially, but the experience dramatically changed my future approach to Android development.