StudyHack's Stretch Churn

Although I am no longer a student, I really enjoy reading Cal Newport's StudyHacks blog. In particular, I like its focus on achieving success through good time management, hard focus, deliberate practice, and obtaining outstanding skill.

In this post on James McLurkin, Cal discusses an interesting concept called Stretch Churn. Paraphrased from Cal's post:

• Stretch Project: A project that requires a skill you don't have at the outset. Importantly, a stretch project is hard enough to stretch your ability but reasonable enough to be completed.
• Stretch Churn: The number of stretch projects you complete per unit time.

The premise is that the higher your stretch churn rate the more likely you are to obtain the kind of skill required to be a leader in your chosen field. As the interview with James demonstrates, highly successful people are adept at maintaining a high stretch churn rate. I suspect this is one of the underlying attributes of Outliers.

I think the stretch churn concept is an important insight because it clarifies how to apply the deliberate practice concept in engineering and research environments. Instead of working on a single problem over a long period of time - a common approach in research - the stretch churn concept suggests that it is better to work on a series of related, hard-but-achievable projects. In a way, this strikes me as the agile development model applied to becoming a domain expert.

On a personal level, I found the stretch churn concept interesting for two reasons. First, it explains why I highly value my advanced development experience - the very nature of the work has allowed me to maintain a high stretch churn rate for years. Second, it helped me realize that if I want to become a real domain expert that I'll have to more tightly focus my stretch projects so that they build upon each other. It's a vector math problem - stretch projects in many different directions result in little change when added together.

I suspect the stretch churn concept will be a valuable addition to my self-development toolbox.

Recovering Deleted JPEGs from a FAT File System - Part 9

Part 9 in a series of posts on recovering deleted JPEG files from a FAT file system.

A month ago (!), in part 8, we looked at the JPEG file format specification to determine if there was sufficient determinism in the on-disk layout to allow the recovery of deleted files through analyzing the residual data in the file system. The answer was mixed:

1. GOOD: Uniquely valued markers, discoverable through data inspection, identify the beginning and type of the segments that constitute a JPEG file.
2. GOOD: the metadata segments have a pre-defined size
3. BAD: the length of the entropy encoded image data is, to the best of my knowledge, unspecified in the START-OF-SCAN segment header. Instead, an END-OF-IMAGE marker is used to identify the end of the entropy encoded data. The theory is that this is done to allow JPEG files to be written as the image is processed.

Essentially, this means that there is no way to determine through data inspection the length or location of the clusters containing the encoded image data. The only clue available is the END-OF-IMAGE marker at the end of the entropy encoded data.

One option is to discover and analyze latent directory entries in the data area - doing so could provide valuable clues to the start and length of erased JPEG files. The downsides to this approach are added complexity (recovering deleted directory entries) and incompleteness (directory entries for deleted JPEG files may not exist due to reuse).

A simpler approach is to inspect each cluster in the data area to see if it begins with a START-OF-IMAGE marker or contains an END-OF-IMAGE marker. Any extent of clusters bounded by START-OF-IMAGE and END-OF-IMAGE markers stands a good chance of being the data for a contiguous JPEG file - the very kind of file we've been trying to recover in this series. In this post, I'll implement this simple method and test the results. Follow the "Read more" think for the rest of the post.

Wait a moment...

The other day, I posted a lament about modern (popular) programming being mostly a matter of connecting pre-existing components or libraries with minimal work. In the post I stated that I didn't find such work satisfying.

The next day, however, I realized something - I've just spent nearly a decade in advanced development roles happily creating prototypes by modifying and combining pre-existing software components with the minimal possible work. Spot the inconsistency?

This perplexed me - why did I react this way to Mike Taylor's post when I have enjoyed such prototyping work so much?

After some thought, I concluded that my prototype work has involved modifying complex systems. This has required first understanding enough about each system's design and software to determine the minimal changes required to implement the desired functionality. So, although the eventual changes were relatively minor (100s to 10Ks LOC), along the way I had to obtain a deep understanding to complete the task. I think this is the material difference between the prototyping work I've enjoyed and the kind of "library gluing" that I dislike (and Mike of course!). If true, then there is no inconsistency after all.

Clearly the issue isn't black-and-white - the amount of work performed doesn't represent the amount of understanding required. To some degree this reminds me of the Simplicity Cycle - after a certain point, enough understanding is achieved to make the solution simpler, not more complex. From this perspective, I suspect that the satisfaction that I - and perhaps others - seek is the result of crossing that complexity-understanding threshold - this may be the "grokking" point that was easier to achieve in simpler times (e.g. 8bit programming).

Grokking and Modern Programming

For this blog, I try not to repost popular links from high traffic aggregation sites - odds are you've already seen them. But Mike Taylor's comments in this post so resonated with me that I felt compelled to discuss it despite the attention it has received.

While reviewing a classic Commodore64 book, Mike segues into a follow-up discussion to his What Ever Happened to Programming post in which he says:

So I think this is part of what I was bemoaning in Whatever happened …: the loss of the total control that we had over our computers back when they were small enough that everything you needed to know would fit inside your head. It’s left me with a taste for grokking systems deeply and intimately, and that tendency is probably not a good fit for most modern programming, where you really don’t have time to go in an learn, say, Hibernate or Rails in detail: you just have to have the knack of skimming through a tutorial or two and picking up enough to get the current job done, more or less. I don’t mean to denigrate that: it’s an important and valuable skill. But it’s not one that moves my soul as Deep Knowing does.

I found this comment insightful as it made me realize that I have been possibly struggling with the same thing.

For as long as I can remember, I've felt compelled to Deeply Know the systems that I work on. Starting with my own adolescent experiments programming 8bit computers, I've enjoyed diving deep into the machine to understand its fundamental operation and then using that knowledge to grok whole-system behaviors. As Mike states in the post, this was possible to do in the 8bit days due to the simple machines and books providing all of the necessary information (my tome was Your Atari Computer which still sits on my bookshelf as a reminder of those happy times).

Unfortunately, two things have happened since then - machines have gotten more complicated and having a deep understanding seems to be less valued.

The first point is obvious, computers have gotten more complicated on every level - hardware architectures, operating systems, applications, and networking. Although the fundamentals can be learned, obtaining deep expertise in any of these areas requires specialization.

Having spent most of my career working on large-scale, enterprise computing and storage systems, I've experienced the leading edge of this expanding complexity first hand. In my first job, I think I did pretty well at understanding large portions of that machine but it's been a losing battle since then. The systems that I now work with are now so numerous and complex that it's impossible to deeply understand them all - but I try.

Lately, I've been trying to do more hobby projects to have fun, and expose myself to domains outside of work. But in doing so I find that I quickly run into the second point that Mike's What Ever Happened to Programming post captures well - the hero of modern programming is the person that can stitch together libraries in the shortest amount of time in the fewest lines of code. The greatest heroes are those that can create and launch a startup in less than 24 hours while on a bus. That's great but it's not for me - I just don't find this form of programming satisfying.

I've been overly nostalgic lately, hence recent posts on clicky keyboards, old computer books, and old computer commercials. At first, I thought a looming birthday was to blame but Mike's post has me thinking that it's really a reaction to the shift in programming. The kind of work I like to do doesn't seem to be where all the "action" (read capital investment) is.

It's unreasonable to expect the world to revert back to the way things were. Therefore, there are two possible reactions:

1. Deal with it and change with the times
2. Pick a niche were it is possible to "grok" the system and specialize. This doesn't mean building everything from scratch but rather picking a stable domain that allows an accumulated understanding of as much of the system as I care to know.

Traditionally I've been a generalist but I must admit that option 2 seems much more attractive than 1. I guess this is something to reflect on while planning out my future career.

Keyboard Madness

Last year, I decided to finally become a competent touch-typist and better Emacs user. An unforeseen result of both decisions is that I am developing a mild keyboard obsession.

For the past couple of years I have been using a Microsoft Natural Ergonomic keyboard which has been good but lately I've felt the urge for a change - an irrational desire I'm sure.

Somewhere along the way in considering a new keyboard, I thought it would be cool to get a mechanical model. Probably due to nostalgia, the sound of a "clicky" keyboard is just one of those aesthetic things that makes me think of "real" programming and helps me enter a flow state.

Not knowing anything about mechanical keyboards, I thought it best to do some research. Through the power of Google, I found:

• this blog post on mechanical keyboards with a video demonstration of various models.
• this video summarizing the attributes of various mechanical switches.
• many, many YouTube videos of people typing on various keyboards

After reviewing this material, I considered four options:

• Kinesis Advantage Contoured: well known for its ergonomics and "cool factor". However, the high price (~$300) and unusual layout made me hesitant.
• Unicomp Customizer 104: based on the same buckling spring technology as the renown IBM Model M and reasonably priced at $70. I decided against it mainly due to the greater key resistance.
• Das Keyboard: a popular keyboard amongst geeks that uses Cherry Blue switches (tactile, and clicky) and priced at $130. Unfortunately the glossy case reportedly attracts an abnormal amount of dust.
• Filco Majestouch Tenkeyless: an imported keyboard available with either Cherry Blue or Cherry Brown (tactile, no click) switches available from elitekeyboards.com for ~$120.

In the end, I went with the Filco after reading many positive reviews. While I came close to buying the brown switches, I decided that I really wanted that aesthetic "clicky" sound of the blues.

It's only been a couple of days but so far I really like the Filco. The keys feel smooth and solid - my MacBookPro keyboard just feels wimpy now. The switches are definitely clicky, this YouTube video provides a pretty good example. The only negative is that I miss the ergonomic design of the Microsoft keyboard.

I'm just using the Filco at home for now to avoid driving my cube neighbors nuts with all the clicking but if things go well perhaps I'll buy another Filco with the brown switches for work.

Join the mechanical keyboard retro-revolution!

The Kahn Academy

About a month ago I discovered The Kahn Academy. Since then, I've found myself spending spare moments watching the video tutorials to refresh my memory on a number of topics including finance, banking, the credit crisis, statistics, linear algebra, and physics. Although short (~10 minutes), the tutorials are very well done and a great resource.

Evidently, the creator, Salman Kahn, started out making video tutorials for a geographically distant niece. Unexpectedly, other people around the world began watching the videos and sending positive feedback. This motivated Kahn to turn the effort into a non-profit venture dedicated to providing high-quality, free educational materials. An inspiring story.

Also impressive is Kahn's simple approach of drawing diagrams on a "black board" in real-time - very similar to back-of-the-napkin visual brainstorming. Like all good teachers, Kahn's deep understanding allows him to discuss complex topics in a readily understandable manner.

If you've got a spare moment, I encourage you to check out a video.