Saving weblinks to org-mode from Safari

Each day, I come across numerous web articles, blog posts, newsgroup posts, etc, that appear interesting. Often, I discover them while working on another task. To avoid distraction, I typically save their links for later review. Sometimes I drag the links to my desktop. Sometimes I bookmark them in my browser. Sometimes I send them to myself via email. Sometimes, I post them to my delicious account. It's time to admit that I need a better process.

In a prior post, I mentioned that I use Emacs's org-mode to organize my notes and tasks. I recently setup org-mode's Capture capability to easily record the deluge of thoughts that come at random throughout the day. While reading the documentation, I discovered the solution to my link saving woes, org-protocol. In particular, the ability to capture links to an org file directly from a web browser as demonstrated in this screencast.

Imitating the screencast on OS X turned out to be harder than I expected. So, I thought it worth wile to post my approach.

To begin, it's worthwhile to understand how org-protocol captures work. org-protocol is based on Emacs server which allows applications to use Emacs for text editing. A common practice is to have shells use an already running Emacs instance rather than starting a new one. This is accomplished by a helper program, emacsclient, that communicates with the primary Emacs instance. For org-protocol captures, emacsclient is launched with a specially formatted argument.

emacsclient org-protocol:/capture:/tname/http://foo.com

By advising the server-visit-files function, org-protocol detects such arguments and creates org-mode entries from them. A powerful template facility is provided to specify how the argument information is transformed into an org entry. Multiple templates are supported and selected by the argument's tname field. The remainder of the argument specifies the URL, in this case http://foo.com, and an optional note (not shown in the example).

Templates are specified by elisp code like the following,

(setq org-capture-templates
      '(("tname" "Link" entry 
        (file+headline org-default-notes-file "Links to Read")
        "* %a\n %?\n %i")))

This template, called tname, tells org to save the entry in the default notes file under the header "Links to Read" with the url as a sub-heading. This results in something like,

* Links to Read
** http://foo.com

See the Emacs documentation for all of the template facility's capabilities and options.

In the screencast, two "tricks" are used to have FireFox call emacsclient with the argument needed to capture the present page. The first is a bookmarklet that creates the appropriate emacsclient argument,

javascript:location.href='org-protocol://capture://tname/'+
      encodeURIComponent(location.href)+'/'+
      encodeURIComponent(document.title)+'/'+
      encodeURIComponent(window.getSelection())

The second trick takes advantage of the fact that the emacsclient argument is formatted as a URI. The topmost component of a URI is called the URI scheme. The standard scheme, http, represents HTTP hyperlinks and is handled directly by the browser. Many other URI schemes exist and most browsers support launching separate programs, sometimes called URI handlers, to process them. In the screencast, FireFox is configured to launch emacsclient when links with the org-protocol scheme are "clicked".

Duplicating this functionality with Safari on OSX turned out to be harder than expected. In an effort to make things easy, OSX provides the Launch Services API to automate the registration of URI schemes and their handlers. Unfortunately, a manual method isn't provided to specify new URI schemes and handlers. This means Safari can't simply be told to launch emacsclient when org-protocol links are "clicked".

While searching for a solution, The worg website led me to the org-mac-protocol project. Although promising, org-mac-protocol uses AppleScripts executed via a drop down menu. Call me lazy but I really like the bookmarklet approach. org-mac-protocol also provides far more functionality than I was interested in. I like to keep things simple.

A second look at the Launch Services documentation revealed that two mechanisms are provided to register URI schemes and handlers. Using a programmatic API, applications can, at execution time, register themselves as the handler for new URI schemes. Alternatively, applications can include the URI scheme and handler information in their application bundle property list. Of the two approaches, the property list approach looked like the best to pursue.

Property lists are used by OSX to store application and user settings. They're essentially Objective-C objects serialized to XML. Every OSX application contains a default property list in its application bundle that the Finder reads after certain events. While processing property lists, the Finder will register any URI schemes and handlers it finds with Launch Services. With this approach, all that is necessary to register a new URI scheme and handler is to edit an XML text file.

Unfortunately, modifying Emacs's application bundle plist didn't seem to be an option. I didn't see a way to specify the helper program emacsclient as the URI handler.

I knew from prior experience that AppleScripts can be packaged as application bundles. I reasoned that I could write an AppleScript to launch emacsclient, save it as an application bundle, and modify its property list to register the script as a handler for org-protocol URIs. I suspected that this had been done before and a quick google search led me to this stackoverflow thread.

Using AppleScript Editor and the stackoverflow thread as an example, I wrote the following script and saved it as an application bundle called EmacsClientCapture.app.

on open location this_URL
   do shell script 
      "/Applications/Emacs.app/Contents/MacOS/bin/emacsclient " 
      & this_URL
end open location

Next, I edited the script's plist at the path,

EmacsClientCapture.app/Contents/Info.plist

And added the following XML elements just before the final </dict></plist> tags.

<key>CFBundleIdentifier</key>
<string>com.mycompany.AppleScript.EmacsClientCapture</string>
<key>CFBundleURLTypes</key>
<array>
  <dict>
    <key>CFBundleURLName</key>
    <string>EmacsClientCapture</string>
    <key>CFBundleURLSchemes</key>
    <array>
      <string>org-protocol</string>
    </array>
  </dict>
</array>

I then moved the application bundle to the /Applications directory. This caused the Finder to read the property list and register EmacsClientCapture with Launch Services as the handler for the org-protocol URI scheme.

I added the bookmarket described above to Safari and viola! I can now click on the bookmarklet and save a link to the current page in an org-mode file. No more disorganization. Of course, it's now easier to collect distractions but that is a problem for a future post.

Book Review: On Writing Well

On Writing Well by William Zinsser

Clarity. Simplicity. Brevity. Humanity. Those are the four attributes of good writing that Zinsser promotes and teaches in this classic book on writing.

The book is structured into four parts. Part one, Principles, covers fundamental topics like simplicity, clutter, style, words, and usage. In part two, Methods, Zinsser teaches how to structure a coherent story. Part three, Forms, provides advice on writing interviews, travel stories, technical articles, business communications, and other types of writing. Part four, Attitudes, discusses common feelings and decisions during the writing process. Throughout the book, Zinsser uses examples to illustrate and reinforce his points.

I really liked this book and agree strongly with Zinsser's values. A lot of modern writing is long-winded, complex, and content-free. Simple and clear writing is not only more effective but also more enjoyable. I've always tried to use a simple, direct writing style. I'm eager to apply the book's advice to further improve my writing.

I was greatly encouraged by many of Zinsser's comments on the writing process itself. For example, I consider myself a slow writer. I often spend a lot of time revising my writing to get it "just right". Even simple things like emails and blog posts seem to take an excessive amount of time. I frequently feel insecure about this so I was glad to read the following,

Writing is hard work. A clear sentence is no accident. Very few sentences come out right the first time, or even the third time. Remember this in moments of despair. If you find that writing is hard, it's because it is hard.

Similarly, I was inspired by the following comment in the chapter "Enjoyment, Fear, and Confidence",

Living is the trick. Writers who write interestingly tend to be men and women who keep themselves interested. That's almost the whole point of becoming a writer. I've used writing to give myself an interesting life and a continuing education. If you write about subjects you think you would enjoy knowing about, your enjoyment will show in what you write. Learning is a tonic.

I started this blog to learn by writing about my many interests. I find that explaining a topic often leads to deeper insights. I hope that writing and blogging will help me to continue learning and leading an interesting life.

Brain fitness videos

The "Thoughts about thinking" post motivated me to improve my brain fitness. To that end, I found the following three talks very informative.

• Authors@Google: Dr. John Medina
• GoogleTechTalks: Think faster focus better and remember more, Rewiring our brain to stay younger…
• GoogleTechTalks: Nutrients for Better Mental Performance

I really appreciate Google posting internal guest lectures to YouTube and GoogleVideo. I have watched many of the computer science talks and learned a lot. It's nice to know that talks on many other valuable topics are also available.

Doing less than your best

I once learned a valuable lesson - never do well a task that you don't want to do again.

The situation started out innocently. A task for the project I was working on needed to get done. I disliked the task but grudgingly did it for the benefit of the team. I did my best to do the task well with the expectation that I would be rewarded with more enjoyable work. Instead, I was asked to keep doing it. I protested unsuccessfully and was soon miserable. I couldn't understand how acting selflessly and doing a good job led to such "punishment".

I discovered that when you do any job well the benefiters want you to keep doing it. If they're confident you'll produce good results, they would rather encourage you than find a replacement (who may not do as well). I eventually realized that I shouldn't have done a good job in the first place.

Initially, I found this insight disturbing. As a child, I was taught to always do my best at everything. As I grew up, I tried to do my best at school, sports, hobbies, and part-time jobs. Gradually, "always doing my best" became a core attribute of my self-image. Therefore, the notion of consciously doing less than my best just seemed wrong.

Then I made a discovery - some of my role models consciously do or did less than their best to avoid undesirable work.

For example, the legendary physicist Richard Feynman discusses "actively acting irresponsible" in this BBC interview. He also cautioned against administrative roles in this letter to Stephen Wolfram. Donald Knuth and Neal Stephenson are well known for being "bad" at correspondence. On a somewhat related note, Paul Graham warns in this essay against distracting thoughts and activities. More personally, some of my mentors have privately admitted to performing badly at tasks that they don't want to do.

From these examples I've formulated the following three guidelines:

1. Say "no" to any undesirable tasks.
2. If unavoidable, only do an adequate job.
3. Focus all remaining time on excelling at the work you most want to do.

Following guidelines 1 and 2 will hopefully provide more time for pursuing guideline 3. The ideal result is to be so highly valued for doing work you enjoy that you won't get asked to do anything else - the opportunity cost for distracting you will be too high.

For myself, I expect a lot of time and practice will be required to put these guidelines into consistent application. After all, a self-image can be hard to change. Thankfully, I have role models to encourage me.

Thoughts about thinking

Challenging work assignments, family matters, and home maintenance projects have kept me very busy lately. Hence, the significant drop-off in blog posts over the past couple of months. I'm hoping to correct that soon.

Not surprisingly, I've been thinking a lot about leisure time - in particular its benefits for creative thinking. My reflections are guided by two lectures on the topic.

The first is a GoogleTechTalk by Dr. David M. Levy, No Time to Think. In the talk, Dr. Levy asserts that deep contemplation not only promotes creativity but also provides a sense of calmness and satisfaction. He further argues that deep thinking cannot be forced directly - instead we must make ourselves available to it by seeking out silence, and sanctuary. Dr Levy observes that this requirement for sanctuary conflicts with modern social pressures to multi-task, remain in constant communication, and solve issues through repetitive searches of existing information.

The second is a lecture, Solitude and Leadership, by William Deresiewicz at West Point in 2009. The talk is deeply insightful and worth reading in its entirety but, for this post, can be fairly summarized by the following three points:

1. Original thinking is a core attribute of leadership.
2. Formulating original thoughts requires long periods of concentration and distance from the thoughts of others.
3. Quiet solitude is necessary to do both.

Like Dr. Levy, Mr. Deresiewicz laments social pressures to multi-task, remain in constant communication, and rely on existing information to solve problems. To quote,

Multitasking, in short, is not only not thinking, it impairs your ability to think. Thinking means concentrating on one thing long enough to develop an idea about it. Not learning other people’s ideas, or memorizing a body of information, however much those may sometimes be useful. Developing your own ideas. In short, thinking for yourself. You simply cannot do that in bursts of 20 seconds at a time, constantly interrupted by Facebook messages or Twitter tweets, or fiddling with your iPod, or watching something on YouTube.

and,

Here’s the other problem with Facebook and Twitter and even The New York Times. When you expose yourself to those things, especially in the constant way that people do now—older people as well as younger people—you are continuously bombarding yourself with a stream of other people’s thoughts. You are marinating yourself in the conventional wisdom. In other people’s reality: for others, not for yourself. You are creating a cacophony in which it is impossible to hear your own voice, whether it’s yourself you’re thinking about or anything else.

Thinking originally about difficult problems is an activity that I deeply enjoy and find satisfying. Like Dr. Levy and Mr. Deresiewicz, I reached similar conclusions about distracting activities years ago and have since guarded my time and attention vigorously. I've often been teased for my resistance to social media. At times I've felt self-conscious about this choice, even deficient or outdated. These talks give me renewed confidence in my choices.

The challenge, as indicated at the outset of this post, is finding the time to think. A hard task given the pace of modern society and the high-tech industry in particular.

Org-mode hack: tasks done last month

I'm a big fan of Emacs's org-mode. Over the past year, I've started using it for everything - tracking tasks, taking notes, and drafting all my reports, papers, and blog posts. Org-mode is the only task-tracking software that I've used for more then a week.

At work, I am required to produce a monthly status report. To automate part of the process, I figured out a way to have org-mode produce a list of the tasks completed during a specific month. Since I couldn't find a similar example through a Google search, I thought I would post my approach for the benefit of others (and as a reminder to myself!).

Below is an example org file containing completed tasks that I'll use to illustrate the approach. The tracking closed items feature has been configured to add a time-stamp when each task is transitioned to the DONE state. The header specifies a category, Foo, that org will associate with all of the tasks in the file.

#+Category: Foo

* DONE Feed the dog
   CLOSED: [2010-04-30 ]

* DONE Mow the lawn
   CLOSED: [2010-05-01 ]

* DONE Take out the trash
   CLOSED: [2010-05-20 ]

* DONE Pay the bills
   CLOSED: [2010-06-01 ]

First, configure org-mode's agenda feature and use the C-c [ command to add the example file to the agenda files list.

At this point, a list of the tasks completed in May can be produced by issuing the agenda tag matching command, C-c a m, and giving it the following match string:

CATEGORY="Foo"+TODO="DONE"+CLOSED>="[2010-05-01]"+CLOSED<="[2010-05-31]"

This should produce the following list (slightly reformatted to fit blog width):

Headlines with TAGS match: CATEGORY="Foo"+TODO="DONE"\
+CLOSED>="[2010-05-01]"+CLOSED<="[2010-05-31]"
Press `C-u r' to search again with new search string
  Foo:        DONE Mow the lawn
  Foo:        DONE Take out the trash

Although this works, entering the search string is a cumbersome task. A better solution would avoid this step.

Agenda provides a way to define custom commands that can perform searches using pre-defined match strings. The following elisp code defines a custom command that performs the above tag search automatically.

(setq org-agenda-custom-commands
  `(("F" "Closed Last Month" 
     tags (concat "CATEGORY=\"Foo\""
                  "+TODO=\"DONE\""
                  "+CLOSED>=\"[2010-05-01]\""
                  "+CLOSED<=\"[2010-05-30]\"")))

After eval-ing this command, typing C-c a F will produce the same list as above without having to enter the match string. This approach is indeed better but uses a hard-coded match string. An even better solution would generate the match string based on the current date.

Although the call to concat in the example above programatically generates the match string, it does so only when the setq is evaluated. If the setq is in an initialization file (e.g. ~/.emacs) the match string will get generated based on the date emacs was started and not the date on which the search is performed. This could produce erroneous searches when using an Emacs instance started before the turn of the month. In such cases, the setq could be manually re-evaluated to generate the correct match string but an automatic solution would be best.

Unfortunately, org doesn't currently support providing a lambda to generate the match string at search time. For instance, this example:

(setq org-agenda-custom-commands
  `(("F" "Closed Last Month" 
     tags
     (lambda ()   
       (concat "CATEGORY=\"Foo\""
               "+TODO=\"DONE\""
               "+CLOSED>=\"[2010-05-01]\""
               "+CLOSED<=\"[2010-05-30]\"")))))

produces the error message "Wrong type argument: stringp, …". Patching org-mode to support lambdas for match strings is an option but I prefer to maintain the stock org-mode code.

Thanks to the near infinite hackability of emacs, it's possible to extend the stock org mode functionality without modifying it directly. The below elisp code defines two new interactive functions that call into org-mode to perform a tag search for a specific month.

(require 'calendar)

(defun jtc-org-tasks-closed-in-month (&optional month year match-string)
  "Produces an org agenda tags view list of the tasks completed 
in the specified month and year. Month parameter expects a number 
from 1 to 12. Year parameter expects a four digit number. Defaults 
to the current month when arguments are not provided. Additional search
criteria can be provided via the optional match-string argument "
  (interactive)
  (let* ((today (calendar-current-date))
         (for-month (or month (calendar-extract-month today)))
         (for-year  (or year  (calendar-extract-year today))))
    (org-tags-view nil 
          (concat
           match-string
           (format "+CLOSED>=\"[%d-%02d-01]\"" 
                   for-year for-month)
           (format "+CLOSED<=\"[%d-%02d-%02d]\"" 
                   for-year for-month 
                   (calendar-last-day-of-month for-month for-year))))))

(defun jtc-foo-tasks-last-month ()
  "Produces an org agenda tags view list of all the tasks completed
last month with the Category Foo."
  (interactive)
  (let* ((today (calendar-current-date))
         (for-month (calendar-extract-month today))
         (for-year  (calendar-extract-year today)))
       (calendar-increment-month for-month for-year -1)
       (jtc-org-tasks-closed-in-month 
        for-month for-year "CATEGORY=\"Foo\"+TODO=\"DONE\"")))

The first function, jtc-org-tasks-closed-in-month, generates an appropriate query string and calls the internal org-mode agenda function org-tags-view. The function defaults to the current month but takes optional arguments for the desired month and year. The function also takes a match-string argument that can be used to provide additional match criteria.

The second function, jtc-foo-tasks-last-month, calculates the prior month and calls jtc-org-tasks-closed-in-month with an additional match string to limit the list to DONE tasks from the category Foo. Executing jtc-foo-tasks-last-month interactively automatically produces a list of the tasks closed in the prior month. For my purposes, this is close enough to the ideal solution. Using the optional match-string argument, I can re-use this solution to search for tasks completed in other categories or with specific tags.

My typical work flow is to archive the closed tasks after my status report is written. Org-mode's agenda makes this an easy task. First I mark all of the tasks for a bulk operation by typing m on each. Then I perform a bulk archive by typing the command B $. This will move the closed tasks to an archive file, typically a file of the same name with an added _archive suffix.

Org-mode is a great productivity tool. Combined with Emacs's hackability, it's possible to create tools optimized for your particular work flow.

Addendum

I found that searching on CLOSED date ranges didn't work in org-mode version 6.34a. The problem appears to be fixed in the 6.36c release so be sure to have the right version if you want to replicate this method.

Book Review: The Quants

The Quants by Scott Patterson

In The Quants, Patterson provides an intriguing account of Wall Street's most successful quantitative analysts (aka quants) and the role they played in the subprime crisis.

The first few chapters introduce the main players and provide a brief introduction to quantitative finance. Patterson begins by describing how Ed Thorp applied his mathematics background and experience pioneering Black Jack card counting techniques to invent various hedging techniques and start the first arbitrage hedge fund. From there, Patterson lightly introduces other important concepts like Brownian Motion, Random Walk Theory, Efficient Market Hypothesis, and statistical arbitrage. The introduction winds down with the October, 1987 market crash which is used as the context to introduce the "fat tail" contrary point of view personified by Benoit Mandelbrot, and Nassim Nicholas Taleb - a not so subtle foreshadow.

The "middle" of the book discusses the background, career, and substantial success of primarily five high-profile quants: Pete Muller, Ken Griffin, Cliff Asness, Boaz Weinstein, and Jim Simmons. Other Wall Street personalities are also mentioned but to a lesser extent. This part of the book more or less establishes that the above quants are very smart, and very rich.

The last part of the book provides a blow-by-blow account of the sub-prime crisis. All of the quants appeared to be caught off guard, perplexed by the market's "irrational" behavior, and unshure of how to adjust their models to prevent further losses. Throughout the ordeal, many of the quants are forced to question the very foundations of their mathematical models and prior success - was it all just luck?

Over all, I thought this book was OK but felt it tried to cover too much ground as a quantitative finance primer, homage to quants, historical account of the sub-prime crisis, and financial mystery-thriller. Since my interest lies more in the technical details, I was a disappointed with those portions of the book and uninterested in the dramatized historical account. Perhaps I simply had the wrong expectations of the book.

It's also possible that my expectations were set artificially high by Poundstone's excellent book Fortune's Formula which provides a detailed historical and technical account of the events that gave rise to the quantitative finance industry. If you're interested in this topic, then I highly recommend Fortune's Formula.

While reviewing the book to write this review, one passage caught my eye on page 250 regarding a study performed by MIT Professor Andrew Lo and his student Amir Khandani:

There was also the worry about what happened if high-frequency quant funds, which had become a central cog of the market, helping transfer risk at lightning speeds, were forced to shut down by extreme volatility. "Hedge funds can decide to withdraw liquidity at a moment's notice," they wrote, "and while this may be benign if it occurs rarely and randomly, a coordinated withdrawal of liquidity among an entire sector of hedge funds could have a disastrous consequences for the viability of the financial system if it occurs at the wrong time and in the wrong sector."

There is some evidence indicating that the withdrawal of high-frequency liquidity was a contributing factor to the May 6, 2010 flash crash. I doubt the story of the quants is over just yet.