How to query an on-disk SearchKit index with PyObjC

OS X comes with a framework called SearchKit that applications can use to index content and support full-text searches. I occasionally run into situations where I want to take advantage of an application’s existing SearchKit index so that I can perform my own full-text searches programmatically. The most recent case arose this weekend while I was attempting to build an Alfred workflow for searching Quiver notes.

Quiver uses SearchKit internally to expose full-text search through its own user interface, but it doesn’t have native Spotlight integration yet. To find notes faster, I wanted to make an Alfred script filter that queries the SearchKit index and displays the results. It only took me a few minutes to figure out where Quiver caches its SearchKit index when I started poking around the ~/Library folder. The only remaining obstacle to querying it myself is that I’d need to figure out the actual name of the index in order to read the file with the SearchKit APIs.

Apple doesn’t provide a standard method for retrieving the index name from the SearchKit index file itself. Fortunately, I’m not the first person to tackle the problem. A quick internet search turned up a very helpful blog post by Tim Schröder that includes a detailed explanation of the issue and a snippet of Objective-C code that manually extracts the data from the correct position in a SearchKit binary index file. Armed with Tim’s snippet, I figured out that “index” is the (rather unsurprising) name of Quiver’s index.

I decided to write my SearchKit query script in Python, using PyObjC to access the SearchKit APIs. Even though PyObjC often results in slightly ugly code, I’ve had good luck with it in the past. It seems to work out of the box in most Mac environments and it lets me use native platform APIs without compiling anything. With the help of Apple’s SearchKit programming guide and SearchKit API reference, I put together the following Python code:

import os.path, Cocoa, SearchKit
from CoreFoundation import CFURLGetString

def search(query, fileName, indexName):
  file = Cocoa.NSURL.fileURLWithPath_(fileName)
  index = SearchKit.SKIndexOpenWithURL(file, indexName, False)
  group = SearchKit.SKSearchGroupCreate([index])

  results = SearchKit.SKSearchCreate(index, query, SearchKit.kSKSearchRanked)
  busy, items, scores, count = \
      SearchKit.SKSearchFindMatches(results, 20, None, None, 1.0, None)

  for score, item in zip(scores, items):
    doc = SearchKit.SKIndexCopyDocumentForDocumentID(index, item)
    url = CFURLGetString(SearchKit.SKDocumentCopyURL(doc))
    name = SearchKit.SKDocumentGetName(doc)
    props = SearchKit.SKIndexCopyDocumentProperties(index, doc)
    print url, name, props

The function performs a search on the specified index using the provided query string. When you write your query, you can use boolean operators (&, |, !) and the wildcard symbol (*). For my Alfred workflow, I chose to enclose the user’s query in wildcard symbols in order to find partial matches. The search returns a score for each item to convey the relevance of the match. The URL that the function displays for each result points to the file that contains the match.

My Alfred workflow reads the JSON metadata file for each Quiver note that matches the user’s query. When the user selects a note in Alfred, the workflow uses the quiver:// URL scheme to open the note. You can download the complete Alfred workflow here.

In addition to Apple’s documentation, I found several other useful resources that proved helpful while I was writing the search code. NSHipster has a great introductory tutorial by Matt Thompson that explains how to use SearchKit. I also learned a lot from a SearchKit example included in the PyObjC documentation.

Weekend Project: Leap Motion and Pixi.js

The Leap Motion Controller is an input device that tracks the position of the user’s hands and fingers in 3D space, taking the first step towards the kind of gesture-based computer interaction that we’ve seen in science fiction films like Minority Report. I built a simple space game demo with the Leap Motion JavaScript library and the Pixi.js 2D graphics framework.

You can find the complete source code on GitHub. The repository also includes a few other examples. If you are interested in learning more, keep an eye out for a tutorial that will appear on Ars Technica in the next month or so.

Watching the Web evolve at Fluent 2014

I attended O’Reilly’s Fluent conference in San Francisco earlier this month. The annual event covers various aspects of the Web platform, particularly topics that relate to frontend development. The talks that I attended covered a wide range of topics, spanning from JavaScript performance improvements to unconventional uses for the CSS border-radius property.

Fluent gave me an opportunity to reflect on the manner in which the Web is evolving and the opportunities that will arise as it continues to mature. As a platform, the Web benefits from the collective investment of many different stakeholders. An incredibly diverse assortment of companies are committing resources to help advance new standards that extend the capabilities of the Web, improve the performance of HTML rendering engines and JavaScript runtimes, and bring Web technology to new devices and platforms.


Brendan Eich, the creator of JavaScript, gave a keynote presentation that highlighted the future of JavaScript. He started by discussing the high-level syntactic improvements that are introduced in ES6, the next generation of the language. In the second half of his presentation, he showed how ASM.JS and SIMD make JavaScript increasingly suitable for low-level programming, paving the way for new kinds of performance-sensitive content to come to the Web platform.


The most compelling demo in Eich’s presentation was Unreal Engine 4, an extremely impressive new generation of Epic’s powerful game engine. Brendan showed sample UE4 content running in Firefox, with incredible atmospheric lighting effects and other stunning visuals. The performance delta between Web and native is shrinking, making it possible for incredibly rich, hardware-accelerated 3D experiences to come to the browser.

Better performance

In a particularly interesting session at the conference, an Intel representative talked about the company’s multifaceted collaboration with browser vendors. They are working on increasing parallelization so that rendering engines can take better advantage of modern multicore CPUs. They are also working to help HTML rendering engines take advantage of hardware acceleration more pervasively so that even more work can be offloaded to the GPU.

Continue Reading ›

Slide deck syntax highlighting made easy with Alfred

I’ve spent a lot of time lately creating slide decks for my MontageJS presentations. When I include source code in a slide, I like to have syntax highlighting to help make it easier for the audience to read. Unfortunately, syntax highlighting isn’t a built-in feature in Powerpoint or Keynote. There are some great browser-based presentation frameworks like reveal.js that offer native syntax highlighting, but I currently need to use a standard tool so that other people can easily edit and consume my decks.

After giving the matter some thought, I decided to solve my problem by building an Alfred workflow. My new workflow (which you can download here) applies RTF syntax highlighting to the contents of the clipboard, making it easy to paste the colored text into Keynote and other applications. The workflow includes several simple scripts written in Python. It uses the popular Pygments library to perform the highlighting.

A slide I made for a recent MontageJS presentation

When the user types the highlight keyword in Alfred, the workflow uses a simple script filter to let the user select the desired programming language. After a language is selected, the workflow runs a script that extracts the current contents of the clipboard, colorizes it with the lexer for the desired language, and then replaces the contents of the clipboard with the RTF output.

Unfortunately, the standard “Copy to Clipboard” mechanism provided by Alfred’s workflow system doesn’t appear to support anything other than plain text. When I used it to place Pygment’s output into the clipboard, pasting would produce a plain text string with all of the RTF escape sequences. To work around that limitation, I ended up handling the clipboard interaction programmatically in my workflow’s Python script.

The PyObjC bridge provides direct access to NSPasteboard, which you can use to manually put RTF content into the clipboard as rich text. The following code is my simple colorization script:

import AppKit
from pygments import highlight
from pygments.lexers import get_lexer_by_name
from pygments.styles import get_style_by_name
from pygments.formatters import RtfFormatter

clip = AppKit.NSPasteboard.generalPasteboard()
content = clip.stringForType_(AppKit.NSStringPboardType)

lexer = get_lexer_by_name("{query}", stripall=True)
result = highlight(content, lexer, RtfFormatter(style="solarized"))

arr = AppKit.NSArray.arrayWithObject_(AppKit.NSPasteboardTypeRTF)

clip.declareTypes_owner_(arr, None)
clip.setString_forType_(result, AppKit.NSPasteboardTypeRTF)

As you can see, all you have to do to indicate that the clipboard data is rich text is use the NSPasteboardTypeRTF option. In my workflow, the “{query}” value passed into the script is the name of the programming language selected by the user. If you want to use the script by itself without Alfred, you can just replace that string so that it specifies the language that you want. I threw in nice a Solarized Dark style for Pygments that I found on GitHub, but you can change the style string in the script if you want a different color scheme.

There are command line tools like Highlight that provide similar functionality, but I find it very convenient to have this feature available through Alfred. The workflow is entirely self-contained, so it works seamlessly on every Mac where I have my Alfred workflows synchronized.

My experience at Xamarin Evolve 2013

Earlier this month, Xamarin hosted its first-ever developer conference. The event, called Xamarin Evolve 2013, exceeded all expectations. It offered a great mix of high-quality technical content, fun social activities, good networking opportunities, and plenty of time for the usual hallway banter that tends to create the most valuable discussions at technical conferences.

Xamarin Evolve 2013 Screen

Xamarin Evolve 2013 ran for four days, including two days of training and two days of conference sessions. There was a ton of energy and enthusiasm during all four days. The tickets completely sold out in the weeks leading up to the event, so we had a massive audience of 600 attendees.


Bryan Costanich and his team put together an amazing training program with an enormous amount of material. I sat in as an assistant on the beginner training during the first day and the advanced training on the second day. Both tracks were outstanding, with great content and excellent presentations.

I particularly liked Nina Vyedin’s advanced training session about app backgrounding. It was an entertaining and well-delivered presentation that genuinely improved my understanding of the subject matter. The application lifecycle is very different on mobile platforms and there are many constraints that aren’t intuitively obvious. It’s an area where some expert training goes a long way, offering useful insights that can help developers materially improve their applications.

The advanced training was exactly the right level for me. I can build applications with Xamarin, but I don’t always fully understand what’s happening under the hood. The segments of advanced training that I attended filled in some of the gaps for me, leaving me with a higher level of confidence in my familiarity with the Xamarin platform.

I’m really looking forward to seeing the rest of the advanced training sessions when the video recordings are finally published. I’m also thrilled with all of the in-depth text resources that the documentation team produced to accompany the training sessions—there’s so much material that it practically constitutes a book.


Nat and Miguel Xamarin Evolve 2013 Keynote

The conference segment of the event was a lot of fun. The highlight was definitely the opening keynote, which featured some really exciting announcements about new Xamarin products.

The audience was totally ecstatic, hanging on every word and applauding enthusiastically for all of the news. The keynote produced such tremendous social media buzz that it propelled Xamarin into Twitter’s trending topics. The following is a summary of the major announcements:

  • Building Xamarin apps with F#: Due to work driven by the F# community, it’s now possible to build native iOS and Android apps with F# and Xamarin. F# melds the best qualities of functional and object-oriented programming paradigms, offering a high level of productivity and expressive syntax.
  • Xamarin adopting Mono 3.0: Xamarin’s products are being updated to run on top of Mono 3.0, which supports the latest features of the C# programming language and .NET runtime. With full support for .NET 4.5 and C# 5, Xamarin developers will be able to take advantage of modern capabilities like the new async and await keywords.
  • iOS Designer for Xamarin Studio: Xamarin has built a new visual design tool for creating iOS user interfaces. It integrates seamlessly with Xamarin Studio, enabling drag-and-drop iOS design and development in a unified environment. Developers who build iOS apps with Xamarin no longer have to rely on Xcode to create their user interface layouts.
  • Xamarin Test Cloud: Xamarin is introducing a completely new product that will greatly simplify automated user interface testing for mobile developers. The Xamarin Test Cloud is a hosted environment that will run an application on hundreds of real mobile devices—making it easy for developers to identify cases where their app isn’t working as expected.

I’m especially excited about the new iOS designer, which will make iOS development with Xamarin much more seamless. The tool has some extraordinary capabilities that make it even better than Xcode, including the ability to render custom controls in the design view.

Continue Reading ›

My first workflows for the Alfred v2 Beta

The marvelous is one of my favorite tools for Mac OS X. At its core, Alfred is a simple and effective app launcher. When used to its full potential, it does much more. Among many other things, Alfred lets you find and manage files, control music playback, manage and search clipboard history, and search the web. It exposes that incredibly rich set of features through a fantastic, keyboard-friendly user interface.

Editing my Markdown workflow in Alfred

Alfred is also extensible, allowing the user to define their own commands and search filters. The latter is especially significant: Alfred leverages the Spotlight search infrastructure under the hood, but provides functionality on top that makes it easy for the user to conduct powerful, targeted searches and perform useful actions on the search result items. According to Alfred’s built-in usage calculator, I use the tool an average of 90 times per day. It’s easily one of the most heavily-used tools in my arsenal.

The Alfred team is currently developing an impressive new version version of the application that introduces a number of new features. They released a private beta of Alfred v2 over the weekend, making it available to Mega Supporters—users who purchased a lifetime license. As a Mega Supporter, I am among the beta testers.

The most exciting feature in v2 is the new “workflow” system, which greatly enhances Alfred’s extensibility. It allows the user to compose complex chains of custom triggers, filters, actions, and outputs to create much more powerful third-party extensions. It also adds an awesome new “scripting filter” mechanism that lets the user create scripts that programmatically generate custom query results.

I’ve already built several of my own custom workflows (using a combination of Python, shell scripting, and AppleScript) that perform useful tasks. I’m making them available here for the benefit of other Alfred v2 users.

GitHub Search (Download)

The scripting filter that I made for this workflow (written in Python) allows you to search for repositories on GitHub. It uses the GitHub REST API to display results directly in Alfred as you are typing. Select a repository from the results and hit enter to visit the repository’s GitHub page in your default web browser. If you hold the “alt” key when you hit enter, the workflow will instead open the official GitHub app for Mac OS X and clone the desired repository.

Markdown Processing (Download)

This workflow can be used to perform Markdown processing on the selected text, clipboard content, or a file. It will put the HTML output into the system clipboard so that you can paste it. The workflow also has a file filter that makes it easy to search for Markdown files and optionally open them in I embedded the MIT-licensed Markdown 2 Python library in this workflow so that it can handle Markdown processing without requiring the user to install any additional software.

Evernote Search (Download)

This workflow includes a file filter that will allow you to search for Evernote notes by title. When you hit enter, the Evernote desktop app will display the selected note. The workflow also includes a fallback search that uses AppleScript to pass the user’s search query directly into the Evernote app’s built-in search box.

Hue Control (Download)

This workflow allows the user to control Philips Hue lightbulbs. It defines a “hue” keyword command that lets you adjust brightness by providing a numerical value between 0 and 255. You can also type “hue on” and “hue off” to turn the lights on and off. In order to use this workflow, you will have to modify several variables at the top of the script: the local network IP address of your Hue base station and an app registration hash. Refer here for details about app registration and Hue hacking.

Chrome Window (Download)

This workflow will use AppleScript to forcibly open a new Chrome window on the current space. It avoids doing annoying things like giving focus to an existing Chrome window or jumping to another space.

Xamarin and MonoGame at BUILD 2012

I represented Xamarin at Microsoft’s BUILD conference last month. I attended the event, which took place at Microsoft’s Redmond campus, with my coworker Craig Dunn. We had a booth on the second floor of building 92, in the section dedicated to Visual Studio partners.

BUILD offered us a great opportunity to spread the word about Xamarin and meet lots of .NET developers. Our booth, where we handed out t-shirts and plush Xamarin monkeys, was very popular. There was so much demand for the monkeys that we had only a few left by the end of the week. We also conducted a daily raffle during the conference, giving one winner a Xamarin license on each day.

The Xamarin booth at BUILD

At our booth, we demoed our Android development add-in for Visual Studio and showed developers how they can use Xamarin to build cross-platform mobile applications that work across Android, iOS, and Windows 8. We also talked about the launch of our Xamarin.Mobile update with Windows support.

Many Xamarin customers came to our booth during the conference to tell us about the apps that they are building with our products. It was gratifying to hear so much positive feedback and enthusiasm from actual users. I’m always impressed by breadth and diversity of the mobile development landscape.

It’s especially interesting to hear about the kind of mobile apps that companies are building for internal use, because it offers a glimpse of the enormous application ecosystem that exists outside the realm of conventional consumer app stores.

One of the highlights of the event was the unofficial MonoGame meetup, which attracted a considerable audience of developers and XNA enthusiasts. At the meetup, Xamarin’s Dominique Louis and Tom Spilman of Sickhead games explained how developers can use MonoGame to make games that will work across multiple platforms.

Tom talking about MonoGame at BUILD

Tom’s team at Sickhead is largely responsible for porting MonoGame to Windows 8, opening the door for porting XNA games to the Windows 8 Modern App environment. Sickhead used MonoGame to build ARMED, which it is now bringing to a number of platforms including Windows and iOS.

BUILD is the first Microsoft event that I’ve attended. Although running a booth for four days at such a busy conference proved exhausting, it was a very rewarding experience.

Open Source and .NET at MonkeySpace 2012

I flew to Boston last week to attend MonkeySpace, a community-driven event dedicated to open source software projects in the .NET technology ecosystem. The Mono stack is obviously a prominent part of that landscape, but the conference attendees came from a wide range of technical backgrounds—reflecting the diversity of the broader .NET community.

I enjoyed many of the talks that were given at the conference, particularly Chris Hardy’s introduction to iOS 6 features, Michael Hutchinson’s MonoDevelop tip walkthrough, Aaron Bockover’s overview of the Vernacular localization system. The MonoGame session, which was presented by Dean Ellis, also included one of the highlights of the event: an XNA demo running on the ARM-based Raspberry Pi Linux computer.

Dean Ellis showing a MonoGame demo on the Raspberry Pi Linux computer

Videos of the conference sessions will eventually be published online and made available for download. You can, however, already get the slides and code samples from many of the sessions. Nic Wise, who sadly wasn’t at the conference, put together a helpful blog post with links to a bunch of the conference material.

Mono 3.0

Miguel de Icaza used the MonkeySpace opening keynote to announce the release of Mono 3.0. The news was warmly welcomed by the conference attendees. Mono 3.0 introduces compatibility with C# 5.0, including much-anticipated support for the language’s new asynchronous programming capabilities. Mono 3.0 also has a number of important enhancements under the hood.

Miguel introducing Mono 3.0

SGen, the generational garbage collector, has seen further improvement and is considered ready for widespread adoption. Mono’s compilation architecture was overhauled, allowing the various C# profiles to be supported in a single, unified compiler. IKVM was adopted as the default code generator, replacing Reflection.Emit. For more technical details about Mono 3.0, you can refer to Miguel’s blog post, the official release notes, or the excellent coverage by Sean Gallagher at Ars Technica.

Open source and the .NET community

Mono 3.0 adds several libraries and frameworks that Microsoft has released under open source software licenses, including Entity Framework and the Razor HTML templating library. Microsoft’s decision to make these libraries available under suitable terms is a big win for the community.

I had an opportunity to meet several noteworthy figures from Microsoft at the event, including F# luminary Don Syme and Bob Familiar, the regional Microsoft evangelism chief. I’ve recently worked with Bob in my capacity as a Xamarin evangelist, so it was a pleasure to meet him in person. Microsoft seems to increasingly understand the importance of fostering and engaging with the open source software community around .NET.

GitHub’s Phil Haack, who I had the pleasure of meeting at the conference, wrote a great blog post last week that captures the importance of collaboration. He highlights the important role that Xamarin, the Mono project, and the open source software community can play in helping .NET and C# achieve their full potential.

A slide from Miguel's keynote

Phil’s blog post also describes MonkeySquare, the non-profit organization that backed the MonkeySpace conference. Phil is a member of the MonkeySquare board, alongside Scott Hanselman, Joseph Hill, David Nielsen, Dale Ragan, and Louis Salin. If the success of the MonkeySpace event is any indication, I think that MonkeySquare will have a very bright future. Dale, who served as the MonkeySpace event organizer, did a fantastic job.

MonkeySpace is the first event that I’ve attended since joining Xamarin as a developer evangelist. I’m looking forward to meeting even more developers next week when I fly to Redmond for Microsoft’s BUILD conference.

Joining Xamarin

After seven fantastic years at Ars Technica, I’ve decided that it’s time for new challenges and new opportunities. I’m really pleased to announce that I’m joining Xamarin’s developer relations team as a technology evangelist.

Some of the best and most exciting innovation in the field of software is happening on mobile devices. Tools that accelerate mobile development and boost the portability of existing code can unlock a ton of value. I’ve seen developers build some great things with Xamarin. In my new role, I’m going to have the opportunity to highlight some of those achievements and engage with the vibrant mobile application development community.

You can read my full farewell note over at Ars Technica. I’m still going to write articles for Ars on a freelance basis as my schedule permits. I’m going to use my personal blog here to write about software development topics and my experiences in the tech industry.

The Wall of Cthulhu

I recently purchased a limited edition print by Florian Bertmer that depicts the eponymous alien entity from Lovecraft’s The Call of Cthulhu. The piece was originally created for the Required Reading show at Gallery 1988, which featured posters inspired by literature. Bertmer sold 60 prints of the gallery edition on the Internet through Moon Editions. They sold out very quickly, but I managed to secure one to hang on the wall of my home office.

That is not dead which can eternal lie,
And with strange aeons even death may die.

I really like limited edition science fiction posters, but I rarely buy any myself. The serious collectors tend to accumulate these things as an affirmation of their commitment to fandom. It often takes luck or significant effort to snag a popular piece before the supply is completely depleted. On several occasions in the past, I’ve taken a liking to a particular print for sale from a vendor like Mondo but found that it was already sold out by the time I decided to actually make the purchase.

When Moon Editions put up Bertmer’s Cthulhu print, I didn’t hesitate for an instant. The piece was absolutely perfect for me—one of my favorite stories rendered by one of my favorite artists. I fell in love with it immediately and decided to buy it for myself as a birthday present. It arrived right around my birthday, but I was in Chicago at the time and didn’t get a chance to open it up until the weekend when I returned home.

It arrived in perfect condition and looks amazing. The detail in the image, particularly the intricate curvature of the vines and tentacles, is stunning. Bertmer’s style is influenced by Giger, but has some Art Nouveau flourishes that make it truly distinctive. The Cthulhu poster is darker and less flamboyant than some of his other works. The lettering and structure of the composition give it the rich feel of a vintage science fiction book cover.

Bertmer's Call of Cthulhu

I had it framed at Aaron Brothers and chose gallery-style glass so that it wouldn’t pick up too much glare from the lights in the room. I selected a subtle black wooden frame and a nice olive green mat that really contrasts nicely with the dark colors of the print. I hung it on the wall opposite of a framed poster print of Dali’s Moment of Explosion.

Now that I have a Great Old One adorning the wall of my home office, I think I’m going to keep an eye open for more Lovecraft-themed artwork so I can start building a collection.