Like the rest of the world, I've been completely obsessed with the presidential election and nonstop news coverage. My drug of choice? Gabe Rivera's Memeorandum, the political sister site of Techmeme, which constantly surfaces the most controversial stories being discussed by political bloggers.

While most political blogs are extremely partisan, their biases aren't immediately obvious to outsiders like me. I wanted to see, at a glance, how conservative or liberal the blogs were without clicking through to every article.

With the help of del.icio.us founder Joshua Schachter, we used a recommendation algorithm to score every blog on Memeorandum based on their linking activity in the last three months. Then I wrote a Greasemonkey script to pull that information out of Google Spreadsheets, and colorize Memeorandum on-the-fly. Left-leaning blogs are blue and right-leaning blogs are red, with darker colors representing strong biases. Check out the screenshot below, and install the Greasemonkey script or standalone Firefox extension to try it yourself.

Note: The colors don't necessarily represent each blogger's personal views or biases. It's a reflection of their linking activity. The algorithm looks at the stories that bloggers linked to before, relative to all other bloggers, and groups them accordingly. People that link to things that only conservatives find interesting will be classified as bright red, even if they are personally moderate or liberal, and vice-versa. The algorithm can't read minds, so don't be offended if you feel misrepresented. It's only looking at the data.

For example, while Nate Silver of FiveThirtyEight may be a Democrat, he has a tendency to link to stories conservative bloggers are discussing slightly more often than liberal bloggers, so he's shaded very slightly red. (Geeks can read on for more details about how this works.)

Install it!

Greasemonkey users: memeorandum_colors.user.js
Standalone Firefox Extension: memeorandumcolors.xpi

After it's installed, go to any page on Memeorandum and wait a second for the coloring to appear. I hope you like it!

How It Works (Nerds Only)

The first challenge was getting the data. I emailed Gabe Rivera, and he graciously gave offered a full dump of every blog listed on Memeorandum. This didn't include relationship data, showing which blogs linked to which stories, so Joshua and I crawled the site instead. Using the historical archives, we took a snapshot of the site's homepage for every six hours for the last three months — about 360 total. With a Python script, Joshua scraped the links from the saved HTML to get the link data.

Armed with the spreadsheet of over 50,000 blogger-to-article relationships, we needed to somehow find correlations in the data. We used a method called Singular Value Decomposition (SVD), a method to break down complex data in matrices to its component parts. It's extremely flexible, used in applications as diverse as weather prediction, movie recommendations, genome modeling, clustering search results, and image compression.

Inspired by GovTrack's use of SVD to visualize the political spectrum for members of Congress, we attempted to do the same thing for political blogs.

Here's how Joshua describes the methodology:

I created an adjacency matrix, with discussion sites as the rows and the discussed articles as the columns. When a site discusses an article on Memeorandum, we fill in a 1 in that cell; everything else is left as zero.

Every site becomes a very high dimensionality vector into link-space. This is very difficult to visualize. (Unless your monitor displays many dimensions. Mine only has two.) Since a bunch of sites tend to link to the same groups in the same way, we don't need all those dimensions. So, very roughly, what SVD lets us do is reproject the points in space into a new coordinate system, so that the points that are similar are near each other and we know which dimensions are most important. We can take just the most significant ones.

We could use two or three for a nifty visualization, but we wanted to show the bias as a spectrum, which is just a single dimension. In this case, the second most significant dimension (v2) ends up corresponding to linking similarity. The first dimension (v1) corresponds to how much linking they do in general.

Curiously, when running the exact same analysis on Techmeme, the second most significant dimension ends up being Business vs. Technology. (The conservatives/liberals of the geek world?)

Did you get all that? If you'd like to try to figure out what the other dimensions represent, take a look at columns v3-v5 on the full spreadsheet below and let us know if you come up with anything. (We didn't have much luck.)

Once we'd realized that the second dimension (v2) highly correlated with political leaning, we uploaded the spreadsheet into Google Spreadsheets and created a new column with a normalized score, scaled between a range of -1 and 1. The spreadsheet, with all of the sources and their respective scores, is below. (Download the Excel document or CSV if you want to sort or filter the data.)

After deriving the scores, writing the Greasemonkey was straightforward. Google offers XML feeds for Spreadsheets, so I queried this public feed of our data using XMLHttpRequest, parsed it, and colored it based on the score.

If you have any improvements to the code, please pass them on by emailing me or IMing me using my contact information at the top of the page.

Conclusion

I'd love to know what dedicated Memeorandum fans think of this. For me, it makes the site much easier to skim. At a glance, I can see what left-wing and right-wing bloggers each find interesting and, more importantly, when there's an article that's of genuine interest to both parties. It's also interesting to quickly see which bloggers cross party lines, willing to link to stories that don't favor their own candidates.

I hope you like it, and please contribute your changes to make it better!

Further Reading

Puffinware's SVD tutorial is one of the most concise, coherent explanations of SVD I could find for the layman. Ilya Grigorik applied SVD to build a recommendation system in Ruby, with great explanations and source code. Simon Funk explains how he used SVD to tie for third in the Netflix Prize leaderboard (for a short time).

This was my first Greasemonkey script, and I found Mark Pilgrim's Dive into Greasemonkey invaluable. I highly recommend writing a couple scripts yourself; it's incredibly empowering to modify other people's websites.

A special thanks to Gabe Rivera for building Memeorandum and Techmeme and for supporting this little project.

Updates

October 10: J. Chris Anderson built a bookmarklet for use with non-Firefox browsers, or by anyone who just wants to test it out without installing an extension. This also has the benefit of working on sites beyond Memeorandum, like Google News. (Though, of course, it will only color sites that appear in our spreadsheet.)

October 11: Brendan O'Connor compared our unsupervised machine-derived rankings to human judgments of political bias on Skewz, and found there's a significant correlation. He released the code and full dataset on his entry.

Somehow, a 22-year-old University of Alaska student named Richard Millay got his hands on a videotape that's eluded the media since John McCain asked Sarah Palin to be his running-mate — original footage of her 1984 Miss Alaska Pageant.

Of course, this is all very frivolous and has nothing to do with the current campaign. But like Barack Obama's high school basketball footage, it's a little glimpse into the early life of a highly-visible national figure.

In the first part added to YouTube, he posted the portion from the swimsuit competition, prefaced by a brief introduction mentioning the demand for the "88 minutes of Alaska Gold."

Update: The original video was removed, but I managed to save a copy of the relevant footage without Richard's original intro. YouTube's removing every copy of this video, so I'm streaming the clip below from my own server. It won't be removed.

As the future vice-presidential candidate parades on stage, an off-screen announcer reads her early biography: "Contestant #8, Sarah Heath. Sarah says that she wants to prepare for a career in television broadcasting by majoring in Telecommunications and Political Science. It's no wonder that she has also been recognized by Who's Who, since she has displayed her leadership in all areas, from academics to student politics to athletics, having led her basketball team to the championship at the state tournament. Ladies and gentleman, contestant #8, Sarah Heath."

I've emailed Richard asking for a brief interview, and will update here if he gets in touch. (Thanks to Jeff Milner for the original tip this morning.)

I wanted to take a moment to announce that I've joined the board of directors for Kickstarter, a brand-new startup based out of Brooklyn and Chicago.

Kickstarter aims to let creative people of all kinds — journalists, artists, musicians, game developers, entrepreneurs, bloggers — raise money for their projects by connecting directly with fans, who receive exclusive access and rewards in exchange for their patronage. More than just a fundraising app, Kickstarter's a publishing platform where project creators can communicate with the people that are supporting them. (Think Jill Sobule, A Swarm of Angels, or Sean Tevis.)

I was introduced to founders Charles Adler, Perry Chen, and Yancey Strickler by Caterina Fake back in June, and sealed the deal after a trip to NYC to meet the team. They're a great group of guys with a strong vision, and I feel lucky to be involved.

Ultimately, everybody should be able to support themselves doing what they love using the web, and I think Kickstarter will be a great way to get there. Expect to hear more on Waxy.org as launch day gets closer.

To help them on their way, they're currently looking for a CTO to join the founding team. I've been helping guide some of the technology decisions and building the development team, but we're looking for a passionate and talented person to devote themselves to the project full-time.

If you're interested, drop me an email or IM and I'll introduce you!

After recording last week's interview, I was left with a 36-minute MP3 and a profound feeling of dread. You see, I hate transcribing audio. I used to transcribe interviews in high school, and it's always tedious, taking upwards of eight times the length of the clip itself.

Bracing for a good four or five hours of rewinding and writing and rewinding, I remembered that this is The Future! So, instead, I tossed the job over to the global anonymous workforce at Amazon Mechanical Turk instead.

The result: my 36-minute recording was transcribed while I slept, in less than three hours, for a grand total of $15.40.

This is a fraction of the cost/time of any other transcription service online, including the Turk-driven Casting Words, though you potentially sacrifice some quality. In my experience, though, there were virtually no errors.

Here's how to do it yourself, with no programming knowledge required. The instructions below are verbose, but using my template, it shouldn't take you more than five minutes of setup per job.

Step 1: Prepare your audio.

First, I split my 35-minute audio file into seven five-minute MP3s. Why? Mechanical Turk workers are all working in parallel, so the more discrete tasks, the faster the job gets done. This also diminishes the risk of one bad worker ruining your whole job. (Though you're always allowed to reject bad submissions, and you'll never have to pay for those.)

I used the open-source Audacity to split the files, but you could just as easily use any audio utility or editing software. Optionally, you might want to make each clip overlap by a few seconds, so you'll be able to easily recognize where each segment of the transcript starts and stops.

Name the files sequentially. In my case, they were interview_1.mp3 through interview_7.mp3. When you're done, upload the files somewhere they can be downloaded publicly. You'll need the full URLs later.

Step 2: Design your HIT template.

Mechanical Turk jobs are called HITs — short for the dystopic-sounding "Human Intelligence Tasks." After you've signed up as a new Requester on Mechanical Turk, you can design a new template from the homepage using one of several samples. Choose the Default Template.

On the Properties screen, we'll write a short description of the task, define how many people we want to work on it, and how much we're willing to pay them.

For a five-minute MP3, I think allotting two hours per assignment is ample time, and I expired the entire HIT in 12 hours because I was in a hurry. As for pay rate, you'll need to determine the "Reward per Assignment" based on the difficulty of the task and what you think is fair. I chose $2.00 per five-minute MP3, or about $0.40/minute. Depending on the difficulty, you might want to try going higher or lower.

I only wanted one worker to attempt each clip, so I changed the "number of assignments per HIT" to 1. (If you want redundant transcripts for each clip, change this to 2 or 3... But be aware your costs will double or triple!)

After entering all this information, here's what my finished Properties screen looked like:

On the Design Layout screen, you design the template that gets displayed to each worker, using basic variables that will be substituted later. For this template, we make up only one variable named "$url." You can call it anything you like.

The basics you'll need are a title, some simple rules, the link to the audio file with a substitution variable, and a text form for the worker to type the transcription into. If you'd like to use my template HTML, here it is. (Make sure you change the path to your own audio files!)

Two things to notice in my example. First, the "${url}" variables will be substituted with values in the "url" column of the spreadsheet we'll create in the next step. Second, any form element you create will end up in your final output from Mechanical Turk, so don't worry about the naming. I called mine simply "transcription." Here's what the relevant part looks like in the final template:

For the worker's convenience, I also added an embedded Flash player for the MP3, but this is entirely optional. When you're done designing your template, it should look something like this:

On the next screen, make sure it looks the way you like, and click "Preview and Finish" to save the HIT template.

Step 4: Upload the data for your HITs.

Once we're done designing our template, we can select it to create a new HIT batch. We'll be creating a simple comma-separated file (.CSV) filled with the data that will be substituted into our template.

On the Publish tab, select the template you just created by clicking the "Select" button:

Now, Amazon generates a sample CSV for you to put the URLs to your MP3s in. Click the link to "Download a sample input file" and open the downloaded CSV in a text editor. If you've done everything right, it should look like this:

Replace the "Hit1_url_data" lines with the full URLs to your own MP3 files. For me, this looked something like:

And so on. Save the CSV file, and upload it to Amazon. When you're done, your uploaded file should appear, with the number of input lines.

Step 5: Publish your HITs.

Select your uploaded input file, and preview the finished batch of HITs. You'll be able to page through each HIT, seeing exactly what workers will see. Use this opportunity to test that your audio files can be downloaded and heard properly. If it all looks good, click "Next" to confirm and publish your batch. This is what the final screen looks like:

If you don't have any money in your Amazon Payments account, you'll be prompted to fund it with a credit card. After you've paid, click "Publish HITs" and you're done!

Your HITs will publish out to the Mechanical Turk workers, who will find and work on your task. Depending on the length and number of your MP3s, expect some work back within an hour.

As they're working, you can browse and approve the results. The final output is an exported CSV, a spreadsheet of all the finished work that can be opened in Excel for your review.

Conclusion

You'd be insane not to use this for your own transcription projects. Absolutely nothing else comes close in price and speed.

One thought: I suspect it'd get even faster if you split clips into more pieces. I'd bet that splitting into one-minute segments would reduce the time by at least half. I'll bet you'd be able to command lower rates with smaller MP3s too, since the time commitment would be lower, driving more competition for the tasks. If anyone experiments along these lines, please let me know!

First, the bad news. Two days ago, I received a polite email from David Winton, the director of Code Rush, asking me to take the out-of-print documentary off of Waxy.org. As promised, I immediately complied.

Now, the good news — In my reply, I asked David if he'd mind being interviewed, and he agreed! He's an accomplished director and producer, the creator of the Big Thinkers series for TechTV, and the cofounder of Winton/duPont Films, located in San Francisco's Presidio.

We had a wonderful conversation about the film, which revealed for the first time that he's planning on not only re-releasing Code Rush digitally, but considering releasing the original outtakes (100 hours of footage) to the public domain on Archive.org.

I wish all my takedown notices were like this! Read on for the full interview, with selected clips from Code Rush, used by permission.