I just got back from a trip to Berlin as part of the Knight-Mozilla learning lab (MoJo). Twenty of the participants from the previous round (the month long lecture series) were invited to spend a week in Germany getting to know each other while attempting to churn out some code for the rest of the world to see.

I arrived Sunday morning and quickly learned why it is never a good idea to get to a country before hotel check in. No recovery naps for me! The first thing I did was meet up with Saleem Kahn, Nicola Hughes, and Laurian Gridinoc and take a trip to the Bauhaus where I learned that people have been making things for a long, long time.

——–
Let me pause to quickly explain. I grew up, like most of you, using lots of things. When it came to making I was stuck with Legos and the like until one day I discovered programming and started making digital things with that. Fast forward 15 years and I’m at MIT taking How to Make Almost Anything and I say “oh awesome! Hardware isn’t just magic!” But once the course ended I, for the most part, reverted back to my comfort zone of software (still empowered with the potential to carve out circuits and molds, but not seizing immediate opportunities to utilize that empowerment just yet).

Bauhaus is the rest of the picture and got me excited about making again. The series of shops (I attended the one in Berlin) which closed with the rise of the Nazis, were basically buildings dedicated to modernist design (i.e. creating objects that are both beautiful and functional). As I walked around the museum I realized that I don’t have to make things that are super high tech and based on circuits to be making almost anything, I just have to be making things with a unique purpose. Hackable life for the win!

No time to worry about making things now (sponsor week and thesis proposal deadlines are looming), but I sure am ready to build stuff instead of buying stuff.
——–

After Bauhaus we went back to the hotel and I crashed and burned (and woke up just in time to meet up with Mark Boas and have some good old fashioned German Asian food since most things were closing up by then!).

The next morning was the beginning of what ended up being an INCREDIBLY packed four-day schedule of programming, talking, eating, walking, and sleeping. The hackathon (a term used to refer to these kinds of get togethers where people sit around and code) itself took place in a building called the Betahaus, located in Moritzplatz (aka “Makerplatz” since it is the hub of Berlin’s Maker community). The room was awesome – the fourth floor of a stark concrete building, full of tables, chairs, soda, lots and lots of wifi, and random posters of wildlife on the walls. Now that I think about it I wouldn’t be surprised if they hosted fight clubs on weeknights.

As the 20 of us pondered statements like “nothing should be anything” we started milling around and getting to know one another. Some people were designers, some were journalists, some were hackers, and some were mutts but sure enough project clusters slowly sprung up and by the second day people were nose deep in their laptops.

It was around this time that I realized that my project shared a very common need with most of the others: the need for metadata extraction from pieces of media! Thus was born the Meta Meta Project.

——–
I’ll write more about Meta Meta in another post somewhere on the Internet, but the basic idea is that there are a lot of tools out there which can extract information from images, videos, and text. For instance maybe you want to know all of the locations mentioned in a news article, or maybe you want to find all the words that appear in an image.

A lot of projects would benefit greatly from having access to this information, but to use the tools out there takes a fair amount of time setting up, implementing logic, and generally re-inventing parts of the wheel. Rather than having everyone need to become an expert in the tools, the Meta Meta Project is an API suite which will make it dead simple to put in a piece of media and get back the information you want.

Like I said, I’ll have to post more on that somewhere else.
——–

By day 3 the news partners arrived – there were representatives from BBC, The Guardian, Zeit Online, Al Jazeera, and The Boston Globe. They were there to get to know our work and us, but more importantly they were there to get to know one another. The idea of open collaboration still seems to be a somewhat foreign concept among the professional news industry. This is a pity because there is surely a lot of room for mutual benefit and it would surely free up lots of resources for one another. (Hey news rooms! Hop on board the Meta Meta Project!)

There is so much more to write about but there is so little time so I’m going to wrap up with some quick points:

• I had never attended a hackathon before this one and I’m now totally hooked.
• I had never attended a Mozilla event before and I’m now totally hooked.
• Berlin, and Germany in general, is a surreal place to be. The whole city is marked with the painful memories of the past, and it is just so interesting and tragically beautiful to walk around and see memorials, broken pieces of walls, and intentional marks designed to ensure that things aren’t forgotten.
• I came to realize that America isn’t really as young as everyone makes it out to be. When you think about how both Germany and Spain have had radical change in government in the past century it’s almost as if they’re the newbies.
• Germany pulls off maker punks.

I want to end with my favorite memory from the trip (I stuck around for three days after the coding portion just to see the city). A small group of fellow stragglers and I were wandering around a part of Berlin that I would never wander around on my own. This was by no means a place for tourists. As we passed by doorways of punk clubs blasting out dance music we crossed a well-lit alley blasting a different kind of music. At the other end of the alley was a small band with a gathering crowd behind it. No vocals, just tones, and the energy slowly built. We got caught in the sounds and just watched as they wailed away and eventually climaxed.

In the words of Chris Keller: if someone did that in Manhattan they would probably be carted away.

ATTN-SPAN Intro.

Part 2: Prototype and Development Plan

The Good News: I created a proof of concept prototype of the ATTN-SPAN platform powered by the Metavid project.

The Bad News: Metavid is having a lot of stability issues right now, so you probably won’t be able to use my prototype. I made a screen cast just in case.

Relying on a 3rd party for the most important aspect of an application is a major risk; one that I must mitigate. This brings me to my first batch of design work: the content scraper.

Scraping, Slicing, and Scrubbing C-SPAN

How do you get from a TV channel to a rich video archive and how do you get there automatically? The goal is to convert C-SPAN into a series of overlapping video segments that are identified in terms of state, politician, topic, party, action, and legislative item. Some of this is straightforward and some of it might be impossible, but here’s an overview of the planned nuts and bolts:

1. DirecTV offers TV content in a format that is easy to record digitally and VLC is a free tool that can do that recording. Combine the two and we can download C-SPAN streams into individual files that are primed and ready for analysis.
2. Once a video file is in our clutches we can use VLC once again to separate out the video from the Closed Captioning transcript.
3. Now we have a transcript and a raw video file. Next we register all of this information (in a database) so that we can look it all up later, and then convert the video file in to streaming-friendly formats and store it alongside the original recording.
4. C-SPAN consistently shows a graphic on the bottom of the screen that says who is talking, their state, their party, and what is being debated. By using a technique called Optical Character Recognition (OCR) we can pull this text out of the video image. Once pulled, we can add that to our database so that we can access all of this information for any moment in the video.
5. At this point we have most of the information we need, but there is still room for fine tuning. We can use audio levels and the closed captioning transcripts to try to identify moments of inactivity, normal dialogue, and heated dialogue.

These steps are enough to split up and categorize C-SPAN footage into an organized video database, but there are still more ways to flag special moments in the footage. For example, we may want to identify changes in speaker emotion in order to give our algorithms the ability to craft more engaging episodes. This is possible through the work of Affective Computing group at the MIT Media Lab, a group which has developed several tools that perform emotional analysis using facial recognition.

We may also want to identify specific legislative action (e.g. “calling a vote”). This could be accomplished by looking for key words in the transcript (e.g. “call a vote”) and possibly through common patterns in the audio signal (maybe there are identifiable sounds, such as a gavel hitting the table). Both of these concepts require additional research.

Creating a Profile and Constructing an Episode

If video events are the building blocks then viewer interests are the glue. The creation of a personalized episode requires two things: A user account, and a context. The user account provides general information like where you live, what issues you have identified as important, and (if you are willing to connect with Twitter or Facebook) what issues your circles have been discussing lately.

The context comes from time and cyberspace. Every night, after congress closes their gates, your profile is used to create a short, rich video experience designed to contain as much relevant content from that day as possible. At this point you might get an email begging you to watch, or maybe you log in on your own because you are addicted to badges and points and you want as much ATTN-SPAN karma as you can get.

There is another way to access this content though, and that is through the web sites you visit anyway. Imagine if you could read an article about the National Debt on the New York Times (or in a chain email) and actually see quotes from your own senators in the report. What if you could supplement the national report with a video widget that lets you browse what your house members had to say when they controlled the floor during the debt debates.

From a technical perspective this isn’t that far fetched. Truth Goggles, one of my other projects, is a bookmarklet that will analyze the web page you are viewing, fact check it, and rewrite the content to highlight truths and lies. This impossible feat is fairly similar to what I’m proposing here.

Adding Rich Information

Once an episode is pieced together we can look up the information surrounding the video to know who is talking and what they are talking about. What else can be added and how do we get it? Existing APIs offer some good options:

• Contact Information – Thanks to the Sunlight Labs Congress API it is possible to get the contact information for any member of congress on the fly. Thanks to VOIP services it is possible to create web-based hooks to call those people with the click of a button.
• Campaign Contributions – The New York Times offers a Campaign Finance API which can help you understand where the person on screen gets his or her money.
• Voting Records – The New York Times also offers a Congress API that will make it possible to know vote outcomes from related bills as well as information about the active speaker’s voting records.
• Truth and Lie Identification – My Truth Goggles project can be easily adapted to work with snippets from video transcripts. This will allow ATTN-SPAN to take advantage of fact checking services like PolitiFact and NewsTrust.

This is a good start, but I would also like to show links to related news coverage and create socially driven events based on community sentiment (for instance to track moments that caused people to get upset or happy). This won’t come for free, but it should be accessible given the right interface design.

Part 3: A Note to the Newsies

So that’s the idea and the plan. What’s the value?

It seems plausible that ATTN-SPAN, a system that analyzes primary source footage and pulls out any content that is related to a particular beat could be useful as a reporters tool, but what about your subscribers? ATTN-SPAN can augment an individual article so that it hits everybody close to home. Suddenly one article becomes as effective as two dozen. Moving past text, for larger organizations with a significant amount video footage ATTN-SPAN can be tweaked to use your programming instead of (or in addition to) C-SPAN.

At this point I have to warn you that this is not the first nor will it be the last project to work with C-SPAN. A 2003 demo out of the Media Lab used C-SPAN as one of several sources of information in a platform aimed to provide citizens with Total Government Awareness. Metavid, the platform I used in my initial prototype, already makes C-SPAN more accessible by enabling searches and filters. The list surely goes on.

So why is this a more powerful project? Well, the real goal of ATTN-SPAN isn’t to get more people watching C-SPAN. In fact I tricked you: this project isn’t about government awareness at all. It’s actually part of an effort to make indisputable fact (“blunt reality” and “primary source footage”) a more prominent part of the media experience without requiring additional effort from the audience. Newsrooms do an amazing job of reporting events and providing insight, but for deeper stories there simply isn’t enough time or money to cover everybody’s niche without going beyond the average person’s attention span.

Thus ends my pitch.

The code for both prototypes mentioned in this post can be found on github: ATTN-SPAN and Truth Goggles. Please forgive any dirty hacks. I would be thrilled if anybody wants to offer suggestions or even collaborate. On that note, please get in touch on Twitter @slifty.

After the legislation was passed I saw a poll on CNN’s front page that I can only assume was a blatant taunt to drive this horrible situation home. The poll read something along the lines of “How satisfied are you with the actions of your elected representatives?” To which I responded by clenching my fists and screaming to the sky: “How the hell should I know?”

Of course I realize this is nobody’s fault. I realize this especially after listening to Mohamed Nanabhay describe the work and challenges faced by the journalists at Al Jazeera. The professionals manning the ships of media corporations must face countless unsolvable challenges involving what content to air, how to craft a message, and how to share information across many diverse communities in a way that makes sense.

ATTN-SPAN is my hopeful attempt to have my cake and eat it too. Don’t let MoJo or MIT fool you: I’m making it for myself. The idea behind this project is that most content out there is a product that was created for the masses – not for me. I can find algorithms and editors that try to pick out articles written for masses that are similar to me, but ultimately those articles are still written for masses. My theory is that the only way to get true personalization is at the source. The primary source.

The reason nobody likes primary sources is that they are a really inefficient way to transfer information. The irrelevant-to-information ratio is simply too high. Worse. The boring-to-anything ratio is too high. I mean seriously, who watches C-SPAN? But what if that primary source can be tagged, catalogued, and marked up in a way that will help generate digestible content on an individual level?

Once the footage of congress can be automatically organized in terms of not just things like who is talking and what is being discussed, but also in terms of when voices get louder or when gavels hit the table… Well, suddenly primary sources can be patched together completely dynamically in a way that tells a story just for you. Your information diet can be augmented with personalized, real world footage. Finally you’ll know for sure that your senator is just as ineffective as you had previously assumed!

The HTML5 goal didn’t take much prodding because it’s something I should have done last year.  I delayed because jQuery was meeting my needs in terms of prototypes and I was cramming my head full of nodejs, arduino, OSX and terminal magic, git and mercurial, Python, Matlab, how to make almost anything, and all sorts of nerdly things that I should have picked up during my undergrad years but somehow avoided.

What convinced me this time – aside from the fact that it is clearly the future of the web – is that my project for the Learning Lab, a C-SPAN analysis and summary tool called ATTN-SPAN, is pretty much exactly the kind of project that HTML5 is supposed to improve.

The fit is probably best explained by going over the three major hurdles I face for this project:

1. How do I collect and store the video content?
2. How do I processes and personalize the video content?
3. How do I present the video content?

Item 1 is partially solved thanks to some of the great researchers here at the Media Lab – and thanks to the metavid project.

Item 2 is going to be tricky, but that’s a can of worms that has very little to do with HTML5.  I’m hoping that I can turn the videos into an “event based” organization, where events are automatically identified things such as “senator A smiles” “senator B says X” “senator C shows a picture of grazing cattle.”  I also want to leave room for user defined events such as “senator D says something I disagree with.”

Item 3 is where this week’s lectures become particularly important, for implementation, HTML5 is where video presentation and interaction suddenly becomes much more flexible.   The events described earlier in this post are going to drive an “personalized episode generation” algorithm which, for a given individual from a given state, will create a series of C-SPAN timestamps associated with video clips and all sorts of metadata.

As for design… Well, lets just say that clearly I’m going to have to spend some time re-watching that lecture by Jesse James Garret.  I’m going to take a few hours this week when I’m not reading up on HTML5 or creating buttons that allow you to discover the truth about anything online (link to this one coming soon) to do some wireframes for the ATTN-SPAN UX.