Run like the wind.

Being an Alumnus

As you approach the end of your fellowship you are going to ask yourself many questions. Will Dan Sinker still love me when I'm old? Is it true that on your last day they brand your inner thigh with a hot iron that says "PROPERTY OF MOZILLA"? Where did I leave my FitBit?

The biggest one is going to be "where the hell should I go from here?" I'll give an example of what a fellow's immediate future can be by describing my current status as a functioning adult.

It's difficult to say what I do for a living. When asked, I usually give up and declare that I am a freelancer. In reality I'm…

1: A Cofounder

I spent this week in San Francisco for the orientation of Mozilla's accelerator program, WebFWD. I'm here as one of three founders of Hyperaudio Inc., an nonprofit organization formed on behalf of my fellow fellow, Mark Boas.

Together, with a few others–including yet another 2012 Fellow, Laurian Gridnoc–we will spend the next year taking Mark's baby and turning it into a sustainable nonprofit ecosystem for remixable, transcribed video and audio.

2: A Teacher

There is a letter from Syracuse University’s Newhouse School sitting on my doorstep right now which offers a part time, remote faculty position. It is very likely that I will spend the next academic year mentoring students and creating a new set of resources to help them learn "how to make almost anything on the web."

3: An Innovator

Last month I worked with an amazing team at an OpenNews hackathon to build CivOmega. CivOmega makes it possible for people to ask questions about their government and get answers powered by open datasets and APIs. This month I'm in the running with 2013 Knight-Mozilla Fellow Mike Tigas to get funding to turn it into a real, contributor-ready open source project.

4: A Greybeard

Last Friday I was in Miami to serve as a judge for the Knight Community Information Challenge. I read many applications from around the country that pitched ideas about how they want to solve a major community issue with digital tools. The month before that I spoke on a panel about newsroom innovation at the MIT-Knight Center Media Conference.

If nothing else, being a Knight-Mozilla Fellow means you can trick otherwise reputable organizations like The Knight Foundation into thinking you know what you're talking about.

5: An Architect

I work part time to help startups build out their technology. This involves spending a few hours a week managing a team of developers and playing the role of architect and tech lead. Not every startup has to do with my immediate interests, but this is a nice way to keep things fresh. For instance, last month I helped make a button that rich people can press to give themselves more money.

Usually just mentioning my relationship with Mozilla is enough to cause people to swoon and faint, but sometimes I decide to go with vague threats instead. "I know some very important people on the internet. If you don't hire us, life could get very 'difficult'."

6: A Fellow (again…?)

In addition to being a Knight-Mozilla Fellow for life, #km4lyfe, I'll be a remote 2013 RJI Fellow starting in September. My project is an effort to flesh out of my good ol' thesis project, Truth Goggles, an automated bullshit detector for the internet.

7: A Trainer

The 2012 fellows have started a collective brand organization called Shape Journalism. It's a loose group of makers who are willing to help media organizations by training, building, or just offering advice. For example last week Nicola Hughes, Mark Boas and I started laying out plans for a week long data viz training we're expecting to run in November.

8: An Advisor

Have a crazy idea related to journalism, new media, or technology? Apparently I'm the guy to talk to to get feedback! But seriously so many people have reached out to pitch ideas, and it has been wonderful to get to help out.

There are so many people getting into this space, and being a Knight-Mozilla fellow is eerily similar to being a leader.

9: A Hired Gun

Organizations reach out to me fairly regularly to help them build out a prototype, apply something I have made in the past to their mission, or otherwise write some code. It's always awesome to get to work on something you love and get paid at the same time.

10: A Hobbyist

The best part about not having a job that provides health insurance is that you can do whatever you want in between other work. This means learning new skills (like professional-level soundscaping) but it also means getting to continue to make things.

For instance I'm working on a forum that lets groups of people talk to each other in a closed community without isolating them. Basically you can share threads between forums (and be part of lots of communities), so you can have conversations spread to the most relevant places without getting inundated with the anonymous jackasses that we lovingly call “the general public.”

The Moral of the Story

Poster masterfully created by Lyla Duey.

While it is technically true I don't have a job, I am here very much by choice. Being a fellow has set me up with a network of amazing people, who I still work with closely to build awesome things and participate in some badass events.

By the time you complete your fellowship you will be an unstoppable force of raw digital power. You will be oxymoronically established as both an outsider and an insider (so your perspective is priceless), and you will have had 10 months to show off what you can do. Following your passion at that point is as easy as breathing, unless you're a fish.

If your dream is a startup, you will come out of this with mentors, collaborators, and understanding. If you want to teach, you have an impressive set of experiences to show off. If you want a full time job, my other fellows have shown that you can absolutely do that too.

But honestly, seriously, not kidding, what are you waiting for.

This post is not about Quizzler, it is about my quest to answer the question “why are we building this from scratch?” It’s about observed realities regarding cross-newsroom collaboration, insights from upper management of The New York Times, and some major hurdles for open source in legacy media organizations. Prepare to explore the deep, dark, and relatively unspoken depths of technological openness in newsrooms.

We want something similar to…

I was introduced to Quizzler back in August. That first meeting was generally uneventful; we sat in a room. I listened to Miranda Mulligan skillfully duke it out with the project’s newsroom sponsor to explain that no, the first version won’t have custom “you are a 95% Vampire” sharable Facebook messages. I listened to the sponsor vocalize concern that there would never actually be a second version. I decided that both of them were probably right.

Eventually someone said something so shocking that I literally spat out my drink and fell out of my chair at the same time. It wasn’t intended to stand out—I don’t even know who said it. Ready? Brace yourself. Here it is: “Have you seen the Academy Awards tool by The New York Times? Eventually we will want something similar to that.” No wait that wasn’t it.

The New York Times is the parent company of The Boston Globe. They own the Globe in the same way humans own their children.

OK here’s the exchange.

Me: “Can we use some of their code?”

Someone: “We would have to pay them for that.”

Me:

Their response implied two things. First, that The New York Times would charge their kid for the digital equivalent of food. Second that the anticipated costs were high enough that it would be cheaper to rebuild this tool from scratch (again) than it would be to explore the possibility of reusing existing code.

Before you call child protection services, hold on. The situation is complex.

EDIT: To be clear, I quickly learned that the Times would not have charged us a dime.

Actually, this sounds completely reasonable

“Meh.” you say, “so The Boston Globe and The New York Times don’t share code, what’s the big deal?” A fair response, but trust me when I say the deal is big. If the deal was a rapper it would be notorious.

Here’s why: Starting from an existing code base instead of starting from nothing is often the difference between “having time to innovate” and “not.” If you are using technology as a core part of your business and you aren’t set up to experiment then you’re doing it wrong and you will become obsolete.

Borrowing code is kind of like being airdropped into the middle of a marathon; sure, you have to take a moment to figure out where you are and what direction to go, but now you have time to run in circles laughing like a crazy person before winning the race.

There’s more! If you borrow code then you are more likely to be familiar with what the rest of the world is doing. If you share code then you are going to build your systems with an emphasis on reuse and extensibility (i.e. correctly). If you regularly borrow AND share code then you are building a community around whatever it is you do.

What I’m trying to say is that if newspapers can buy into the mantra of openness—even just internal openness—they can kill about thirty birds with one stone.

But they usually don’t.

Why not? Are they idiots?

There are many reasons these organizations don’t trade bytes, none of which have to do with the original “we would have to pay for it” claim.

Reason 1: Wildly Different Technology Stacks

I lied to you earlier when I said the Globe was like a child to the Times–they’re more like middle-aged lovers. They didn’t grow up together or meet in college. They are two independent entities that recognized their love later in life, which means they have fundamentally different infrastructures.

One uses Java and PHP, the other uses Python, Ruby, and NodeJS. They have incompatible content management systems. They disagree on deployment policies, quality control processes, needs, and third party libraries. It’s like they come from two stubborn families that speak completely different languages and eat very different foods. They aren’t going to start casually sharing cook books.

Reason 2: Internal Politics

If a full team dedicates three months to creating a new public-facing interactive, will they want to just give it away? If you are a manager do you want to rely on favors from an external team to accomplish your goals? If you are a coder do you want to be judged for the quick last minute hacks you had to throw into the project?

The answer to these questions, and many more like them, is “hell no.”

Reason 3: Moving Costs and Learning Curves

Most technologies are dirty piles of duct tape with a shiny chrome finish. This makes them difficult to deploy and hard to understand. This is especially true among newspapers.

Packaging code in a way that strangers can use could take hours, days, or weeks depending on how much the developers cared about portability when they built it. I’m basically describing the difference between moving a campsite and a home. Newsroom developers don’t tend to have camping on the brain when rushing to meet looming deadlines.

Words from On High

Fine, so there are real reasons that code sharing between the Globe and the Times is a lost cause, but what does that mean for the industry? If financial allies with serious resources don’t share code, what are the chances that other newsrooms around the world will look outside their walls for help? Maybe this is why so many open source journalism tools are gathering dust.

I talked to Rajiv Pant (CTO) and Marc Frons (CIO) of The New York Times about code sharing and the role of open source in their company. For context: the Times is very progressive compared to other newsrooms when it comes to innovation and openness. They have a blog dedicated to their open source inititatives, there is a suite of APIs that provide civic data, and they do a good job of telling people about what they do.

Unfortunately they are also leading an industry that is forced into “deadline driven technology” and without a supportive institutional strategy, open source and reusable code are just nice-to-haves. Developers must ask themselves if they have time to meet the organization’s needs while also contributing to open source. Sometimes this means the same tools get built multiple times, but such is the nature of deadlines. Plus, as Marc was quick to point out, reinventing the wheel can be a good thing so long as the new one is slightly different.

Source: The Wheel Superstore. (Illustration by Lyla Duey)

But wouldn’t it be nice if all these new wheels could be used again and improved upon over time? Rajiv identified three factors that a project needs in order to be realistically used again by an organization like the Times.

Your code has to be…

1. Established – Is it safe to rely on your creation? How long will your project stay active, and how long after you move on will it stay useful?
2. Extensible – Your solution won’t meet all needs. How easy is it to improve? What kinds of features can be added?
3. Easy to Integrate – Will this play with existing systems and tools? Can it be skinned to look like it belongs?

In short, it doesn’t matter how powerful you think your code is: if it is difficult or risky to adopt, it will stay an orphan.

None of those points should come as a surprise, but they should probably be considered gospel to anyone developing anything—open or closed—in any newsroom. Just ask yourself “would the Times use this if they needed it?” If the answer is yes then you’ve made something that will last; otherwise you might as well get out the broom now.

Note: Since I’m sure you are worried, the Times doesn’t actually charge the Globe for code. And yes, we are writing Quizzler from scratch.

At face value it just looks like a live feed of C-SPAN’s Closed Captions. This alone is actually pretty cool if you think about it, especially if you are a deaf political junkie who sits far away from the TV and can’t read the closed captions.

Of course there is more. The real excitement comes when you contemplate what’s happening to get those words to appear on your screen.

This system unlocks and syndicates a real-time dataset that used to be a pain in the ass to access. Now anyone can build applications and visualizations that update before those crafty politicians have even finished making their points. This post explains why Opened Captions is worth hacking with, what it takes to use it, and how it works.

What is it Good For?

The Internet is filled with real-time updates triggered by online activity, but it still feels like magic when we see automatic updates driven by the real world. Opened Captions makes it easy for programmers to use live TV transcripts as an input.

Note: version .001 only supports a single channel (and my server is pointed to C-SPAN). Eventually the protocol should expand to allow multiple channels.

Let’s consider C-SPAN. If a computer knows what is being said on C-SPAN this very second, it can do things like:

• Change the background of your email client to reflect the issues being debated right this moment on the senate floor.
• Generate modified, more amusing, transcripts by replacing key words and phrases with Tolkien lore (i.e. C-SPAN for Middle Earth)
• Search through lyrics and generate a C-SPAN medley for you to rock out to while voting.
• Send SMS messages 24/7 commanding you to “drink” when certain phrases are spoken on air.

There are also possibilities that aren’t ridiculous. For instance, you could make tools that…

• Improve the transcript by automatically adding contextual information, such as definitions and histories thefted from Wikipedia.
• Send emails with transcript snippets whenever a specific representative or state is being discussed on TV so you know what’s going on.
• Parse out paraphrases of known fact checks and insert a credibility layer over the transcript feed (real time fact-checking).
• Draw parallels between what is being said on TV and what is being said on Twitter.

I could go on and on and on. There is just so much potential!

The Backend

Behind the stream is a first stab at a distributed architecture for Closed Captioning live-feeds. Opened Captions servers can pull a CC stream over a serial port, or (more likely) they will connect to an existing Opened Captions server and pull the stream from there. What that means in de-jargon is that anybody can set up a server that does exactly what mine is doing, even if they don’t have access to hardware, software, or a live TV stream.

When I say exactly, I mean it — your new project runs the same code as mine, and will serve the feed too. People can connect their servers to yours in the same way you connected yours to mine. Practically speaking this architecture means a few things:

1. Once your amazing mashup gets popular it won’t break my server. Your application is syndicating the captions to your users. I serve the captions to you, you serve them to the world!
2. Your server creates a fork of my stream. Want to modify the text so the politicians sound drunk? Add extra layers of information to the message payload? Translate the captions to Klingon? Go for it. If your tweaks happen server side then others can build their apps from your stream to modify it further.
3. You don’t have to rely on anyone else for the Closed Captions. If you want to spend some extra time setting up your own scraper you can point your server to that source instead of a third party. You have total control.

Check ‘Em

Wondering if this is worth your time? Well, it doesn’t require much of it. The service takes about two minutes to set it up if you already have Node.js and Git installed on your computer. Here’s a video to prove it:

Installation instructions can be found in the readme and you can always get in contact with me through the blog or on twitter.

By now you may have read four different accounts of my program, but if you make it through this one you will be rewarded with Internet gold. My backstory is fast. I started this process later than the others because I had to graduate first. I’m only two months in, which means I’m about 20% complete.

In that short amount of time I have already:

• Blown my tech stipend on the new hotness and become the envy of everyone around me.
• Moved to an entirely different state and bought a kitten.
• Toured Europe for three weeks with my wife (and it wasn’t considered vacation).
• Been monitored by the CIA at the Tor Hackathon in Florence.
• Broken the law in Berlin by infiltrating an old abandoned spy building under the leadership of Cole Gillespie.
• Watched In Bruges in Bruges.
• Raided The Guardian, The BBC, and Mozilla’s London HQ.
• Drank a Tactical Nuclear Penguin.
• Worked with the other fellows to dream up complete iterations on two home-brewed projects (Torwolf and Newsquest).
• Dreamed with my peers and started moving on dozens of ideas including Raspberry Pi crisis cams, time-lapse Internet radio, and magical data extraction APIs.
• Contributed to the infrastructure that supports The Boston Globe.
• Challenged and questioned policies and processes at The Globe.
• Learned first hand about the politics and hurdles within news organizations (I call this “the real world.”)
• Regularly met with VIPs at The Globe, The Guardian, The MIT Media Lab, and The New York Times, to absorb their insights about the state of journalism.
• Maintained strong ties with the people of my past by arranging a formal ongoing relationship with the Media Lab.
• Not been sued or fired yet.

You’ll notice that this list contains a mixture of productivity, fun, and life. This is because fellowships are not just paychecks, they are about personal growth as much as personal output. This one is no different, as shown by the fact that I’ve spent almost half of my time traveling around the world.

If you are saying to yourself “holy crap that’s sweet” you are absolutely correct. In fact, one point of this post is to help you understand why The Knight-Mozilla Fellowship is one of the most rewarding jobs on earth right now and that you should apply this week.

There are some things you should expect if you make it in.

1) You will understand why the news industry is struggling to survive, and why there is hope

I decided not to use this post to talk about my observations and insights about journalism. I already write about that subject on this blog. This item still gets top billing because it defines our mission. We are trying to publicly understand, question, observe, and create in the context of news. There are so many chances to do all four of those things. Not a day has gone by where I haven’t been exposed to something new — a new idea, a new problem, or a new opportunity.

2) You will become unemployable (in a good way)

This now makes two positions in a row where I’ve heard my colleagues say that our work has made us unemployable. This isn’t because employers won’t want to hire us, it’s that we are being spoiled. We’re getting so used to creative freedom, security, and special treatment that simply doesn’t usually come with a traditional job. It’s a good problem to have.

3) You will be challenged, and you will be special

You are being thrown into an organization that may have a vision for you to work with, or may expect you to invent a vision of your own from scratch. Either way your time is going to be your own and you will be expected to make great use of it. This kind of freedom is difficult to cope with, especially when people have high hopes for you. People will throw you questions to ponder, ideas to critique, and problems to solve and you will need to prove yourself.

In return you get to ask anyone anything. You will get to bend the rules and do things that other people around you might have to fight hard to accomplish. If you are interested in something, you will be able to work on it. If you have a question or concern you will be able to get an audience with the CTO or the chief editor. Nobody else at your organization has your title.

4) You will make friendships that last the rest of your life

It’s a cliche, but it’s for real: one of the most rewarding parts of this is the people. You won’t just be part of a community, you will be part of creating a community. It started last year during a week long hackathon in Berlin, where I met dozens of people who I still see all the time. That sense of comradery continues to dominate this experience, and of course it also includes the people in your newsroom.

For the next round of fellows there won’t be a Berlin event, but you will still get to be a part of an 8 person group for almost a year in addition to becoming immersed in a young community of passionate people. You will drink whiskey in foreign countries with friends, share stories of trials and tribulations in workplaces half a world away from you, and find yourself in areas you never would have entered alone.

And with that I think I promised you a reward… I present to you the Amazing Spinning Gridinoc!

ALSO: Please come talk to the folks in the OpenNews community by signing into on our chat room below. Just come in and say hello!

At least I’m not the weird new guy wearing dress shirts that don’t really fit any more; I’ve upgraded to button down T’s.

Update: Matt Stempeck has told me “it isn’t your fault, but nobody in the world can pull off the button down T.” This is why I usually let the Internet choose my wardrobe.

The Trilogy

Over the next ten months I’ll rotate between the three key technology groups at the Globe. I’ve given them battle-clad nicknames for the purposes of this post.

• Round 1: Front Lines. I’m starting on the product team, working under Mike. This is where people work on longer term product-level projects. This includes things like the two web sites (BostonGlobe.com and Boston.com), the Content Management System (what people actually use to write articles), mobile experiences, and comment systems.
• Round 2: Special Ops. My second rotation will be for the Interactive News team (NewsDev for short), working under Miranda Mulligan. They work on more targeted technology projects such as reporter tools or interactive interfaces.
• Round 3: R&D. I’ll end my run working in the Boston Globe’s Media Lab / R&D department under Chris Marstall. This is where the more “cutting edge” / “insane” / “impractical” / “highly creative” work gets done. I really like the idea of this being my capstone rotation since it feels the most in line with my darker past.

Each of these groups has a different set of responsibilities, constraints, and skills. It seems like the Chris’s group (R&D) has the most freedom but isn’t as likely to ship full blown features and systems that get used by millions. Mike’s team (Front Lines), on the other hand, pushes out code that will support the newsroom and the world. As a result they need to keep their eyes on reality — they can’t take as many risks.

Miranda’s (Special Ops) gets to be somewhere in the middle simply because of the nature of user-facing interactivity and the fact that many of their projects are “reusable one-offs.” For instance, if an interactive feature didn’t scale it would be very unfortunate, but at least the entire organization wouldn’t collapse in a fiery heap of paper. The same could not be said for the code that renders the front page of Boston.com.

Free Developer… like Free Beer?

I was jovially introduced to most people as the free developer (“You’re not free any more! Mwahahah”), although I’m hoping to do more than just hang out and crank out code. I’m coming in with some ideas in mind and a general mission, but narrowing that down into something actionable became a lot easier after seeing my fellow fellows this weekend at the Knight Civic Conference.

• Understand pain points, identify opportunities, and communicate. I heard battle stories from the other fellows and immediately understood that one purpose of the OpenNews experiment is to unearth those stories and put them into the open for discussions. The entire industry needs to know examples of what works and what doesn’t, even if there are a few worms under the rocks.
• Stay creative, pursue wild and crazy projects, and share that creative spirit with everyone around me. I’ve just come from one of the most forward thinking places in the world, but the Lab isn’t defined by skill sets and brains. What makes the lab great is the attitude, approach, and the ability to fail effectively. There is no reason why some of that spirt can’t be transported to the globe (in fact some of it is already here.)
• Stay grounded and write awesome code. At the end of the day I am also a developer! I want to learn some new technologies and create some great projects. The key for me here is that I want to make stuff that will have applications both inside and outside of The Boston Globe.

And so this is my hopeful breakdown of the next 10 months:

1. 30% on immediate needs of the Boston Globe, like new platforms and systems.
2. 30% on my own fleshed out ideas, like Truth Goggles, or The Meta Meta Project.
3. 30% on wacky ideas like Newsquest or ATTN-SPAN.
4. 10% on reflection, in blog posts like this.

I see a parallel between the three groups I’ll be working in and those three major chunks, but I’ll be aiming to do it all, all the time. If you were at last weekend’s OpenNews hackathon at MIT you know that I’m feeling multithreaded.

It takes three screens and two laptops to be multi threaded.

I started the process by meeting with Mike Krolak. He is the man responsible for the technology related to product at The Boston Globe. This means web sites, but also things like printing presses, delivery trucks, and employee management systems. I’ll be starting my fellowship in one of his teams, which is the reason why I ended my first day with an ID badge, credentials, and access to most systems. Impressive, most impressive.

When we talked over lunch he was clearly curious to learn more about my purpose and personal goals, at which point I said “you and me both, bub” and stared awkwardly past his head. What I actually did was share my high level understanding of the Fellowship: it’s an experiment, but the goals are to help technologists learn what it is like to work in the journalism industry and to infiltrate and destroy newsrooms get newsrooms more comfortable with external collaboration and open development. Really we’re figuring out the details as we go along.

I got to learn a little more about Mike, what motivates him, and why he has spent more than a decade working for The Boston Globe. Here are three of my favorite points:

• There are so many simple, unsolved problems. Mike has a math background (I’m wary to tell people with actual math backgrounds that I was a math minor, but I was immediately able to relate), and one of the things that motivate math geeks is unsolved problems. In fact, the quest for the perfect proof can cause them to actually go insane! Mike sees the news industry as a great way to go mad, with tons of challenges ripe for the picking. A prime example: “did you know that there isn’t one major newspaper in the world that can tell you how many papers they distributed today?”
• Technology done wrong can get in the way of good journalism. I feel silly admitting this, but the idea that technology could actually hinder the way a newsroom produces content had never crossed my mind. For example many papers have publication processes that force articles through up to 50 different systems. Each transition leaves room for accidental content modification which results in more edits. What you get by the end is a professional version of whisper down the lane (or “telephone” for you non-Pennsylvanians).
• Newspapers are getting more comfortable with innovation. As you probably know already, when the Internet got popular the News industry sat around fat and happy and tried to use it in the same way they used any medium: to publish their content. In the past 10 years we have seen the advent of R&D departments and research spaces in newsrooms. There is a lot of catchup to do, but at least they’re trying.

For me these points frame the essence of the OpenNews initiative. Now that newspapers are innovating we want them to take a page from the mathematicians and make sure unsolved problems get solved in a way that everyone can learn from, expand upon, and contribute to (e.g. by publishing them to the world). As fellows we are also trying to understand what technology means for journalism and to share our lessons about where it can help and where it can hurt.

I just got back from a trip to Berlin as part of the Knight-Mozilla learning lab (MoJo). Twenty of the participants from the previous round (the month long lecture series) were invited to spend a week in Germany getting to know each other while attempting to churn out some code for the rest of the world to see.

I arrived Sunday morning and quickly learned why it is never a good idea to get to a country before hotel check in. No recovery naps for me! The first thing I did was meet up with Saleem Kahn, Nicola Hughes, and Laurian Gridinoc and take a trip to the Bauhaus where I learned that people have been making things for a long, long time.

——–
Let me pause to quickly explain. I grew up, like most of you, using lots of things. When it came to making I was stuck with Legos and the like until one day I discovered programming and started making digital things with that. Fast forward 15 years and I’m at MIT taking How to Make Almost Anything and I say “oh awesome! Hardware isn’t just magic!” But once the course ended I, for the most part, reverted back to my comfort zone of software (still empowered with the potential to carve out circuits and molds, but not seizing immediate opportunities to utilize that empowerment just yet).

Bauhaus is the rest of the picture and got me excited about making again. The series of shops (I attended the one in Berlin) which closed with the rise of the Nazis, were basically buildings dedicated to modernist design (i.e. creating objects that are both beautiful and functional). As I walked around the museum I realized that I don’t have to make things that are super high tech and based on circuits to be making almost anything, I just have to be making things with a unique purpose. Hackable life for the win!

No time to worry about making things now (sponsor week and thesis proposal deadlines are looming), but I sure am ready to build stuff instead of buying stuff.
——–

After Bauhaus we went back to the hotel and I crashed and burned (and woke up just in time to meet up with Mark Boas and have some good old fashioned German Asian food since most things were closing up by then!).

The next morning was the beginning of what ended up being an INCREDIBLY packed four-day schedule of programming, talking, eating, walking, and sleeping. The hackathon (a term used to refer to these kinds of get togethers where people sit around and code) itself took place in a building called the Betahaus, located in Moritzplatz (aka “Makerplatz” since it is the hub of Berlin’s Maker community). The room was awesome – the fourth floor of a stark concrete building, full of tables, chairs, soda, lots and lots of wifi, and random posters of wildlife on the walls. Now that I think about it I wouldn’t be surprised if they hosted fight clubs on weeknights.

As the 20 of us pondered statements like “nothing should be anything” we started milling around and getting to know one another. Some people were designers, some were journalists, some were hackers, and some were mutts but sure enough project clusters slowly sprung up and by the second day people were nose deep in their laptops.

It was around this time that I realized that my project shared a very common need with most of the others: the need for metadata extraction from pieces of media! Thus was born the Meta Meta Project.

——–
I’ll write more about Meta Meta in another post somewhere on the Internet, but the basic idea is that there are a lot of tools out there which can extract information from images, videos, and text. For instance maybe you want to know all of the locations mentioned in a news article, or maybe you want to find all the words that appear in an image.

A lot of projects would benefit greatly from having access to this information, but to use the tools out there takes a fair amount of time setting up, implementing logic, and generally re-inventing parts of the wheel. Rather than having everyone need to become an expert in the tools, the Meta Meta Project is an API suite which will make it dead simple to put in a piece of media and get back the information you want.

Like I said, I’ll have to post more on that somewhere else.
——–

By day 3 the news partners arrived – there were representatives from BBC, The Guardian, Zeit Online, Al Jazeera, and The Boston Globe. They were there to get to know our work and us, but more importantly they were there to get to know one another. The idea of open collaboration still seems to be a somewhat foreign concept among the professional news industry. This is a pity because there is surely a lot of room for mutual benefit and it would surely free up lots of resources for one another. (Hey news rooms! Hop on board the Meta Meta Project!)

There is so much more to write about but there is so little time so I’m going to wrap up with some quick points:

• I had never attended a hackathon before this one and I’m now totally hooked.
• I had never attended a Mozilla event before and I’m now totally hooked.
• Berlin, and Germany in general, is a surreal place to be. The whole city is marked with the painful memories of the past, and it is just so interesting and tragically beautiful to walk around and see memorials, broken pieces of walls, and intentional marks designed to ensure that things aren’t forgotten.
• I came to realize that America isn’t really as young as everyone makes it out to be. When you think about how both Germany and Spain have had radical change in government in the past century it’s almost as if they’re the newbies.
• Germany pulls off maker punks.

I want to end with my favorite memory from the trip (I stuck around for three days after the coding portion just to see the city). A small group of fellow stragglers and I were wandering around a part of Berlin that I would never wander around on my own. This was by no means a place for tourists. As we passed by doorways of punk clubs blasting out dance music we crossed a well-lit alley blasting a different kind of music. At the other end of the alley was a small band with a gathering crowd behind it. No vocals, just tones, and the energy slowly built. We got caught in the sounds and just watched as they wailed away and eventually climaxed.

In the words of Chris Keller: if someone did that in Manhattan they would probably be carted away.

After the legislation was passed I saw a poll on CNN’s front page that I can only assume was a blatant taunt to drive this horrible situation home. The poll read something along the lines of “How satisfied are you with the actions of your elected representatives?” To which I responded by clenching my fists and screaming to the sky: “How the hell should I know?”

Of course I realize this is nobody’s fault. I realize this especially after listening to Mohamed Nanabhay describe the work and challenges faced by the journalists at Al Jazeera. The professionals manning the ships of media corporations must face countless unsolvable challenges involving what content to air, how to craft a message, and how to share information across many diverse communities in a way that makes sense.

ATTN-SPAN is my hopeful attempt to have my cake and eat it too. Don’t let MoJo or MIT fool you: I’m making it for myself. The idea behind this project is that most content out there is a product that was created for the masses – not for me. I can find algorithms and editors that try to pick out articles written for masses that are similar to me, but ultimately those articles are still written for masses. My theory is that the only way to get true personalization is at the source. The primary source.

The reason nobody likes primary sources is that they are a really inefficient way to transfer information. The irrelevant-to-information ratio is simply too high. Worse. The boring-to-anything ratio is too high. I mean seriously, who watches C-SPAN? But what if that primary source can be tagged, catalogued, and marked up in a way that will help generate digestible content on an individual level?

Once the footage of congress can be automatically organized in terms of not just things like who is talking and what is being discussed, but also in terms of when voices get louder or when gavels hit the table… Well, suddenly primary sources can be patched together completely dynamically in a way that tells a story just for you. Your information diet can be augmented with personalized, real world footage. Finally you’ll know for sure that your senator is just as ineffective as you had previously assumed!

The HTML5 goal didn’t take much prodding because it’s something I should have done last year.  I delayed because jQuery was meeting my needs in terms of prototypes and I was cramming my head full of nodejs, arduino, OSX and terminal magic, git and mercurial, Python, Matlab, how to make almost anything, and all sorts of nerdly things that I should have picked up during my undergrad years but somehow avoided.

What convinced me this time – aside from the fact that it is clearly the future of the web – is that my project for the Learning Lab, a C-SPAN analysis and summary tool called ATTN-SPAN, is pretty much exactly the kind of project that HTML5 is supposed to improve.

The fit is probably best explained by going over the three major hurdles I face for this project:

1. How do I collect and store the video content?
2. How do I processes and personalize the video content?
3. How do I present the video content?

Item 1 is partially solved thanks to some of the great researchers here at the Media Lab – and thanks to the metavid project.

Item 2 is going to be tricky, but that’s a can of worms that has very little to do with HTML5.  I’m hoping that I can turn the videos into an “event based” organization, where events are automatically identified things such as “senator A smiles” “senator B says X” “senator C shows a picture of grazing cattle.”  I also want to leave room for user defined events such as “senator D says something I disagree with.”

Item 3 is where this week’s lectures become particularly important, for implementation, HTML5 is where video presentation and interaction suddenly becomes much more flexible.   The events described earlier in this post are going to drive an “personalized episode generation” algorithm which, for a given individual from a given state, will create a series of C-SPAN timestamps associated with video clips and all sorts of metadata.

As for design… Well, lets just say that clearly I’m going to have to spend some time re-watching that lecture by Jesse James Garret.  I’m going to take a few hours this week when I’m not reading up on HTML5 or creating buttons that allow you to discover the truth about anything online (link to this one coming soon) to do some wireframes for the ATTN-SPAN UX.

Sponsor Week -- thanks to glemak on Flickr for the photo.

But demos are hard, and not because of technology.  How does a person take a big, beautiful, perfect idea sitting in Plato’s land of forms and force it into the ugly, imperfect, duct-taped monstrosity that is a “prototype?”  Even with all the technical knowledge in the world, that first step from mental to reality is the single most difficult moment in the process of idea realization.  It isn’t difficult because of time, or risk, or giant R&D brick walls that need hurdling (although all of those can be a real pain in the ass).  Whether you realize it or not, it is an issue of humility.

It’s an incredibly humbling experience to sit down with your best ideas and crank out something quick and dirty for the world to see.  In order to make something, you have to get your head out of the clouds and come careening down to earth.  I knew this coming into the Media Lab, and quite honestly it is why I joined; I came here to force myself to demo or die.  (So far I’ve done a bit of both, but that’s what learning is all about!)

At this point I want to mention the Knight-Mozilla learning lab, which is actually the reason I wrote this particular blog post.  The program is essentially a miniature lecture-based course being run online as a joint effort between the Knight Foundation and Mozilla.  I managed to sneak in!  The first lecture, by Aza Raskin, was about exactly this concept: prototyping.  In particular, the value of.  For me the most important takeaway (aside from “Prototypes are the tits of software design”) was a direct order from the Mozilla head designer himself: sit down with an idea, give yourself a day, and make as much as possible. By forcing yourself to cram you get the benefits of procrastination with none of the negatives.

I’m hoping that this mentality will numb the harshness of the reality descent just enough to bash through that subconscious mental block.  Once a day is burned, if the idea is still worth pursuing I’ll give myself a weekend. Then a week if still needed.  The point is that I need to just !@#$%& do it!