Run like the wind.

Being an Alumnus

As you approach the end of your fellowship you are going to ask yourself many questions. Will Dan Sinker still love me when I'm old? Is it true that on your last day they brand your inner thigh with a hot iron that says "PROPERTY OF MOZILLA"? Where did I leave my FitBit?

The biggest one is going to be "where the hell should I go from here?" I'll give an example of what a fellow's immediate future can be by describing my current status as a functioning adult.

It's difficult to say what I do for a living. When asked, I usually give up and declare that I am a freelancer. In reality I'm…

1: A Cofounder

I spent this week in San Francisco for the orientation of Mozilla's accelerator program, WebFWD. I'm here as one of three founders of Hyperaudio Inc., an nonprofit organization formed on behalf of my fellow fellow, Mark Boas.

Together, with a few others–including yet another 2012 Fellow, Laurian Gridnoc–we will spend the next year taking Mark's baby and turning it into a sustainable nonprofit ecosystem for remixable, transcribed video and audio.

2: A Teacher

There is a letter from Syracuse University’s Newhouse School sitting on my doorstep right now which offers a part time, remote faculty position. It is very likely that I will spend the next academic year mentoring students and creating a new set of resources to help them learn "how to make almost anything on the web."

3: An Innovator

Last month I worked with an amazing team at an OpenNews hackathon to build CivOmega. CivOmega makes it possible for people to ask questions about their government and get answers powered by open datasets and APIs. This month I'm in the running with 2013 Knight-Mozilla Fellow Mike Tigas to get funding to turn it into a real, contributor-ready open source project.

4: A Greybeard

Last Friday I was in Miami to serve as a judge for the Knight Community Information Challenge. I read many applications from around the country that pitched ideas about how they want to solve a major community issue with digital tools. The month before that I spoke on a panel about newsroom innovation at the MIT-Knight Center Media Conference.

If nothing else, being a Knight-Mozilla Fellow means you can trick otherwise reputable organizations like The Knight Foundation into thinking you know what you're talking about.

5: An Architect

I work part time to help startups build out their technology. This involves spending a few hours a week managing a team of developers and playing the role of architect and tech lead. Not every startup has to do with my immediate interests, but this is a nice way to keep things fresh. For instance, last month I helped make a button that rich people can press to give themselves more money.

Usually just mentioning my relationship with Mozilla is enough to cause people to swoon and faint, but sometimes I decide to go with vague threats instead. "I know some very important people on the internet. If you don't hire us, life could get very 'difficult'."

6: A Fellow (again…?)

In addition to being a Knight-Mozilla Fellow for life, #km4lyfe, I'll be a remote 2013 RJI Fellow starting in September. My project is an effort to flesh out of my good ol' thesis project, Truth Goggles, an automated bullshit detector for the internet.

7: A Trainer

The 2012 fellows have started a collective brand organization called Shape Journalism. It's a loose group of makers who are willing to help media organizations by training, building, or just offering advice. For example last week Nicola Hughes, Mark Boas and I started laying out plans for a week long data viz training we're expecting to run in November.

8: An Advisor

Have a crazy idea related to journalism, new media, or technology? Apparently I'm the guy to talk to to get feedback! But seriously so many people have reached out to pitch ideas, and it has been wonderful to get to help out.

There are so many people getting into this space, and being a Knight-Mozilla fellow is eerily similar to being a leader.

9: A Hired Gun

Organizations reach out to me fairly regularly to help them build out a prototype, apply something I have made in the past to their mission, or otherwise write some code. It's always awesome to get to work on something you love and get paid at the same time.

10: A Hobbyist

The best part about not having a job that provides health insurance is that you can do whatever you want in between other work. This means learning new skills (like professional-level soundscaping) but it also means getting to continue to make things.

For instance I'm working on a forum that lets groups of people talk to each other in a closed community without isolating them. Basically you can share threads between forums (and be part of lots of communities), so you can have conversations spread to the most relevant places without getting inundated with the anonymous jackasses that we lovingly call “the general public.”

The Moral of the Story

Poster masterfully created by Lyla Duey.

While it is technically true I don't have a job, I am here very much by choice. Being a fellow has set me up with a network of amazing people, who I still work with closely to build awesome things and participate in some badass events.

By the time you complete your fellowship you will be an unstoppable force of raw digital power. You will be oxymoronically established as both an outsider and an insider (so your perspective is priceless), and you will have had 10 months to show off what you can do. Following your passion at that point is as easy as breathing, unless you're a fish.

If your dream is a startup, you will come out of this with mentors, collaborators, and understanding. If you want to teach, you have an impressive set of experiences to show off. If you want a full time job, my other fellows have shown that you can absolutely do that too.

But honestly, seriously, not kidding, what are you waiting for.

You may want to skip this boring post and just check out the site.

Over the past 24 hours I worked with an amazing team to start building a Siri for government. Well, Wolfram Alpha is more like it, but you probably have a better sense of what Siri is. The site is called CivOmega and it allows you to ask any question you want about civics. The system will do its best to get you an answer.

I can’t speak for the team, but I’ll let you know why I proposed this idea at a hackathon about open data. I’ll even use big letters:

Open Data Sucks

People have talked about making government data more accessible for approximately 500 years. The hope is that if you can find data about the way your government operates, you can shed light on interesting patterns and stories. It’s all about transparency and accountability. It’s a beautiful concept. It’s wonderful for society.

But actually data is pretty crappy. It’s dirty and boring: just a bunch of numbers and rows and tables. This kind of stuff doesn’t usually tell you much without a lot of very laborious prodding and exploration. Don’t believe me? Fine. Go find out for yourself. If you managed to get anything interesting out of that link then you have too much time on your hands.

The ONLY thing that civic data has going for it is that programmers tend to build cool hacks using it. I guess every once in a while you get a groundbreaking piece of journalism out of it too but I’ll ignore that for the sake of argument.

It’s Also Elitist

Here’s another problem: programmers have awesome, special tools to access data. These tools are called “Application Programming Interfaces” (also known as an APIs). An API is just a standard way for computers to ask each other for information.

A human version of this plays out every time you go to a restaurant and order from a menu. You look at the list of what you can ask for, you ask for what you want, and eventually you either get your food or you get impatient and start throwing your silverware at other patrons.

In my analogy the food is data and you and the chef are computers. The waiter is the API and the menu is the documentation. I guess the restaurant is the Internet and the restaurant’s manager is the NSA or something. The silverware don’t really fit in.

The point is that the COOL stuff happens because of these APIs. Too bad nobody real knows what the hell an API is or how they could possibly go about using it. Don’t believe me? Go find out for yourself. If the stuff on that page gave you access to data then you’re a nerd.

If nerds and people who have too much time on their hands are the only ones who can use government data then it won’t change the world. Plus, why should those people get to decide what is and isn’t important?

Humanizing Government Data

And so we come back to CivOmega. This is an attempt to give people with normal, human questions the ability to benefit from the data that so many have worked their asses off to expose. It makes it possible for a human to interact with an API in the same way they might interact with their waiter: by asking questions. Users can type in questions about the government and it attempts to provide answers.

It is built on a programming language called Python and the way it works is pretty simple. A programmer who understands an API can write some code that knows how to answer certain question patterns. For instance I made it possible to ask the question “What bills are about [X]” where X can be any phrase you want. If you ask that, CivOmega will talk to the appropriate APIs to get you the answer you want. Then it will tell you what it learned.

The beauty of this setup is that any other programmer can spend a few minutes teaching the system to answer new kinds of questions. For instance maybe someone knows about an environmental dataset and wants you to be able to ask questions about natural disasters (how many forest fires happened in California last year?). That person could easily unlock that resource.

If you’re a developer, go take a look at the repository and consider adding a module. If you are a master of NLP please get in touch with me so we can improve the way people ask questions. If you don’t know what either of those sentences meant, please just go check out the site.

This post is not about Quizzler, it is about my quest to answer the question “why are we building this from scratch?” It’s about observed realities regarding cross-newsroom collaboration, insights from upper management of The New York Times, and some major hurdles for open source in legacy media organizations. Prepare to explore the deep, dark, and relatively unspoken depths of technological openness in newsrooms.

We want something similar to…

I was introduced to Quizzler back in August. That first meeting was generally uneventful; we sat in a room. I listened to Miranda Mulligan skillfully duke it out with the project’s newsroom sponsor to explain that no, the first version won’t have custom “you are a 95% Vampire” sharable Facebook messages. I listened to the sponsor vocalize concern that there would never actually be a second version. I decided that both of them were probably right.

Eventually someone said something so shocking that I literally spat out my drink and fell out of my chair at the same time. It wasn’t intended to stand out—I don’t even know who said it. Ready? Brace yourself. Here it is: “Have you seen the Academy Awards tool by The New York Times? Eventually we will want something similar to that.” No wait that wasn’t it.

The New York Times is the parent company of The Boston Globe. They own the Globe in the same way humans own their children.

OK here’s the exchange.

Me: “Can we use some of their code?”

Someone: “We would have to pay them for that.”

Me:

Their response implied two things. First, that The New York Times would charge their kid for the digital equivalent of food. Second that the anticipated costs were high enough that it would be cheaper to rebuild this tool from scratch (again) than it would be to explore the possibility of reusing existing code.

Before you call child protection services, hold on. The situation is complex.

EDIT: To be clear, I quickly learned that the Times would not have charged us a dime.

Actually, this sounds completely reasonable

“Meh.” you say, “so The Boston Globe and The New York Times don’t share code, what’s the big deal?” A fair response, but trust me when I say the deal is big. If the deal was a rapper it would be notorious.

Here’s why: Starting from an existing code base instead of starting from nothing is often the difference between “having time to innovate” and “not.” If you are using technology as a core part of your business and you aren’t set up to experiment then you’re doing it wrong and you will become obsolete.

Borrowing code is kind of like being airdropped into the middle of a marathon; sure, you have to take a moment to figure out where you are and what direction to go, but now you have time to run in circles laughing like a crazy person before winning the race.

There’s more! If you borrow code then you are more likely to be familiar with what the rest of the world is doing. If you share code then you are going to build your systems with an emphasis on reuse and extensibility (i.e. correctly). If you regularly borrow AND share code then you are building a community around whatever it is you do.

What I’m trying to say is that if newspapers can buy into the mantra of openness—even just internal openness—they can kill about thirty birds with one stone.

But they usually don’t.

Why not? Are they idiots?

There are many reasons these organizations don’t trade bytes, none of which have to do with the original “we would have to pay for it” claim.

Reason 1: Wildly Different Technology Stacks

I lied to you earlier when I said the Globe was like a child to the Times–they’re more like middle-aged lovers. They didn’t grow up together or meet in college. They are two independent entities that recognized their love later in life, which means they have fundamentally different infrastructures.

One uses Java and PHP, the other uses Python, Ruby, and NodeJS. They have incompatible content management systems. They disagree on deployment policies, quality control processes, needs, and third party libraries. It’s like they come from two stubborn families that speak completely different languages and eat very different foods. They aren’t going to start casually sharing cook books.

Reason 2: Internal Politics

If a full team dedicates three months to creating a new public-facing interactive, will they want to just give it away? If you are a manager do you want to rely on favors from an external team to accomplish your goals? If you are a coder do you want to be judged for the quick last minute hacks you had to throw into the project?

The answer to these questions, and many more like them, is “hell no.”

Reason 3: Moving Costs and Learning Curves

Most technologies are dirty piles of duct tape with a shiny chrome finish. This makes them difficult to deploy and hard to understand. This is especially true among newspapers.

Packaging code in a way that strangers can use could take hours, days, or weeks depending on how much the developers cared about portability when they built it. I’m basically describing the difference between moving a campsite and a home. Newsroom developers don’t tend to have camping on the brain when rushing to meet looming deadlines.

Words from On High

Fine, so there are real reasons that code sharing between the Globe and the Times is a lost cause, but what does that mean for the industry? If financial allies with serious resources don’t share code, what are the chances that other newsrooms around the world will look outside their walls for help? Maybe this is why so many open source journalism tools are gathering dust.

I talked to Rajiv Pant (CTO) and Marc Frons (CIO) of The New York Times about code sharing and the role of open source in their company. For context: the Times is very progressive compared to other newsrooms when it comes to innovation and openness. They have a blog dedicated to their open source inititatives, there is a suite of APIs that provide civic data, and they do a good job of telling people about what they do.

Unfortunately they are also leading an industry that is forced into “deadline driven technology” and without a supportive institutional strategy, open source and reusable code are just nice-to-haves. Developers must ask themselves if they have time to meet the organization’s needs while also contributing to open source. Sometimes this means the same tools get built multiple times, but such is the nature of deadlines. Plus, as Marc was quick to point out, reinventing the wheel can be a good thing so long as the new one is slightly different.

Source: The Wheel Superstore. (Illustration by Lyla Duey)

But wouldn’t it be nice if all these new wheels could be used again and improved upon over time? Rajiv identified three factors that a project needs in order to be realistically used again by an organization like the Times.

Your code has to be…

1. Established – Is it safe to rely on your creation? How long will your project stay active, and how long after you move on will it stay useful?
2. Extensible – Your solution won’t meet all needs. How easy is it to improve? What kinds of features can be added?
3. Easy to Integrate – Will this play with existing systems and tools? Can it be skinned to look like it belongs?

In short, it doesn’t matter how powerful you think your code is: if it is difficult or risky to adopt, it will stay an orphan.

None of those points should come as a surprise, but they should probably be considered gospel to anyone developing anything—open or closed—in any newsroom. Just ask yourself “would the Times use this if they needed it?” If the answer is yes then you’ve made something that will last; otherwise you might as well get out the broom now.

Note: Since I’m sure you are worried, the Times doesn’t actually charge the Globe for code. And yes, we are writing Quizzler from scratch.

At face value it just looks like a live feed of C-SPAN’s Closed Captions. This alone is actually pretty cool if you think about it, especially if you are a deaf political junkie who sits far away from the TV and can’t read the closed captions.

Of course there is more. The real excitement comes when you contemplate what’s happening to get those words to appear on your screen.

This system unlocks and syndicates a real-time dataset that used to be a pain in the ass to access. Now anyone can build applications and visualizations that update before those crafty politicians have even finished making their points. This post explains why Opened Captions is worth hacking with, what it takes to use it, and how it works.

What is it Good For?

The Internet is filled with real-time updates triggered by online activity, but it still feels like magic when we see automatic updates driven by the real world. Opened Captions makes it easy for programmers to use live TV transcripts as an input.

Note: version .001 only supports a single channel (and my server is pointed to C-SPAN). Eventually the protocol should expand to allow multiple channels.

Let’s consider C-SPAN. If a computer knows what is being said on C-SPAN this very second, it can do things like:

• Change the background of your email client to reflect the issues being debated right this moment on the senate floor.
• Generate modified, more amusing, transcripts by replacing key words and phrases with Tolkien lore (i.e. C-SPAN for Middle Earth)
• Search through lyrics and generate a C-SPAN medley for you to rock out to while voting.
• Send SMS messages 24/7 commanding you to “drink” when certain phrases are spoken on air.

There are also possibilities that aren’t ridiculous. For instance, you could make tools that…

• Improve the transcript by automatically adding contextual information, such as definitions and histories thefted from Wikipedia.
• Send emails with transcript snippets whenever a specific representative or state is being discussed on TV so you know what’s going on.
• Parse out paraphrases of known fact checks and insert a credibility layer over the transcript feed (real time fact-checking).
• Draw parallels between what is being said on TV and what is being said on Twitter.

I could go on and on and on. There is just so much potential!

The Backend

Behind the stream is a first stab at a distributed architecture for Closed Captioning live-feeds. Opened Captions servers can pull a CC stream over a serial port, or (more likely) they will connect to an existing Opened Captions server and pull the stream from there. What that means in de-jargon is that anybody can set up a server that does exactly what mine is doing, even if they don’t have access to hardware, software, or a live TV stream.

When I say exactly, I mean it — your new project runs the same code as mine, and will serve the feed too. People can connect their servers to yours in the same way you connected yours to mine. Practically speaking this architecture means a few things:

1. Once your amazing mashup gets popular it won’t break my server. Your application is syndicating the captions to your users. I serve the captions to you, you serve them to the world!
2. Your server creates a fork of my stream. Want to modify the text so the politicians sound drunk? Add extra layers of information to the message payload? Translate the captions to Klingon? Go for it. If your tweaks happen server side then others can build their apps from your stream to modify it further.
3. You don’t have to rely on anyone else for the Closed Captions. If you want to spend some extra time setting up your own scraper you can point your server to that source instead of a third party. You have total control.

Check ‘Em

Wondering if this is worth your time? Well, it doesn’t require much of it. The service takes about two minutes to set it up if you already have Node.js and Git installed on your computer. Here’s a video to prove it:

Installation instructions can be found in the readme and you can always get in contact with me through the blog or on twitter.

He goes on to explain that he is a journalist writing about Tor. He also tells us that he bets that the CIA and the Italian Secret Service are going to have moles there. What he obviously meant to say was “I work for the CIA and I’ve been watching you now for quite some time.”

It’s possible that he didn’t actually work for the CIA. His name and photo checked out under the website he claimed to write for. It was probably just a one-time job. Even if this isn’t true, even if a network of government spies didn’t track my position across Europe just to meet us in a restaurant, his comment set the tone for my weekend in Florence.

Tor is serious business.

What the hell is Tor?

Did I mention Tor yet?

Tor is a program that makes you anonymous. This means that, for better or for worse, the big brothers, neighborhood hackers, and ad agencies of the world can’t tell what you are doing on the Internet without going through a lot of effort and expense.

Is that too abstract? Here are some illustrative statements. *Taps the microphone*

• A Tor user walks into a bar, the bartender says “who are you?”
• How many Tor users does it take to screw in a light bulb? Only a few, but you’ll never know who did it.
• I used Tor last night and now my wife says that she doesn’t even know who I am any more.

I’ll be here all night.

If you use Tor you become Spartacus. Tor takes everything you do, makes it look exactly like what everyone else is doing, and gets random computers on their network to do the talking for you. Ta-da! Now it is practically impossible to pin an action on you.

The Original Need

I bet you wouldn’t have guessed that this idea was invented by The U.S. Navy. You would have? Oh.

Put on your paper sailor hat and I’ll explain. Imagine you are the king of the Navy and you’re going to war with your fleet of a thousand brand new Navy cars (I don’t really know how the Navy works). Being king, you are in the most important car of all because you’re calling the shots. You don’t want the enemy to know which vehicle is yours. You also don’t want them to know who is receiving orders because that could give away your tactics.

“I know,” you say, “I’ll encrypt everything so that they can’t see the content. Then they won’t be able to tell that my broadcasts are more important than others.”

Unfortunately for you, the enemy has fancy technology. They can’t decrypt messages but they are able to track where everything comes from and where it is going. They can’t tell what you’re saying, but they have all they need.

After about 5 minutes you think you’re doing well. Half of the enemy cars are already on fire! Yours explodes. “How did they do that?” you say in the afterlife. “Easy,” responds god, “they were able to see that your car was sending out the most messages. They knew exactly where you were.” Then he slaps you with a piece of linguini and drifts away.

To prevent this from ever happening again the Navy decided to invent the concept of an “Onion Network” (not to be confused with The Onion Network). Now instead of having packets go directly from point A to point B, each one randomly hops around the fleet first. Because of encryption, the enemy can’t tell the difference between a new message and a “hop” message — they all look the same. It’s like running an invisible sprinkler in a thunderstorm.

Suddenly nobody but the sender and the recipient can figure out the end points of a message chain. Even the middle men (the ones doing the hops) don’t know the path. Each piece of the hop — each “layer” of the message — is encrypted with a different key, so the only thing a relay knows is who gave them the package and where it should go next.

Onions have layers too, that’s why this setup is called an Onion Network. Get it? It’s like Shrek!

Trolls use the Internet, Ogres use Tor. (Illustration by Anne Buckwalter)

What’s it Good For?

Tor has applications in the real world. You can buy drugs and guns, share illegal pictures, and hire assassins. Oh wait, I’m just describing Tor’s reputation (more on that later). Seriously, there are a lot of important situations where people have moral and compelling reasons to want anonymity.

Here are a few:

• Protecting witnesses and victims of domestic abuse. Anyone who wants to be able to access the internet without being discovered by a third party can use Tor to defend against their stalkers.
• If you don’t like being tracked by your government, Internet Service Providers, or search engines.
• Providing truly anonymous tips. There are times when people need or want to share information against the wishes of powerful and potentially dangerous forces (e.g. mafias, governments, or corporations).
• Safely bypassing censorship. If you live in Syria, China, or The United States of RIAA/MPAA, you might use Tor to access content from the outside world more safely.

These kinds of reasons explain why organizations with very good reputations, like the Knight Foundation, are devoting resources to Tor.

The Dark Web

What I’ve just described is a spin on the way people access normal information online. If you point Tor Browser to Google you will see the same old Google, it’s just that now Google doesn’t know who you are. That’s powerful enough, but there’s more: Tor also lets you see hidden content on the Internet.

Using Tor is like entering a cheat code into real life and playing the lost levels. It is the digital equivalent of platform 9 and 3/4. This secret section of the Internet is possible because Tor users can serve content anonymously too.

If you don’t know much about how the Internet works, believe me when I say that if a web site’s location is hidden it becomes essentially impossible to access. It would be like trying to visit someone’s house without knowing anything about where they live — not even the country. Tor gives you a blindfold and leads you there. You still don’t know where the house is, but at least you can visit.

Anonymous sites are accessed through something called an “onion address,” which is made up of a series of random letters and numbers. For instance, this is a “clean” version of Tor’s wikipedia: 3suaolltfj2xjksb.onion. Feel free to try clicking the link, it won’t work (Unless, of course, you are using the Tor browser).

Note: even if that link worked you wouldn’t see any terrible images. However, you need to use your brain before you start actually clicking around if you don’t want to get really upset.

That random looking string is used to find the server within the Tor network. Because the addresses don’t point to a real address on the Internet, there is no way to fully access this content without Tor. There are services you can use to get there without using Tor, but you lose all benefits of anonymity and content is often censored.

Onion addresses are the most fascinating part of Tor, albeit the most potentially disturbing. Rest assured that they don’t all lead to child porn, guns, and drugs. For example there is a secret version of Twitter, a bunch of blogs, a search engine, and an email service. There is even a secret version of 4chan (called Torchan), which I won’t link to because that one does lead to child porn and drugs.

These types of content networks—ones that are served on top of the normal web so that you need special programs to reach them—are known as the Dark Web. Not necessarily because the content is darker (it is), but because it is hidden from view and can’t really be searched and scraped as reliably.

Implications of The Dark Web

Most uses for Tor become more potent with onion addresses. Anonymous servers are just as protected from higher powers as anonymous users. If Amazon suddenly started selling illegal drugs they would get in trouble. If a Tor marketplace started selling illegal drugs, the law would have to figure out a way to find them first.

This power applies to legitimate uses as well. If a government official wanted to contact The Boston Globe with a corruption leak, he or she could use Tor to create a gmail account anonymously. The government could then subpoena Google, and Google might be willing to give away the information they have. They won’t know much, but now things like account access patterns and full email logs would be fair game.

If the official had used Tormail then even Google wouldn’t know what happened. The government would have no course of action because there would be no service provider to ask. Every journalist in the world should be able to agree that there is no good reason for a watchdog to trust the organizations they are watching. Why should you trust in corporations and governments to keep sources safe?

Tor has a reputation because it has a lot of criminal content, but the social good that it supports is just so important (criminals will always be criminals). I’m working on a game called Torwolf to simulate a few situations where Tor would be effective (if you have played Werewolf or Mafia, you can start to imagine what the game will be like). In the mean time, read up on Tor if you’re curious. Better yet, go try it out.

IMPORTANT EDIT: while Tor is much better than nothing, it is neither foolproof nor perfect. If maintaining anonymity could be a matter of life / death / imprisonment, then you need to know more than what I could fit into the scope of this overview. This paper is a good starting point, but seriously, spend some time researching on your own.

By now you may have read four different accounts of my program, but if you make it through this one you will be rewarded with Internet gold. My backstory is fast. I started this process later than the others because I had to graduate first. I’m only two months in, which means I’m about 20% complete.

In that short amount of time I have already:

• Blown my tech stipend on the new hotness and become the envy of everyone around me.
• Moved to an entirely different state and bought a kitten.
• Toured Europe for three weeks with my wife (and it wasn’t considered vacation).
• Been monitored by the CIA at the Tor Hackathon in Florence.
• Broken the law in Berlin by infiltrating an old abandoned spy building under the leadership of Cole Gillespie.
• Watched In Bruges in Bruges.
• Raided The Guardian, The BBC, and Mozilla’s London HQ.
• Drank a Tactical Nuclear Penguin.
• Worked with the other fellows to dream up complete iterations on two home-brewed projects (Torwolf and Newsquest).
• Dreamed with my peers and started moving on dozens of ideas including Raspberry Pi crisis cams, time-lapse Internet radio, and magical data extraction APIs.
• Contributed to the infrastructure that supports The Boston Globe.
• Challenged and questioned policies and processes at The Globe.
• Learned first hand about the politics and hurdles within news organizations (I call this “the real world.”)
• Regularly met with VIPs at The Globe, The Guardian, The MIT Media Lab, and The New York Times, to absorb their insights about the state of journalism.
• Maintained strong ties with the people of my past by arranging a formal ongoing relationship with the Media Lab.
• Not been sued or fired yet.

You’ll notice that this list contains a mixture of productivity, fun, and life. This is because fellowships are not just paychecks, they are about personal growth as much as personal output. This one is no different, as shown by the fact that I’ve spent almost half of my time traveling around the world.

If you are saying to yourself “holy crap that’s sweet” you are absolutely correct. In fact, one point of this post is to help you understand why The Knight-Mozilla Fellowship is one of the most rewarding jobs on earth right now and that you should apply this week.

There are some things you should expect if you make it in.

1) You will understand why the news industry is struggling to survive, and why there is hope

I decided not to use this post to talk about my observations and insights about journalism. I already write about that subject on this blog. This item still gets top billing because it defines our mission. We are trying to publicly understand, question, observe, and create in the context of news. There are so many chances to do all four of those things. Not a day has gone by where I haven’t been exposed to something new — a new idea, a new problem, or a new opportunity.

2) You will become unemployable (in a good way)

This now makes two positions in a row where I’ve heard my colleagues say that our work has made us unemployable. This isn’t because employers won’t want to hire us, it’s that we are being spoiled. We’re getting so used to creative freedom, security, and special treatment that simply doesn’t usually come with a traditional job. It’s a good problem to have.

3) You will be challenged, and you will be special

You are being thrown into an organization that may have a vision for you to work with, or may expect you to invent a vision of your own from scratch. Either way your time is going to be your own and you will be expected to make great use of it. This kind of freedom is difficult to cope with, especially when people have high hopes for you. People will throw you questions to ponder, ideas to critique, and problems to solve and you will need to prove yourself.

In return you get to ask anyone anything. You will get to bend the rules and do things that other people around you might have to fight hard to accomplish. If you are interested in something, you will be able to work on it. If you have a question or concern you will be able to get an audience with the CTO or the chief editor. Nobody else at your organization has your title.

4) You will make friendships that last the rest of your life

It’s a cliche, but it’s for real: one of the most rewarding parts of this is the people. You won’t just be part of a community, you will be part of creating a community. It started last year during a week long hackathon in Berlin, where I met dozens of people who I still see all the time. That sense of comradery continues to dominate this experience, and of course it also includes the people in your newsroom.

For the next round of fellows there won’t be a Berlin event, but you will still get to be a part of an 8 person group for almost a year in addition to becoming immersed in a young community of passionate people. You will drink whiskey in foreign countries with friends, share stories of trials and tribulations in workplaces half a world away from you, and find yourself in areas you never would have entered alone.

And with that I think I promised you a reward… I present to you the Amazing Spinning Gridinoc!

ALSO: Please come talk to the folks in the OpenNews community by signing into on our chat room below. Just come in and say hello!

There are many reasons why he might have said this, but two stand out. For one thing I had just given a presentation about NewsJack, a media manipulation platform that I created from Mozilla’s Hackasaurus with Sasha Costanza Chock. When NewsJack was released it was immediately met with a Cease and Desist from the New York Times (note that The Times is the parent company of The Boston Globe).

It is also possible that he was inspired because I had just confessed on stage that one of my first thoughts when walking into The Globe’s headquarters was “I wonder what it would take to bring down this organization.” I’m betting it was the juxtaposition.

The Backstory

An evil newspaper editor?

Assuming you’re a competent villain, whatever you end up targeting should be important. Not only that, but the target will reflect your personal passions and expertise. Try the mental exercise yourself and share the results. I dare you.

For example, to take down a newspaper you could…

• Open up their paywall (if it exists), steal their content, and make it freely visible to the world without giving them any form of recognition or compensation.
• Eliminate their productivity, either by instigating a massive strike or by hiring away all of their employees.
• Scare away their advertisers so they lose a significant revenue stream and can no longer pay their bills.
• Destroy their infrastructure (printing presses, websites, etc), thus disabling their ability to ship product.
• Corrupt their editors and slowly replace key actors with your henchmen so that the paper becomes your mouthpiece.
• Buy sharks with laser beams attached to their heads.

A super villain’s master plan needs to be intricate enough to be interesting and difficult enough to be impressive. Blunt ideas like “take down their website” or “steal all their money” are a bit too obvious. It must also be simple enough for a diverse audience to understand. If nobody can figure out what you did, why it was sinister, or how it actually worked then it is hardly going to make headlines. Finally, it can’t be a series of bee stings; the evil needs to be condensed enough that it could fit in a tweet.

The Plan

My evil plan didn’t take long to imagine (given my recent work). If I were evil and wanted to destroy a newspaper I would ruin their brand’s credibility. This could be accomplished in many interesting and convoluted ways, but the “how” isn’t the point, the important question is “why?”

A media product will die miserable and alone unless it differentiates itself from the rest of the Internet. Luckily, newspapers have something that the chaff doesn’t: they have the capacity to create trustworthy information experiences. They are the ones with paid reporters asking the hard hitting questions, they have the editors and the internal fact-checkers, they don’t have an agenda and aren’t trying to manipulate me! right?

You could tie yourself to a bungee cord, close your eyes, and jump off a cliff… or you could read the New York Times.*

Well, maybe. As a reader I don’t know where content comes from or how much journalism went into it. All I have is faith in their brand. I trust that the sources I read are doing their jobs. That faith didn’t come from nowhere. I might have liked what they had to say in the past, or I saw my parents reading their paper, or their brand has a strong reputation. Regardless, I am now far more likely to trust what they have to say than I am to trust, for example, what my crazy friends like to read.

Just to drive this home: given the way content is presented today I could read the exact same article on the front page of the New York Times, Fox News, or the Huffington Post and my decision to trust it would be more strongly influenced by my opinions of the publisher than by the content itself.

To drive it home a different way: hijacking a newspaper’s credibility is as simple as imitating their brand.

Save the Day

The wheels are turning and it is already out of my control! IP lawyers are powerless compared to the forces of the anonymous web! But seriously, brand is a really fragile way to differentiate on the Internet. So what’s a newspaper to do?

Take a page from Apple and redefine the way people consume content. Train your readers to expect a certain experience not just from your website, but from every source of news. Make sure that experience is either expensive or impossible for alternative sources to replicate. Newspapers need to make their readers expect proof of everything. People should feel uncomfortable trusting information without explicit, functional credibility.

Newspapers have journalists doing research, checking facts, and taking names. They have multiple people and multiple systems touching every piece of content before it gets published, so why does the product usually end up being a bunch of words with prose-based evidence?

News organizations need to make the world hold information to their standards.

Like I said earlier, it makes sense that this particular plot and solution are coming from me. I dedicated my thesis to credibility layers — interfaces that lead to credible information experiences based on more than faith and trust. There are many paths to differentiation. Some are evil, some are entertaining, and some could even change the world.

* Drawing courtesy of Lyla Duey

At least I’m not the weird new guy wearing dress shirts that don’t really fit any more; I’ve upgraded to button down T’s.

Update: Matt Stempeck has told me “it isn’t your fault, but nobody in the world can pull off the button down T.” This is why I usually let the Internet choose my wardrobe.

The Trilogy

Over the next ten months I’ll rotate between the three key technology groups at the Globe. I’ve given them battle-clad nicknames for the purposes of this post.

• Round 1: Front Lines. I’m starting on the product team, working under Mike. This is where people work on longer term product-level projects. This includes things like the two web sites (BostonGlobe.com and Boston.com), the Content Management System (what people actually use to write articles), mobile experiences, and comment systems.
• Round 2: Special Ops. My second rotation will be for the Interactive News team (NewsDev for short), working under Miranda Mulligan. They work on more targeted technology projects such as reporter tools or interactive interfaces.
• Round 3: R&D. I’ll end my run working in the Boston Globe’s Media Lab / R&D department under Chris Marstall. This is where the more “cutting edge” / “insane” / “impractical” / “highly creative” work gets done. I really like the idea of this being my capstone rotation since it feels the most in line with my darker past.

Each of these groups has a different set of responsibilities, constraints, and skills. It seems like the Chris’s group (R&D) has the most freedom but isn’t as likely to ship full blown features and systems that get used by millions. Mike’s team (Front Lines), on the other hand, pushes out code that will support the newsroom and the world. As a result they need to keep their eyes on reality — they can’t take as many risks.

Miranda’s (Special Ops) gets to be somewhere in the middle simply because of the nature of user-facing interactivity and the fact that many of their projects are “reusable one-offs.” For instance, if an interactive feature didn’t scale it would be very unfortunate, but at least the entire organization wouldn’t collapse in a fiery heap of paper. The same could not be said for the code that renders the front page of Boston.com.

Free Developer… like Free Beer?

I was jovially introduced to most people as the free developer (“You’re not free any more! Mwahahah”), although I’m hoping to do more than just hang out and crank out code. I’m coming in with some ideas in mind and a general mission, but narrowing that down into something actionable became a lot easier after seeing my fellow fellows this weekend at the Knight Civic Conference.

• Understand pain points, identify opportunities, and communicate. I heard battle stories from the other fellows and immediately understood that one purpose of the OpenNews experiment is to unearth those stories and put them into the open for discussions. The entire industry needs to know examples of what works and what doesn’t, even if there are a few worms under the rocks.
• Stay creative, pursue wild and crazy projects, and share that creative spirit with everyone around me. I’ve just come from one of the most forward thinking places in the world, but the Lab isn’t defined by skill sets and brains. What makes the lab great is the attitude, approach, and the ability to fail effectively. There is no reason why some of that spirt can’t be transported to the globe (in fact some of it is already here.)
• Stay grounded and write awesome code. At the end of the day I am also a developer! I want to learn some new technologies and create some great projects. The key for me here is that I want to make stuff that will have applications both inside and outside of The Boston Globe.

And so this is my hopeful breakdown of the next 10 months:

1. 30% on immediate needs of the Boston Globe, like new platforms and systems.
2. 30% on my own fleshed out ideas, like Truth Goggles, or The Meta Meta Project.
3. 30% on wacky ideas like Newsquest or ATTN-SPAN.
4. 10% on reflection, in blog posts like this.

I see a parallel between the three groups I’ll be working in and those three major chunks, but I’ll be aiming to do it all, all the time. If you were at last weekend’s OpenNews hackathon at MIT you know that I’m feeling multithreaded.

It takes three screens and two laptops to be multi threaded.

I started the process by meeting with Mike Krolak. He is the man responsible for the technology related to product at The Boston Globe. This means web sites, but also things like printing presses, delivery trucks, and employee management systems. I’ll be starting my fellowship in one of his teams, which is the reason why I ended my first day with an ID badge, credentials, and access to most systems. Impressive, most impressive.

When we talked over lunch he was clearly curious to learn more about my purpose and personal goals, at which point I said “you and me both, bub” and stared awkwardly past his head. What I actually did was share my high level understanding of the Fellowship: it’s an experiment, but the goals are to help technologists learn what it is like to work in the journalism industry and to infiltrate and destroy newsrooms get newsrooms more comfortable with external collaboration and open development. Really we’re figuring out the details as we go along.

I got to learn a little more about Mike, what motivates him, and why he has spent more than a decade working for The Boston Globe. Here are three of my favorite points:

• There are so many simple, unsolved problems. Mike has a math background (I’m wary to tell people with actual math backgrounds that I was a math minor, but I was immediately able to relate), and one of the things that motivate math geeks is unsolved problems. In fact, the quest for the perfect proof can cause them to actually go insane! Mike sees the news industry as a great way to go mad, with tons of challenges ripe for the picking. A prime example: “did you know that there isn’t one major newspaper in the world that can tell you how many papers they distributed today?”
• Technology done wrong can get in the way of good journalism. I feel silly admitting this, but the idea that technology could actually hinder the way a newsroom produces content had never crossed my mind. For example many papers have publication processes that force articles through up to 50 different systems. Each transition leaves room for accidental content modification which results in more edits. What you get by the end is a professional version of whisper down the lane (or “telephone” for you non-Pennsylvanians).
• Newspapers are getting more comfortable with innovation. As you probably know already, when the Internet got popular the News industry sat around fat and happy and tried to use it in the same way they used any medium: to publish their content. In the past 10 years we have seen the advent of R&D departments and research spaces in newsrooms. There is a lot of catchup to do, but at least they’re trying.

For me these points frame the essence of the OpenNews initiative. Now that newspapers are innovating we want them to take a page from the mathematicians and make sure unsolved problems get solved in a way that everyone can learn from, expand upon, and contribute to (e.g. by publishing them to the world). As fellows we are also trying to understand what technology means for journalism and to share our lessons about where it can help and where it can hurt.

My first full day was yesterday and I’m just about finished a post about my first impressions and lessons learned. Before I hit publish on that, though, I want to give a nod to the option I decided not to pursue: a PhD at the MIT Media Lab.

Choosing a Path

I ultimately decided not to apply to the PhD program at the Media Lab for now. This wasn’t easy (and doesn’t mean I won’t come back and apply later), but it came down to a few points:

I’m not really a great academic. Don’t get me wrong, the motto of “Demo or Die” generally trumps “Publish or Perish” at the Media Lab, but this doesn’t change the fact that a PhD is ultimately an academic endeavor. When it comes to technology my natural inclination is to code and blog, not to design studies and write scientific papers.

I got a huge amount out of my time at the lab. Yes, the awesome ability of the Media Lab to jump start lives is actually part of why I’m moving on. In two years I feel empowered to create anything I want, I’ve redefined my understanding of the world, quadrupled my confidence, developed an amazing network of friends and connections, and become part of a wonderful community. I’m sure that four more years at the lab would continue improvement on all of those fronts, but I doubt that it would be nearly as saturated.

Four years is much longer than 10 months. Call me an opportunist but I’m married, and who knows where Lyla and I will want to be over the next four years. Actually it turns out we do know (Lyla got into RISD and starts her graduate program in September), but that isn’t the point. Locking myself into something for four years seems dangerous at a time when there are so many problems to solve and ideas to pursue. What would have happened if Luke just chilled out with Yoda for four years?

All of that being said, if I ever commit to a PhD it will be at the Media Lab. I already miss the place! Luckily I’ll stay an official part of it for at least another year as a “zero cost visiting researcher.” This means I don’t get paid, but I get to keep access to resources and have a good explanation as to why I’m still skulking around the hallways at night.

This leaves me here, living the Tolkien dream and beginning a brand new fellowship.