This post is not about Quizzler, it is about my quest to answer the question “why are we building this from scratch?” It’s about observed realities regarding cross-newsroom collaboration, insights from upper management of The New York Times, and some major hurdles for open source in legacy media organizations. Prepare to explore the deep, dark, and relatively unspoken depths of technological openness in newsrooms.

We want something similar to…

I was introduced to Quizzler back in August. That first meeting was generally uneventful; we sat in a room. I listened to Miranda Mulligan skillfully duke it out with the project’s newsroom sponsor to explain that no, the first version won’t have custom “you are a 95% Vampire” sharable Facebook messages. I listened to the sponsor vocalize concern that there would never actually be a second version. I decided that both of them were probably right.

Eventually someone said something so shocking that I literally spat out my drink and fell out of my chair at the same time. It wasn’t intended to stand out—I don’t even know who said it. Ready? Brace yourself. Here it is: “Have you seen the Academy Awards tool by The New York Times? Eventually we will want something similar to that.” No wait that wasn’t it.

The New York Times is the parent company of The Boston Globe. They own the Globe in the same way humans own their children.

OK here’s the exchange.

Me: “Can we use some of their code?”

Someone: “We would have to pay them for that.”

Me:

Their response implied two things. First, that The New York Times would charge their kid for the digital equivalent of food. Second that the anticipated costs were high enough that it would be cheaper to rebuild this tool from scratch (again) than it would be to explore the possibility of reusing existing code.

Before you call child protection services, hold on. The situation is complex.

EDIT: To be clear, I quickly learned that the Times would not have charged us a dime.

Actually, this sounds completely reasonable

“Meh.” you say, “so The Boston Globe and The New York Times don’t share code, what’s the big deal?” A fair response, but trust me when I say the deal is big. If the deal was a rapper it would be notorious.

Here’s why: Starting from an existing code base instead of starting from nothing is often the difference between “having time to innovate” and “not.” If you are using technology as a core part of your business and you aren’t set up to experiment then you’re doing it wrong and you will become obsolete.

Borrowing code is kind of like being airdropped into the middle of a marathon; sure, you have to take a moment to figure out where you are and what direction to go, but now you have time to run in circles laughing like a crazy person before winning the race.

There’s more! If you borrow code then you are more likely to be familiar with what the rest of the world is doing. If you share code then you are going to build your systems with an emphasis on reuse and extensibility (i.e. correctly). If you regularly borrow AND share code then you are building a community around whatever it is you do.

What I’m trying to say is that if newspapers can buy into the mantra of openness—even just internal openness—they can kill about thirty birds with one stone.

But they usually don’t.

Why not? Are they idiots?

There are many reasons these organizations don’t trade bytes, none of which have to do with the original “we would have to pay for it” claim.

Reason 1: Wildly Different Technology Stacks

I lied to you earlier when I said the Globe was like a child to the Times–they’re more like middle-aged lovers. They didn’t grow up together or meet in college. They are two independent entities that recognized their love later in life, which means they have fundamentally different infrastructures.

One uses Java and PHP, the other uses Python, Ruby, and NodeJS. They have incompatible content management systems. They disagree on deployment policies, quality control processes, needs, and third party libraries. It’s like they come from two stubborn families that speak completely different languages and eat very different foods. They aren’t going to start casually sharing cook books.

Reason 2: Internal Politics

If a full team dedicates three months to creating a new public-facing interactive, will they want to just give it away? If you are a manager do you want to rely on favors from an external team to accomplish your goals? If you are a coder do you want to be judged for the quick last minute hacks you had to throw into the project?

The answer to these questions, and many more like them, is “hell no.”

Reason 3: Moving Costs and Learning Curves

Most technologies are dirty piles of duct tape with a shiny chrome finish. This makes them difficult to deploy and hard to understand. This is especially true among newspapers.

Packaging code in a way that strangers can use could take hours, days, or weeks depending on how much the developers cared about portability when they built it. I’m basically describing the difference between moving a campsite and a home. Newsroom developers don’t tend to have camping on the brain when rushing to meet looming deadlines.

Words from On High

Fine, so there are real reasons that code sharing between the Globe and the Times is a lost cause, but what does that mean for the industry? If financial allies with serious resources don’t share code, what are the chances that other newsrooms around the world will look outside their walls for help? Maybe this is why so many open source journalism tools are gathering dust.

I talked to Rajiv Pant (CTO) and Marc Frons (CIO) of The New York Times about code sharing and the role of open source in their company. For context: the Times is very progressive compared to other newsrooms when it comes to innovation and openness. They have a blog dedicated to their open source inititatives, there is a suite of APIs that provide civic data, and they do a good job of telling people about what they do.

Unfortunately they are also leading an industry that is forced into “deadline driven technology” and without a supportive institutional strategy, open source and reusable code are just nice-to-haves. Developers must ask themselves if they have time to meet the organization’s needs while also contributing to open source. Sometimes this means the same tools get built multiple times, but such is the nature of deadlines. Plus, as Marc was quick to point out, reinventing the wheel can be a good thing so long as the new one is slightly different.

Source: The Wheel Superstore. (Illustration by Lyla Duey)

But wouldn’t it be nice if all these new wheels could be used again and improved upon over time? Rajiv identified three factors that a project needs in order to be realistically used again by an organization like the Times.

Your code has to be…

1. Established – Is it safe to rely on your creation? How long will your project stay active, and how long after you move on will it stay useful?
2. Extensible – Your solution won’t meet all needs. How easy is it to improve? What kinds of features can be added?
3. Easy to Integrate – Will this play with existing systems and tools? Can it be skinned to look like it belongs?

In short, it doesn’t matter how powerful you think your code is: if it is difficult or risky to adopt, it will stay an orphan.

None of those points should come as a surprise, but they should probably be considered gospel to anyone developing anything—open or closed—in any newsroom. Just ask yourself “would the Times use this if they needed it?” If the answer is yes then you’ve made something that will last; otherwise you might as well get out the broom now.

Note: Since I’m sure you are worried, the Times doesn’t actually charge the Globe for code. And yes, we are writing Quizzler from scratch.

By now you may have read four different accounts of my program, but if you make it through this one you will be rewarded with Internet gold. My backstory is fast. I started this process later than the others because I had to graduate first. I’m only two months in, which means I’m about 20% complete.

In that short amount of time I have already:

• Blown my tech stipend on the new hotness and become the envy of everyone around me.
• Moved to an entirely different state and bought a kitten.
• Toured Europe for three weeks with my wife (and it wasn’t considered vacation).
• Been monitored by the CIA at the Tor Hackathon in Florence.
• Broken the law in Berlin by infiltrating an old abandoned spy building under the leadership of Cole Gillespie.
• Watched In Bruges in Bruges.
• Raided The Guardian, The BBC, and Mozilla’s London HQ.
• Drank a Tactical Nuclear Penguin.
• Worked with the other fellows to dream up complete iterations on two home-brewed projects (Torwolf and Newsquest).
• Dreamed with my peers and started moving on dozens of ideas including Raspberry Pi crisis cams, time-lapse Internet radio, and magical data extraction APIs.
• Contributed to the infrastructure that supports The Boston Globe.
• Challenged and questioned policies and processes at The Globe.
• Learned first hand about the politics and hurdles within news organizations (I call this “the real world.”)
• Regularly met with VIPs at The Globe, The Guardian, The MIT Media Lab, and The New York Times, to absorb their insights about the state of journalism.
• Maintained strong ties with the people of my past by arranging a formal ongoing relationship with the Media Lab.
• Not been sued or fired yet.

You’ll notice that this list contains a mixture of productivity, fun, and life. This is because fellowships are not just paychecks, they are about personal growth as much as personal output. This one is no different, as shown by the fact that I’ve spent almost half of my time traveling around the world.

If you are saying to yourself “holy crap that’s sweet” you are absolutely correct. In fact, one point of this post is to help you understand why The Knight-Mozilla Fellowship is one of the most rewarding jobs on earth right now and that you should apply this week.

There are some things you should expect if you make it in.

1) You will understand why the news industry is struggling to survive, and why there is hope

I decided not to use this post to talk about my observations and insights about journalism. I already write about that subject on this blog. This item still gets top billing because it defines our mission. We are trying to publicly understand, question, observe, and create in the context of news. There are so many chances to do all four of those things. Not a day has gone by where I haven’t been exposed to something new — a new idea, a new problem, or a new opportunity.

2) You will become unemployable (in a good way)

This now makes two positions in a row where I’ve heard my colleagues say that our work has made us unemployable. This isn’t because employers won’t want to hire us, it’s that we are being spoiled. We’re getting so used to creative freedom, security, and special treatment that simply doesn’t usually come with a traditional job. It’s a good problem to have.

3) You will be challenged, and you will be special

You are being thrown into an organization that may have a vision for you to work with, or may expect you to invent a vision of your own from scratch. Either way your time is going to be your own and you will be expected to make great use of it. This kind of freedom is difficult to cope with, especially when people have high hopes for you. People will throw you questions to ponder, ideas to critique, and problems to solve and you will need to prove yourself.

In return you get to ask anyone anything. You will get to bend the rules and do things that other people around you might have to fight hard to accomplish. If you are interested in something, you will be able to work on it. If you have a question or concern you will be able to get an audience with the CTO or the chief editor. Nobody else at your organization has your title.

4) You will make friendships that last the rest of your life

It’s a cliche, but it’s for real: one of the most rewarding parts of this is the people. You won’t just be part of a community, you will be part of creating a community. It started last year during a week long hackathon in Berlin, where I met dozens of people who I still see all the time. That sense of comradery continues to dominate this experience, and of course it also includes the people in your newsroom.

For the next round of fellows there won’t be a Berlin event, but you will still get to be a part of an 8 person group for almost a year in addition to becoming immersed in a young community of passionate people. You will drink whiskey in foreign countries with friends, share stories of trials and tribulations in workplaces half a world away from you, and find yourself in areas you never would have entered alone.

And with that I think I promised you a reward… I present to you the Amazing Spinning Gridinoc!

ALSO: Please come talk to the folks in the OpenNews community by signing into on our chat room below. Just come in and say hello!

At least I’m not the weird new guy wearing dress shirts that don’t really fit any more; I’ve upgraded to button down T’s.

Update: Matt Stempeck has told me “it isn’t your fault, but nobody in the world can pull off the button down T.” This is why I usually let the Internet choose my wardrobe.

The Trilogy

Over the next ten months I’ll rotate between the three key technology groups at the Globe. I’ve given them battle-clad nicknames for the purposes of this post.

• Round 1: Front Lines. I’m starting on the product team, working under Mike. This is where people work on longer term product-level projects. This includes things like the two web sites (BostonGlobe.com and Boston.com), the Content Management System (what people actually use to write articles), mobile experiences, and comment systems.
• Round 2: Special Ops. My second rotation will be for the Interactive News team (NewsDev for short), working under Miranda Mulligan. They work on more targeted technology projects such as reporter tools or interactive interfaces.
• Round 3: R&D. I’ll end my run working in the Boston Globe’s Media Lab / R&D department under Chris Marstall. This is where the more “cutting edge” / “insane” / “impractical” / “highly creative” work gets done. I really like the idea of this being my capstone rotation since it feels the most in line with my darker past.

Each of these groups has a different set of responsibilities, constraints, and skills. It seems like the Chris’s group (R&D) has the most freedom but isn’t as likely to ship full blown features and systems that get used by millions. Mike’s team (Front Lines), on the other hand, pushes out code that will support the newsroom and the world. As a result they need to keep their eyes on reality — they can’t take as many risks.

Miranda’s (Special Ops) gets to be somewhere in the middle simply because of the nature of user-facing interactivity and the fact that many of their projects are “reusable one-offs.” For instance, if an interactive feature didn’t scale it would be very unfortunate, but at least the entire organization wouldn’t collapse in a fiery heap of paper. The same could not be said for the code that renders the front page of Boston.com.

Free Developer… like Free Beer?

I was jovially introduced to most people as the free developer (“You’re not free any more! Mwahahah”), although I’m hoping to do more than just hang out and crank out code. I’m coming in with some ideas in mind and a general mission, but narrowing that down into something actionable became a lot easier after seeing my fellow fellows this weekend at the Knight Civic Conference.

• Understand pain points, identify opportunities, and communicate. I heard battle stories from the other fellows and immediately understood that one purpose of the OpenNews experiment is to unearth those stories and put them into the open for discussions. The entire industry needs to know examples of what works and what doesn’t, even if there are a few worms under the rocks.
• Stay creative, pursue wild and crazy projects, and share that creative spirit with everyone around me. I’ve just come from one of the most forward thinking places in the world, but the Lab isn’t defined by skill sets and brains. What makes the lab great is the attitude, approach, and the ability to fail effectively. There is no reason why some of that spirt can’t be transported to the globe (in fact some of it is already here.)
• Stay grounded and write awesome code. At the end of the day I am also a developer! I want to learn some new technologies and create some great projects. The key for me here is that I want to make stuff that will have applications both inside and outside of The Boston Globe.

And so this is my hopeful breakdown of the next 10 months:

1. 30% on immediate needs of the Boston Globe, like new platforms and systems.
2. 30% on my own fleshed out ideas, like Truth Goggles, or The Meta Meta Project.
3. 30% on wacky ideas like Newsquest or ATTN-SPAN.
4. 10% on reflection, in blog posts like this.

I see a parallel between the three groups I’ll be working in and those three major chunks, but I’ll be aiming to do it all, all the time. If you were at last weekend’s OpenNews hackathon at MIT you know that I’m feeling multithreaded.

It takes three screens and two laptops to be multi threaded.

I started the process by meeting with Mike Krolak. He is the man responsible for the technology related to product at The Boston Globe. This means web sites, but also things like printing presses, delivery trucks, and employee management systems. I’ll be starting my fellowship in one of his teams, which is the reason why I ended my first day with an ID badge, credentials, and access to most systems. Impressive, most impressive.

When we talked over lunch he was clearly curious to learn more about my purpose and personal goals, at which point I said “you and me both, bub” and stared awkwardly past his head. What I actually did was share my high level understanding of the Fellowship: it’s an experiment, but the goals are to help technologists learn what it is like to work in the journalism industry and to infiltrate and destroy newsrooms get newsrooms more comfortable with external collaboration and open development. Really we’re figuring out the details as we go along.

I got to learn a little more about Mike, what motivates him, and why he has spent more than a decade working for The Boston Globe. Here are three of my favorite points:

• There are so many simple, unsolved problems. Mike has a math background (I’m wary to tell people with actual math backgrounds that I was a math minor, but I was immediately able to relate), and one of the things that motivate math geeks is unsolved problems. In fact, the quest for the perfect proof can cause them to actually go insane! Mike sees the news industry as a great way to go mad, with tons of challenges ripe for the picking. A prime example: “did you know that there isn’t one major newspaper in the world that can tell you how many papers they distributed today?”
• Technology done wrong can get in the way of good journalism. I feel silly admitting this, but the idea that technology could actually hinder the way a newsroom produces content had never crossed my mind. For example many papers have publication processes that force articles through up to 50 different systems. Each transition leaves room for accidental content modification which results in more edits. What you get by the end is a professional version of whisper down the lane (or “telephone” for you non-Pennsylvanians).
• Newspapers are getting more comfortable with innovation. As you probably know already, when the Internet got popular the News industry sat around fat and happy and tried to use it in the same way they used any medium: to publish their content. In the past 10 years we have seen the advent of R&D departments and research spaces in newsrooms. There is a lot of catchup to do, but at least they’re trying.

For me these points frame the essence of the OpenNews initiative. Now that newspapers are innovating we want them to take a page from the mathematicians and make sure unsolved problems get solved in a way that everyone can learn from, expand upon, and contribute to (e.g. by publishing them to the world). As fellows we are also trying to understand what technology means for journalism and to share our lessons about where it can help and where it can hurt.