It is 1999, you’re 13, and your mother just walked through the front door carrying a large, vibrantly colored blue and yellow bag. Based on the heft and the "Best Buy" logo on the side, you know it holds something interesting—something related to electronics—but what? A Nintendo 64? A Dell computer? 1,000 free hours of AOL?

There is only one way to find out.

Depending on your maturity and cowhand status you walk, mosey, or scamper up to take the bag. In a single, excited motion you reveal a nondescript cardboard box. It has no clear branding, just three letters: "DVR". On the opposite face, an illustrated TV depicting endless suited silhouettes.

"This is going to change everything." She says calmly, looking you in the eyes. You nod, and set it up.

A few moments later you are both sitting on the floor staring at… Al Gore. He is talking about a lock box. The screen fades, and George Bush appears. This continues for hours. Political ad after political ad. No interruptions. For days. For years.

In fact, you are still watching now.

Presenting the Political Ad Archive

Oh ***.

Witness the Internet Archive's Political Ad Archive. Our mission is to provide a free and open resource for citizens, journalists, and researchers who want to understand the paid messages from their politicians, and to archive billions of dollars worth of democracy.

We record and track political ads. Through our service you can find out when, where, and how often a given ad was played across the channels we are recording. It also tells you who the ads are about, who paid for them, what is said in them, whether they have been fact checked, and plenty of other odds and ends.

This post is about how the service works, but I'll start with the punch line: we watch tons of TV—probably literally, our servers are heavy—and filter out the noise, leaving only the political ads. Then our DVR robots (DVRRs for short) activate and count all copies of those ads, keep track of when and where they were played, toss in a little human contributed metadata, and share the DVRR results (DVRRRs) and code base with you, our DVRRR recipients.

Three key pieces

There are three pieces to the Political Ad Archive:

1. The Internet Archive collects, prepares, and serves the TV content as it comes. It’s trying to archive the entire internet too, so their infrastructure is set up to be able to store, you know, all of human knowledge.
2. The Duplitron 5000 is an open source system responsible for taking video, smooshing it all into smaller, searchable files called audio fingerprints, and then finding copies of known ads. It reports the results back to the archive.
3. The Political Ad Archive is a wordpress site that takes our data and our videos and presents it to the rest of the world.

Look, here’s a fun flow chart of the entire process:

Step 1: Recording Television

All of our artisanal grass-fed TV has been locally sourced from super-premium, organic hardware distributed around the country. We do process it though. A lot. Sorry about that.

The ad counts we publish are based on actual airings, as opposed to reported airings. Because we are working from the source, we know we aren’t being misled by anything but our own algorithms. On the flip side this means that we can only report counts for the channels we actively record.

We have a few ways to collect TV content. In some cases, like the San Francisco market, we own and manage the hardware that records local cable. In other cases, like New Hampshire and Philadelphia, the content is provided to us by third party services or academic partners.

Regardless of how we get the data, the pipeline takes it to the same place. We record in minute long chunks of video and stitch them together into programs based on what we know about the station’s schedule. This results in video segments of anywhere from 30 minutes to 12 hours. Those programs are then put into a high pressure cooker and turned into all sorts of file formats for archival purposes (mp3, mp4, MBA, PST, Apollo 13, banana).

A lot can go wrong here. Storms can affect satellite reception, packets can be lost or corrupted before they reach our servers (resulting in time shifts or missing content), small children can disappear in our server rooms. It all happens, but most of the time the data winds up sitting comfortably on our hard drives unscathed.

Step 2: Searching Television…

Illustration by Lyla Duey.

Remember that time you were watching Netflix and you blacked out because your cat sucker-punched you? Wasn’t it a huge pain the next day when you had to try to figure out where you had stopped watching? You kept clicking, waiting for it to buffer, being too early, then too late, then closer but too early, then somehow back to the first try, until you finally gave up and just started from the beginning again?

This is a great example of how terrible video is when you’re trying to look for a specific piece of it. It’s slow, it’s heavy, it is far better suited for watching than for working with.

What if you had no choice, and you really did need to search video for something. Worse, maybe you have to search millions of minutes of video for an arbitrary number of somethings. Welcome to my world.

There are a few things to try. One is transcription; if you have a transcript you can do anything. Like create a text editor for video, or search for key phrases, like “I approve this message.”

The problem is that most television is not transcribed. Closed Captions exist sometimes, but there is a shocking amount of content—especially political ads—without captions. There are a few open source tools out there for automated transcript generation, but the results usually “love match Tobey desert ire” (… leave much to be desired).

So what do we do? Our nation’s future is at stake and we don’t have time to be able to do it all manually. Say hello to audio fingerprinting.

… Using Audio Fingerprinting …

We use a free and open tool called audfprint to convert our audio files into audio fingerprints. An audio fingerprint of a file is just what it sounds like. Get it? AUDIO fingerprint… SOUNDS like? Ha!

An audio fingerprint is a summarized version of an audio file, one that has removed everything except the most interesting pieces of every few milliseconds. The trick is that the summaries are formed in a way that makes it easy to compare them, and because they are summaries they’re a lot smaller and faster to work with than the original.

“Summary” is a pretty vague term. There are lots of ways you can summarize a piece of audio. For instance, I could summarize a song in terms of its chord progression (G major -> C minor -> D major …). If I heard the same song twice it would have the same chord progressions both times, so I could flag it as a match and be correct.

But what if two different bands played the same song? Or what if you compared two pop songs? Those would also have the same chord progressions even though they are obviously different audio files. Also what about spoken word? Or long, loud, sensual recordings of fog horns. No chords in either case. This clearly isn’t going to work.

… Based on Frequency

The audio fingerprints we use are based on a thing called frequency. Sounds are made up of waves, and each wave repeats (oscillates) at different rates. Faster repetitions are linked to higher sounds, lower repetitions are lower sounds.

Don’t believe me? Go drop a small pebble in a lake and you will see a bunch of quickly repeating tiny ripples. Next drop a boulder and you will see a few larger ripples. There are also sounds generated in both cases. The boulder creates a loud and deep “KERPLUNK” — the ripples have a lot of space between them less often, which is true of the ripples through the air as well. The pebble has a lot more ripples closer together which results in a higher pitched “pleep!” How cute!

That number of waves you see can be measured in terms of frequency—as in how frequently does the wave repeat per second. Most sounds you hear are a crazy combination of thousands of waves of different frequencies. Each of the waves get turned into vibrations in your ear that travel down a magical cone covered in tiny hairs which then turns into electric signals to your brain which you then hear in your head as sound. For instance, most people hear an “A” when a wave that repeats 440 times per second (440 Hz) hits our ear drum.

Try waving your hand 440 times per second. If you do, you will hear that “A.”

Computers don’t have ears, so they just take these frequencies at face value. An audio file contains instructions that tell a computer how far to push the inside of a speaker in or out (generating a wave). Audfprint breaks those audio files into tiny chunks (around 20 chunks per second) and runs a mathematical function on each fragment to identify the most prominent waves and their corresponding frequencies.

The rest is thrown out, the summaries are stored, and the result is an audio fingerprint.

If the same sound exists across two files (not the same song, or the same words, or same voice, but literally the exact same set of frequencies), a common set of dominant frequencies will be seen in both fingerprints. Audfprint makes it possible to compare the chunks between two sound files, count how many they have in common, and how many appear in roughly the same distance from one another.

This is what we use to find copies of political ads.

Step 3: Cataloguing Political Ads

When we discover a new political ad the first thing we do is register it on the Internet Archive, kicking off the ingestion process. The person who found it types in some basic information such as who the ad mentions, who paid for it, and what topics are discussed. This is all called metadata.

The ad is then sent to the system we built to manage our fingerprinting workflow, called the Duplitron 5000—known locally as “DT5k”. This uses audfprint to generate fingerprints, organizes how the fingerprints are stored, process the comparison results, and allows us to scale to process across millions of minutes of television.

An artistic rendition of the Duplitron 5000 (by Lyla Duey).

DT5k generates a fingerprint for our new ad, stores it in the list of known political ads, and then compares that fingerprint with hundreds of thousands of existing fingerprints for the shows we have already ingested into the system. It takes a few hours for all of the results to come in. When they do, the Duplitron makes sense of the numbers and tells the archive which programs contain copies of the ad and what time the ad played.

All of these steps end up being pretty darn accurate, but not perfect. The matches are based on audio, not video, which means we face trouble when the same soundtrack is used in a political ad as has been used in, for instance, an infomercial.

We are working on improving the system to filter out these kinds of mistakes, but even with no changes these fingerprints have given us reasonable accuracy across the markets we track.

Step 4: Enjoying the Results

And so you understand the fundamentals behind the amazing futuristic technology that we used to build a system that records only political ads. You can download our data, and watch the ads, all day every day at the Political Ad Archive.

Over the coming months we are working to make the system more accurate, and exploring ways to get it so that it can automagically identify newly released political ads without any need for manual entry.

P.S. we’re also working to make it as easy as possible for random strangers to download all of our fingerprints to use in their own local copies of the Duplitron 5000. Would you like to be a random stranger? If so, contact me on Twitter at @slifty.

You may want to skip this boring post and just check out the site.

Over the past 24 hours I worked with an amazing team to start building a Siri for government. Well, Wolfram Alpha is more like it, but you probably have a better sense of what Siri is. The site is called CivOmega and it allows you to ask any question you want about civics. The system will do its best to get you an answer.

I can’t speak for the team, but I’ll let you know why I proposed this idea at a hackathon about open data. I’ll even use big letters:

Open Data Sucks

People have talked about making government data more accessible for approximately 500 years. The hope is that if you can find data about the way your government operates, you can shed light on interesting patterns and stories. It’s all about transparency and accountability. It’s a beautiful concept. It’s wonderful for society.

But actually data is pretty crappy. It’s dirty and boring: just a bunch of numbers and rows and tables. This kind of stuff doesn’t usually tell you much without a lot of very laborious prodding and exploration. Don’t believe me? Fine. Go find out for yourself. If you managed to get anything interesting out of that link then you have too much time on your hands.

The ONLY thing that civic data has going for it is that programmers tend to build cool hacks using it. I guess every once in a while you get a groundbreaking piece of journalism out of it too but I’ll ignore that for the sake of argument.

It’s Also Elitist

Here’s another problem: programmers have awesome, special tools to access data. These tools are called “Application Programming Interfaces” (also known as an APIs). An API is just a standard way for computers to ask each other for information.

A human version of this plays out every time you go to a restaurant and order from a menu. You look at the list of what you can ask for, you ask for what you want, and eventually you either get your food or you get impatient and start throwing your silverware at other patrons.

In my analogy the food is data and you and the chef are computers. The waiter is the API and the menu is the documentation. I guess the restaurant is the Internet and the restaurant’s manager is the NSA or something. The silverware don’t really fit in.

The point is that the COOL stuff happens because of these APIs. Too bad nobody real knows what the hell an API is or how they could possibly go about using it. Don’t believe me? Go find out for yourself. If the stuff on that page gave you access to data then you’re a nerd.

If nerds and people who have too much time on their hands are the only ones who can use government data then it won’t change the world. Plus, why should those people get to decide what is and isn’t important?

Humanizing Government Data

And so we come back to CivOmega. This is an attempt to give people with normal, human questions the ability to benefit from the data that so many have worked their asses off to expose. It makes it possible for a human to interact with an API in the same way they might interact with their waiter: by asking questions. Users can type in questions about the government and it attempts to provide answers.

It is built on a programming language called Python and the way it works is pretty simple. A programmer who understands an API can write some code that knows how to answer certain question patterns. For instance I made it possible to ask the question “What bills are about [X]” where X can be any phrase you want. If you ask that, CivOmega will talk to the appropriate APIs to get you the answer you want. Then it will tell you what it learned.

The beauty of this setup is that any other programmer can spend a few minutes teaching the system to answer new kinds of questions. For instance maybe someone knows about an environmental dataset and wants you to be able to ask questions about natural disasters (how many forest fires happened in California last year?). That person could easily unlock that resource.

If you’re a developer, go take a look at the repository and consider adding a module. If you are a master of NLP please get in touch with me so we can improve the way people ask questions. If you don’t know what either of those sentences meant, please just go check out the site.

At face value it just looks like a live feed of C-SPAN’s Closed Captions. This alone is actually pretty cool if you think about it, especially if you are a deaf political junkie who sits far away from the TV and can’t read the closed captions.

Of course there is more. The real excitement comes when you contemplate what’s happening to get those words to appear on your screen.

This system unlocks and syndicates a real-time dataset that used to be a pain in the ass to access. Now anyone can build applications and visualizations that update before those crafty politicians have even finished making their points. This post explains why Opened Captions is worth hacking with, what it takes to use it, and how it works.

What is it Good For?

The Internet is filled with real-time updates triggered by online activity, but it still feels like magic when we see automatic updates driven by the real world. Opened Captions makes it easy for programmers to use live TV transcripts as an input.

Note: version .001 only supports a single channel (and my server is pointed to C-SPAN). Eventually the protocol should expand to allow multiple channels.

Let’s consider C-SPAN. If a computer knows what is being said on C-SPAN this very second, it can do things like:

• Change the background of your email client to reflect the issues being debated right this moment on the senate floor.
• Generate modified, more amusing, transcripts by replacing key words and phrases with Tolkien lore (i.e. C-SPAN for Middle Earth)
• Search through lyrics and generate a C-SPAN medley for you to rock out to while voting.
• Send SMS messages 24/7 commanding you to “drink” when certain phrases are spoken on air.

There are also possibilities that aren’t ridiculous. For instance, you could make tools that…

• Improve the transcript by automatically adding contextual information, such as definitions and histories thefted from Wikipedia.
• Send emails with transcript snippets whenever a specific representative or state is being discussed on TV so you know what’s going on.
• Parse out paraphrases of known fact checks and insert a credibility layer over the transcript feed (real time fact-checking).
• Draw parallels between what is being said on TV and what is being said on Twitter.

I could go on and on and on. There is just so much potential!

The Backend

Behind the stream is a first stab at a distributed architecture for Closed Captioning live-feeds. Opened Captions servers can pull a CC stream over a serial port, or (more likely) they will connect to an existing Opened Captions server and pull the stream from there. What that means in de-jargon is that anybody can set up a server that does exactly what mine is doing, even if they don’t have access to hardware, software, or a live TV stream.

When I say exactly, I mean it — your new project runs the same code as mine, and will serve the feed too. People can connect their servers to yours in the same way you connected yours to mine. Practically speaking this architecture means a few things:

1. Once your amazing mashup gets popular it won’t break my server. Your application is syndicating the captions to your users. I serve the captions to you, you serve them to the world!
2. Your server creates a fork of my stream. Want to modify the text so the politicians sound drunk? Add extra layers of information to the message payload? Translate the captions to Klingon? Go for it. If your tweaks happen server side then others can build their apps from your stream to modify it further.
3. You don’t have to rely on anyone else for the Closed Captions. If you want to spend some extra time setting up your own scraper you can point your server to that source instead of a third party. You have total control.

Check ‘Em

Wondering if this is worth your time? Well, it doesn’t require much of it. The service takes about two minutes to set it up if you already have Node.js and Git installed on your computer. Here’s a video to prove it:

Installation instructions can be found in the readme and you can always get in contact with me through the blog or on twitter.

There are many reasons why he might have said this, but two stand out. For one thing I had just given a presentation about NewsJack, a media manipulation platform that I created from Mozilla’s Hackasaurus with Sasha Costanza Chock. When NewsJack was released it was immediately met with a Cease and Desist from the New York Times (note that The Times is the parent company of The Boston Globe).

It is also possible that he was inspired because I had just confessed on stage that one of my first thoughts when walking into The Globe’s headquarters was “I wonder what it would take to bring down this organization.” I’m betting it was the juxtaposition.

The Backstory

An evil newspaper editor?

Assuming you’re a competent villain, whatever you end up targeting should be important. Not only that, but the target will reflect your personal passions and expertise. Try the mental exercise yourself and share the results. I dare you.

For example, to take down a newspaper you could…

• Open up their paywall (if it exists), steal their content, and make it freely visible to the world without giving them any form of recognition or compensation.
• Eliminate their productivity, either by instigating a massive strike or by hiring away all of their employees.
• Scare away their advertisers so they lose a significant revenue stream and can no longer pay their bills.
• Destroy their infrastructure (printing presses, websites, etc), thus disabling their ability to ship product.
• Corrupt their editors and slowly replace key actors with your henchmen so that the paper becomes your mouthpiece.
• Buy sharks with laser beams attached to their heads.

A super villain’s master plan needs to be intricate enough to be interesting and difficult enough to be impressive. Blunt ideas like “take down their website” or “steal all their money” are a bit too obvious. It must also be simple enough for a diverse audience to understand. If nobody can figure out what you did, why it was sinister, or how it actually worked then it is hardly going to make headlines. Finally, it can’t be a series of bee stings; the evil needs to be condensed enough that it could fit in a tweet.

The Plan

My evil plan didn’t take long to imagine (given my recent work). If I were evil and wanted to destroy a newspaper I would ruin their brand’s credibility. This could be accomplished in many interesting and convoluted ways, but the “how” isn’t the point, the important question is “why?”

A media product will die miserable and alone unless it differentiates itself from the rest of the Internet. Luckily, newspapers have something that the chaff doesn’t: they have the capacity to create trustworthy information experiences. They are the ones with paid reporters asking the hard hitting questions, they have the editors and the internal fact-checkers, they don’t have an agenda and aren’t trying to manipulate me! right?

You could tie yourself to a bungee cord, close your eyes, and jump off a cliff… or you could read the New York Times.*

Well, maybe. As a reader I don’t know where content comes from or how much journalism went into it. All I have is faith in their brand. I trust that the sources I read are doing their jobs. That faith didn’t come from nowhere. I might have liked what they had to say in the past, or I saw my parents reading their paper, or their brand has a strong reputation. Regardless, I am now far more likely to trust what they have to say than I am to trust, for example, what my crazy friends like to read.

Just to drive this home: given the way content is presented today I could read the exact same article on the front page of the New York Times, Fox News, or the Huffington Post and my decision to trust it would be more strongly influenced by my opinions of the publisher than by the content itself.

To drive it home a different way: hijacking a newspaper’s credibility is as simple as imitating their brand.

Save the Day

The wheels are turning and it is already out of my control! IP lawyers are powerless compared to the forces of the anonymous web! But seriously, brand is a really fragile way to differentiate on the Internet. So what’s a newspaper to do?

Take a page from Apple and redefine the way people consume content. Train your readers to expect a certain experience not just from your website, but from every source of news. Make sure that experience is either expensive or impossible for alternative sources to replicate. Newspapers need to make their readers expect proof of everything. People should feel uncomfortable trusting information without explicit, functional credibility.

Newspapers have journalists doing research, checking facts, and taking names. They have multiple people and multiple systems touching every piece of content before it gets published, so why does the product usually end up being a bunch of words with prose-based evidence?

News organizations need to make the world hold information to their standards.

Like I said earlier, it makes sense that this particular plot and solution are coming from me. I dedicated my thesis to credibility layers — interfaces that lead to credible information experiences based on more than faith and trust. There are many paths to differentiation. Some are evil, some are entertaining, and some could even change the world.

* Drawing courtesy of Lyla Duey

Now that the study has officially concluded, the numbers have been crunched, and the thesis has been submitted, I want to share what I learned from the resulting data and feedback.

I’ll warn you upfront: All conclusions drawn here should be taken with a grain of salt. The participants were not a random sample of the Internet, and as such, the results don’t reflect the general population. I think they are quite exciting nevertheless!

The Questions

There are many ways that a tool like Truth Goggles could be considered successful. A bare minimum is that users should prefer it to the non-augmented consumption experience (you know, the kind you have normally). Another measure of success might reflect the number of claims that users explored when the tool was enabled, or maybe the quality of that exploration.

These questions are interesting, but they all require different study designs. Here is what I considered when putting the study together.

1. Did people use Truth Goggles? It is difficult to accurately measure the use of a tool when working with a “captive” audience (i.e., study participants). Truth Goggles does not yet contain enough facts to be regularly useful in the real world, so the study had to simulate a reading experience and present articles with known fact-checked claims, so this question wasn’t explored too deeply.
2. Did people enjoy using Truth Goggles? This study was run online, so the only way to get direct feedback was by asking users directly. I also gave participants a chance to choose to enable or disable Truth Goggles for the final few articles after the tool has been completely exposed to them. Presumably, if they hated the interface they would have disabled it.
3. Were users exposed to more fact checks? In order to compare a change we must have a baseline and the ability to measure exposure. This study wasn’t quite comprehensive enough to address this directly, although I did keep track of how often users chose to view “More” information about a fact check (which took them directly to PolitiFact’s site).
4. Did users engage with the fact checks? To understand levels of engagement, the tool would need to keep track of what content was actually read and comprehended, as opposed to what content was simply rendered on a screen. Once again, tracking the use of the “More” button was a good indication of engagement.
5. How well did Truth Goggles enable critical thinking? Although critical thinking doesn’t require a change of opinion, it seems reasonable to believe that a change of opinion does indicate thought. By measuring the drift in beliefs about fact-checked claims after using Truth Goggles, it was possible to better understand the interface’s ability to facilitate updated beliefs.
6. Did Truth Goggles affect levels of trust in consumption experiences? This question is deeply relevant, but given the format of this study I did not attempt to measure trust in a robust way. I did give users an opportunity to comment on how they felt Truth Goggles affected their trust.

The final study design reflected aspects of each of these questions; however, “Did people enjoy using Truth Goggles,” “Did users engage with fact checks,” and “How well did Truth Goggles enable critical thinking” ended up getting the most focus.

The Preparation

Before the study began I selected, tagged, and pre-processed 10 political articles to create a pool of content that I knew would have fact-checked claims in them. For the most part, this involved going through PolitiFact, Googling the phrases, and hoping that some good articles would show up. Most of what I used was published in 2012 and came from a variety of sources with varying degrees of credibility.

I also added some tracking features to Truth Goggles in order to better understand what was clicked and explored. This meant I would know when users viewed a fact check or when they would interact with other parts of the interface. Finally, I had to create the actual study website, which added some randomization and guided participants through the process.

The Participants

The study was conducted online over the course of five days. Participants were recruited through email and Twitter. When I did the initial number crunching there were a total of 219 participants, 88 of whom completed the entire process. These numbers increased to 478 and 227, respectively, before the study officially concluded. This analysis reflects my thesis work, and only considers results from the original 88 participants who completed the entire study.

Unfortunately, the participant pool contained a disproportionate number of friends, individuals familiar with the concept of Truth Goggles, and professionals already aware of the challenges surrounding media literacy. The vast majority (about 90%) of those who actually completed the process were strong and moderate liberals. All of these biases were anticipated, but nevertheless they significantly limit the potential impact of the study.

The Process

From start to finish, the study took each participant around 20 or 30 minutes. After being shown the initial instructions, people were asked to rate 12 claims on a truth scale from 1 to 5. They only had 10 seconds per claim to answer, so this was really trying to get at a person’s gut reaction based on the information sitting in his or her head.

After the survey was completed, the treatments began. Everyone was shown a series of 10 articles which contained the previously rated claims. The first two articles were always shown with Truth Goggles disabled. The next six were presented with different Truth Goggles interfaces to help call out fact-checked phrases. For the final two articles, participants were able to choose one of the four interfaces (including “None”).

Once the article reading ended, participants were asked to re-rate the claims from the beginning of the study. At this point, they had been exposed to explanations and context for most of them, so this time they were supposedly providing “informed” answers as opposed to gut feelings. After the second round of ratings, the study wrapped up with a short exit survey, where participants had a chance to yell at me in the comments and tell me what they thought about the experience.

The Irony

Before going any further, I want to be clear that Truth Goggles does not assume that fact-checking services are correct. To the contrary, the hope is that users will question fact checks just as much as they would question any source, and consider all evidence with scrutiny. This philosophy is problematic for evaluation, because it is difficult to measure belief accuracy without considering something to be “true.”

Lacking a better metric, the source verdicts (i.e., PolitiFact’s ratings) were used as grounding for accuracy for this analysis. This means that from an evaluation perspective, I considered interfaces to be more effective if users ended up with beliefs in line with PolitiFact’s verdicts. Since belief dissemination is not the goal of Truth Goggles, the system must eventually use more sources (e.g., Factcheck.org and Snopes) to keep users on their toes.

The Results

In my thesis, I slice and dice the study data in more ways than I care to think about. But this isn’t my thesis, so I’m going to spare everyone a lot of pain and stick to the high-level observations.

• Truth Goggles increased accuracy and decreased polarization. Participants changed their beliefs about the fact-checked claims after reading the articles, regardless of whether or not a credibility layer was rendered. But without Truth Goggles those updates resulted in more polarization and less accuracy. In particular, when Truth Goggles was disabled people tended to become overly trusting of claims that appeared in articles. With Truth Goggles active, however, beliefs became nuanced and more accurate.
• When using credibility layers, people became less incorrectly skeptical but they remained just as incorrectly trusting. Truth Goggles was able to help skeptics become more trusting when trust was appropriate, but was not as effective at convincing false believers that they should become more doubtful. This means that participants who were not already overly trusting of a claim would tend to update their beliefs in a way that resulted in more accuracy when using a credibility layer. If you incorrectly believed a claim, however, you weren’t likely to correct yourself.
• Normal reading caused people to become more incorrectly trusting but they remained just as incorrectly skeptical. Without a credibility layer, participants who were not already overly distrusting of a claim would tend to overly trust that claim after reading its related article. This means that if someone was highly skeptical of a claim before reading the article, they wouldn’t change their minds. But if they were more neutral or already trusting, then seeing the claim in an article would cause them to believe it more strongly.
• Almost everyone enabled Truth Goggles when given a choice. Only two out of the 88 participants who completed the study chose to view their final articles without using some variation of Truth Goggles. The vast majority of participants (70%) selected “highlight mode,” the least obtrusive of the three possible interfaces. These numbers unfortunately don’t mean much because it is entirely possible that participants simply wanted to play with the tool. They could be far worse, though.
• There were virtually no significant differences between the three interface types. It was no surprise that “Highlight Mode” was the most popular, since it did nothing but highlight text and didn’t bully people into clicking things. Less anticipated was the fact that “Safe Mode” and “Goggles Mode,” which force exploration, did not outperform Highlight Mode. I suspect that this was a study artifact — forced interaction was unnecessary during the study because the novelty of Truth Goggles meant people might be curious enough to click regardless of the interface — but it was interesting nonetheless.

The short version of these results is that Truth Goggles helped combat misinformation, but there is still plenty of room for improvement. There also clearly needs to be a more comprehensive, longer-term user study.

For me, the big surprise was that that people were so prone to trusting content just because it appeared in an article or opinion piece. I was absolutely thrilled to see that effect get completely squelched through credibility layers. The results from the exit survey are also incredibly exciting, but that is a post for another day.

On Friday I officially handed in my thesis, titled “Truth Goggles: Automatic Incorporation of Context and Primary Source for a Critical Media Experience.” For those who don’t know already, it was about an automated bullshit detector for the Internet / an interface to help people think carefully called Truth Goggles. The final version weighed in at a nice round 145 pages.

I’ll let the dust settle before putting this monstrosity online. I also want to write some more condensed posts about the interesting parts because I know nobody is ever going to read the damn thing. Those will come later. For now I give you a few bullet points.

The Gist

Here’s the basic story of the document:

• I learned about the millions and millions of reasons why my idea could never work.
• Not having a strong sense of self preservation I kept on going anyway and tried to create “Truth Goggles!”
• I worked really hard to design and implement an interface that people could value even if they didn’t trust the sources behind the tool.
• I ran a user study and learned that the interfaces worked pretty well when it came to protecting people from misinformation, and that almost everyone who took the study really wants to be able to trust information again.

The Gems

I’ll give a quick preview of some lessons learned. Each of these points deserves a post of its own but since this isn’t my thesis I’m going to just put out my own observations and thoughts. The posts later will probably be more “scientific” and “explanatory” (i.e. “boring” and “less quotable”).

• When people consume information they are struggling hard to maintain their identity. That’s all there is to it. There is plenty of evidence that people consume information with ideological motivations. Those motivations often cause them to accept or reject information based on how well it aligns with what they already believe. I have a theory that if you could just remind someone that there’s nothing to fear — that you aren’t trying to change who they are — you will suddenly be able to actually communicate with them.
• Trying to tell people what to think is a losing battle. When the first round of press for Truth Goggles came out back in 2011 I paid attention to every single comment on every single report about the idea I could find. Lots of people liked it, but a lot of people were instantly dismissive due to concerns about bias. I heard their point, agreed with it, and realized what journalists saw ages ago: there is no way to create a universally respected system that also tells people what to think. I changed course and settled for a system that would remind people when to think instead. I think that is a better mission anyway.
• Credibility breeds respect, and respect breeds open minds. Several participants in the Truth Goggles user study commented that having a credibility layer made them more willing to consider perspectives and messages that they might have normally ignored completely. Think about that for a second. It makes sense, right? It is much easier to respect what a person is saying if you can trust them. Usually “respect” and “trust” are like “chicken” and “egg”, but if you’re using something like Truth Goggles it is possible to develop trust and let the respect follow if it ends up being deserved.

This entire experience has given me a lot of hope about information online and the people who consume it. I’ve said before that credibility was the future of journalism and I’m half tempted to expand that statement to say that credibility could save the world. I’ll probably need to run a few more tests though.

As for the next steps for Truth Goggles, that is to be determined! I’m going to at least keep exploring some of the processes and technologies behind phrase detection, but once I graduate and start my fellowship at the Boston Globe in June I’ll need an explicit way to keep it alive. Stay tuned.

It turns out only part of that quote is accurate, but you’ll never know which one for sure! This is why, before I can graduate from MIT, I have to create an automated bullshit detector. The basic premise is that we, as readers, are inherently lazy. It isn’t just that we’ll believe almost anything — remember that time in 1938 when we believed aliens were invading the planet just because someone on the radio said so? Yeah. That happened. The real problem is that we’ll often believe what we want to believe (or disbelieve what we don’t want to believe).

It’s hard to blame us. Just look at the amount of information flying around every which way. Who has time to think carefully about everything? Not me, that’s who’nt. This is why I’m working on a tool called Truth Goggles that will help hone our critical abilities; one that will help us identify pieces of information that are worth inspecting a little bit more closely before deciding how it fits into our world views.

Thesis Goggles

When I wrote “before I can graduate from MIT” earlier in this post I wasn’t lying; I have decided to pursue Truth Goggles for my thesis. I’m definitely not the first person to explore this problem space but there is a lot of room to contribute. New technology has opened up new possibilities, needs have become clearer, and there is a wide variety of possible solutions and unanswered questions just sitting around waiting to be explored.

In November I presented the idea to the Media Lab community using the following slides:

The feedback I got was mixed, but what can you expect from a day called “Crit Day” which is short for “Critically Injure Pride, Hopes, and Dreams of Graduating Day.” Here were the main questions asked:

This doesn’t seem like it will scale considering Politifact only has a few thousand fact checked claims. Why aren’t you using the crowd to fact check?

My time at MIT will be spent focusing on the interface and user interaction rather than the generation and aggregation of source information. There are enough difficult questions surrounding the interaction layer. I don’t think it is worth complicating things further by trying to create a crowd-based journalism platform (which is essentially what crowd sourced fact checking amounts to).

Isn’t this just a mashup of technologies and data sets? How is what you are doing novel?

It’s true that I’m not inventing new algorithms. I’m applying existing algorithms in novel ways. Credibility layers aren’t robust right now, and they come with their own sets of interesting questions in terms of user experience and system design. My contribution will be to frame those questions, answer some of them, create a prototype, and test that prototype. This won’t be as trivial as just throwing more information on a screen and calling it a day, the interface has to be designed with care.

Do you expect to incorporate primary source data?

My initial prototype probably won’t pull from sources other than Politifact and other fact checking services, but I will definitely be thinking about ways to use other sources of data. Primary source content will eventually help with information scalability since raw footage and raw data could help computers find potentially dubious claims (and help readers make determinations about those claims).

Bullshit, This is Clearly Science Fiction

There are a lot of hard questions lurking behind corners here. In fact, most of them aren’t even trying to hide; they’re just sitting obnoxiously in the middle of the room. Some are technical, some are philosophical, but all of them need to be addressed intelligently for something like Truth Goggles to actually have a chance of working. I’ll rattle off a few of them.

• Who determines the truth? Journalists? Experts? Crowds? Individuals? Algorithms?
• Sometimes there is a right answer and sometimes there is room for debate. Can you tell which is which? How do you reflect the difference?
• How does the tool account for bias in sources?
• How does the tool account for bias in users?
• Will the system actually know enough to be regularly useful?
• This could easily just make consumers more lazy, how do you prevent that?
• What happens when the tool is wrong?
• How will this change the way people produce content?
• Where do Journalists fit into the picture?

As I’ve pondered these questions I’ve come to the following absolute conclusion: Credibility layers need to empower critical ability. I’ve also decided that it’s OK for the system to make mistakes but it is never allowed to lie. This means the interface should be less focused on telling the reader what to think and much more focused on reminding (and helping) the reader to think at times when thinking is most important.

I’ve also come up with a list of weaker claims to throw out there for discussion:

• Credibility layers don’t have to speak to everyone, but they need to empower the open minded.
• Journalists are our best bet for deep analysis and identifying truth that requires lots of time and effort (e.g. investigation and concept synthesis).
• Algorithms are our best bet for identifying contextual evidence (e.g. data, trends, and sources of sound bytes).
• Mobs can’t be trusted to decide what is true and false, but they are the key to figuring out what is worth thinking about.

Over the coming months I’ll be cranking out interfaces, prototypes, and eventually some good old fashioned boring academic papers about this idea. In the mean time if you’re interested in Truth Goggles I’ll be trying to post updates as regularly as possible on my blog, on twitter (@slifty), and eventually on the newly registered truthgoggl.es.

I feel odd writing too many separate posts in one day. My solution is a merger: the post on Anonymous and Hacktivism is going to buy out my project update. The terms of the buyout haven’t been made public but money has already exchanged hands between the 1% so there is no going back.

Part 1: Anonymous

My stalking of Anonymous and 4chan has always been an equal blend of scientific, hilarious, and disturbing. I pay just enough attention to know what’s going on, but not enough to actually be part of the community. Life is a lot better when you don’t visit 4chan. Of course, I can’t help the fact that the entire culture is absolutely fascinating.

Not to be hipster or anything, but I wrote about Anonymous before it was cool in the age of protesting Scientology. If you are curious about how anon functions as a hive mind then I highly suggest clicking that link, not to read the article but to read the comments.

This one, in particular, summarizes a significant portion of the Anonymous mentality: “one thing you may not understand about us, is our drive. We all crave one thing, the lulz. That which produces the highest amount of said lulz will be where our efforts go into. Any real anon will fight for the death for the lulz and the creation of more lulz. We are a hive minded organization that can be described as chaotic neutral. In lulz we trust.”

Looking back at this (and spending 20 minutes on 4chan along with the rest of the class last week) reminded me that all of the mainstream coverage of Anonymous often misses this aspect of the core personality. If Anonymous were a Shakespearian character it would be Puck from A Midsummer Night’s Dream. Think that everyone at 4chan would be mad at pepper spray cop? Absolutely not, his actions upset lots of people – a very potent form of lulz indeed.

I need to be more careful, of course, when trying to describe something as complicated as Anonymous. It isn’t an organization, it is a collective, which means trying to pin down a single motivation is a fruitless effort. To be sure there are more things that drive the group than laughs. Freedom of information, for instance (which is part of what set off the initial rebellion against Scientology — taking down that tom cruise video was both an attack on lulz AND an attack on information freedom).

Anon aside, last week’s readings opened up my eyes to the much longer history of digital disruption. Who knew that people used digital tools to cause collective trouble before 4chan? Not me, that’s for sure.

Part 2: Page One Remix

As for my project (a system designed to make it easy to re-mix and share the front page of the new york times), I have a few updates. I’ve forked, cloned, diced, and spliced Hackasaurus – a tool that is designed to help non-techies better understand how web pages work by making it easy to modify the code under the hood on the fly. They even have a built in sharing mechanism!

So far I have focused on changing the interface and interaction side of things. I made modifications to put less emphasis on “learning HTML” and more emphasis on remixing. This meant stripping talk of HTML tags, simplifying interactions where possible, and making it a bit easier to trigger the editing window (on the original tool you had to hover over an element and type “r.” On my tool clicking the element will do the trick).

The next step is to match some of the styling of the New York Times. Once that is done I’ll set up my own version of the sharing and boom! Tool complete and anyone can create their own news!

Once the technical side is complete (which will happen over the next few days) I’ll get crackalackin’ on the associated write-up. Here is the world premiere of the planned sections:

• Tool Introduction (Explaining the concept)
• Previous Remix Cultures (YTMND, 4Chan)
• Previous News Remixes (Yes Men, Others?)
• Page One Remix Overview (Description of the tool and how it works)
• Plans and Future Work (What I hope the tool will enable and how to add to it)

Here’s my new plan: I am going to make it possible for anyone to control the content of front page of the New York Times. Want your kid’s little league game in the local news? That’s cool, but you know what’s cooler? Having your kid’s fame story smack dab front and center next to the article about Osama Bin Laden’s assassination. Suddenly little Billy is the talk of more than just the town, he’s the talk of the entire world!

Interested? Well hang onto your hats because I’m about to teleport to a completely different topic.

How to Manipulate the Masses: A Simple Guidebook

People say the Internet is liberating and I suppose that can be true; however, as a wise man named Ethan Zuckerman once said, it isn’t enough to have a voice. What you really need is an audience. For the average digital Joe or Janet that audience is probably something between zero and maybe a few thousand people. If you didn’t realize it from that last sentence I’m saying your audience is smaller than a colony of ants. Hell, what you have isn’t even captive, so good luck getting more than a few minutes of collective attention across your entire network in a given day!

How does it feel to know that your personal media power quotient, even with access to the latest and greatest forms of communication in all of human history, is pretty close to zilch? Feels bad, right? Kind of makes you not want to bother trying to do anything at all?

Well suck it up because you aren’t alone. In fact, “you aren’t alone” is exactly why so many grassroots messages have spread in a land of noise and tweets: they go viral. If everyone gets two minutes of daily attention from a network then the only way for to spread a message is to hijack your network’s airtime too. More importantly, you have to do so in a way that equips all of those people to hijack THEIR networks too.

You can get your network to share by either:

• Pushing something that they will agree with (curse you filter bubble.)
• Pushing something that is amazing
• Pushing something that is hilarious
• Cats

Keep in mind that comments on your content will do almost nothing for spreading messages beyond one network step, which is why “pushing something that will piss them off” isn’t on the list. “Cats” is a placeholder for the type of content that, as of now, is the only way to get to consistently get a network’s network to share.

Anyone who was holding a hat can let go now because I’m going to talk about my project again. For those who didn’t care before but whose interested has piqued now it’s your turn to hat hold. For everyone else, why are you still reading this?

Taming the Meme

I met Ben Huh, the owner of ICanHasCheebzburger.com, last week at a 2012 election coverage summit. If that sentence meant nothing to you, I’m basically saying I met a viral god. The nearby newsfolk peppered him with questions about how to use memes to spread their own messages. His response was simple: you probably can’t. It is so difficult to harness a meme because they come from a digital version of whisper down the lane (or “Telephone” if you’re from that other part of the country). People add twists, there is no central control, and this is almost tautologically part of why the thing becomes popular to begin with.

Memes spread because they easy to shape, which is how people can use them in ways that are exciting enough to share. Boom, viral content achieved

My proposed system is one that will hijack an existing tool to make it easy to twist and turn the front page of the New York Times, to share those twists with a network, and to have members of that network add (and share) further twists of their own. I also want this system to track the changes; I want to build a conversation around the evolution of a given strain of modifications. I would even like to help incorporate some real content into the picture since I have the real estate; maybe some actual news can slip or fade its way in sometimes.

The end result is more than a simple stage for content sharing. Through very minor forms of control (in the form of history and tracking and known context) it becomes possible to infuse the adapting content with useful information layers. Boom, meaningful viral content achieved.

Oh, and for the record, remix-and-share services do exist (consider Startup Spirit vs Citrus Flavoring) but they are systems designed to provide a technology. I’m proposing a system designed to empower viral conversations.

This has been cross posted on the civic blog.