You may have heard of closed captions before: they’re the little words that show up at the bottom of TVs in airports, bars, or maybe at home when you mute the TV. Opened Captions scrapes those captions and regurgitates them into something called a web socket (web sockets take data and pass it along to anything willing to listen.) The service can take in caption data directly from a USB, or from another opened captions server.

This project came from my Knight-Mozilla fellowship at the Boston Globe during the 2012 elections. We were brainstorming innovative ways to cover the presidential debates, and quickly realized that the best of them required a way to know what was being said in real time. I got to work looking for solutions and, finding none, decided to make my own. A few days later and Opened Captions was born.

Although we didn’t end up building any official Boston Globe coverage, I did manage to throw together a wonderful project called DRUNK-SAPN. The application streamed the debates and told the world to drink on key phrases. Most importantly, the transcript took shots along with the players. You can view the transcript here.

Papers, Posts, and Press

• Introducing Opened Captions (slifty.com)
• Opened Captions: Turning the spoken words on TV screens into streams of hackable data (Nieman Lab)
• Introducing Opened Captions: A SocketIO API for live TV closed captions (Source)
• Opened Captions (Slideshare)

Technologies

• Node.js
• Socket.io

Testimonials

h*hic*Tey brougttth*hic* su binder*hic*s ffflul fo wmoen.
– Mitt Romney

NewsJack was built as a class project for Sasha Costanza Chock‘s “Introduction to Civic Media.” It enables détournement using web technologies. For those of you who don’t speak French / Chinese / Whatever that is, it means “turning expressions of the capitalist system and its media culture against itself.” It’s a very specific form of satire that takes subversive messages and wraps them in a skin that you are used to seeing (this might mean brand, it might mean medium, it might mean something completely different). Good détournement forces the viewer to question their world and their expectations.

The specific inspiration behind NewsJack is the Yes Men’s New York Times Special Edition. This fake paper was actually printed and handed out on the streets of New York City in 2009. It looked like a real copy of the Times, but had headlines like “IRAQ WAR ENDS” and “Maximum Wage Law Succeeds.” Imagining picking up a newspaper that you believed was the New York Times and seeing that type of headline. For a moment, until you realized what was going on, it might change the way you see your world.

That’s the experience that I ultimately wanted to enable online, where content is far harder to modify and spread for anyone who doesn’t know how to code. It is built using a modified version of a Mozilla tool called Hackasaurus. The original code was designed to help people learn HTML (the core building block of the Internet). I stripped out all that hippie foo foo learning crap and left the essence: code that makes it possible to edit a website by pointing and clicking.

Once you’re done you can hit publish, and a copy of the page you just modified is uploaded to some server out in the universe. You get a URL to share around, and suddenly your remix is alive and kicking.

You might say to yourself: “… how is this legal?” to which I would respond by picking up a copy of the first amendment, making a paper airplane out of it, and throwing it at your head. Of course, that didn’t stop the New York Times from sending a cease and desist on the first day we launched the site! I suppose that’s a story for another time.

Papers, Posts, and Press

• NewsJack launches to let you remix, edit news websites (Poynter)
• All The News You’d Love to See: NewsJack Launches To Allow You To Remix & Spoof The Media (BostInno)
• Jack This Site: NewsJack Lets Regular Users Remix Websites (Media Bistro)
• Workshopping NewsJack with Press Pass TV (C4CM)

Technologies

• jQuery
• MySQL
• PHP

Testimonials

I have a feeling our lawyers will be particularly interested in this project.
– Every Mainstream Media Source

Truth Goggles was my master’s thesis at the MIT Media Lab. This means that I spent a lot of time exploring the many challenges behind an idea like this, and attempting to solve at least some of them. There are three gigantic hurdles to jump:

1. Fact Database – What should be used as ground truth? How do you identify it? Is there enough of it? I ended up realizing that there aren’t many universally believed truths, and I want the system to be accessible to a diverse audience. It needs to contain as many well-explained and thoughtful verdicts as possible. I decided that for the first round a truth source needs to do things: have a reputation of neutrality (i.e.both sides call them biased or neither side calls them biased), and explain the reason behind their verdicts.
2. Paraphrase Detection – There are thousands of ways to say the same thing, how do you identify known fact-checks with slightly different phrasing? This is a challenging problem because it mans computers need to understand language. Luckily there are a lot of smart people exploring this space, so I can use existing tools to get part of the way there.
3. Human Brains – Assume we have a perfect system that is able to identify fact checked phrases 100% of the time. Would you trust it? Would you use it? What if it told you that you were wrong, would it change your mind? A system that isn’t usable isn’t worth building.

I focused on this third problem (human brains) because I like thinking about people more than algorithms. I tried to design the system to make it easy to swap out better algorithms and data sets down the line, but my experimentation revolved around the user experience.

The result of my thesis was a prototype and a gigantic document. The front end is written in JavaScript and jQuery. It scrapes the page and sends it to a credibility API written in PHP. That API checks against known instances of paraphrases and also sends the text to a “fuzzy matching” API that is currently written in Python.

The prototype can be used anywhere online, but it doesn’t do any intelligent paraphrase detection at this point so chances are it won’t be useful in most places. The study results were promising, and indicated that credibility layers could very well help people think more carefully and in more nuanced ways.

There is still a lot to do for Truth Goggles, and the project is not dead! It’s also clear that people love the idea of an automated bullshit detector.

Papers, Posts, and Press

This got a huge amount of coverage, here are some highlights.

• The first piece by Nieman Lab
• El Tiempo
• CBC Interview
• La Vanguardia
• The Register
• Tech Crunch
• NPR Interview
• The second piece by Nieman Lab
• Boing Boing
• New Scientist (Volume 215, Issue 2882, 15 September 2012, Pages 44–47)
• Wired UK (November 2012)
• Introducing Truth Goggles
• Achievement Unlocked: Thesis
• Truth Goggles Study Results

Technologies

• jQuery
• MySQL
• Python
• PHP

Testimonials

Oh shit.
– Politicians

Long Description

This project was a collaborative effort by myself, Julia Ma, and our undergraduate researcher Nat Atnafu. It started off as a course project for “Eccescopy” taught by Ken Perlin, which was about social augmented reality. Our argument was that looking into the past is totally social, it’s just time lapsed!

The system is made up of a web server (written in PHP and MySQL), where you can upload and annotate panoramas, and an iPad client (written using OpenFrameworks) that accesses and renders those panoramas. The user must calibrate the panorama by orienting the iPad so that the space being displayed matches the location of that space.

After calibration, the iPad tracks rotation and updates the portion of the panorama that appears on screen. This creates the illusion that the iPad acts as a looking glass into the past. You can see people and objects that used to exist in the space in front of you standing right there.

Although Ghost panoramas are never perfectly aligned, it gets “close enough” for your eye to believe it. We can get away with this because you aren’t able to focus on the iPad screen and the real-world background at the same time. The fact that the scale might not be quite right or that the image is offset by a few degrees won’t actually disrupt the experience.

It also turns out that this approach can be used to create other AR experiences. Normally you need to figure out where the camera is actually pointing with very high levels of accuracy if you want to create a compelling effect. By using an existing panorama you don’t need to be pixel perfect in the real world any longer, you can just use the pixels from the photo and be done with it.

We imagined several use cases for Ghosts. The original intent was to make it easy to capture and re-live community or personal events. If you took a panorama of a wedding, or of a block party, you could use Ghosts of the Past years later to remember what the event was like. This is pretty much supported in full.

You could also use the AR potential of Ghosts to provide more information, or to show hidden objects. We worked with Jim Vrabel, a local historian, to create a few panoramas with “information points.” Users could then stand in special locations in downtown Boston and learn more about the buildings by pointing their iPad at the right spot.

You can also manipulate the panorama itself using photo editing tools to create some interesting effects. We worked with the MIT Museum to create a few panoramas that spliced in old photos of artifacts working in their original space. As you viewed the panorama in the museum, you would see the objects sitting in the lab with scientists working around them.

There are some obvious next steps. The first is geofencing using the iPad’s GPS, so that only nearby panoramas can be picked, and so that people can be guided to stand in the optimal spot for viewing. The second is auto calibration using the iPad’s compass, so that images are oriented properly without any effort on the part of the user.

Technologies

• C++ / OpenFrameworks
• PHP
• MySQL

Testimonials

Hasn’t this been done already?
– My Advisors