Truth Goggles was my master’s thesis at the MIT Media Lab. This means that I spent a lot of time exploring the many challenges behind an idea like this, and attempting to solve at least some of them. There are three gigantic hurdles to jump:

1. Fact Database – What should be used as ground truth? How do you identify it? Is there enough of it? I ended up realizing that there aren’t many universally believed truths, and I want the system to be accessible to a diverse audience. It needs to contain as many well-explained and thoughtful verdicts as possible. I decided that for the first round a truth source needs to do things: have a reputation of neutrality (i.e.both sides call them biased or neither side calls them biased), and explain the reason behind their verdicts.
2. Paraphrase Detection – There are thousands of ways to say the same thing, how do you identify known fact-checks with slightly different phrasing? This is a challenging problem because it mans computers need to understand language. Luckily there are a lot of smart people exploring this space, so I can use existing tools to get part of the way there.
3. Human Brains – Assume we have a perfect system that is able to identify fact checked phrases 100% of the time. Would you trust it? Would you use it? What if it told you that you were wrong, would it change your mind? A system that isn’t usable isn’t worth building.

I focused on this third problem (human brains) because I like thinking about people more than algorithms. I tried to design the system to make it easy to swap out better algorithms and data sets down the line, but my experimentation revolved around the user experience.

The result of my thesis was a prototype and a gigantic document. The front end is written in JavaScript and jQuery. It scrapes the page and sends it to a credibility API written in PHP. That API checks against known instances of paraphrases and also sends the text to a “fuzzy matching” API that is currently written in Python.

The prototype can be used anywhere online, but it doesn’t do any intelligent paraphrase detection at this point so chances are it won’t be useful in most places. The study results were promising, and indicated that credibility layers could very well help people think more carefully and in more nuanced ways.

There is still a lot to do for Truth Goggles, and the project is not dead! It’s also clear that people love the idea of an automated bullshit detector.

Papers, Posts, and Press

This got a huge amount of coverage, here are some highlights.

• The first piece by Nieman Lab
• El Tiempo
• CBC Interview
• La Vanguardia
• The Register
• Tech Crunch
• NPR Interview
• The second piece by Nieman Lab
• Boing Boing
• New Scientist (Volume 215, Issue 2882, 15 September 2012, Pages 44–47)
• Wired UK (November 2012)
• Introducing Truth Goggles
• Achievement Unlocked: Thesis
• Truth Goggles Study Results

Technologies

• jQuery
• MySQL
• Python
• PHP

Testimonials

Oh shit.
– Politicians

Long Description

The Media Lab gets a lot of visitors. People bring in friends and family, sponsors send representatives, and others just wander in off of the street. The Glass Ingrastructure is designed to help anyone quickly and effectively explore the kind of work being done in the building without having to bother us. Basically, grad students really just don’t like talking to people if they can avoid it.

The system is made up of about 30 giant touch screens, wired to mac minis and RFID readers. They are located throughout the Media Lab’s old and new buildings, and display maps, projects, and other applications.

When I arrived the Glass Infrastructure had a fairly linear interface. Each screen displayed a rotating list of nearby projects based on where it was placed. Our goal was to redesign the entire thing, building on the systems already set up. We had a pretty large team too (about nine people) working on various parts of the product.

We wanted to reflect the relationship between people and projects without overwhelming the user. There also needed to be a way for people carrying RFID tags to “favorite” different projects, and view a list of their favorites on the bottom of the screen. After weeks of discussion and thrown out designs we came up with the molecule interface.

I spent most of my time working on client side middleware, middle learning a bit more about jQuery at the same time. The backend was a RESTful API that worked with the database in addition to some fancy Natural Language Processing (NLP) tech which automatically found project relationships based on their descriptions.

The design that we came up with is still in use today, and the Glass Infrastructure has become a core part of the visitor experience at the Media Lab. We have also set up a few screens in other parts of the world (Spain, for instance) so that sponsor companies can have a window into the work being done here.

Technologies

• jQuery
• CSS3

Papers, Posts and Press

• The Glass Infrastructure: Using Common Sense to Create a Dynamic, Place-Based Social Information System (Havasi, 2011)

Testimonials

You know what would be better than asking me? Going over and prodding that screen over there.
– Media Lab Grad Students

Long Description

ATTN-SPAN was my final project in a future-of-TV course called Social Television. My goal was to figure out a way to make C-SPAN not completely suck. Fun fact: this was almost my thesis project.

The plan was to take C-SPAN, processes it, tag the moments of video with as much metadata as possible, and use those moments to create a compelling consumption experience. Where would the data come from? Three places:

• Closed Captioning – C-SPAN is closed captioned. The captions are full of typos, but they can be used for general analysis (e.g. guessing what topics are being discussed).
• Video Processing – OCR (Optical Character Recognition) is a process that takes an image and looks for words. On C-SPAN, text consistently appears in the lower left hand corner of the screen to state names of politicians, bills, states, and political parties. Face recognition is another possible technique.
• Audio Processing – there are a few algorithms that can detect different voices in a conversation. They might not be able to tell you what is being said, but you can at least get a sense of who is talking.

From these streams I can do things like find all content that was spoken by YOUR senator. It might not sound that interesting but it actually does make the content significantly more relevant.

I made a quick prototype using an existing project called Metavid. This service handled most of the above processing for me, so I could focus on creating a fast hack to show what it would be like to have an episode of only your senator. I eventually hoped to do all the heavy lifting myself (rather than rely on a third party).

The prototype was a reasonable success, although it didn’t end up being insanely compelling. This was partially due to implementation and partially due to the fact that Metavid started acting unstable and was slow to process new footage.

Had I continued to work on ATTN-SPAN I would have designed an API so that newsrooms request custom generated episodes. Those episodes would be displayed along-side their political coverage. Imagine if you could read about new policies and see exactly what ~~your~~ senator said on the floor about the topic, right there, embedded in the article. This concept of automatically filtered primary source content almost made its way into Truth Goggles. It still might some day.

Technologies

• PHP
• MySQL
• jQuery

Posts and Press

• My learning lab proposal

Testimonials

I had no idea that C-SPAN was so interesting! Oh wait no this is still really boring.
– The American Public

Long Description

This project was a collaborative effort by myself, Julia Ma, and our undergraduate researcher Nat Atnafu. It started off as a course project for “Eccescopy” taught by Ken Perlin, which was about social augmented reality. Our argument was that looking into the past is totally social, it’s just time lapsed!

The system is made up of a web server (written in PHP and MySQL), where you can upload and annotate panoramas, and an iPad client (written using OpenFrameworks) that accesses and renders those panoramas. The user must calibrate the panorama by orienting the iPad so that the space being displayed matches the location of that space.

After calibration, the iPad tracks rotation and updates the portion of the panorama that appears on screen. This creates the illusion that the iPad acts as a looking glass into the past. You can see people and objects that used to exist in the space in front of you standing right there.

Although Ghost panoramas are never perfectly aligned, it gets “close enough” for your eye to believe it. We can get away with this because you aren’t able to focus on the iPad screen and the real-world background at the same time. The fact that the scale might not be quite right or that the image is offset by a few degrees won’t actually disrupt the experience.

It also turns out that this approach can be used to create other AR experiences. Normally you need to figure out where the camera is actually pointing with very high levels of accuracy if you want to create a compelling effect. By using an existing panorama you don’t need to be pixel perfect in the real world any longer, you can just use the pixels from the photo and be done with it.

We imagined several use cases for Ghosts. The original intent was to make it easy to capture and re-live community or personal events. If you took a panorama of a wedding, or of a block party, you could use Ghosts of the Past years later to remember what the event was like. This is pretty much supported in full.

You could also use the AR potential of Ghosts to provide more information, or to show hidden objects. We worked with Jim Vrabel, a local historian, to create a few panoramas with “information points.” Users could then stand in special locations in downtown Boston and learn more about the buildings by pointing their iPad at the right spot.

You can also manipulate the panorama itself using photo editing tools to create some interesting effects. We worked with the MIT Museum to create a few panoramas that spliced in old photos of artifacts working in their original space. As you viewed the panorama in the museum, you would see the objects sitting in the lab with scientists working around them.

There are some obvious next steps. The first is geofencing using the iPad’s GPS, so that only nearby panoramas can be picked, and so that people can be guided to stand in the optimal spot for viewing. The second is auto calibration using the iPad’s compass, so that images are oriented properly without any effort on the part of the user.

Technologies

• C++ / OpenFrameworks
• PHP
• MySQL

Testimonials

Hasn’t this been done already?
– My Advisors

Long Description

This started as simply a fun way to play with a laser cutter. Actually, it ended that way too. In fact, it never really drifted from that status. However, carving pumpkins with a laser had some interesting challenges.

Laser cutters usually burn away material like acrylic or cardboard to slice pieces out of a flat material. This is great because the laser has a focal point where the clean cut will happen. If you are out of focus then the burned hole will either be too big or it won’t cut the material at all. If you’re working with a flat plane you can focus it in one spot and be fine.

If you are carving into a cylinder (for instance a can of baked beans), then you don’t have a common height to focus onto! Fortunately for me, the laser cutter at the Media Lab has a lathe attachment. With a lathe you simulate a flat plane by converting the y axis into rotation (meaning you could “spin” an object instead of having the laser actually move forward and backward).

Pumpkins aren’t cans of baked beans, pumpkins are spherical. The lathe only lets me rotate on one axis, which means that the X axis varies in height too. I found a really good solution to the problem, which was to ignore it and carve anyway. It worked, although I lost detail the further I went from the middle. Meh.

Before you can even pick out a pumpkin you need to take an image and break it into four layers (I used Adobe Illustrator to do this). Each layer will represent a different brightness, since it will be a different cut depth. You don’t need to use four layers — you could be boring and just use one, for example. I used four because deep down it just felt right.

It’s hard to predict the depth of a cut once you do your second pass, because the area around the laser point will often vary in height depending on how many cuts have happened. For instance if you were to carve the shape of a doughnut, the middle part of the doughnut might begin to sag, which means that additional cuts could accidentally hit the outskirts of the center. Tiny details have to be deeper (e.g. brighter) or they will be blasted away on later cuts.

More could be done — e.g. automatic generation of layers based on pumpkin color calibration (carve the pumpkin with varying depths, take a photo of the lit pumpkin, and match it to the photo you want to carve). We’ll see what happens!

Posts and Press

• My overview
• New Scientist

Technologies

• Lasers
• Adobe Illustrator

Testimonials

Halloween, it’s about time
– Tychus Findlay

Long Description

This project was done as a 48 hour hack with a team of three people (myself, Julia Ma, and Boris Kizelshteyn) for an HTML5 game creation competition. We didn’t win, but that’s only because I went insane at 5:00 in the morning and tried to fix our collision detection algorithm with only 15% mental capacity. I broke it even more. I think they forgave me, but I still kick myself.

REGARDLESS!!! QRTcullis is awesome. Our goal was to think about what mobile gaming could look like. We wanted to incorporate things into the game that simply couldn’t exist using more traditional technologies (e.g. consoles or PCs). We started from scratch and used technologies that we had never used before (NodeJS and MongoDB) because we hate ourselves.

Levels are built with tiles. Tiles can have attributes, modifiers, and scripts (e.g. “move up 500 points over two seconds”). In order to prevent cheating, everything is processed on the server side and animation commands are sent to the client using sockets. We came up with a set of attributes and scriptable actions a tile could have, so as the player explores she finds deadly pigs wandering around, and tornados.

Less than half of the vision was actually implemented — for instance we wanted to be able to tie tile-scripts to sensors (so, for instance, a door opening in the real world would trigger a door opening in the fort). There is still a lot of potential for this idea, and I’m hoping some day there will be a reason to develop it again.

Technologies

• HTML / CSS
• jQuery
• NodeJS
• SocketIO
• MongoDB

Testimonials

I don’t get it, what am I supposed to do?
– Anyone, after being shown a QR code

Wall Paper allows users to explore information by navigating through physical space.

This demo has undergone several iterations but the overall architecture has remained consistent: there is a browser-based client (JavaScript / HTML / CSS) separated across eight windows, a server (NodeJS) which listens for position updates and manages state, and a sensor component (Python / C) which does the actual position tracking.

The first iteration used infrared (IR) sensors and Arduino to track about 10 points evenly distributed across the eight displays. This provided approximately one data point per screen. A “newspaper” was spread across the screens so that each one displayed a section title. To read a given section you would walk up to the appropriate screen. The sensors would detect you, pass that detection onto the client, and an article from the screen’s section would fade in an article for you to read.

The second iteration replaced the IR sensors with a Microsoft Kinect, resulting in a much higher resolution depth map. Instead of the 10 depth points I now had access to closer to 1000, and I could track positions far more consistently. The interface was also replaced to display a bar-chart representing 20 years of reporting by the New York Times. Each month had a bar broken into four colors representing coverage of “Afghanistan,” “Iraq,” “Wall Street,” and “Protests.” You could walk up to a part of the screen to learn about a specific month, and as you got even closer you could see headlines for each of the four content types scrolling across your section of the display.

Of course, it was also used to mess with people. There was a time when the content would get smaller as people walked closer.

Technologies

• Arduino
• C
• HTML / CSS
• jQuery
• NodeJS
• Python

Testimonials

So what you’re saying is that this project has failed both in terms of user experience, and in terms of graphic design?
– Henry Holtzman