Steve Mann – General Chair Address

My grandfather taught me to weld when I was 4 years old, so early in my life I became aware of what the world looked like when “Seen Through the Glass, Darkly”.  Darkglass (the welder’s glass) diminishes reality.  Diminished Reality is the logical opposite of Augmented Reality.

A childhood vision of mine was to use television cameras and miniature displays, with contrast adjustment, to be able to

see-in-the-dark while still being able to clearly see the electric welding arc without hurting my eyes.

In my childhood, back in the 1970s, as an amateur scientist and amateur inventor, my experiments in contrast reduction became experiments in a general-purpose wearable computer system, using hybrid analog and digital computing equipment.  I also began to add overlays of text and graphics on top of my visual reality.

I started to think of the “Digital Eye Glass” as something to be used to help people see better in everyday life, not just while welding. Around that time others started putting electronics in glass, e.g. 3M’s Speedglas/Speedglass was introduced in 1981.

Wearing “Glass” in everyday life back then scared people — first because it looked strange — which later was to become something people thought was “cool” rather than strange — but ultimately because of the sensors/camera(s).

As I refined the Glass into a sleek and slender eyeglass form-factor in the 1980s and early 1990s the appearance became acceptable to the general public, but I began to find myself harassed by police and security guards afraid that the “Glass Eye” might be recording them.

How ironic it was that the very people who were installing and monitoring surveillance cameras — watching us — were the people most afraid of being watched!

Glass (Digital Eye Glass, GlassEye, Mannglas, Speedglas, etc.) became a metaphor for a fragile and transparent society.  We can learn a lot from a society by how readily its police accept Glass — i.e. how readily the society accepts reciprocal transparency and the mutual vulnerability we share when the “Glass Eye” becomes as commonplace as the surveillance cameras of the authorities.

This year’s ISTAS theme, therefore, is not Surveillance (Watching from above, in a hierarchy, e.g. police watching suspects) but, rather, Veillance (watching in a politically neutral sense).

 Please join us as we explore this half-silvered world of two-way transparency.

Insight from tracking the Boston Marathon bombing suspects

PBS Nova recently broadcast (with amazingly short turn around) a show on the technology used to track the Boston Marathon bombing suspects.  (Since NOVA is produced in Boston via WGBH that is not too surprising, but they also pulled off a NOVA the same night on the Oklahoma Tornado)

The show outlined the types of technology used, and examples from other cities of technology that might have been used if it were available.  It presents an interesting, but perhaps a bit disturbing view of current and emerging police systems in the U.S.

Within minutes, Boston police were starting to get access to the many corporate video cameras recording in the area, and started an intensive evaluation. Boston does not have the fully integrated, real time video feeds from private and public sources that New York does.  The NYC system is called the “Domain Awareness System” (DAS)  and is integrated with many other elements.  For example, a 911 call can be triangulated to the cell phone connection, cameras in that area highlighted so even before the phone discussion starts, the dispatcher can be viewing the source of the call. Integration with face recognition allows for identifying suspects (although that process failed in the Boston case, in part due to the poor quality of pictures available.)

The DAS system also tracks the license plates of every vehicle entering or leaving Manhattan via cameras associated with every bridge and tunnel. It appears from the show that at least 30 days of video are saved for every one of the rumored 3000+ video feeds being tracked in the system. Interestingly, there is very little information about this system on the NYC web site, a bit on guidelines and authority, press releases, but not a lot of government transparency.

While standard face recognition technology did not yield results, two other systems did. Marios Savvides at CMU is developing an advanced facial recognition system that can operate off lower quality images, and when a good picture was included in a million plus face data base, it was number twenty in the selected group.  So we can expect new generations of this capability. There was a surge of crowd-sourcing in terms of providing the police with information and pictures, but also in terms of reviewing posted pictures (REDDIT) being a key community for this.  While the Reddit process did not ID valid suspects (it did surface innocent folks), it did force the Boston Police into posting their photos of the suspects.  This also did not result in an ID, but it did appear to force the suspects into activities that ultimately killed a police officer, hijacked a car and resulted in the death of one and capture of the other.

Cell phone triangulation played a role in locating the suspects, just with the phone turned on, no call required.  (I wonder if an OnStar or similar system could also have been used.) The Boston Police also used plane and helicopter based infra-red cameras to try to locate the final suspect.  This proved quite successful as the individual was hiding in a boat with a covering that was IR transparent, providing a clear indication of his location.

The NOVA program ends with an often heard commentary about how some of the technology really helped, some didn’t deliver… but we must consider the implications for the privacy of citizens as this technology emerges.

Meme Propagation

Deb Roy in his TED presentation on “The Birth of a Word” gives us a glimpse at a technology with potentially high impact.  His primary talk discusses how, over 5 years, with a 18/7 audio/video recording from every room in his house, his MIT team is able to trace the word acquisition of his son. I will let you contemplate the pros and cons of having 100% of your household activities recorded for posterity.

However, his team applied the software they used to capture every use of specific words by his son, then connecting these with every word from members of the household in the proximity of his son, to analyse other interactions.  One source was the feed from every major television network. The second was to track emerging phrases from these sources via the blogosphere/twitterverse.  Their result is the ability to obtain near-real time measurement of the impact that a given source is currently having on the population at large.

A popular TV show may trigger social media flow with a positive feedback loop bringing more viewers into the show.  The proliferating comments may provide analysis of what works best in the show, what is not working, where viewers want the story to go.  One can envision a program driven impromptu by viewer responses measured in the Twitterverse.

However, a second example was President Obama’s State of the Union address. This showed much broader distribution than any single TV show, with massive response and interaction in the Twitterverse.  One can envision real time AI analysis (Deb Roy has been working with Bluefin Labs which does this commercially) that is used to critique a political speech or event.  In the extreme, a presenter may get coaching feedback from real time evaluation, altering the presentation spinning up on the teleprompter in real time.

Consider a political debate where candidates are receiving real time talking points based on analysis of the Blogosphere, and altering their apparent positions based on this.  The good news is that it would require some fairly smart candidates to pull that off, or perhaps ones with nothing in their heads except the words that are being feed to them anyway.  But now the kicker. Activating a zombie army of previously captured devices, all of which have twitter accounts, and you can re-direct the discussion. The discussion migrates from starvation in Iran to education in Latvia. Both candidates arguing the strategic value of Latvia and how their proposals will yield the best educational outcomes.  And the only indication that this discussion has been hijacked is that neither candidate knows where Latvia is. But the good news is that they are supporting education …..somewhere.

The Citizen Surveillance State

Like everyone, I listened to the news about the Boston Marathon bombings on Monday afternoon with horror. I’m not from the area myself, but I have a lot of friends who live there, and a lot of friends who run marathons. Luckily for my personal peace of mind that afternoon, those two groups don’t intersect for me, and I was so grateful as one by one, so many people posted to Facebook and twitter that they were OK. I know many other people weren’t so lucky.
It’s also been fascinating to watch over the last several days as the FBI has asked anyone with photos or video of (or before) the bombings to send them in. There must be tens of thousands of submissions for them to sift through, and yesterday the FBI posted several pictures taken from various sources of two men they are seeking information about in connection with the blasts. It’s remarkable how quickly law enforcement has been able to pinpoint suspects given the huge volume of evidence they must have had to sift through, but it was virtually inevitable that they would have photos and video of whomever did this.
I’ve heard it said that this is probably the most photographed and recorded terrorist event in history, and I’m sure that’s true. Sure, this is partially because the Boston Marathon is a huge public event, but it’s also because we are living in an era of citizen surveillance. Very simply, if you go out in public and are around other people, there’s a pretty decent chance you are being recorded. Some of this is because of the ever increasing use of security cameras and cctv, but it’s also because nearly everyone is carrying a recording device around with them in the form of a phone.
As a society, we are still figuring out how to deal with this. I’m sure they meant well, but redditors publicly misidentified two subjects, likely putting them in danger. Reddit eventually stepped in to stop things, but if you had been one of the two guys who’s personal information had been posted identifying you as a suspect, that probably too way to long to happen.
So we live in an era of mass citizen surveillance – that is to say, (mostly unintentional) surveillance by other citizens. Of course, most of us think of this data as not being organized in any cohesive way, but actually, a lot of it is. Most people don’t realize it, but almost all smart phones by default geo-tag all photos they take. When you post them, you’re not just posting the image, you’re posting where and when it was taken. Users can turn this feature off, but since most of them don’t realize it exists, they don’t. Facebook, Instagram, and Flickr all have APIs that allow searches by geographic area. Right now this isn’t something most people can actually do, but for law enforcement or anyone with programming skills, it isn’t difficult at all.
What does all this mean? I’m not really sure, actually. But it did all remind me to switch off geo-tagging on my phone.

You have to work pretty hard to make something private these days

Oh Google, I love a lot of your products and services, but this was pretty stupid. If you hadn’t heard of this before, basically Google Street View cars were scanning open wireless networks and scraping them for any information they could get. Google apologized and paid a big fine, but correctly note that what they did wasn’t actually illegal in any way.
I’ve thought for a long time that obscurity was actually a pretty good privacy protection – for years my home wifi didn’t have a password because I was set quite far back from the main road and it made it so much easier when guests came over. I finally added a password a few years ago, but in general I’m not one of the privacy paranoid. Efforts like this however, and the fact that they are not, nor is it likely that they ever will be, are changing that. My personal information is valuable. And these days there are more an more companies like Google that have the resources to gather that information en mass.
We are less and less obscure every year.
Some of this is totally out of our hands. The ever-falling cost of processing power and better and better data crunching algorithms mean that it’s feasible to find and store a lot of information about, literally, everyone. But, a lot of this we give away ourselves – without even realizing it. Anyone who sends un-encrypted data over open wi-fi in the eyes of the law has zero expectation of privacy – but I bet that’s not what those users thought. In the U.S., the courts have held that law enforcement can trace you via your smartphone without a warrant. Since your phone is constantly broadcasting a GPS signal, the logic goes, it’s akin to using a dog to trace your scent. But I seriously doubt many people know this.
Some of this, of course, will be fixed by time. Technology is a disrupter, and we are all, collectively, constantly learning how to use it. Criminals are figuring out they have to switch off their cell phones. People like me have added passwords to their home wifi networks. At some point maybe Harvard deans will realize their work email isn’t private and figure out how to use Tor – then we’ll know the future has arrived!

Everybody knows where you are?

A recent paper in IEEE Computer Magazine asked the question “Can a Phone’s GPS ‘Lie’ Intelligently ”   As you may have been aware, many mobile aps use the location information of the phone to …well … stalk you.  For an ap that is going to suggest nearby places to eat, that is not all that surprising, but what is surprising is how many actually include this in their “rights” for installation.  (It took me some time to find a FreeCell ap that didn’t want to know my location, who I was talking to on the phone, who my contacts were, etc. etc.)

The  article suggests that for many such applications, there is a “close enough” — which is likely to have a margin of error circa one mile.  This is not enough for your friends (or enemies) to locate you in a shopping mall, but is enough to let you know a friend is near, or a store with a product you might value.  Ergo the title of the article.

Mobile device tracking of location is fairly impressive.  A few years ago I got an Apple iTouch — WiFi (aka IEEE Std. 802.11)  but no cell and no GPS.  I turned it on, and it was able to pinpoint my location on the map!  Google, as you probably know,  has been tracking WiFi MAC addresses as they shoot their street view pictures.  Between this information, cell tower triangulation (needed for 911 calls), and integrated GPS devices my more recent (Android) cell phone has my location nailed very quickly — much faster than my Garmin devices.

The camera in the device can incorporate Longitude/Latitude information into the JPG metafile.  Something that upset some celebrities when they realized that photos being posted to the web not only showed who was at their party, but the date, time and specific location of said party.  I’m not sure if most cell phones or similarly enabled cameras come with the “include location in information” on or off. But I suspect many users haven’t got a clue how to control this — or even that it exists.

In any case … a little paranoia may be justified. … that and much more information for the public so folks suspect that the Angry Birds might actually know where they are!