Data Mining the Electorate

Obama Data Mining Team

The New York Times Magazine had a cover piece on the Obama data mining team that used modern data-mining techniques to more efficiently target the undecided voters that they needed to bring across the fence to win the election. Check out the last line (emphasis mine) on their clever use of Facebook photo tags as a way to further refine their targeting to determine who your real friends were. If they identified any of your close friends as potential voters that were on their “undecided” list, they would then put them on a list of friends for you to ask to vote for Obama.

They started with a list that grew to a million people who had signed into the campaign Web site through Facebook. When people opted to do so, they were met with a prompt asking to grant the campaign permission to scan their Facebook friends lists, their photos and other personal information. In another prompt, the campaign asked for access to the users’ Facebook news feeds, which 25 percent declined, St. Clair said.

Once permission was granted, the campaign had access to millions of names and faces they could match against their lists of persuadable voters, potential donors, unregistered voters and so on. “It would take us 5 to 10 seconds to get a friends list and match it against the voter list,” St. Clair said. They found matches about 50 percent of the time, he said. But the campaign’s ultimate goal was to deputize the closest Obama-supporting friends of voters who were wavering in their affections for the president. “We would grab the top 50 you were most active with and then crawl their wall” to figure out who were most likely to be their real-life friends, not just casual Facebook acquaintances. St. Clair, a former high-school marching-band member who now wears a leather Diesel jacket, explained: “We asked to see photos but really we were looking for who were tagged in photos with you, which was a really great way to dredge up old college friends — and ex-girlfriends,” he said.

Data You Can Believe In

Unintended use for Microsoft Kinect

I ran across this interesting use for Microsoft Kinect by an MIT Researcher who was featured in the latest issue of The Red Bulletin, a print sports lifestyle magazine that magically shows up on my doorstep every month. The article (I cannot link to it because I cannot find it online), which profiles urban planner Carlo Ratti, had this description of an upcoming urban hack he’s releasing in Madrid, Spain.

Spanish cities have what they call puntos limpios, collection points where you can drop off bulky waste like old sofas or a washing machine. We have converted a Microsoft Kinect sensor, which is usually used to operate a game console, to scan this waste. Then the system automatically tweets: “I’m a couch. Come get me!” This allows other city residents to see what is being thrown away, and they can save it from the dump. In this way, everyone is better off – it’s free furniture, the waste management company saves on costs, the local authorities promote cohesion in the community. The waste disposal service we are working with wants to try it in Madrid.

The Future of Our Cities, The Red Bulletin, July 2013

Wall-E

If you live in the US, there’s Freecycle – a Yahoo Groups community split up into local chapters that is similar. It’s usually the quickest way to clear out your garage. Go online and list something you want to give away for free and, usually within hours, someone will reply and take it off your hands. Carlo Ratti has managed to automate the listing part of it.

Digital Contrails

What do you do when you have access to the twitter firehose and a top notch geo-visualization artist? Make beautiful maps of course! Gnip and Eric Fischer got together with MapBox and plotted millions of tweets by location, language, and device to come up with some fantastic interactive maps.

twitter-geo-tokyo

 

The map above is Tokyo and the blue dots represent the location of geo-stamped tweets by people identified by their tweet history as locals while those dots in red are “tourists” who normally tweet from somewhere outside the region. The map tells you a couple of things.

  1. Most tourists are tweeting (photo-sharing?) from the major city centers. I can recognize Shibuya, Shinjuku, Marunouchi, Yokohama, Ueno, Ikebukuro, maybe the Rainbow Bridge?
  2. If you’re familiar with Tokyo, you can see that people tend to tweet while on the train.

This second point reminds me of something I read in Wired a couple of years ago. In an experiment, researches placed oat flakes in a pattern that resembled the major city centers in Tokyo. Then they place a culture of slime mold in the middle and let the culture figure out how best to harvest or “move around” the oat flakes across the pattern. What they found was that the mold grew a series of tunnels that matched the patterns found on the metropolitan rail system.

What works on a large scale also fits a pattern at a much smaller scale.

slime mold

I’ve written about Location Traces as Art before. Even before the crazy NSA/Snowden tracking scandal broke it was a well-known fact that the phone companies had a wealth of data about us. Aggregated en-mass in platforms such as twitter, this data can paint an pretty amazing picture of the world around us. A couple more maps from the Gnip/Fischer/MapBox collaboration.

twitter-devices

It’s a little hard to see but this is a map of the world that shows which type of twitter client is used when a tweet is made. The Red is iPhone, Green is Android, and Purple is Blackberry. Looks like Spain is big on Android (for twitter anyway) while Saudi Arabia, Mexico, and Southeast Asia are Blackberry strongholds (where BBM is huge).

bayareadevices

If we look at my neighborhood, you can see that I mostly live in an iPhone town except for a Oakland/San Leandro which is more into Android. I know what you’re thinking, The Atlantic already wrote about it. When you see lots of green, it usually signifies a less affluent area.

Fascinating stuff.

Patently Ridiculous

I had a listen to the second part to This American Life’s excellent two part series on patent trolls (the first part is here) and how the use of weaponized software patents is squashing innovation.

The podcast outlined the plight of entrepreneurs too afraid to start business for fears of being sued out of existence by shell companies that own broadly defined patents for no other purpose than to shake down founders too cash strapped to defend themselves. While listening, I recalled an episode that took place while at Yahoo where we were the victim of exactly this type of attack. Here’s how it went down.

It was late-2007 when Todd Sampson, the co-founder of MyBlogLog, the small social network acquired by Yahoo earlier that year, contacted me to tell me that we (Todd and I worked together at MBL while I was at Yahoo) had been contacted by a small company in Israel called Girafa notifying us that the screenshots used on the site were in violation of a patent owned by Girafa.

It was a classic shakedown. Girafa explained that they were preparing a lawsuit in the amount of $5 million but there was an out. We could avoid litigation if we licensed their software. The price? $5 million.

MyBlogLog Thumbnails

You can imagine our surprise. Todd has tied together a couple of open-sourced software packages to create a screenshot process that ran on an old PC that literally sat under one of the engineer’s desks. The software was a bit temperamental and would sometimes fall over. We would get complaints for users that their website needed a refreshed screenshot which was our queue to go over and restart the screenshotter machine. It happened enough times to be a minor pain in the neck so we were open to licensed software or web service that was a bit more stable but $5 million was certainly more than we were willing to pay.

Yahoo legal was on the case and they told us that not only MyBlogLog but also delicious and Yahoo Bookmarks were named in the suit along with Alexa (part of Amazon) as well as a few other companies were also part of the suit. All during the course of the lawsuit, we had to have several meetings with the legal team at Yahoo (who were great) but also had to keep copies of all our correspondence and take care when we eventually moved the machine to a Yahoo data center to ensure the screenshot software was not integrated with any other parts of the Yahoo infrastructure to ensure that other divisions in Yahoo couldn’t get ensnared in the lawsuit. At some point Todd had to give a deposition and he even spent an afternoon over at the Internet Archive looking through the Wayback Machine archives to locate old screenshots from HotWired which we remembered used to take screenshots of sites back in late-90’s.

I left Yahoo before I heard what happened so, after listening to the This American Life podcast, I poked around to find out the rest of the story.

I am not a lawyer but from what I can tell, the lawsuit was initially thrown out in late-2008 but documents and paperwork continued to be filed (in hopes of continuing the suit?) all the way into late 2010.Thge Girafa nonsense tied up the courts for a solid three years. Think of the untold wasted hours!

In June of 2011, Google swooped in and purchased Girafa’s patents, supposedly as insurance against any action against Google’s use of thumbnails in their instant preview search results feature. Let’s hope the Girafa patent will stay dormant where it is and not be unleashed again by its new owners to cause a new round of havoc and hand-wringing.

If you go to girafa.com there’s a sad notice that the service has been discontinued. A sad final chapter for a service that caused nothing but pain and consternation.

girafa

Autonomous Robotic Weapons

Daniel Suarez is one of my favorite SciFi writers because he takes the technology trends happening today and draws a line into the future to warn us of what dystopian nightmare awaits if we take the humanity out of technology. I have recommended his book, Daemon to many people as a must read if you’re involved in the world of news feeds, reputation systems, and social media.

There’s now a TED talk where Daniel introduces the latest in drone technology and talks about the need to reign in the march towards automating the use of these drones in warfare. Tracing the history of weapons to the evolution of the modern nation state, he views drones as a corrosive catalyst that allows power to aggregate back to small, powerful, and ultimately anonymous organizations.

Suarez’s latest book, Kill Decision is about this very topic.

How Design Can Defeat Technology

The Offline Glass from Mauricio Perussi on Vimeo.

Maybe not the most practical if you’re worried about condensation getting all over your phone but it certainly does make a statement. The Offline Glass needs your phone to keep it from tipping over, thus forcing you to put your device down and engage with those around you.

Developed for a bar in Brazil by an advertising agency, the offline glass campaign states, “Isn’t it boring to be at a table in a bar with someone who just plays with their mobile? We resolved this problem.”

via: The Telegraph

Visualizations – Foursquare and Twitter

Foursquare and Twitter both released experiments that let you look at a visualization of your activity on that service over time. One is art that happens to convey information and tell a story of your travels thru time and space. The other is a functional dashboard that is designed to give you and idea of how effective you are in getting your message out to others.

Foursquare Visualization

The Fourquare visualization is sponsored by Samsung and is clearly branded so. It’s a well done, if slightly solipsistic, eye-candy. Foursquare got their money up front on this one and used it so their users (including me) would have something beautiful that they wanted to share along with their sponsor.

Twitter Timeline

The Twitter timeline (read The Next Web for details on how to get to yours) shows mentions and follow/unfollow activity along with details about specific tweets and how they performed.  There’s also a screen that shows your follower growth over time along with some basic demographic information. It’s all business though, a reason for the normal folks to login and poke around their ad platform and think about spending money on some of twitter’s ad products. Tucked at the bottom of their graph is a sober message reminding us that we’re browsing through a business tool.

And there, my friend, is the difference between these two efforts:

Foursquare:  Zoom through time and space as you visualize all your check-ins

Twitter: The data reported on this page is an estimate, and should not be considered official for billing purposes.

NSA Everywhere

NSA Ops Center

They set about remotely penetrating communications systems and networks, stealing passwords and data by the terabyte. Teams of “vulnerability analysts” searched hundreds of computers and servers for security holes, according to a former senior CIA official involved in the Stuxnet program. Armed with that intelligence, so-called network exploitation specialists then developed software implants known as beacons, which worked like surveillance drones, mapping out a blueprint of the network and then secretly communicating the data back to the NSA. (Flame, the complex piece of surveillance malware discovered by Russian cybersecurity experts last year, was likely one such beacon.) The surveillance drones worked brilliantly. The NSA was able to extract data about the Iranian networks, listen to and record conversations through computer microphones, even reach into the mobile phones of anyone within Bluetooth range of a compromised machine. (emphasis mine)

– from James Bamford for Wired, The Secret War

I posted last year about Stellar Wind, an NSA data-mining program and an upcoming film by Laura Poitras who helped break the Snowden story in Hong Kong. Collection of data is happening. They can listen to you from anywhere. Imagine how happy the NSA is to be able to tap into everyone’s Google Glass and get another set of eyes on the ground. It’s the dark side of the Internet of Things. Connected devices can be correlated across devices to learn more about you.

In fact, if you think about your mobile sensor platform, there’s a really cool little app – Activity Tracker. It’s a little Android app – have you guys seen this anywhere? What they’ve discovered is fundamentally they take your 3-axis accelerometer on your phone. . . What happens is, they discovered that just simply by looking at the data what they can find out is with pretty good accuracy what your gender is, whether you’re tall or you’re short, whether you’re heavy or light, but what’s really most intriguing is that you can be 100% guaranteed to be identified by simply your gait – how you walk.

CIA CTO Ira “Gus” Hunt on stage at GigaOM Structure:Data earlier this year

With enough data sitting around, you can know just about anything. The truth is coming out as we put together the pieces. Most those in the tech industry know this kind of stuff is possible but now congressional hearings are taking places and governmental minds are being blown.

The federal surveillance programs revealed in media reports are just “the tip of the iceberg,” a House Democrat said Wednesday.

Rep. Loretta Sanchez (D-Calif.) said lawmakers learned “significantly more” about the spy programs at the National Security Agency (NSA) during a briefing on Tuesday with counterterrorism officials.

“What we learned in there,” Sanchez said, “is significantly more than what is out in the media today.”

The Hill, June 12, 2013