Ghost in the Machine

There are at least two sides to every story. The Planned Parenthood videos were a polarizing topic that monopolized the news cycle several weeks ago. How do you teach an algorithm a point of view? How do you optimize for discovery and strike the right balance for diversity while avoiding duplication?

SmartNews is a news aggregation app driven by machine learning algorithms. The platform is tuned for discovery (as opposed to personalization). After using it regularly, I began collecting screenshots of my favorite examples when the app taught me something new or showed me two items side-by-side that suggested a subtle intelligence.

Two candidates and their technology.

The science and application of artificial intelligence to personalization is well understood. From Amazon’s people-that-bought-this-also-bought-that to Pandora’s Music Genome Project, software has been recommending what you’ll like next best based on what you’ve liked so far for years.

The new frontier in artificial intelligence is machine learning. Companies such as Spotify and Netflix are hard at work trying to predict future tastes based on an evolving understanding of collective tastes. Sure, learning assumes knowledge of the past, but projecting that learning into the future is much harder as you build a model based on an understanding of something that does not exist. Rather than showing you something we know you’ll like based on what you liked in the past, machine learning discovers things you didn’t know you would like.

First a little context. SmartNews, while deceptively simple, has a lot going on under the hood. At any time, the SmartNews app shows around 250 headlines across 8 categories. These headlines are selected from millions of stories that are scanned each day. In order to ensure that the stories featured in the app are the most important and interesting, a number of things must take place.

SmartNews Engine

After harvesting URLs, the text of each article is run through a classifier that examines things such as the headline, author byline, publication date, images and video embeds. These pieces are analyzed by a semantic engine that extracts data so the algorithm can map the article to a topic cluster and place it into the appropriate subject category. (I wrote about how this is done in an earlier post)

Importance estimation is where we rank an article and determine where it will go in the app relative to other articles. Does it go towards the top of a section or towards the bottom? If the top, does it deserve featured treatment? Maybe it’s so topical it needs to be pushed to the Top page, which is reserved for only the most important stories of the moment.

Finally, diversification ensures there is a good mix of stories in each category. If there are 40 stories about guacamole and peas, here’s where we determine which to show and which to push to the background. If there’s a new development on a story, the update will push its way in and take prominence over an older story.

These are just details to give you context. The most amazing thing to me is when the app surfaces a “hidden gem” that I would not normally run across if I were using an RSS reader hard-coded to a collection of feeds, or a social network that is limited to news shared by my friends.

The best way to appreciate SmartNews as a discovery engine is to use it daily, but if you haven’t had a chance, here are a few more of my favorite Gems below:


While the Center for Medical Progress’ undercover video interviews with Planned Parenthood staffers may have been shocking, the representation of two points of view helped me see both sides of the issue. What was interesting was the Cosmopolitan article (a source I normally do not read) had the best measured rebuttal.

alternate science

Much of the climate change news ends up in the Science category. As that story grows in relevance to us all, more publications dig into it. If you haven’t read this terrifying Rolling Stone piece, read it now.

Hulk Hogan

Here’s an example of a developing story getting an update. ESPN reports that WWE is cutting its relationship with Hulk Hogan his comments that were offensive. People Magazine follows up with the story of his apology. Oh, also notice that the algorithm put both stories into the Entertain section.

Lion Hunter

As news of the killing of Cecil the Lion went viral, the algorithm was smart enough to surface a side of the story from a local Minnesota paper.


The screenshot above, more than any of the others, shows the freaky intelligence working behind the scenes. Like those times when an algorithmically generated playlist just nails the transition of one song into the next, drawing the causality between gun violence in the US to how such an environment might have prepared an off-duty soldier to do the right thing shows how a well-designed system can be greater than just the sum of its component parts.

Do you use SmartNews? Have you had the same experience? Send along some of your own Hidden Gems and I’ll add them to the gallery.

SmartNews Pro Tip: Save it for Later

smartnews_iconSmartNews is focused on today’s news. Because of this the app is optimized for showing you the most important stories of the moment. The idea is to get you up to speed on what’s going on and then on with your day. If we do our job well there, the thinking goes, you’ll be back.

That said, there are times when you’re glancing at the latest headlines and you run across a meaty profile in Vanity Fair or a lengthy speech transcript in Medium. I’ve seen comments in the App Store where people are looking for a way to save articles for later. There are a couple of options that I’d like to share.

Read it later with Pocket

SmartNews is integrated with Pocket. Create an account at Pocket or login with your existing one. When you share from the article page on SmartNews (another pro tip, a long press on any headline will go directly to the save menu), you have the option to Save to Pocket. Once you’ve saved it here you can go back to Pocket on the Web and read the full text of the article later. If you upgrade to Pocket Premium, they will even download, index, and archive the full text of anything you save to Pocket making later retrieval easier.

Hear it later with Pocket

Pocket TTS

Pocket recently added Text-to-Speech to their mobile app. I ride my bike to work so sometimes it’s better to have a long article read to me. This afternoon I listened to the transcript of Jennifer Granick’s excellent keynote at Black Hat 2015, The End of the Internet Dream which was posted on Medium.

It somehow seemed appropriate to have the same voice that speaks to me as Siri explaining how important it is to keep the internet open and decentralized.

Show more, is that an archive?

SmartNews Read More

Well, kinda. While we try as much as possible to keep things lightweight in the SmartNews app, we recognize that you might sometimes go more than several hours in between SmartNews fixes. We hear you. But if you’re hearing about that great story in the morning and it’s no longer there, we’ve got your back!

Scroll to the bottom of any tab other than Top and you’ll see a “Show more” link that will show you more articles in the channel. We can’t store everything but it’ll at least extend your horizon a few more hours if you want to dig in a little further.

Back when News was Physical

etaoin shrdlu are the first line of letters on a linotype keyboard, arranged based on frequency. The phrase is used to mark the end of a column. It is also the title of a short documentary about the last run of the linotype machines at the New York Times on July 2, 1978.

There are all sorts of wonderful details in this 30-minute film. We learn the origin of words such as hot type and mattress and are shown how a “pig” of lead is melted down to cast type forms.

The mechanical crank and whirl of the linotype machines are wonderful sound, especially when contrasted with the castanet-like crackle of the new chicklet keyboards on the the new mainframe terminals shown later in the film. As the 9pm first edition deadline approaches, the “make up men” hunch over their tables side-by-side with page editors physically laying out the paper on full page forms. There’s a wonderful exchange as they figure out how to make the page work, a construction project of words.

Farewell – ETAOIN SHRDLU – 1978 from Linotype: The Film

These days are long gone now but I’m glad for this film which captures a technology that was a wonder of its day.

h/t Open Culture

Recycled Gear Goes to Africa

One of my volunteer activities is to serve as the Webmaster (I love that retro-cool title) for the Alameda Soccer Club. It’s a pretty large organization serving the 1,300+ kids in our neighborhood, completely staffed by unpaid volunteers and it does a great job of getting the kids excited about the sport.

This Summer we received a request by an Alameda native who is serving in the Peace Corps in West Africa and was home for the holidays. Many kids in her village loved to play soccer but often didn’t have any equipment so she wondered if we could donate any used gear for her to take back to Africa upon her return.

I put the word out and Alameda responded. There was too much for her to take back with her so her father packed it up and shipped it off. The other day she sent back photos and in with all the photos was a one of a kid in West Africa trying out my son’s old cleats!

Hiroshima, 70 years on

Last year, on July 4th, I visited Hiroshima with Izumi and the kids to see first-hand the city and the memorial.

Hiroshima (literally means flat island) was one of two sites bombed by atomic weapons. It was chosen because it was relatively unscathed by previous bombings and military scientists wanted to measure the effectiveness of an aerial detonation. Nagasaki (long cape), the other target, was on different terrain and that bomb was detonated on the ground. In each case, military personnel were sent to each city to carefully measure the destruction they had wrought.

The lone standing building survived the blast because it occurred directly overhead by several hundred meters.

Hiroshima Aerial Detination

The museum in Peace Park has many artifacts from August 6, 1945 including section from a stone bridge that still has the shadow of a person that was literally vaporized by the blast. There are many personal stories told by survivors posted next to items salvaged from the wreckage. How one man saw his hand melt off because it was on a windowsill and a schoolgirl who was home sick from school and was the only one from her entire class to survive.

There is also a stopwatch frozen at the exact moment the bomb went off.

Hiroshima Pocket Watch

What really struck me were some of the documents preserved which captured the debate on the US side over the deployment of this terrible weapon and presaged the arms race that would follow. There is a letter from Albert Einstein to FDR, minutes from a meeting with some of the contractors that worked on the bomb (General Electric, duPont), and declassified Top Secret notes from the Target Committee.

But most disturbing was an excerpt from the Franck Report from June 11, 1945 written by some of the scientists involved with the Manhattan Project that built the bomb. In it they plead with military leadership to consider demonstrating the power of the atomic bomb on an uninhabited island as a way to coerce Japan into surrender.

Franck Report

Highlighted above is a passage that would come to haunt our world for many years hence.

We believe that these considerations make the use of nuclear bombs for an early, unannounced attack against Japan inadvisable. If the United States would be the first to release this new means of indiscriminate destruction upon mankind, she would sacrifice public support throughout the world, precipitate the race of armaments, and prejudice the possibility of reaching an international agreement on the future control of such weapons.

Something to think about as we debate our current negotiations with Iran.

Further Reading:

Italian Foo Fighters Fans Wishes Granted

1,000 Foo Fighters fans gathered in a field in Cesena, Italy to play Learn to Fly to get the band’s attention and to listen to their plea to play a concert in Italy. Maybe you’ve seen the video which was all over the ‘nets. It’s pretty awesome.

The band noticed, later posting Ci vediamo a presto, Cesena…. xxx Davide from their twitter account. Google Translate tells me it says, “See you soon, Cesena”

Today Dave Grohl, the band’s leader posted this video response in Italian.

In awe.