Ghost in the Machine

There are at least two sides to every story. The Planned Parenthood videos were a polarizing topic that monopolized the news cycle several weeks ago. How do you teach an algorithm a point of view? How do you optimize for discovery and strike the right balance for diversity while avoiding duplication?

SmartNews is a news aggregation app driven by machine learning algorithms. The platform is tuned for discovery (as opposed to personalization). After using it regularly, I began collecting screenshots of my favorite examples when the app taught me something new or showed me two items side-by-side that suggested a subtle intelligence.

Two candidates and their technology.

The science and application of artificial intelligence to personalization is well understood. From Amazon’s people-that-bought-this-also-bought-that to Pandora’s Music Genome Project, software has been recommending what you’ll like next best based on what you’ve liked so far for years.

The new frontier in artificial intelligence is machine learning. Companies such as Spotify and Netflix are hard at work trying to predict future tastes based on an evolving understanding of collective tastes. Sure, learning assumes knowledge of the past, but projecting that learning into the future is much harder as you build a model based on an understanding of something that does not exist. Rather than showing you something we know you’ll like based on what you liked in the past, machine learning discovers things you didn’t know you would like.

First a little context. SmartNews, while deceptively simple, has a lot going on under the hood. At any time, the SmartNews app shows around 250 headlines across 8 categories. These headlines are selected from millions of stories that are scanned each day. In order to ensure that the stories featured in the app are the most important and interesting, a number of things must take place.

SmartNews Engine

After harvesting URLs, the text of each article is run through a classifier that examines things such as the headline, author byline, publication date, images and video embeds. These pieces are analyzed by a semantic engine that extracts data so the algorithm can map the article to a topic cluster and place it into the appropriate subject category. (I wrote about how this is done in an earlier post)

Importance estimation is where we rank an article and determine where it will go in the app relative to other articles. Does it go towards the top of a section or towards the bottom? If the top, does it deserve featured treatment? Maybe it’s so topical it needs to be pushed to the Top page, which is reserved for only the most important stories of the moment.

Finally, diversification ensures there is a good mix of stories in each category. If there are 40 stories about guacamole and peas, here’s where we determine which to show and which to push to the background. If there’s a new development on a story, the update will push its way in and take prominence over an older story.

These are just details to give you context. The most amazing thing to me is when the app surfaces a “hidden gem” that I would not normally run across if I were using an RSS reader hard-coded to a collection of feeds, or a social network that is limited to news shared by my friends.

The best way to appreciate SmartNews as a discovery engine is to use it daily, but if you haven’t had a chance, here are a few more of my favorite Gems below:

Planned-Parenthood

While the Center for Medical Progress’ undercover video interviews with Planned Parenthood staffers may have been shocking, the representation of two points of view helped me see both sides of the issue. What was interesting was the Cosmopolitan article (a source I normally do not read) had the best measured rebuttal.

alternate science

Much of the climate change news ends up in the Science category. As that story grows in relevance to us all, more publications dig into it. If you haven’t read this terrifying Rolling Stone piece, read it now.

Hulk Hogan

Here’s an example of a developing story getting an update. ESPN reports that WWE is cutting its relationship with Hulk Hogan his comments that were offensive. People Magazine follows up with the story of his apology. Oh, also notice that the algorithm put both stories into the Entertain section.

Lion Hunter

As news of the killing of Cecil the Lion went viral, the algorithm was smart enough to surface a side of the story from a local Minnesota paper.

too-many-guns

The screenshot above, more than any of the others, shows the freaky intelligence working behind the scenes. Like those times when an algorithmically generated playlist just nails the transition of one song into the next, drawing the causality between gun violence in the US to how such an environment might have prepared an off-duty soldier to do the right thing shows how a well-designed system can be greater than just the sum of its component parts.

Do you use SmartNews? Have you had the same experience? Send along some of your own Hidden Gems and I’ll add them to the gallery.

SmartNews Pro Tip: Save it for Later

smartnews_iconSmartNews is focused on today’s news. Because of this the app is optimized for showing you the most important stories of the moment. The idea is to get you up to speed on what’s going on and then on with your day. If we do our job well there, the thinking goes, you’ll be back.

That said, there are times when you’re glancing at the latest headlines and you run across a meaty profile in Vanity Fair or a lengthy speech transcript in Medium. I’ve seen comments in the App Store where people are looking for a way to save articles for later. There are a couple of options that I’d like to share.

Read it later with Pocket

SmartNews is integrated with Pocket. Create an account at Pocket or login with your existing one. When you share from the article page on SmartNews (another pro tip, a long press on any headline will go directly to the save menu), you have the option to Save to Pocket. Once you’ve saved it here you can go back to Pocket on the Web and read the full text of the article later. If you upgrade to Pocket Premium, they will even download, index, and archive the full text of anything you save to Pocket making later retrieval easier.

Hear it later with Pocket

Pocket TTS

Pocket recently added Text-to-Speech to their mobile app. I ride my bike to work so sometimes it’s better to have a long article read to me. This afternoon I listened to the transcript of Jennifer Granick’s excellent keynote at Black Hat 2015, The End of the Internet Dream which was posted on Medium.

It somehow seemed appropriate to have the same voice that speaks to me as Siri explaining how important it is to keep the internet open and decentralized.

Show more, is that an archive?

SmartNews Read More

Well, kinda. While we try as much as possible to keep things lightweight in the SmartNews app, we recognize that you might sometimes go more than several hours in between SmartNews fixes. We hear you. But if you’re hearing about that great story in the morning and it’s no longer there, we’ve got your back!

Scroll to the bottom of any tab other than Top and you’ll see a “Show more” link that will show you more articles in the channel. We can’t store everything but it’ll at least extend your horizon a few more hours if you want to dig in a little further.

Back when News was Physical

etaoin shrdlu are the first line of letters on a linotype keyboard, arranged based on frequency. The phrase is used to mark the end of a column. It is also the title of a short documentary about the last run of the linotype machines at the New York Times on July 2, 1978.

There are all sorts of wonderful details in this 30-minute film. We learn the origin of words such as hot type and mattress and are shown how a “pig” of lead is melted down to cast type forms.

The mechanical crank and whirl of the linotype machines are wonderful sound, especially when contrasted with the castanet-like crackle of the new chicklet keyboards on the the new mainframe terminals shown later in the film. As the 9pm first edition deadline approaches, the “make up men” hunch over their tables side-by-side with page editors physically laying out the paper on full page forms. There’s a wonderful exchange as they figure out how to make the page work, a construction project of words.

Farewell – ETAOIN SHRDLU – 1978 from Linotype: The Film

These days are long gone now but I’m glad for this film which captures a technology that was a wonder of its day.

h/t Open Culture

Recycled Gear Goes to Africa

One of my volunteer activities is to serve as the Webmaster (I love that retro-cool title) for the Alameda Soccer Club. It’s a pretty large organization serving the 1,300+ kids in our neighborhood, completely staffed by unpaid volunteers and it does a great job of getting the kids excited about the sport.

This Summer we received a request by an Alameda native who is serving in the Peace Corps in West Africa and was home for the holidays. Many kids in her village loved to play soccer but often didn’t have any equipment so she wondered if we could donate any used gear for her to take back to Africa upon her return.

I put the word out and Alameda responded. There was too much for her to take back with her so her father packed it up and shipped it off. The other day she sent back photos and in with all the photos was a one of a kid in West Africa trying out my son’s old cleats!