SmartNews Hidden Gems from 2015

We all remember the biggest stories of 2015, El Chapo’s escape, Ronda Rousey’s KO, and who can forget The Dress? In the spirit of discovery, we at SmartNews would like to highlight the stories that you might have missed. Following on the hidden gems theme, I took a look at each of the SmartNews categories and looked for the outliers. My somewhat unscientific methodology looked for stories from sources that would not normally appear in the category but were picked up and featured based on a topic analysis, hopefully introducing a source to a new audience that would not normally be exposed to that publication.

cats-in-box

GQ describes itself as a men’s fashion and style magazine. When Marshall Sella tests the Bitcoin waters, SmartNews puts his piece in front of the Business readers. Marshall describes his time with Charlie Shrem, an early Bitcoin entrepreneur (bitrepreneur?) whose LinkedIn profile now shows him cooling his heels at Lewisburg Federal Prison.

It’s not often that Scientific American shows up in the Entertainment section. Cindi May’s The Problem with Female Superheroes took a look at how characters such as Storm and Dazzler in the recent X-Men films may be adversely affecting the young audiences who watch them. “Saving the world in spiked heels” may not be giving young girls a realistic expectation of their abilities. We hope the upcoming Dawn of Justice does a better job.

We all cringed when we saw the video of the 12-year-old boy who tripped and punched a hole in the 350-year-old painting valued at $1.5 million. Oliver Holms of The Guardian covers the restoration effort (thankfully it was insured) and points to other mishaps such as when a pair of Qing dynasty Chinese vases and a Picasso did not fare as well. SmartNews placed this one in the Lifestyle section which is where our Art & Culture are featured.

“I thought it was a CIA surveillance device,” said Brett McBay in Modesto, California after instructing his son to shoot his neighbor’s drone from out of the sky with a 12-gauge shotgun. Cyrus Farivar at Ars Technica brought up a number of issues including the right to privacy (the skies around your home) and the respect for private property, (Eric Joe’s homemade hexacopter drone), and of course the right to fire off buckshot into the sky. SmartNews to placed this story into the US category where much of our gun violence stories have been running. Inquiring minds want to know if this Brett McBay of Modesto is the same Brett McBay whose twitter profile states he is the District Representative for a California State Assemblymember.

SB Nation covers sports and, yes, there is a basketball in this bit but it’s used to explain the Magnus Effect from physics and, for that reason, this article showed up in Science.

The Nation likes to dig (and sometimes poke) which usually lands them in the US section for political coverage. Back in May, Dave Zirin asked why mainstream sports sites were not covering the case of NBA player Thabo Sefolosha, who was tackled by NYPD outside a nightclub, injured, and subsequently missed the playoffs with his team. This story introduced Sports readers to The Nation style of media inquiry. Seven months later ESPN published an in-depth investigative piece on this same story.

Finally, in Technology, there’s this fun piece blowing the cover on arcade claw machines. Did you know that they were rigged? Phil Edwards at Vox posts PDFs from the operator manuals to prove his point and caused great consternation in the gaming industry.

We hope you had a great 2015 and learned something new. Stay curious and see you in 2016.

Cracking the Code of Local News

In his opening remarks, Rich Jaroslovsky, VP for Content and Chief Journalist at SmartNews, said that Local News is something that everyone has tried but, “nobody has fully cracked the code.” Backfence, Topix, Patch, and countless other valiant efforts have taken a run at it but failed to sustain. “Local News is still a huge opportunity because of the intensity for what people want,” Rich continued. Sharing a datapoint from SmartNews, we see almost double the CTRs on our local channels (the SmartNews app has 23 existing with several more in development).

With this context, SmartNews and Pocket hosted the first New News meetup. The founding principal is to discuss the evolution or tell the story about a particular aspect of the News business. The inaugural meetup held last week was about Local News, its opportunities and challenges.

the new news meetup

Sponsors from Say Bubble and Townsquared generously supplied the food and drinks. The panel of guests included:

the new news local news pannel
From left, Lila LaHood of SF Public Press, Jess McCuan of SF Chronicle, Sunil Rajamaran (speaking) of The Bold Italic, and Eric Eldon of Hoodline

Each panelist presented a distinct business model for local news.

After writing for the tech press (VentureBeat, TechCrunch) Eric joined Hoodline where he is the Editor-in-Chief. Hoodline has built their own CMS and is using this technology to differentiate themselves. They have soft-launched an events listing service and will be using hyper-focused local reporting (i.e. Change is afoot at Rising Star Laundromat) to draw attention to those listings.

Sunil found out from a Facebook comment thread that The Bold Italic was abruptly shutting down last April. Someone from TBL reached out to him and within 8 weeks he made an offer that was accepted by the previous owners, Gannett. Sunil, a co-founder of Scripted.com, a freelance writer marketplace, took on The Bold Italic as  new project. The challenge is to maintain high quality content at “1/25th the budget” which he aims to do by focusing on “evergreen” pieces (i.e. An Open Letter to Anyone Moving to San Francisco for a Tech Job) and leaving the resource-heavy “news” stories to other sites. Sunil comes across as very business-savy and he has brought the site back up to a half-million pageviews/month with just 3 editors on staff. If he can source writers from his freelance community at a price that makes economic sense, he could come up with a model that can be replicated in other metro areas.

Jess stated clearly that the 150-year old SF Chronicle, “is not a start-up” but with it’s deep archive and extensive resources plays to its strengths. Jess’ team of 10 reporters, 2 editors and dedicated copy-editors & designers cover tech in the Bay Area and balance the spark and flash of the beat with the sober need to inform the public. Not many people know that SF Gate and SF Chronicle are not only two different sites but also two different organizations with different missions. “If you want to read about Justin Bieber’s butt, come to SF Gate, it’s the candy.” SF Chronicle is the premium content site, “the medicine” where they feature high-quality journalism about money, politics, tech, and other important forces changing the landscape of the Bay Area.

Lila describes the non-profit SF Public Press as an online & print version of NPR. They sell memberships to their quarterly newspaper but also raise funding from grants and donors. They are admittedly “slow news” and focus on big investigative reports which take many weeks to put together pulling data from FOIA document requests, interviews, and data analysis to create meaty articles featuring beautiful visualizations and maps (i.e. Building on the Bay, Sea Level Rise Threatens Waterfront Development). As a non-profit they need to work with like-minded organizations such as the local public radio stations, Mother Jones magazine, Bay Nature, Earth Island Institute, and the Center for Investigative Reporting to amplify their stories and get the word out about their stories.

the new news meetup - local news

Geography defining communities

A member of the audience wanted to know how each site covered issues which cut across geographic boundaries. Hoodline has a “density formula” to determine which neighborhoods to cover but Eric was quick to note that when it came to certain issues such as gentrification it was important to maintain objectivity to build trust.

Sunil noted the monetary challenge of local which limits The Bold Italic’s attractiveness to national advertisers. Jess touted the Chronicle’s deep archive of coverage of big national stories such as the LGBT community which helped them put together an impressive package of historical pieces for their Marriage Equality Act issue which went on to become one of their best-selling issues.

The focus on a pieces that resonate with a local audience but tell a good story about an important national trend is what makes Jess most proud. The Chronicle has ambitions to be a national paper and she is aware that, in many aspects, the world is paying disproportionate attention to the Bay Area as a leading indicator of national trends that are impacting the rest of the United States. Tech touches on important issues regarding privacy, security, First Amendment rights, and the sharing economy. Many of the global players are here.

When a new batch of census data came out showing the rise in median income from 2009 to 2014 there were numerous local angles to the story. The piece that struck home was a piece by local reporter Joaquin Palomino about a local bakery in the Mission that had been selling the local community empanadas but recently switched to vegan carrot cake and organic coffee in order to stay in business.

Events or No?

Because media companies are in the business of aggregating audiences and connecting them with their sponsors, events are a logical way to bring together their audience in real life. Events were an important part of The Bold Italic’s previous business model and were the #1 source of revenue. In reality, Sunil admitted, the margins were “razor thin” so, going forward, events will not be part of “the new iteration” of TBL.

Eric from Hoodline also has experience with events in his previous life at VentureBeat and TechCrunch where large events such as The Crunchies and Disrupt were important but ultimately a “distraction” from the reporting and building product.

Jess stated emphatically that the SF Chronicle, “is not in the events business” but did share a preview of an upcoming documentary film they are producing about men living with HIV that will have a opening night event at the Castro Theatre.

By contrast, the SF Public Press looks at events as an important fundraising activity and has the occasional open house and has hosted hackathons such as the recent Hack the Housing Crisis conference to engage with the local community to address a specific concern.

Is Local Profitable?

To everyone’s surprise the local papers are turning a profit. The Bay Area News Group, with 14 daily and 32 weekly newspapers, is an important targeting vehicle for local business.

Showing that he’s well aware of what he’s getting into with reviving The Bold Italic, Sunil said that you need to build a site that is profitable from the first pageview. “ADD is the challenge,” you cannot rely on your readers for that elusive second click and certainly should not predict revenues based on that. Social platforms such as Pinterest have added “buy” buttons in attempts to monetize the attention they are getting but with mixed results as their audiences have not yet migrated their intentions when they visit the site from browsing to buying. The big ticket acquisitions of Bleacher Report and HuffPo from the last up cycle show that there is an appetite for media acquisitions but he is patient to wait out this current downturn and is prepared to patiently build up a sustainable business for the long term.

The SF Public Press is having fun with experimentation. Because they have less to lose, they have a larger appetite for risk. “Constraints inspire creativity but more resources would be better!”

The SF Chronicle knows they will always be outgunned by the national tech press in sheer resources but celebrate their victories which effect real change. Jess shared the story of “soft-spoken Wendy Lee” who worked on a story about bus drivers for Apple, eBay, and Genetech who were forced to sleep in their minivans because they didn’t earn enough to live in the Bay Area. The front page story was a wake up call to the PR departments of the high tech giants “scared to death” of it becoming a national story and quickly turned things around and gave their drivers raises.

Strong local stories draw attention and the paper is cashing in on its readership. Poytner recently reported that the San Francisco Chronicle is profitable thanks to an in-house ad agency that provides, “Madison Avenue-type marketing consultancy and advertising agency services at a palatable price for a large regional brand.”

the new news meetup crowd

Who else is doing good work?

Each of the panelists shared other companies and organizations doing good work.

  • DNA Info for New York and Chicago local news.
  • Corner Media Group for their coverage of  Brooklyn.
  • Institute for Non-profit News for their support of investigative reporting.
  • LION (Local Independent Online News publishers) is a great resource for local news providers to meet each other.
  • ONA (Online News Association) is another organization with many local chapters.
  • The Atlantic was highlighted for their top notch journalism and Quartz was mentioned as having mobile particularly well done.
  • ProPublic is also doing great investigative journalism.
  • Sunil pointed to Mic as brilliantly designed to draw you deeper into the site and view more pages. He went on to note that while Buzzfeed does great with its viral content, it’s news vertical is not getting traction.

Other Coverage:

Recommended from SmartNews

SmartNews has built a sophisticated duplicate content filter so that when the latest press release from a presidential candidate, disaster, crime, or culinary sensation hits the proverbial viral loop that breaking news from multiple outlets does not overwhelm the app and crowd out other stories. SmartNews strives to promote only the best unique stories to our readers.

But there are times when you want to dig deeper on an issue or read an alternative take. Introducing the Recommended widget.

You can find the Recommended widget at the bottom of the SmartView of any article in the SmartNews app. Swipe left on any article to get to the simplified SmartView of that article. Scroll down to the bottom to get to the Recommended widget.

Take, for example, the recent Rolling Stone story about Ringo Starr auctioning off his personal copy of The Beatles’ White Album.

smartnews-recommended-widget

The first three headlines (in purple) are from the publisher of the original piece (Rolling Stone). If the publisher has continued coverage of the story, you’ll see past stories about the topic giving you deeper context around what you just read. In this example there is a link to an earlier story about the auction followed by an interview with Ringo Starr and then a piece about The Beatles and their album Rubber Soul.

The bottom two headlines (in green) are culled from our daily crawl of 10 million+ headlines and matched entirely based on a custom SmartNews algorithm. Here we see two other stories about the Ringo Starr auction, one from The Guardian and the second from NME.

How do we do it? That sophisticated de-dupe filter we built to reduce articles that are too much alike? Turn it around and it makes a fantastic related articles algorithm!

Each article is automatically “read” and key terms, companies, people, and other entities are extracted along with data around the author, publisher, length of the piece and many other factors are used to make a data representation of the article. When two representations overlap significantly we give them a similarity score. The higher the score, the more similar the two articles are for the purposes of filtering or recommending.

I like to think of the Recommended widget as a jumping off point for further exploration. Headlines 1-3 go deeper into the past with a specific source while headlines 4 & 5 go broader along the same topic but across different publications. Choose your adventure.

The similar articles feature is not new. I use a WordPress plugin on this blog to power the Related box you see below each post. Most news sites have something similar, usually driven by keyword or tag matching, against a limited content set. SmartNews has a more sophisticated matching algorithm across a much broader universe of articles and I think you’ll notice the difference right away.

Download SmartNews have fun exploring and let me know what you think!

Because. . .Baseball

Post Season Cubs

It’s not often that the Cubs make it to the post season so when they do, something’s gotta give. University of Illinois at Urbana-Champaign student Ben Larson asks his professor if he could take a make up exam and the professor’s response is priceless.

via twitter

Ghost in the Machine

There are at least two sides to every story. The Planned Parenthood videos were a polarizing topic that monopolized the news cycle several weeks ago. How do you teach an algorithm a point of view? How do you optimize for discovery and strike the right balance for diversity while avoiding duplication?

SmartNews is a news aggregation app driven by machine learning algorithms. The platform is tuned for discovery (as opposed to personalization). After using it regularly, I began collecting screenshots of my favorite examples when the app taught me something new or showed me two items side-by-side that suggested a subtle intelligence.

Two candidates and their technology.

The science and application of artificial intelligence to personalization is well understood. From Amazon’s people-that-bought-this-also-bought-that to Pandora’s Music Genome Project, software has been recommending what you’ll like next best based on what you’ve liked so far for years.

The new frontier in artificial intelligence is machine learning. Companies such as Spotify and Netflix are hard at work trying to predict future tastes based on an evolving understanding of collective tastes. Sure, learning assumes knowledge of the past, but projecting that learning into the future is much harder as you build a model based on an understanding of something that does not exist. Rather than showing you something we know you’ll like based on what you liked in the past, machine learning discovers things you didn’t know you would like.

First a little context. SmartNews, while deceptively simple, has a lot going on under the hood. At any time, the SmartNews app shows around 250 headlines across 8 categories. These headlines are selected from millions of stories that are scanned each day. In order to ensure that the stories featured in the app are the most important and interesting, a number of things must take place.

SmartNews Engine

After harvesting URLs, the text of each article is run through a classifier that examines things such as the headline, author byline, publication date, images and video embeds. These pieces are analyzed by a semantic engine that extracts data so the algorithm can map the article to a topic cluster and place it into the appropriate subject category. (I wrote about how this is done in an earlier post)

Importance estimation is where we rank an article and determine where it will go in the app relative to other articles. Does it go towards the top of a section or towards the bottom? If the top, does it deserve featured treatment? Maybe it’s so topical it needs to be pushed to the Top page, which is reserved for only the most important stories of the moment.

Finally, diversification ensures there is a good mix of stories in each category. If there are 40 stories about guacamole and peas, here’s where we determine which to show and which to push to the background. If there’s a new development on a story, the update will push its way in and take prominence over an older story.

These are just details to give you context. The most amazing thing to me is when the app surfaces a “hidden gem” that I would not normally run across if I were using an RSS reader hard-coded to a collection of feeds, or a social network that is limited to news shared by my friends.

The best way to appreciate SmartNews as a discovery engine is to use it daily, but if you haven’t had a chance, here are a few more of my favorite Gems below:

Planned-Parenthood

While the Center for Medical Progress’ undercover video interviews with Planned Parenthood staffers may have been shocking, the representation of two points of view helped me see both sides of the issue. What was interesting was the Cosmopolitan article (a source I normally do not read) had the best measured rebuttal.

alternate science

Much of the climate change news ends up in the Science category. As that story grows in relevance to us all, more publications dig into it. If you haven’t read this terrifying Rolling Stone piece, read it now.

Hulk Hogan

Here’s an example of a developing story getting an update. ESPN reports that WWE is cutting its relationship with Hulk Hogan his comments that were offensive. People Magazine follows up with the story of his apology. Oh, also notice that the algorithm put both stories into the Entertain section.

Lion Hunter

As news of the killing of Cecil the Lion went viral, the algorithm was smart enough to surface a side of the story from a local Minnesota paper.

too-many-guns

The screenshot above, more than any of the others, shows the freaky intelligence working behind the scenes. Like those times when an algorithmically generated playlist just nails the transition of one song into the next, drawing the causality between gun violence in the US to how such an environment might have prepared an off-duty soldier to do the right thing shows how a well-designed system can be greater than just the sum of its component parts.

Do you use SmartNews? Have you had the same experience? Send along some of your own Hidden Gems and I’ll add them to the gallery.