everwas

a blog by Ian Kennedy

Year: 2017

  • Reuters Tracer combs Twitter for news

    According to internal research, Reuters determined that 10-20% of news broke first on Twitter.

    Reuters, the news agency that first scooped its rivals with the use of carrier pigeons, is seeing good results from an algorithm to sift through Twitter (over 12 million tweets/day,  2% of total volume) to search for signal in the noise. Reuters Tracer is the system summarized in MIT’s Technology Review, How Reuters’s Revolutionary AI System Gathers Global News

    The first step in the process is to siphon the Twitter data stream. Tracer examines about 12 million tweets a day, 2 percent of the total. Half of these are sampled at random; the other half come from a list of Twitter accounts curated by Reuters’s human journalists. They include the accounts of other news organizations, significant companies, influential individuals, and so on.

    The next stage is to determine when a news event has occurred. Tracer does this by assuming that an event has occurred if several people start talking about it at once. So it uses a clustering algorithm to find these conversations.

    Of course, these clusters include spam, advertisements, ordinary chat, and so on. Only some of them refer to newsworthy events.

    So the next stage is to classify and prioritize the events. Tracer uses a number of algorithms to do this. The first identifies the topic of the conversation. It then compares this with a database of topics that the Reuters team has gathered from tweets produced by 31 official news accounts, such as @CNN, @BBCBreaking, and @nytimes as well as news aggregators like @BreakingNews.

    At this stage, the algorithm also determines the location of the event using a database of cities and location-based keywords.

    Once a conversation or rumor is potentially identified as news, an important consideration is its veracity. To determine this, Tracer looks for the source by identifying the earliest tweet in the conversation that mentions the topic and any sites it points to. It then consults a database listing known producers of fake news, such as the National Report, or satirical news sites such as The Onion.

    Finally, the system writes a headline and summary and distributes the news throughout the Reuters organization.

    Three recent events and their corresponding Tracer’s and Reuters alerts.

    More details (and attached screenshots) sourced from the paper, Reuters Tracer: Toward Automated News ProductionUsing Large Scale Social Media Data

  • The week that was

    Trying something new. A weekly wrap-up of sorts.

    The Austin headquarters for the Texas Health and Human Services Commission was overrun by “several hundred” rats.

    A Russian rocket launch went awry when the wrong directions were loaded into it’s flight plan and sent it to the bottom of the ocean.

    Meanwhile, the Voyager team dusted off the thruster controls to spacecraft now 21 billion kilometers away. After waiting 20 hours for the instructions to get there and waiting another 20 hours for the response, the damn thing worked!

    Finally, Mike Hughes’ rocket launch into the “atmosflat” where he planned to take photographs to prove the earth is flat and “expose the conspiracy” has been delayed until next week.

    Los Angeles’ free street paper, the LA Weekly, has a post asking if anyone can tell them who owns them. The next day, the paper answered it’s own question.

    And for another year, Kate Dries has still not made it into Hendrik Pöhler’s Erotic Carp Fishing Calendar. The Amazon reviewers are not pleased.

  • ProAm Used Car Listing

    This is what happens when your boyfriend is a professional director, and you want to sell your used ’96 Honda Accord.

  • The World We Live In

    The World We Live In

    Things have gotten so crazy and divisive lately that it’s now increasingly obvious that people are living in alternate realities. Our filter bubbles have become so completely isolating that they are impervious to commonly understood baseline facts such as science and reason. Some examples:

    And today Vox has a terrifying piece which imagines a world where even a well-documented case of malfeasance by the current administration might be rejected out of hand by a misinformed and untrusting public.

    That insular partisan far-right media is also full of nonsense like Pizzagate that leaves the base continuously pumped up — outraged, infuriated, terrified, and misled. At this point, as the stories above show, the conservative base will believe anything. And they are pissed about all of it.

    At least the stock market is still breaking new records. I shudder to think what will happen if the economy takes a turn for the worse.

  • Story behind iconic Steve Jobs portrait

    How do you get a portrait subject to relax so you can get their true essence? What if your subject is Steve Jobs? You know the photo – here’s the story behind how it was captured.

    the pair got to work and Watson began to approach the process like he was conducting a passport photoshoot; within minutes, Jobs was told to “slightly lean forward towards the camera” and give the expression he would give if he was at a table with a small group of people that didn’t agree with him, even though he knew he was right.

    – via Hypebeast

  • Carl Bernstein at ONA 17

    Carl Bernstein at ONA 17

    Recently, SmartNews (my employer) hosted Carl Bernstein, the distinguished, Pulitzer Prize-winning journalist, at the Watergate Hotel during the annual Online News Association conference in Washington, DC.

    Under a full moon on a warm DC evening, Mr. Bernstein inspired us all to strive harder to make use of the powerful “reportorial platform” the digital era provides us to seek out, “the best obtainable version of the truth.”

    Mr. Bernstein reminded us that it is not the job of the press to undo a candidate or knock a one out of office. The mission of the press is to uncover the truth, in context and developed through great investigative reporting so that the people may make the important decisions that drive our democracy.

    But lately the press has been,

    disfigured by celebrity, celebrity worship, gossip, sensationalism, manufactured controversy (particularly in the press), denial of our society’s real conditions (good and bad), and by a political and social discourse that we (the press) the politicians, and the people have allowed to devolve into a cacophony of name-calling, idealogical warfare and, especially, easy answers to tough questions

    He pleaded for all journalists to keep their focus on the larger story. Even with the new tools available today, it is the unique perspective and context that comes from knocking on doors and “shoe leather” reporting that will unearth the truth required by our democracy. To gather that truth we all needed to be better listeners.

    His remarks concluded with a quote from his editor at the Washington Post, the late Ben Bradlee.

    The more aggressive our search for the truth, the more some people are offended by the press. So be it. I take great strength knowing that in my experience the truth does emerge. It takes forever sometimes but it does emerge. Any relaxation by the press will be extremely costly to democracy.

    Amen to that.

  • Perspective

    When your job brings you in regular contact with tragedy on a mass scale (as it does when you work in a newsroom) the rush & tumble of getting the news out gets in the way of stopping to feel the personal impact of these events. As I’m certain will happen with the Napa/Sonoma fire situation, I am now reading personal stories out of Las Vegas.

    This from the NY Times’ Reporter’s Notebook hit me hard.

    I checked out of that Mandalay Bay suite on Saturday morning, excused from reporting duties, and flew home in the hopes of making my daughter’s soccer game. I found the red rose from the vigil, starting to fade and wilt, in a vase on the kitchen counter. When we got to the game, we and the other parents were somewhat surprised to see Stacee’s husband and extended family there, too. Warming up with the girls was No. 8, with her long ponytail.

    We all wore orange ribbons, attached by safety pins, including the girls on both teams. The Novato team wore orange armbands with the initials “S.E.” Before kickoff, both squads came across the field to the spectator side and lined up in straight lines. Our team’s coach asked the parents to stand for 30 seconds of silence. And then two of the league’s better teams played a rather meaningless soccer game, only this one felt about as meaningful as anything I’ve ever watched.

    And it was late in the second half when the ball suddenly swung from one end to the other, and Stacee’s daughter gave chase through three retreating opponents and beat them all to the ball. And in one blink-and-you-missed-it moment, she booted the ball into the corner of the net for what held on as the winning goal.

    Her teammates chased her and swarmed her, and they and she looked as free and happy as girls can be on a sunny fall Saturday afternoon with their friends. The parents jumped and cheered as loudly as I’ve heard parents cheer at a kids’ soccer game. Behind my sunglasses, I was bawling. It was the first time I’d cried all week.

    – The Girl in the No. 8 Jersey

  • Preventing Harry Tuttle

    Preventing Harry Tuttle

    My last post, Democracy’s Soft Underbelly warned how algorithms for content distribution and advertising have been weaponized to alter public opinion.

    The most disturbing aspect of the affair is there is no public evidence of the ad campaigns so there is literally nothing to talk about. When there is no public record of what ad creatives were used, there can be no debate over the of the veracity of the claims made or appropriateness of the imagery used. This recently changed as Facebook has now turned over evidence (most likely because of a warrant) to Special Counsel Robert Mueller who is looking into Russian hacking of the election.

    Mueller’s investigation has received copies of the Russian-bought ads and details about the specific account information and targeting criteria the buyers used to distribute their ads, according to the (Wall Street) Journal, citing people familiar with the matter. TechCrunch

    “Democracy dies in darkness” is the new Washington Post tagline. The only way to have an open civic debate about the new threat to public discourse posed by advertising on social media platforms is to review the evidence. There needs to be transparency about how advertisers are paying their way into the hearts and minds of their target audiences. With transparency comes accountability.

    If you’re going to target me, I deserve to know.  All ads and ad copy must be stored, shared, and freely available to anybody in a searchable database. So if I want to see ads targeted at African-Americans in a swing state, I can just type “African-Americans aged 35-60 with a college degrees” and show me all the ads that were done. – Jason Calcanis, This Week in Startups

    This database of ads, freely available to the public, would expose any questionable or unethical activity by advertisers and hopefully prevent heavy-handed regulation of this industry which has been such an important driver of economic growth.

    A democratic society evolves through transparency and public debate. Advertisers that operate in the open will be forced to self-regulate themselves in full view of public opinion. Without public exposure, advertising standards and interpretation of these standards will remain behind closed doors.  In such a world, reform will come through an opaque and inevitably bureaucratic regulations that will stifle innovation.

    Terry Gilliam’s 1985 film  Brazil cast a satirical eye on a world  where bureaucracy and regulation have run amok. Nothing can be done without the appropriate paperwork. Harry Tuttle (played by Robert De Niro) is the terrorist heating engineer is our hero of common sense.

    Have you got a 27B/6?

    * It’s been pointed out that the title of this post is misleading. We do not want to “prevent” Harry Tuttle, he’s the hero and one of my favorite characters. We want to avoid being “Brazilled” – Thanks Stephen Davidson!

  • Democracy’s Soft Underbelly

    Democracy’s Soft Underbelly

    Let me explain.

    It’s now well-documented that outside forces took advantage of social media platforms to spread rumors in order to swing the 2016 presidential election. Journalists digging into the story are looking more closely at the tools used to purchase advertising that helped amplify these rumors, and are horrified by what they are discovering.

    Last week Pro Publica discovered you can target “jew haters” and BuzzFeed News found that on Google you could target phrases such as “blacks ruin everything”

    In the days of print, each advertisement was reviewed by multiple people from both the organization that bought the ad and the publication that ran it. Extreme care was taken to make sure the advertising complemented the editorial and the message was the right fit for the audience, not only to maximize effectiveness but also to avoid instances such as the one below.

    Despite careful review, print ad placements sometimes backfire

    Online advertising is a delicate balance between scale and quality. The dream is to serve a perfectly targeted ad to as many people as possible. But because of the scale, it is impossible to manually review each and every ad creative for quality and fit. In the online world, people “optimize” and let the algorithms do the work.

    While at Yahoo, I met with an advertiser who wanted to learn about our behavioral targeting options. I was working with a team that was thinking about exposing detailed facets of the massive Yahoo audience that would help advertisers reach very specific segments. When I walked into the room, the client had a spreadsheet he was using to allocate his million-dollar budget. After asking a few questions about his goals, I proposed a few very targeted criteria to build his target audience. Unfortunately, he grew frustrated because the total audience was too small and we were going to have to run hundreds of queries to build up the reach he needed. He didn’t have the time to continue the exercise nor appetite to keep track of all the data to show ROI to his client. The meeting wrapped up with four very broad buckets into which he poured roughly $250k each and called it a day.

    He couldn’t be bothered with the details.

    This is the state of online advertising today. The tools available to reach massive scale are even more sophisticated but to do it right, with quality, requires manual oversight. Ad units can be configured to dynamically swap out ad copy and assets depending on the target audience, which can also be built algorithmically. Ad spend adjusts automatically and APIs monitor trending keywords to take early advantage of trending topics and get broad reach on the cheap.

    “Programmatic Advertising” is a blanket term for techniques used to automatically generate thousands of “personalized” ads at massive scale. Because it’s automated, generating ad copy variants and target segments is inexpensive. The downside is that quality suffers if you take out the human element, leaving the robots to mind the store.

    This slideshow requires JavaScript.

    There are many examples that show us that resulting matches still need regular review. As long as ad matching algorithms such as Google’s and Facebook’s remain black boxes, a regular human review is necessary to prevent the unexpected.

    Which brings me back to what we’re learning today. Last week Facebook shared that ad placements made to “amplify divisive messages” were used to influence the 2016 elections. We are slowly uncovering the extent of information warfare that uses social media platforms to weaponize fake news. Using programmatic advertising to draw attention to and amplify these campaigns is a natural extension.

    More careful review of editorial content posted to social networks is important to verify facts and prevent the spread of “fake news.” On the flip side, advertising platforms need review as well because ad targeting is also rife with repugnant audience segments automatically suggested by the algorithms. As they should, Facebook and Google have both said that more rigorous review is on the way but there will always be the tension of profit motives to discourage too rigorous a lens.

    In the late-90’s, the movie Wag the Dog spins a tale of how a Washington “spin doctor” (Robert De Niro) hires top Hollywood producer (Dustin Hoffman) to engineer public opinion. The film made light of the gullible public but there was a broader, cynical message about how media (and the press) can be used to manipulate public opinion.

    Today we are seeing this same scenario played out, but instead of manipulating public opinion through TV and Hollywood, public opinion is bought and sold using social networks and online advertising.