Spotify’s Mixtape Algorithm

mixtapes

With the launch of Apple Music’s “For You” feature, Spotify hand has been forced to unveil it’s own personalization engine in response. Discover Weekly was launched today via a series of well-timed pieces published today across the tech press. The PR push is on to explain to everyone currently evaluating Apple Music on a 3-month trial.

Spotify describes Discover Weekly as, “like having your best friend make you a personalised mixtape every single week.” More specifically,  “Updated every Monday morning, Discover Weekly brings you two hours of custom-made music recommendations, tailored specifically to you and delivered as a unique Spotify playlist.”

Spotify, to date, has relied mostly on the social sharing of tracks and manually curated playlists (more than 2 billion!) to enhance the experience of the Spotify subscriber. The coverage today highlights the contribution of Echo Nest, an music intelligence and data platform acquired by Spotify in March of 2014. Reading a number of posts we learn the following:

Inside Spotify’s Hunt for the Perfect Playlist – Wired

Spotify’s internal tool that they use to build playlists has the wonderful moniker, Truffle Pig.

and,

The Echo Nest’s job within Spotify is to endlessly categorize and organize tracks. The team applies a huge number of attributes to every single song: Is it happy or sad? Is it guitar-driven? Are the vocals spoken or sung? Is it mellow, aggressive, or dancy? On and on the list goes. Meanwhile, the software is also scanning blogs and social networks—ten million posts a day, Lucchese says—to see the words people use to talk about music. With all this data combined, The Echo Nest can start to figure out what a “crunk” song sounds like, or what we mean when we talk about “dirty south” music.

Smart.

maybe some of the songs are bad, or the leadoff song isn’t representative of the rest of the playlist—we’ll try to refine that and give it a shot.” Playlists are made by people, but they live and die by data.

This is another way of underlining the best practices of machine learning. An algorithm is really only as good as it’s training set.

In order to keep a burst of listens from drifting your taste profile towards a fleeting interest, something re/code’s Kafka calls, “the Minions Problem“, Spotify isolates isolated wandering from the core.

Spotify says it solves the Minions Problem by identifying “taste clusters” and looking for outliers. So if you normally listen to 30-year-old indie rock but suddenly have a burst of Christmas music in your listening history, it won’t spend the next few weeks feeding you Frank Sinatra and Bing Crosby. The same goes for kids’ music, which is apparently why Spotify knows I didn’t really like “Happy” that much — it was just in the “Despicable Me 2” soundtrack.

Spotify has built its discovery algorithm on the listening behaviors of its 75 million users while Apple has advertised a more top-heavy approach using designated curators that publish playlists for a mass audience. I have to wonder what happened to all the Genius data that has been gathered after analyzing everyone’s iTunes collections and wonder if we’ll see that being used to balance out Apple’s approach.

I’ve heard that Spotify is working on a “family plan” that would let me break out the collective profile built up on my Spotify account that I share with my kids. That will yield more relevant personal recommendations so I don’t get the hip-hop heavy playlist that greeted me due to my son’s heavy rotation.

I think it’s still very early days and consumers will ultimately benefit in the music recommendation race that has just begun.

 

Fun with Google Trends Real Time

Google Trends announced last week that they’ve upgraded their service to be real time. Using their tools, I created a dashboard so you can quickly see who’s trending in Google Search for the past 7 days. If I did this right, this page should continually update.

I’ll manually add/remove names as the list of candidates change.

Democratic Party

Republican Party*

*There are too many in the field to fit on Google’s graph (Google Trends only takes up to five terms to compare). I took the top five announced candidates in the polls.

Presidential Candidate 404 Pages

As the 2016 US Presidential Election gets going, each candidates website 404 pages give us a rare glimpse into the personality behind each campaign. Let’s have a look.

Democrats

Hillary Rodham Clinton – her page looks like a misplaced snapshot and speaks to the basic, simplistic style of her Middle American base. Look what we found!

hillaryclinton404

Martin O’Malley – great if you think your candidate is going to be the next George Washington.

martinomalley404

Bernie Sanders – bonus points for the down-home folksy message on the embedded video. “The good news is you’re on the right web site . . . the bad news is you’re on-the-wrong-page. Just scoot down . . . and you’ll find your way back home.”

berniesanders404

Republican

Jeb Bush – his “wat” page has been localized – you can also access the page in Spanish.

jebbush404

Ben Carson – plain and direct, no-nonsense and frankly boring.

bencarson404

Chris Christie – updated with a clip of the Governor dancing with Jimmy Fallon

Ted Cruz – redirects back to the homepage, there are no wrong answers.

Carly Fiorina – looks real professional but it’s really just window dressing

carlyfiorina404

Lindsey Graham – Oops indeed. It looks naked.

lindseygraham404

Mike Huckabee – includes share buttons just in case you want to share a cute photo of Mike with his dog.

mikehuckabee404

Bobby Jindal – blames Obama for the missing page and then asks for a donation.

Bobby Jindal 404

John Kasich – Lost? Brother, can you spare a dime?

johnkasich 404

George Pataki – at least you know who to call to blame

georgepataki404

Rand Paul – updated with a dig at the NSA.

Rick Perry – still hasn’t found what he’s looking for.

rickperry404

Marco Rubio – slick promotional video recovers the mistake

marcorubio404

Rick Santorum – always looking for an opportunity to make fun of Hillary.

ricksantorum404

Donald Trump – Error.

Trump404

Scott Walker – thaar be pirates there to steal your loot.

Scott Walker 404

Further Reading:

Which 2016 Candidate Has the Worst Logo? – Politico

My Pushcart in Yokohama

When I was a sophomore in college, I took a year off to teach English in Japan to help pay for tuition. Having experienced taco truck street food while attending Occidental College in Los Angeles, I yearned for the late night snack and saw an opportunity to introduce a decent street taco to the Tokyo late-night crowd. Japan had not discovered Mexican food back in the 80s. Tacos were a caricature of the real thing, cabbage was often substituted for lettuce and the only tortillas you could find were El Paso’s fried hard shells at the expat grocery store. 

There is a long tradition of street food in Japan. Scores of ramen or oden pushcarts (yatai) could be found at every railway station with groups of salarymen grabbing a bite after a long day at the office or a night of karaoke. The yatai business is hyper-competitive and you would often spot dusty old pushcarts under railroad bridges, abandoned by someone who had tried and failed to bootstrap their career. My thought was to reclaim one of these old carts, fix it up, and introduce my tacos in the streets of Tokyo after my evening English lessons. Encouraged by my fellow English teachers, I felt I had a core group of regular customers with which to start and it seemed easy enough to fix up an old yatai, find a spare spot, and set up shop. Boy was I wrong.

Locating an old yatai that was abandoned and in decent condition was easy enough. During my spare time between classes I found some carts that were parked a good distance away from any station that appeared no longer in use. There were several of them in what looked like an old yatai graveyard. I found an old one that clearly had not been used for a long time and left a note asking if I could take it. I came back several days in a row to check. When I was confident the owner was long gone, I set to work refurbishing it for service.

I visited the local government office to ask what was required to open a yatai and received a document with several pages of requirements. My basic Japanese translation of the document listed out things such as a source of running water, refrigeration, and a way to keep the food warm. All these would be examined at an inspection by the health authorities before I could get a license.

Perhaps it was my naiveté but I chose a creatively liberal interpretation to meet these requirements:

  • Running water? A 20-gallon plastic jug with a hose connected to a spigot running into a “sink” which was a plastic bucket with a hole drilled out of the bottom.
  • Refrigeration? I cut a series of styrofoam panels and glued them to the walls of a cabinet and fashioned a door to seal it shut.
  • Burner? I placed two portable table-top gas burners into a rack I fashioned to hold them and the pots of taco fillings I would serve.

I think the old health inspector took pity on the innocent gaijin kid with a capitalistic gleam in his eye because when I rolled my contraption to the government office for my appointment, he took a cursory glance, listened patiently to my explanation of how I satisfied all the checkboxes on the form, grunted his approval, and stamped my certificate! All I needed now was a location, this is when I learned about the yakuza.

The yatai business in Japan, is lightly regulated by the government (as demonstrated by my easy approval), but strictly controlled by the yakuza. The area immediately around each train or subway station, while clear during the day, hosts a warren of cozy little noodle stands that crop up around 8 or 9 pm each evening. Every open spot is “controlled” by the local gang who takes a share of the revenues as “protection money” to keep things running smoothly. The yakuza runs interference with the local government, police, and rival gangs to keep them out of the hair of the proprietors and for this service, each yatai owner pays the gang a cut of their profits. A noodle stand can clear between $1000 – $3000 a night so it’s a good business so long as everyone’s happy and things are running smoothly.

My English school was near Yokohama station, a major terminus south of Tokyo so the streets leading to the station were already crowded with many yatai, all with their own specialties and faithful regulars. Of course no one was serving tacos so I felt that was my in. I had scouted out a location on a street under a nearby highway overpass that was in between a busy nightclub district and the station so I felt I could grab some people on their way home or going out for the evening. It was also on the way home from the English school so convenient for the teachers and students as well. It appeared that the space was open so I asked the nearby yatai owners if it would be ok if I opened my stand in that spot.

There is a clear pecking order among all the yatai and I was directed to one stand lording over a fork in the road, a prime spot at the nexus of two major walking routes to the station. The yatai was a ramen cart and the broad shouldered proprietor had such a steady stream of clientele that he required two satellite tables to serve all his customers. I bought a bowl of his excellent tonkotsu and after making small talk about the secret of his broth, explained my street taco concept and how I might go about securing a spot down the road and how much the “protection money” would cost. He explained to me that I would have to meet “the man,” explain my business, and this person would negotiate a fee and grant his permission. I asked when I might meet this mysterious person who didn’t have a name and was told that he would introduce me when appropriate.

Several nights passed and each time I would stop by the ramen stand and ask if “the man” was around and each time was told I had either just missed him or that he had not come by that night. Finally, after about two weeks, I decided that the only way I was going to meet this guy was just to set up my stand and he would come by sooner or later, the chips would fall and I would then know what I was working with. I made final preparations, cooked up a batch of taco fillings to open for business after my last lesson.

Opening night was a roaring success. All the English teachers and several students came by to try out my 300 yen tacos. I sold cans of beer out of a cooler and all the activity attracted a few curious barmaids on their way to work and they promised to tell their friends. No “man” came by on the first night, neither on the second. Just as I was beginning to wonder if such a guy existed, on the third night a cheery guy just looking for a good time stopped by and was full of questions about what I was doing.

yatai1

My imagination prepared me for some scarface gangster in a black suit and red tie but I was totally thrown off by this guy with the fashion sense of a carnival barker. He was such a bundle of joy I showered him with free beers and tacos and me and my English teacher friends quickly won him over. He clearly enjoyed the music we were playing (I think it was Aretha Franklin) and loved the vibe. Later that evening, as I was cleaning things up and he was on his 5th can of Asahi Dry , I broached the topic of “rent” and he shrugged it off and patted me on the shoulder and said that he wanted to do his bit for international relations and sponsor my stand gratis!

yatai2

I continued the stand the rest of that year, later bringing on a partner who ran the stand on alternate evenings so I could get some time off. One evening it was pouring rain so my partner and I decided not to open for business. The next night, when I was setting up, he was waiting for me, sitting on the curb. He scolded me saying that if I wanted to establish regular customers it was imperative that I be open, rain or shine. A fundamental rule of business that I have never forgotten.

My yatai adventure ended that Summer as I got ready to return to college but I’ll never forget the lessons I learned. The hardest part of starting a business is starting. Once you gain momentum, people and promise have a way of materializing and turning your dream into reality. And with the right attitude, barriers that you imagine for yourself are just that, your imagination. Chip away at anything and it’s all just people.

yatai-pinto

The importance of context

Surprisingly, the YouTube recommendation algorithm doesn’t draw inputs from far beyond the confines of YouTube itself. You might think that mining our Google search histories for clues about what videos we’d like would pay off. Nope, Goodrow says.

“The challenge is that web search history is very very broad.” Just because you Googled for help with your taxes does’t mean you want to watch YouTube videos about the ins and outs of U.S. tax law.

To Take on HBO and Netflix, YouTube had to Rewire Itself, Fast Company

Not surprising at all actually. Just because everything on the internet can  be connected doesn’t mean it has to be connected. When the internet is your world, zooming in on contexts and measuring behaviors in those contexts becomes paramount.

Popping filter bubbles at SmartNews

It’s now just over a month since I joined SmartNews and I am digging into what’s under the hood and the mad science that drives the deceptively simple interface of the SmartNews product.

smartnews

On the surface, SmartNews is a news aggregator. Our server pulls in urls from a variety of feeds and custom crawls but the magic happens when we try and make sense of what we index to refine the 10 million+ stories down to several hundred most important stories of the day. That’s the technical challenge.

The BHAG is to address the increased polarization of society. The filter bubble that results from getting your news from social networks is caused by the echo chamber effect of a news feed optimized to show you more of what you engage with and less of what you do not. Personalization is excellent for increasing relevance in things like search where you need to narrow results to find what you’re looking for but personalization is dangerously limiting for a news product where a narrowly personalized experience has what Filter Bubble author Eli Pariser called the “negative implications for civic discourse.”

So how do you crawl 10 million URLs daily and figure out which stories are important enough for everyone to know? Enter Machine Learning.

I’m still a newbie to this but am beginning to appreciate the promise of the application of machine learning to provide a solution to the problem above. New to machine learning too? Here’s a compelling example of what you can do illustrated in a recent presentation by Samiur Rahman, and engineer at Mattermark that uses machine learning to match news to their company profiles.

Samiur Rahman on Machine Learning

The word relationship map above was the result of a machine learning algorithm being set loose on a corpus of 100,000 documents overnight. By scanning all the sentences in the documents and looking at the occurrence of words that appeared in those sentences and noting the frequency and proximity of those words, the algo was able to learn that Japan: sushi as USA : pizza, and that Einstein : scientist as Picasso : painter.

Those of you paying close attention will notice that some the relationships are off slightly – France : tapas? Google : Yahoo?  This is the power of the human mind at work. We’re great with pattern matches. Machine learning algorithms are just that, something that needs continual tuning. Koizumi : Japan? Well that shows you the limitations of working with a dated corpus of documents.

But take a step back and think about it. In 24 hours, a well-written algorithm can take a blob of text and parse it for meaning and use that to teach itself something about the world in which those documents were created.

Now jump over to SmartNews and understand that our algorithms are processing 10 million news stories each day and figuring out the most important news of the moment. Not only are we looking for what’s important, we’re also determining which section to feature the story, how prominently, where to cut the headline and how to best crop the thumbnail photo.

The algorithm is continually being trained and the questions that it kicks back are just as interesting as the choices it makes.

The push and pull between discovery, diversity, and relevance are all inputs into the ever-evolving algorithm. Today I learned about “exploration vs. exploitation”. How do we tell our users the most important stories of the day in a way that covers the bases but also teaches you something new?

This is a developing story, stay tuned!

Francis Ford Coppola on the Amateur

in the documentary Hearts of Darkness, A Filmmaker’s Apocalypse Francis Ford Coppola’s wife, Eleanor, chronicled the filming and production of her husband’s masterpiece Apocalypse Now. It’s an fascinating film, a meta-commentary of the American entertainment industry as a metaphor for American imperialism and the war in Vietnam. I highly recommend it.

The clip above comes right before the credits start rolling. A weary Francis looks forward to the return of the amateur who practices film making purely for the art. It’s a prescient glimpse to the world of YouTube and Snapchat artists where we find ourselves today, a refreshing support of new art forms from a lion of the old.

To me, the great hope is that now these little 8mm video recorders and stuff have come out, and some… just people who normally wouldn’t make movies are going to be making them. And you know, suddenly, one day some little fat girl in Ohio is going to be the new Mozart, you know, and make a beautiful film with her little father’s camera recorder. And for once, the so-called professionalism about movies will be destroyed, forever. And it will really become an art form. That’s my opinion.