Category Archives: Digital Journalism

Digital journalism

Natural Language Processing: Reading Project Gutenberg

This post is an process exploration of my most recent project, Reading Project Gutenberg, a proof of concept for a content-analysis recommendation engine. I’m going to go deep on process for this one, more so than the other projects. Still trying to get the hang of explaining all this, so reach out with questions.

For our third project at Metis, we dipped into Natural Language Processing, a way of machine learning that deals with reading and interpreting text. Google Translate, for example, uses machine learning to automatically translate text, which is an insanely hard thing to do.

Another caveat of the project: we needed to do unsupervised machine learning. (A primer on those topics here.) Basically, we had to write a program that would read text and do something with it, and we couldn’t provide the program with examples of what we wanted.

Is that vague enough for you?

My idea: Project Gutenberg is a great site dedicated to creating and distributing free e-books. Compared to Amazon, Gutenberg has little to no data about what users like what books.

So how can I help them with data science? By building a basic recommendation engine, which will offer books similar to a chosen selection. This can help guide readers to books they might like.

Here’s the process.

Structure

Logically, it makes sense that books that discuss similar topics and are written similarly would be enjoyed by similar readers. Knowing this, we can read and categorize each book by the words in it. Using NLP, we can separate out topics, ideas and themes, and find books that are alike.

This is called “clustering.” If we can prove that books can be distributed into similar clusters, we can begin sharpening those clusters and defining them. Perhaps we’ll see a “sci-fi” or a “war” cluster. For the recommendation engine, this will help us identify similar books.

So time to grab the data.

Process

NLP is a computationally-expensive endeavor, so it takes a long time if we have a lot of books. I used this ISO creator to get the full text of 93 random books from fantasy, science fiction, drama and other fiction genres. However, the books aren’t labeled by genre, so I don’t know what category they’re in exactly.

After removing stopwords (common words like, they, there, that) and choosing the most important words, I was ready to run analyses on the text. A snapshot of my vectorizer, for those interested:

Using K-Means and Cosine Similarity, we can start clustering the books and see the actual structure. The machine doesn’t know what clusters exist – it just reads the books, finds similarities, and compares them to one another.

After that, I created a visualization of all the books. By looking at them mapped out, we can see if there are patterns of similar books.

Each dot represents a book, and each color represents a cluster. The distance of each dot from all the other dots shows how similar they are. Closer dots are more similar in topic, and dots that are the same color are in the same cluster.

(See the code here.)

Screen Shot 2016-03-08 at 4.54.57 PM

Clearly, clusters have appeared. By looking at some of the top words in each cluster, we can determine their topics. For example:

Green Cluster: ships, boats, sailed, deck, captain, board, vessels
Orange Cluster: London, England, honor, chapter, poet
Purple Cluster: princess, princes, king, colonel, queen, palaces, majesty, royal
Pink Cluster: Captain, aunt, hotel, garden, doctor
Turquoise Cluster: Jack, spiked, wagon, dollars, allies

So you can clearly see some clusters appearing, while others are less clear. The Green Cluster is obvious books about the high seas, and the three closest titles back up this claim:

-The Cruise of the Cachalot Round the World After Sperm Whales –  Frank T. Bullen

-Ned Myers, Or a Life Before the Mast – James Fenimore Cooper

-The Rover’s Secret: A Tale of the Pirate Cays and Lagoons of Cuba – Harry Collingwood

So that’s it. We proved it could be done, so now if any of you want to help me build this thing for Gutenberg, let me know. Also, if this post was unclear in any way, please comment so I can clear things up or answer any questions.

As always, fork me. 

Cover picture is Stranger in a Strange Land, a painting by James Warhola for a cover of the book written by Robert A. Heinlein. A great classic sci-fi read.

 

 

Mapping NYC subway traffic: an interactive

Ever wondered if you could count how many people go through the subway every day?

Okay, probably not. But bear with me here. No code this time.

For our first project in the Metis Data Science Bootcamp, we were given a hypothetical data science project by a company. Our team was asked to use data to help a nonprofit. In an email from an organization created to advocate women in tech, we got our assignment. A quote:

Where we’d like to solicit your engagement is to use MTA subway data, which as I’m sure you know is available freely from the city, to help us optimize the placement of our street teams, such that we can gather the most signatures, ideally from those who will attend the gala and contribute to our cause.

Basically, calculate where people are. But it wasn’t so simple. Our team, made of Ingrid, Ben, Ken and myself, thought through it like this:

1. The busiest turnstiles aren’t necessarily the best. We’re looking for a demographic here – young, progressive and interested in tech. Thousands of pissed off people at Penn Station won’t be any good. We crunched census, income and community data to identify the best neighborhoods.

2. Sometimes the data you’re given isn’t enough. We had to look for lots of extra resources beyond simple MTA turnstile data. Some of this helped us make the map below.

3. When you’re doing data science, make something useful. It’s easy to get lost in “We could do this…” and “But what if…” What if what your client actually cares about is something they can use, not all the stuff you discovered? Never forget your end goal.

And so, what we presented to the company is the below map. We selected five places for their street teams to hang out. The heat flashes show the busiest subway stops over the dates you can see in the bottom corner. Notice how they change throughout the week?

Next step is to plot hourly movements over a day.

Queer Sensibilities: News Xchange Berlin, 2015

Below is a video of our talk at the 2015 News Xchange conference in Berlin in October. Sara Trimble and I discussed LGBT coverage and style guides in modern journalism. News Xchange is a conference of news executives from all over the world. I’m pretty honored they invited us. It was a blast, and a major shout out to Amy Selwyn for putting the whole thing together. Here’s to hoping I can visit the conference again soon.

QUEER SENSIBILITIES from News Xchange on Vimeo.

5 Stupid things journalism students say

Journalism students, I have good news and bad news.

Good news: you’re all talented storytellers with amazing, in-demand publishing skills, especially in digital mediums.

Bad news: your educations have brainwashed you into saying and believing stupid things that have absolutely no truth in the real world.

It’s time you learn the truth before you take that first job. Below are some stupid things you’ve probably said. I’ve said them before. If you say them in the real world, people will laugh at you.

Or you can take your dose of reality now and feel your full potential.

Will you find this list offensive? Probably. Should we talk about that? Definitely.

1. “I don’t care about my salary.”
How admirable of you. Unfortunately, our society runs almost exclusively on money. “Personal conviction” isn’t an acceptable currency you can use to pay rent. Everyone needs money – it pervades everything we do, even journalism. Money buys food, health, time, freedom, education. You will need money, whether you admit it or not. Instead, try saying “I will take a job where I can make enough money to support my chosen lifestyle without violating my personal moral imperatives.” Or just keep couchsurfing and blogging, you douche.

2. “Working for PR is selling your soul.”
Job titles alone do not prescribe moral integrity to individuals. Just as a prison executioner might actually be a nice guy, journalists can be terrible, shitty, lying, totally mentally insane people.
Public relations is often more like brand advocacy: helping journalists and bloggers know about your company, spreading the truth about your organization, or helping customers use your products. The best public relations professionals are completely transparent and genuinely good, much like their companies.

Tylenol, for example. After the tampering of their drugs in the 80s, Tylenol went on an aggressive recall and safety campaign, inventing the modern tamper-proof seal. As you will soon discover, PR jobs outnumber journalism jobs 3-1 and pay to the same ratio. The explosion of branded, native and sponsored content are making a journalist’s skills more demanded in PR than ever. So maybe it’s time you considered that a person’s (or organization’s) ethics are much more complex than titles describe.
Or you can stick to your standard narrative:
Once upon a time, at a dirt crossroads in rural Georgia, Edward Bernays shook hands with a dark, ominous figure…

3. “There aren’t any jobs out there.”
This is complete bull. There are jobs everywhere for people with your skills, you just won’t take them. Let me rephrase your complaint: “Because all I want to do is work in one specific city, and because I don’t look very hard on the internet, and because I spend most of my time watching Netflix and eating in bed, I can’t find a job.” Want a job? Imagine murderers have taken one of your loved ones hostage. Their only demand: you have to get a job in 60 days. What would you do?

4. “Advertising is the dark side.”
Advertising is the lifeblood of good journalism. If you believe otherwise, you have a fundamental misunderstanding of the journalism industry. Good content leads to more readers. More readers leads to more advertisers. More advertisers means more money for your company. More money for you company means better content. Get it? Ignoring your advertising department is like putting an expensive paint job on a car without an engine: you’ll look like an asshole AND you won’t have a ride.

5. “I just want to be a full time freelancer.”
Oh yeah? Go for it, kid. Just be prepared for the worst hours, for zero benefits, for terrible bosses, and no consistent income. So much for that luxurious travel writing lifestyle you expected. Welcome to the real world – time to get a full-time job.

ICYMI: Vox Magazine at the RJI Tech Innovation Showcase

I clearly need to work on my public speaking, but for a last second presentation, this wasn’t half bad. For a quick rundown of Vox’s digital innovation this semester, watch this video:

[9:49] Our Blue Highways — Ride along with Vox reporters and digital editors as they discuss this award-winning multimedia project from Fall 2014.
Members: Atiya Abbas, Bryan Bumgardner, Jenna Fear and Carson Kohler

[16:44] Vox on social media — There’s a right way and a wrong way for publications to use social media. Vox social gurus share some of our success stories, including Renz prison, the Antlers and CoMo cups.
Members: Christine Jackson and Dan Roe

[1:22] Vox’s new website — Publication websites are in a state of constant development. Vox students took a leadership role in the relaunch of Vox’s spiffy new site last summer.
Members: Laura Heck and Justin Paprocki

[26:38] Q&A

More information about this event: http://rjionline.org/events/tech15

The Spring 2015 Mizzou Mag Club Trip, in Quotes

Select quotes from the annual Mizzou Magazine Club trip to New York, where we toured magazines, quizzed editors and mingled with alumni. These quotes tell the story of what we learned.

“Nobody gets hired on GPA, where you went to school, how you structure your resume… It’s who you are.”

Ryan D’Agostino, Editor-in-Chief of Popular Mechanics and former manager of the band Dispatch

“If you end up working at a smaller publication and doing a lot, that can sometimes be better.”

Sara Gaynes Levy, Features Editor of SELF Magazine, talking about summer internship opportunities

“Look for where there is a need and fill it. Every magazine has a blind spot.”

Jesse Kissinger, Assistant Editor of Esquire giving internship advice

Touring SELF Magazine with Tova Diamond.
Touring SELF Magazine with Tova Diamond.

“If you’re not up for a wild adventure for the next ten years, find a different career.”

Richard Dorment, Senior Editor of Esquire Magazine

“Your ideas should always outsize your resources.”

Andrew Del-Colle, Senior Editor of Popular Mechanics and WVU and MU grad

“It’s sexist, it’s disgusting… It makes more money than anything we publish.”

Mark Godich, Senior Editor of Sports Illustrated and MU graduate, talking about the annual Swimsuit Edition of SI

 

Real talk with Mara Reinstein of Us Weekly
Real talk with Mara Reinstein of Us Weekly

“Your magazine must always be evolving – as an editor, that’s your role.”

Lindsay Schallon, Features Editor of Seventeen Magazine and MU grad

“I think there’s a lot more value in personal experiences than people realize.”

Tova Diamond, Senior Designer of SELF Magazine and MU grad, talking about independent passion projects

“Don’t be afraid to tell people what you’re gonna do – don’t just have them tell you what to do.”

Joe Bargmann, Special Projects Director of Popular Mechanics

“I suggest you live life beyond your wildest dreams.”

– Allyson Torrisi, Director of Photography at Popular Mechanics

 

 

How to Destroy Everyone At Trivia Crack

This post first appeared on Vox Magazine. 

If you’re reading this, you’re obviously betrothed to the latest social-gaming app Trivia Crack.

The name “Trivia Crack” is a morbid but accurate association — you’ll be hopelessly addicted within minutes.

Now all you have to do is win. Here are some shamelessly competitive tips from a level 60 addict with a 2-1 winning ratio.

1. Focus on your weakest categories. There are six categories, and nobody is an expert at them all. Check your profile to discover your weakest categories, and try to win in those categories first. You might always pick Entertainment first, but is that actually a smart move? If you can get your weakest out of the way first, you can make a swift coup de grâce when it counts.

2. Go with the answer you recognize. Oftentimes the only name that’s familiar is the correct choice. There are few trick questions in this terrible game, so don’t expect them.

3. Eliminate the wrong answers. You could save yourself some coins if you eliminate the clearly wrong answers first. This could turn a 25 percent chance guess into a 50 percent chance. Answers that use “all of the above” or “none of the above” are rarely the right choice — when they are, you’ll know for sure.

4. Trust your gut. Your unconscious often pulls you toward the correct answer at first glance.

5. If you’re dealing with numbers, choose one toward the middle of the pack. As said before, absolutes are rarely the right answers.

6. Play a lot. This goes without saying. I’ve had the same question about Einstein probably ten times now. The more you play, the better you’ll get — and the more coins you’ll win.

7. Play against people you consistently beat and who you consistently lose against. Easy wins will generate coins, losses will make you better.

8. Be smart with your coins. Don’t waste three coins on a skip right in the middle of a game. Save your coins for defining moments, such as when you need to get one question right to win. Same with your spins.

9. Give and ask for spins on social. It isn’t just a cheap marketing idea by Trivia Crack, it’s a way to help you win.

10. QUIT PLAYING. If you manage to reclaim your life from this wretched addiction, you’re the real winner. These wildly popular apps come and go about every six months, conquering the country then quickly fading into obscurity. Remember Words With Friends? Flappy Bird? Candy Crush?

Yeah, me neither. Be part of a movement and break free of the digital specter that haunts you.

5 Reasons Journalists Should Move to Seattle

Seattle. The name invokes imagery: a temperate rainforest, permanent rain, (God forbid) the Twilight novels, legal marijuana.

But actually visiting the city reveals something else: Seattle is poised to be the tech utopia of the future. Of course, this comes with problems. As journalists, here’s why you should get your ass to Seattle immediately and the challenges you’ll face:

Seattle is beautiful. A coastal temperature keeps the ever-present flora permanently green. Beautiful ivy and moss cover every corner of the futuristic, minimalist office and apartment buildings. Moody rainclouds part for glorious rays of sunshine. There’s no trash to be seen anywhere. Unlike the urbane vintage flair of New York or the historical weight of D.C., Seattle is building itself from the ground up as a city of the future. The downside: the Mediterranean climate means the weather doesn’t change often. No snow, no hot summers.

It’s perfect for millennials. Think West Coast values: widespread recycling, locally-driven grocery places like Pike Place Market, a chill music scene. Hardly anybody wears ties, yet there’s a clear frontier-meets-office-space fashion trend. Everyone you see on the street is young, hip, smart and attractive – why don’t you join them? The downside: Rent is wildly expensive, experiencing one of the steepest hikes in recent years. It’s only a few notches below New York.

Seattle is the new Silicon Valley. It’s a major shipping lane for Asian companies. It doesn’t have the same problems as SoCal. It’s been called the “Detroit of Tech Companies.” The Pacific Coast location makes Asia more accessible than New York. It’s home to giants like Amazon, Boeing and Microsoft, serious players in the tech scene. It refuses the fly-or-die tech startup blueprint popular in Silicon Valley. There’s none of the crazy Silicon Valley drama, either. In Seattle the vibes are chill and the tech progress is real. The downside: The tech industry has filled Seattle with tons of young, single men. Even though Seattle has a vibrant LGBT community, it could be hard for women to find equal upwards mobility.

Seattle has an evolving but distinguished journalism community. The Seattle Times is a hero of the digital future. Marketing agencies like Edelman are growing their presence in response to the technology industry. Legacy organizations like KING5, The Seattle P.I. and Amazon are picking up tons of journalists, especially Mizzou grads. Where the communication technology goes, journalism follows. The downside: Journalism jobs are disappearing, same as across the industry. This doesn’t mean you won’t get a job – it just means you might not end up as a journalist.

COFFEE. And fresh fruit, vegetables, and seafood daily at the Pike Place Market. And legal weed, whenever you like. And awesome craft beers. And awesome mountain views. And the awesome American Northwest waiting for adventure, if you’re into that sort of thing. Seattle has a variety of things to offer, so isn’t it time you tried it out?

 

 

Edelman Worldwide: The Dark Side?

This post is part of a series about a class trip to Seattle. 

Day 2 and it still hasn’t stopped raining. We wake up too early and destroy complimentary breakfast and head downtown to the Seattle newsroom of Edelman, a worldwide public relations and marketing firm. Absolutely nobody has done research on Edelman, but by God we know where to find doughnuts. 

After a parking fiasco we file into a swanky office building near the Belltown part of Seattle. Marty directs us to go to the 24th floor, and we discover it’s the top. What!

We stumble out of the elevator and are immediately struck by the gorgeous view. In the style of modern buildings, mixed concrete and carpet floors lead to full-length windows on all sides. Maude and superfluous office furniture is everywhere, and the office has a futuristic vibe. There are no cubicles.

 

Young attractive people work at standing desks and laugh together. None of the guys are wearing ties and most women aren’t wearing high heels. There’s a huge community kitchen with a free Keurig. We drool.

In retrospect, this was a defining moment. Look at the place you could work. Look at the pretty people. Look at the pretty view. This could all be yours. 

Mizzou alum Cherylynne Crowder greets us and takes us into a side room for a frank discussion. We stand around half-constructed desks as she explains the office space is new. That’s probably why she didn’t lead us to a conference room.

So what do all these young people do, exactly?

In discussions about storytelling, brand trust and understanding trends, we discover Edelman is a PR firm through and through.

“I’m always thinking about the next new trend,” Crowder says, “Just the same as journalists, we’re always looking for the next new story.”

Her lessons sound vaguely familiar. “We need utility infielders – you have to be the next Renaissance person,” she says to a question about skills. “I have about 12 seconds to get the journalist’s attention.”

She goes on to discuss the changing nature of communication.

“For decades people yelled at the TV – now we’re closing the loop,” she says. Direct to consumer marketing is bringing people closer to the brands they love and boy, does it sound a whole lot like journalism.

She discusses the Trust Barometer.

“Media isn’t very highly trusted,” she says.

Two guys walk up to the group. One turns around a computer to reveal a MS Paint drawing of “Go Mizzou Tigers!” The mafia is everywhere. 

Rob, a young account executive in a plaid shirt, steps forward. He goes on to describe the storytelling he gets to do, all the multiplatform and emerging technology he uses. A story is told about a photographer that gets to swim with Great White sharks.

“What are the new ways we can tell familiar stories?” he says, referencing brand identities on various platforms.

He sounds like a journalist, but he isn’t. It’s perplexing. I challenge him.

“A lot of people, especially in our newsrooms at Mizzou, consider what you do to be the “dark side” of journalism,” I say. “What would you say to those people?”

He grins through frustration, like this is an argument he’s tired of making.

“I take pride in my skill set,” he says, “I’m a storyteller at heart, and I gotta make a living. If journalism can’t help you tell stories, find the channels that can.”

Another executive, Tyler, who just finished describing helping Microsoft start an Instagram, shares his perspective.

“In this job you get to work for stuff you believe in,” he says. “I don’t pitch stuff I don’t believe in.”

Rob closes his part with some strong words.

“Journalism is not the only place to do good.”

Exhausted from standing, we go for a tour around the building and head out. Quotes ring through my head.

Journalism is not the only place to do good. If Journalism can’t help you tell stories, find the channels that can. I take pride in my skill set. Media isn’t very highly trusted. 

edelmanview

 

 

Changing Media Business Models: Seattle Trip

I’m in a class called Changing Media Business Models, a class run by Marty Steffens, a respected professor in the Missouri School of Journalism.

Each semester this class takes a trip to Seattle and tours various newsrooms, including nonprofits, marketing companies and digital-only publications.

The trip was wildly complex and a blur of shattered expectations and great coffee. It was more than the educational experience we expected, and less of the vacation we thought we got.

Check the links below as they come.

I Can Has Cheezburger: Ben Huh

Edelman Worldwide: The Dark Side?

5 Reasons Journalists Should Move To Seattle

CrossCut: Survival Mode

Storytelling and Getting Paid for It

LIONDigitalMedia: The Grand Plan

The Seattle Times: Journalism 2.0?

Losing My Virginity Twice: Uber and Legal Marijuana

Seattle PI: A Kingdom Fallen

KING5 and Broadcast journalists

Zillow and the Public Service Hackathon