Tag Archives: cartodb

Mapping NYC subway traffic: an interactive

Ever wondered if you could count how many people go through the subway every day?

Okay, probably not. But bear with me here. No code this time.

For our first project in the Metis Data Science Bootcamp, we were given a hypothetical data science project by a company. Our team was asked to use data to help a nonprofit. In an email from an organization created to advocate women in tech, we got our assignment. A quote:

Where we’d like to solicit your engagement is to use MTA subway data, which as I’m sure you know is available freely from the city, to help us optimize the placement of our street teams, such that we can gather the most signatures, ideally from those who will attend the gala and contribute to our cause.

Basically, calculate where people are. But it wasn’t so simple. Our team, made of Ingrid, Ben, Ken and myself, thought through it like this:

1. The busiest turnstiles aren’t necessarily the best. We’re looking for a demographic here – young, progressive and interested in tech. Thousands of pissed off people at Penn Station won’t be any good. We crunched census, income and community data to identify the best neighborhoods.

2. Sometimes the data you’re given isn’t enough. We had to look for lots of extra resources beyond simple MTA turnstile data. Some of this helped us make the map below.

3. When you’re doing data science, make something useful. It’s easy to get lost in “We could do this…” and “But what if…” What if what your client actually cares about is something they can use, not all the stuff you discovered? Never forget your end goal.

And so, what we presented to the company is the below map. We selected five places for their street teams to hang out. The heat flashes show the busiest subway stops over the dates you can see in the bottom corner. Notice how they change throughout the week?

Next step is to plot hourly movements over a day.

Why I’m Becoming a Data Scientist

First  of all, this post exists because I’m currently in New York, studying the art of Data Science with Metis. It’s a 12-week boot camp focused on training us to use all the tools needed to pull insights from massive mountains of data.

I’m here because I want a job as a Data Scientist.

Usually when I say this, people respond in three ways:

  1. But I thought you wanted to be a journalist?
  2. Aren’t you getting your masters degree in journalism?
  3. What the hell is data science, and why would you want to do it?

My reasons are both philosophical and practical, so here’s a short explainer.

What is Data Science?

Our entire world is recorded in bits of information. Thanks to technology and the internet, human beings create and store more information now than at any point in the history of our species. From birth to death, our entire lives are recorded with paper documents, Google searches, emails, pictures and Facebook statuses. Every day trillions of data are created by billions of humans.

For example, this is how much information people create every minute of every day:

DataNeverSleeps_2.0_v2

That’s a monstrous amount of information, and we’re only looking at a slice! Many firms across hundreds of industries are also recording their own information, as more of our world goes digital. Thanks to constantly improving server memory, it’s cheaper than ever to save all this information, so most of this stuff is just sitting around, unused.

But what if we could use all this information?

What if the data revealed patterns? What if we could look deeply at the information and discover the who, what, where, why and how of our world, to empower people, companies and governments to make better decisions?

People are starting to do just that, and they’re already making waves. Mandatory reading, for examples: Big Data, A Revolution.

That’s data science. And it sounds hella awesome.

Why do Data Science?

While I was doing my masters degree in Data Journalism at Mizzou, I realized data scientists and data journalists are basically the same thing. We learned lots of amazing tools like D3.js, CartoDB and Highcharts to tell stories with data. One major difference is that professional data scientists have stronger backgrounds in mathematics, programming and statistics, which would seriously help data journalists. As I created data-driven projects and infographics, I thought a lot about how I could use data to tell stories. Soon enough, it wasn’t satisfying to make a chart or a graph here or there. I wanted to do something bigger with data.

I also realized my journalistic skills – analysis, research, inherent curiosity and storytelling – were a perfect fit for a job as a data scientist. This inspired me to think outside the box and join this bootcamp, where I could get the programming and statistics needed to complete my education. This Venn Diagram explains the rare and challenging mix of skills needed to be a great Data Scientist.

data science

There’s also the practical motivation: Data Science as a profession is exploding, and every industry, from entertainment to healthcare, is hiring. Demand is high, and so is the pay: the median salary of a data scientist is around $107,000. Companies are hiring people right and left. Compare that to journalism as a profession, where median salaries usually sit around $31,000 a year for newspaper reporters, and layoffs loom around every corner. The storytelling opportunities could be deeper if I was involved with data science research.

Data science is still emerging, and with it, the potential for good or evil. I want to put the skills and ethics I learned as a journalist to use in this industry, so I can help establish responsible, ethical and useful uses of data to improve our world.

That’s it, pretty much. Right now, I’m looking to join a data science team that echoes those values, so I can learn better the skills of the trade. Thanks for reading this far and letting me explain this. Every week or so I’ll be blogging here about this camp, if you’re interested in learning more.

For more on Data Science, read the groundbreaking Booz Allen Hamilton Field Guide to Data Science, online for free.

Also, yes – I’m still working on my thesis. I haven’t forgotten about you, Mizzou.