IMDB Score Bias: It's Temporal

This post was originally published on 2/7/2011.

I'm a nerd, economist, and a movie snob. Sometimes it makes me hard to deal with.

My brother has a natural fear of picking movies with me. During our holiday visits home we invariably try to watch a movie and it ends in eyes being rolled in my direction. My family has taken to calling any movie I pick as a "depressing indie drama."

I don't think of this as being difficult, I think of it as getting the most out of my time. Having seen thousands of great movies, I have trouble committing two hours to a movie of dubious quality. In an effort to avoid wasting time on bad and mediocre flicks, I am on a quest to better predict how much I will enjoy a given movie. I've rated more than 700 movies on Netflix, I visit IMDB about 25 times week, I've tried Flixster, Rotten Tomatoes, and the blogs of well-known critics. The goal is to accurately correlate my movie-watching happiness with the ratings provided by these sources. So far the results are disappointing. No one source accurately predicts my preferences. Even inter-comparing and creating composite indexes frequently leads to contradictory predictions. To date, the best predictor I've found is a film's IMDB rating, but this number is far from perfect.

IMDB ratings are worst when movies are newly released. For a film like Citizen Kane, the IMDB score is accurate, and no wonder: enough people have seen it to decide how good it is. In fact, Orson Welles' masterpiece has 145,319 ratings on IMDB, a score of 8.6/10, and is listed by the American Film Institute as the best movie ever made. [1] Citizen Kane is pretty similar to other critically acclaimed films on IMDB. Among the top ten films, the average number of IMDB votes is 152,073 and the median score is 8.45. With so many ratings, my guess is that these movies are more accurately rated than a movie with 1% as many reviews that was released last week.

Take Inception for example. When it was released it had a rating of 9.3 on IMDB and thousands of reviews. But how could this be? Was Inception actually a better movie than Citizen Kane, Casablanca, The Godfather, Gone with the Wind, Lawrence of Arabia, The Wizard of Oz, The Graduate, On the Waterfront, Schindler's List, and Singin' in the Rain?  Having seen all of these films, I had a hard time believing it.

So, I hypothesized that IMDB ratings were biased upwards for young movies. When new movies come out, the first people to see them are early adopters and critics. As an example, someone disinterested in a new film may see it eventually [2], but they are unlikely to see it the first day it comes to their local theater. My contention was that seeking out such pre-releases, in combination with marketing and release hype, would select and reinforce overly-positive movie reviews.

To test this theory, I spent three months sampling a randomly-selected group of 21 new releases. I started sampling on November 9th by finding IMDB's list of upcoming movies and recording the first data point for all of them [3]. I then checked the ratings once weekly to see if my prediction about prerelease hype held up to a little empirical rigor. My sample was surprisingly diverse. Among the movies I sampled there were big budget Hollywood films like Tron: Legacy as well as indie films like Rare Exports [4]. Because some of the films were slated to release later in the month of December and some had pre-screeners who rated the movies before a popular release, I didn't have an equal number of data points for each film. Almost every film did reach score equilibrium; the score remained stable for at least three sampling periods (three weeks). Here is a time series for the films. I've omitted the titles since it would clutter the graph too much:

Looking at the graph is a bit confusing, and there isn't a clear trend. So I turned to the numbers. With a little statistical crunching I found that the average movement in rating was -.2125, significant at 95% confidence. In other words, new movies do have inflated IMDB ratings, on average those ratings are .2 points above where they will eventually settle.

The greatest volatility in rating was in the first two sample periods, which is to be expected. The Tempest and Casino Jack were the biggest losers (shedding 1.6 points in the 3 month period). There were several films that appear to have been correctly assessed from the get-go and had no rating change after 12 weeks: I Love You Philip Morris, The Tourist, The Fighter, Little Fockers, and a French film by the name of The Illusionist. The rest suffered small declines in score that are consistent with my theory. 

The takeaway here is that if you are asked to watch a new release, assume that the IMDB rating is overly-optimistic by about a fifth of a point, then go anyway and have a good time.

[1] Even a film snob like me must admit that it is ridiculous to make such a claim but it sure sounds definitive.

[2] I suspect the biggest reason that disinterested people see films is social pressure.

[3] The equivalent page for this week would be here.

[4] I tracked all of the following films: Black Swan, I Love Your Phillip Morris, Rare Exports, The Warrior's Way, The Tourist, The Tempest, The Chronicles of Narnia: Voyage of the Dawn Treader, The Company Men, The Fighter, Tron: Legacy, Yogi Bear, How do you Know, All Good Things, Rabbit Hole, Casino Jack, Little Fockers, True Grit, Somewhere, The Illusionist, Gulliver's Travels, and Country Strong.

Increase Your Productivity by Getting Busier

This post was originally published on 12/22/2011.

At the beginning of 2011, my work schedule was light. I could stop work on the startup early without stressing out, I took longer lunch breaks, and I spent the evenings recreating. I watched a bunch of movies that had been on my list for years, I caught up on correspondences, and I thought about and wrote more blog posts. I went to sleep when I wanted, woke up when the sun rose and generally lived it up. From the outside, I was living the lifestyle business dream, if only temporarily, while other projects wrapped up.

I was, however, deeply unhappy. After the first week or two, I found that watching a lot of movies at once was kind of boring, that I could correspond with others much faster than they could reply, and that it didn't take long to write all the blog posts I had been planning. But much worse, I discovered that for me, being productive had a large impact on my happiness and I simply wasn't efficient when I wasn't busy. Without hard deadlines and time limits, coworkers and customers, it was always acceptable to dally and watch an episode of Community. Or stay up late reading Reddit. Or hit the snooze button more than once.

Just to be clear, I didn't have motivation problems at first. I found that the less I had to do, the less I had to worry about using my time efficiently, and this led to a rapid decline in productivity. The whole slowdown process took 2 weeks.

Starting in the summer, work started picking up again. Even though I was working more than full time, I realized I was accomplishing more on my personal projects than I had when a two hour lunch break was no problem and my days ended at 4PM.

In short, I've found that the right kind of productivity (as in, working on projects/work that matters to you personally) tends to increase as you get busier. There is a linear relationship between how much you have to get done in a day and how much work will get done on personal projects.

There's an old saying that goes "if you want to get a job done, give it a busy person." This isn't a new or revolutionary thought, but the inverse is new and revolutionary: "if you want to get a job done yourself, get busier." I can't speak for everyone, but I've definitely found this to be true.

Have others found this to be true? If so, have you found any hacks that increase productivity even more?

Games: The Anti-MVP

This was originally posted on 6/6/2013.

My cofounders and I were sitting in the Mission on a sunny morning, eating crepes and talking with a survey and testing guru about testing business assumptions for our new startup.

"I would be hesitant to sink more than a week or two into the idea without data from customers."

My cofounders and I traded sideways glances. The truth was that we had been working on our new startup for 4 months already. We had been play testing our game for more than a month, but  it would be a while before it was ready for a beta launch .

Our meeting left us feeling uncertain, our previous startup had a minimum viable product ready to ship inside a month; why was this taking so long? It didn't hit us until later on when we were speaking with more experienced game devs.

"You have something ready to play in 4 months? That's great."

We had been operating on the assumption that like a B2B startup website, we could throw something down in a week or two, go to customers, test it, and begin iterating. But games defy MVP release schedules. Nowhere is this better illustrated than in 2D Boy's illuminating blog series about developing World of Goo. It's true that they churned out their first version in a week, but their first game tests (which occurred several months into development) were a mess. At CodeCombat we understand that all too well, unfortunately.

Games must be fun. The primary business assumption of any game is "users will like playing it." It's almost guaranteed that users won't be satisfied playing a two week old MVP. In fact, that's sort of the definition of a minimum viable product: it's rough, unpolished, and easy to change. If you're doing it right, the MVP is difficult to take seriously. By contrast, the quality standard for games has been raised so high that it's often not enough to slap something together to gauge user interest.

The takeaway we've learned is that startups with experience building websites need to step back and redefine their MVP expectations when building a game. It takes longer to make something fun rather than just functional, and unfortunately games are defined by the first term, not the second.

It's Sad to See Your Startup Turn into a Business

This post was originally published on 2/1/2013.

When I founded Skritter in 2008 with Nick and Scott, we called it a startup. We raised three rounds of funding, hired developers to help scale our team, and attended startup summits, venture capital panels, and meetups filled with aspiring entrepreneurs working on the next big thing. As with all young startups seeking capital, our business plan growth model had us making 30M in profit in 3-5 years as we took the language learning world by storm.

Four and a half years later, Skritter has become a viable, successful, growing company. We have three employees in addition to the founding team, and have provided employment for twice that many along the way. We've proven that our business model generates profit, that it adds value to customer's lives, and that we can achieve product-market fit.

But somewhere along the line, Skritter stopped being a startup and became a business. And while I am deeply proud of our achievements, the change makes me sad.

When you run a startup, you dream big, you think in terms of conquering entire new markets, challenging entrenched competitors, and changing the world in a big way. You work hard, play harder, and forge lifelong relationships with your co-founders.

Businesses, by comparison, are more modest and mundane. Businesses tend to know whom their customers are, they have a good sense of what makes money and what doesn't, and they don't make a habit of re-investing every penny to try and shoot the moon with a new product. Businesses are like middle aged fathers who just want things to run smoothly without too much fuss. Startups are their star-struck sons spouting poetry to their lovers in moonlit gardens.

Startups are just more exciting, vibrant, and entertaining.

But they also have this frustrating tendency to fizzle out, fail, or explode catastrophically. Founders lose their shirts, relationships are ruined, investors are burned, and once stable, gainfully employed founders end up in their parent's basements applying for jobs to cover their credit card debt.

I'm proud of what Nick, Scott, and I have built at Skritter. I'm proud we achieved the dream of building a profitable company. But if you've ever been there for the startup part, the irrational giddiness you get from building something new, you'll know instinctively what I mean when I say it's sad to see your startup turn into a business.

Let's Partner Into Prosperity

Most business partnerships are a waste of time. Guy Kawasaki says so, Paul Graham says so (see the section at the bottom), and I have learned from personal advice that both men speak truth. The thing is, partnering is most appealing and dangerous to a startup early on. In those critical months and years where credibility is scarce, partnerships seem to offer a quick path to legitimacy and (your partner will lead you to believe) wealth. So it's imperative to develop resistance and skepticism to partnership offers. But how?

Well, one method stumbled right into my lap recently. This is a spam message that I received last week:

"LET'S PARTNER INTO PROSPERITY:  Kudos!!! You've got a very good work going here. I've been contracted to develop a website and a phone application that can help people in a particular Country to learn their three different dialects. It's a multimillion $ Project to be funded by the Government. I understand that a lot of scamming bullshit is going on online but you won't need to spend a dime of yours, all we need is the service of a person that has the knowledge required which would be magnanimously remunerated. I don't know much about language software design, if you do or if you know anyone that can partner with me on this please mail me now without any delay: address@yahoo.com Do you have a website? If yes, what's your website? I'm waiting ... Success!!!"

It's got all the elements of a bad partnership: vague intentions, an appeal to the legitimacy of some large organization (the Government!), a nod to skeptics, and call to action. My advice to you: the next time someone proposes a partnership, simply tack "... Success!!!" to what they say to remind yourself that most partnerships are a waste of time. What's scary is that many seemingly legitimate partnership offers are more dangerous than this example because they lure you into wasting time on them. At least in this case I can just click delete and get on with my day.