IMDB Score Bias: It's Temporal

This post was originally published on 2/7/2011.

I'm a nerd, economist, and a movie snob. Sometimes it makes me hard to deal with.

My brother has a natural fear of picking movies with me. During our holiday visits home we invariably try to watch a movie and it ends in eyes being rolled in my direction. My family has taken to calling any movie I pick as a "depressing indie drama."

I don't think of this as being difficult, I think of it as getting the most out of my time. Having seen thousands of great movies, I have trouble committing two hours to a movie of dubious quality. In an effort to avoid wasting time on bad and mediocre flicks, I am on a quest to better predict how much I will enjoy a given movie. I've rated more than 700 movies on Netflix, I visit IMDB about 25 times week, I've tried Flixster, Rotten Tomatoes, and the blogs of well-known critics. The goal is to accurately correlate my movie-watching happiness with the ratings provided by these sources. So far the results are disappointing. No one source accurately predicts my preferences. Even inter-comparing and creating composite indexes frequently leads to contradictory predictions. To date, the best predictor I've found is a film's IMDB rating, but this number is far from perfect.

IMDB ratings are worst when movies are newly released. For a film like Citizen Kane, the IMDB score is accurate, and no wonder: enough people have seen it to decide how good it is. In fact, Orson Welles' masterpiece has 145,319 ratings on IMDB, a score of 8.6/10, and is listed by the American Film Institute as the best movie ever made. [1] Citizen Kane is pretty similar to other critically acclaimed films on IMDB. Among the top ten films, the average number of IMDB votes is 152,073 and the median score is 8.45. With so many ratings, my guess is that these movies are more accurately rated than a movie with 1% as many reviews that was released last week.

Take Inception for example. When it was released it had a rating of 9.3 on IMDB and thousands of reviews. But how could this be? Was Inception actually a better movie than Citizen Kane, Casablanca, The Godfather, Gone with the Wind, Lawrence of Arabia, The Wizard of Oz, The Graduate, On the Waterfront, Schindler's List, and Singin' in the Rain?  Having seen all of these films, I had a hard time believing it.

So, I hypothesized that IMDB ratings were biased upwards for young movies. When new movies come out, the first people to see them are early adopters and critics. As an example, someone disinterested in a new film may see it eventually [2], but they are unlikely to see it the first day it comes to their local theater. My contention was that seeking out such pre-releases, in combination with marketing and release hype, would select and reinforce overly-positive movie reviews.

To test this theory, I spent three months sampling a randomly-selected group of 21 new releases. I started sampling on November 9th by finding IMDB's list of upcoming movies and recording the first data point for all of them [3]. I then checked the ratings once weekly to see if my prediction about prerelease hype held up to a little empirical rigor. My sample was surprisingly diverse. Among the movies I sampled there were big budget Hollywood films like Tron: Legacy as well as indie films like Rare Exports [4]. Because some of the films were slated to release later in the month of December and some had pre-screeners who rated the movies before a popular release, I didn't have an equal number of data points for each film. Almost every film did reach score equilibrium; the score remained stable for at least three sampling periods (three weeks). Here is a time series for the films. I've omitted the titles since it would clutter the graph too much:

Looking at the graph is a bit confusing, and there isn't a clear trend. So I turned to the numbers. With a little statistical crunching I found that the average movement in rating was -.2125, significant at 95% confidence. In other words, new movies do have inflated IMDB ratings, on average those ratings are .2 points above where they will eventually settle.

The greatest volatility in rating was in the first two sample periods, which is to be expected. The Tempest and Casino Jack were the biggest losers (shedding 1.6 points in the 3 month period). There were several films that appear to have been correctly assessed from the get-go and had no rating change after 12 weeks: I Love You Philip Morris, The Tourist, The Fighter, Little Fockers, and a French film by the name of The Illusionist. The rest suffered small declines in score that are consistent with my theory. 

The takeaway here is that if you are asked to watch a new release, assume that the IMDB rating is overly-optimistic by about a fifth of a point, then go anyway and have a good time.

[1] Even a film snob like me must admit that it is ridiculous to make such a claim but it sure sounds definitive.

[2] I suspect the biggest reason that disinterested people see films is social pressure.

[3] The equivalent page for this week would be here.

[4] I tracked all of the following films: Black Swan, I Love Your Phillip Morris, Rare Exports, The Warrior's Way, The Tourist, The Tempest, The Chronicles of Narnia: Voyage of the Dawn Treader, The Company Men, The Fighter, Tron: Legacy, Yogi Bear, How do you Know, All Good Things, Rabbit Hole, Casino Jack, Little Fockers, True Grit, Somewhere, The Illusionist, Gulliver's Travels, and Country Strong.

How to Overcome Your Fear of Hustling

The Horror

You know the feeling well: you've decided to attend a networking Meetup, you get to the location and only see one other car in the parking lot. You go into the building and your stomach sinks as you enter the appointed room. There's only one other person there, and it looks like the organizer. You quickly check your phone: it's 20 minutes past the starting time. How embarrassing. You're the only schmuck who turned up.  There  are unopened drinks on the table next to uneaten food. You look desperately around:  this poor organizer looks ready to talk to anyone and you see the next 2 hours of your life vaporizing in a mist of uncomfortable conversation as you try to politely leave.

You're Doing it Wrong

Most people I know would be uncomfortable with the situation I described above. In fact, I would have been uncomfortable with it until very recently, when I realized how much it was holding me back. I don't like being witness to, much less a part of a failing enterprise as much as the next person, but if you are going to start something new, you better get used to that feeling and stop letting it affect you.

Almost every new venture starts with discomfort. It's easy to forget that when you are starting a company because coding a site allows people to hide from that sort of discomfort for far too long. I would wager that if every web startup required the founders to knock on a 1,000 doors and pitch their product to the resident inside before being allowed to write code, very few businesses would get started, but those that did would be 1,000 times better off.

I'm talking about being willing to physically put yourself in socially uncomfortable situations to push your vision forward. I'm talking about hard sales, cold calls, meet and greets, and asking people for favors. I'm talking about being willing to politely ignore the unwritten rules of social decorum and get down to business. I think the more you are able to do that in your life, the more successful you will be at achieving your goals.

How to Do it Right

It's all well and good to talk about being brassy and hustling for your vision, but how do you go from being a shy, socially-hypersensitive person (like I used to be) to a self-determined success? Here are two tricks that I use:

1) Prediction Boxing

When you are faced with something uncomfortable, you will find yourself making excuses to avoid it. What you are doing is trying to avoid encountering the worst case scenario. Let's say you are going to cold call a potential customer. You have been putting it off for a week, but why?

There's a quick mental exercise I do which helps me put possible outcomes into perspective. It works like this: think about what you are going to do, and generate a scenario in which everything turns out impossibly well, and I mean push this to the extreme. To use the cold calling example, the extreme positive situation is that your customer not only buys what you are selling, but they 100 copies of it and refers 100 of their corporate friends. Bam, your company is now a success.

Now, think about the absolute worst case, and I mean the most god-awful worst way this could turn out. This time, your customer screams curses at you for the unsolicited sales call, tells all their corporate friends you're an idiot, and adds you number to their block list. Okay, that would suck pretty hard.

Finally, think about a situation in the middle. Maybe you get a curt hangup, or a polite denial. Maybe they tentatively agree to try your software for free with no obligation, maybe they try to get you off the phone by making up an excuse not to talk with you.

I have found time and time again that the middle situation is the most likely, and so long as it doesn't contain any truly horrible results, you should probably do what you are contemplating. Remember: think of the best, then the worst, then the middle situation, and if the middle doesn't seem bad, go for it.

2) Pre-commitment

This is a motivational hack that one of my good friends shared with me, and has proven effective time and time again: tell people what you are going to do, and tell them when you are going to do it. You will get the most out of this if you tell people you respect, and since we are talking about intimidating challenges in a business context, I would recommend your co-founders. Here's how I was able to push myself out of my comfort zone repeatedly to get stuff done:

Whenever my co-founders and I would come up with a marketing or sales suggestion that was uncomfortable, I would volunteer to do it regardless of my comfort level with the task and set a concrete deadline. This part is easy: it makes you appear valuable and important to your cofounders, and that feels good. The hard part comes later and you can build up the courage when that time comes. The most important thing is that you tell them exactly what you are going to do "I will cold-call 5 of our potential customers" and give it a deadline "and I will call them by this coming Monday."

By making a formal, concrete, public commitment, you have raised the stakes of failure, and made it that much harder to back out. Now if you fail, you can't hide: the people you respect most will know. You've put yourself in the position of having to succeed. Make no mistake, you don't end up hustling accidentally. I recently saw a talk by a guy talking about customer development in which he said that he would never have done it if the alternative hadn't been firing his best friends. One cold call may not push you to that extreme, but it's a good indicator of how difficult it is to make a habit of hustling.

If you're interested in learning more about the tips and tricks I use, you should read my cofounder's book: The Motivation Hacker. 

Conclusion

Starting something new is scary because failure is scary. You maximize your chances of success by improving your hustle and overcoming that fear. To use the example at the beginning of this post, there is no reason to be scared of an empty room. Worst case scenario: you lose a bit of time, learn what doesn't work, pack up early and try again.

The Manly Art of Making Guy Friends

When I was growing up, my parents moved our family 5 times. Then I went off to college (+1 move). Then I moved to be near my now wife. Then I moved to the west coast for my startup. And Rebecca and I eventually plan to move back east.

In my relatively short lifespan, I've moved 11 times in 5 cities in 4 states across more than 4,000 miles. Every time I move, I'm back to square one with regard to in-person friendships, and it gets harder to make friends the older you get. You get busy, you have a family, you have kids, a house, 2 cars, inlaws, a garden, and a slew of other responsibilities which make it difficult to find the time for other new friendships.

What's worse, as a guy it's taboo to want to make more guy friends. American men aren't supposed to talk about their feelings, much less ask for relationship advice about how to befriend other guys. We are supposed to be silent islands of emotionless self-sufficiency like Stallone in Rocky, only less talkative.

But let's get real: most humans need friends, and men are humans. There's a massive amount of research that suggests that happy, lucky, content people tend to have larger social networks and deeper friendships. Even if you don't believe that, think of it more pragmatically: if you want to find co-founders for a company, get promoted, or even get job offers, you need to make friends with people and it's a lot more socially acceptable to do so among your own gender.

Three Pointers

1) Find groups of men close to your age, intelligence, education, and socioeconomic background.

If this sounds harsh, get over it. You want to meet people who are similar enough to you that you feel comfortable forming long term friendships with them. If you are a middle aged guy of above average intelligence with three children and a wife, you probably won't feel comfortable throwing back cans of PBR in a parking lot with a bunch of unemployed 22-year-olds. And that's good. Even if you seek out men who are similar to you, you will still find an enormous amount of variety in the people you meet, so you don't need to feel like an elitist bastard for seeking folks like yourself. It's natural to do so, and it's the basis of long lasting, trustful friendships.

2) Participate in groups that are creating something or working towards a common goal.

My wife used to participate in the #1 nationally-ranked small women's barbershop chorus. They met weekly to do something they love (sing) and polished an arrangement of songs for competition. This aspect of creating and working towards a common goal is even more important for men. Guys bond by doing things together: building a fort in the woods when you're 10, fixing your car for prom when you're 17, helping a friend move when you're 20, and planning backpacking trips when you're 25. Let me say it again: guys connect with one another by working together. The opposite of man-bonding is sitting around and talking. You want to find a group of dudes like you that are doing something you can join in.

3) Find groups of guys comprised of members that are more successful than you.

Don't misread this one: I don't mean "hang out with guys who make a lot money." I'm talking about personal success, which is something you have to define for yourself. For me, someone who barely scrapes by on the earnings from the company he started is a big success. So is one of my Dad's friends who has built up a support network for homeless people in Southeastern Ohio. Being friends with my startup co-founders has proven how valuable it is to surround yourself with successful friends who challenge and force you to grow. It's difficult to find people like that, but don't give up searching. Being around a peer group that is motivated and successful is critically important to achieving your own goals.

I don't assign equal weight to these criteria and you shouldn't either. I generally assume that finding men like me (#1) is about 50% of the search problem, find a group that is creating stuff is 30% of the challenge, and the rest is finding successful peers. I weight my search to stay sane because if you try to meet all three criteria without compromising you'll be sad and lonely.

Actionable Steps

1. After moving somewhere new, find a job working with smart people you enjoy, even if it's not 100% ideal.
2. Start attending meetups that vaguely match your interests. You probably won't end up going to them for very long, so it's not important that they match your schedule or hobbies perfectly.
3. Talk to both coworkers and folks at meetups about the groups you really want to join. Maybe you joined an outdoors meetup but what you really want to do is train to climb a local 5,000 ft peak. Chances are if you start asking, you'll discover other people who want to do similar things. Surprisingly, even in our hyper-connected internet world, I've found the coolest groups of people are NOT organized around meetup groups.
4. Commit and contribute. The easiest way to show you're interested in what other guys are doing is to offer to help them or their organization. Agree to host the group, organize travel logistics, manage the gear inventories, research trail conditions, or any other number of tasks that active groups need to make them successful.
5. Wash, rinse, repeat, and be patient. My experience has been that it takes me a long time to start making friends in a new city, normally about a year. By years 2-3 I typically have ONE or TWO guy friends I might be comfortable hanging with on a Saturday afternoon, and most of my best friends only became a large part of my social life after more than 3-4 years. The point here is that building genuine, healthy friendships takes time, don't expect otherwise.

Conclusion

So, whether you are reading this because you want to make more friends to hang out with, you want to find a cofounder for your startup, or you want to build your professional network, I've found the above steps to be effective and rewarding. If you follow the guidelines earnestly, you'll end up with life-enriching friendships with other guys that make you into a better man.

Quit Your Job Now, Before It's Too Late

This was originally posted on 4/7/2013.

Six weeks ago I handed my employer my formal resignation. Two weeks ago I waved goodbye and left the office to begin work on my second startup.

At this stage in my life I have no children, no mortgage, a small passive income from my first startup, two rock solid technical cofounders, a working prototype for our idea, a monetization strategy, and enough cash saved up to bankroll some small business expenses. In short, there was very little risk to quitting. Yet I still struggled with the decision and on that Thursday as I walked into the chilly North Carolina air, I couldn't help but wonder if I was making the right decision.

Quitting is hard for social reasons. I worked at a small Django consultancy with an excellent group of folks who I respect and admire. But even there, among independent-minded, intelligent, professional, driven peers, most were confused why I would want to quit an excellent job. One coworker was frustrated that I was helping perpetuate the Silicon Valley brain drain, "don't move out there and be one of those guys," he told me during my last week. But by far the most common reaction to my news was "why?" accompanied by what I perceived to be quiet pity. "Oh dear, he thinks he's going to start the next Facebook."

To everyone yearning to quit their job and pursue their startup dreams, do your homework, then quit now. I'm not sure I would be capable of quitting my job with $100k in a home loan, 2 children, and a car payment. Quitting gets more difficult the longer you wait, so brace yourself and jump soon.

This is Why You Don't Want Nice Things

A while back I wrote about how I think renting is a superior form of ownership. As I keep reading and trying new secondary markets, from Uber to Getaround to AirBnb, I am even more convinced that owning things is ill-advised. Right now every internet user can rent cars, tools, bikes, and dozens of other products and services which used to be enjoyable only by the people who ponied up the cash to purchase them [1]. Distributed economies have changed the value of owning things, but most people are still paying too much to own too many things and it makes us unhappy - I call this over-ownership.

A classic example of over-ownership would be buying a home when early in life.

According to Bureau of Labor Statistics, 50% of people between the ages of 20 and 24 have been with their current employer for less than 2 years [2]. The only age group in which people have worked for their current employer longer than 8 years is 55+. An 18 year old person can expect to move 9.1 more times in their life. By age 45, people can expect to move another 2.7 times [4]. That's approximately 1 move every 4.2 years for people between the ages of 18 and 45.

Using the New York Time's "Is it better to rent or own?", I calculated that if you live in Raleigh (my previous city of residence), pay less than $1,000 in rent, and want to purchase a $200,000 home [5] on a favorable 2.8% mortgage and 1% property tax rate, it only makes sense to purchase if you expect to stay put for more than 6 years. If you live somewhere like San Francisco (where I currently live), and want to buy the average $1M home, and you currently pay $3,000 per month in rent, it will take 18 years before it makes sense to buy.

So most young people shouldn't buy houses, yet according to Calculated Risk and The Atlantic [6], more than 38% of people between the ages of 25 and 29 do own their houses. This says that me that a fair number of those buyers are going to lose time, money, and sleep on their housing purchases.

Now, think about all the items in your home and think about how much you use them. Think especially of expensive items; KitchenAids, belt sanders, laptops, etc. I would be willing to wager that it would be cheaper for you to rent items you use infrequently instead of owning them. I would go even one step farther: renting would not just be cheaper, it would solve a myriad of other lifestyle problems. If you had 50% fewer objects in your house, think about how much easier it would be to clean, how much less time you would have to spend maintaining possessions, and how much easier it would be to move to the next home/apartment.

There aren't currently enough online services and they are still too inconvenient for my vision of utopian rentals to be practical tomorrow, next month, or perhaps even next year [7]. But in the near future, renting will make our lives universally better by allowing us to own less stuff.

[1] Car rentals, boat rentals, parking space rentals, and especially service rentals like Taskrabbit. And these are just a few examples in each category.

[2] http://www.bls.gov/news.release/tenure.t01.htm">http://www.bls.gov/news.release/tenure.t01.htm

[3] Surprisingly, the old adage about 7 career changes in a lifetime is also bunk: Seven Careers in a Lifetime? Think Twice, Researchers Say.

[4] Calculating Migration Expectancy Using ACS Data

[5] Trulia housing demographics for Raleigh puts the median value of a house in Raleigh NC at $200,000.

[6] http://www.calculatedriskblog.com/2009/04/us-homeownership-by-age-group.html, The end of ownership: why aren't young people buying more houses?" 

[7] One comment I received on the original version of this post was that in making my recommendation for renting, I forgot to include time costs. The basic argument is that there is too much friction in renting, which makes renting a $300 Kitchenaid, for instance, a bad deal. Existing services would require you to find a nearby person renting the item, perhaps drive to their home, sign a paper, drive back to your home, use the mixer, and repeat the process when returning it. But I'm not thinking about existing services. When I first wrote this post, Getaround didn't offer instant rentals, but now they do. The same is true for AirBnb and a host of other services. The instant acceptance peer to peer rental markets are already thriving, we just haven't hit saturation with all the stuff we could possibly rent.

Special thanks to my good friend Ben for checking my numbers on this post and keeping me honest.