How should companies judge a game’s success?

by pastapadre
Posted February 18th, 2008 at 7:20 am

The interview held with Peter Moore has sparked some good discussion and the reaction has been interesting to follow. One of the more frequently commented upon topics is Moore’s use of Metacritic as a major basis for considering Madden 08 to be an excellent game. While in some ways it may have been, everyone would agree that there is a lot of room for improvement in the series.

That got me to thinking. I’m of the belief that Metacritic is not the best method for determining how good a game was, how well it has been received, or what would need to be improved on going forward for reasons I’ll go into shortly. The Wall Street Journal touched on this topic with a piece about how important the scores are to companies (a great read that can be found here). What other ways could the overall success and reaction to a title be determined if not by reviews?

Metacritic and Reviews

Metacritic grabs review scores from a pool of established sources to present a final average. In theory this should be a perfect way to compare it to other games and give a relative picture of how good it is. Unfortunately it depends solely on reviewers, most of which have shown to be unreliable in the area of sports games.

When it comes to preparing reviews simply enough time is not put into the games to complete a fair analysis. Anyone with experience with sports games knows that in many cases for the first few hours, days, or sometimes even weeks that new game can feel fresh and exciting. However then the problems start to creep in that can ruin the experience. MLB 2K7 and Madden 08 would be two good examples of that (although I was even somewhat down on Madden right away at Community Day). So most reviewers sit down with a game and play it for a few hours before writing up their thoughts.

With Madden it just simply isn’t plausible to expect reviewers to spend hour upon hour playing through one of the deepest sports games ever made while getting the review prepared and out prior to the release date. You can’t expect them to get through a ton of play now to be able to discuss the gameplay, play through years of franchise mode, play through a bunch in Superstar mode, and find other reviewers to play against online. So obviously they generally don’t reach the point where the negatives start to overshadow the positives or have an opportunity to include a detailed overview on everything included in the game.

In many cases reviews end up reading like previews with scores attached. They’ll go down the line of features but there is no way they played through Madden’s franchise mode long enough to realize it was broken and required a patch. We know they didn’t play MLB 2K7′s online leagues to find out they didn’t work. But of course both are praised within reviews for those games and factor into the scores. You can even look at MLB 2K6 on the 360, a game that froze before completing three innings rendering it unplayable and went patch-less for weeks, yet that game somehow remarkably has an average score of 64%.

That brings me to the area of online play, which as I’ve written about before is completely ignored in these games (read through that link for some good examples). The most you’ll get is a recap of the fact sheet for online and some generic comments. It’s obvious they don’t even play online given they’ve bypassed discussing broken 2K leagues or NCAA Football and Madden’s horrendous online experiences. How a full mode could be ignored in reviews and that be acceptable is beyond me.

One thing I’ve noticed with reviews in general is that the fun factor has almost completely been ignored. To me that is the most important thing in gaming yet it gets the least representation in reviews. In the end fun factor is going to be subjective yet that shouldn’t mean it isn’t considered. Was NFL Tour a deep enriching experience? No. Did it have a good single player mode? No. Was it fun as a multiplayer game? Absolutely. But if you read the reviews you’d think no one out there enjoyed the game which simply isn’t the case. When I get more play time in with NFL Tour than many of the sim games of the past year I couldn’t care less whether someone thinks the game deserves a three or a nine and I know I wouldn’t have had any idea how to label that with a score of my own.

While there are a few reviewers out there I would take under consideration, the vast majority don’t give the proper care and respect to sports games. So as a whole an average score isn’t the be-all-end-all on how good a game really is.

Sales

Companies tend to look at sales as another factor in how well a game was received. However in many cases this may in part be attributed to the previous year’s title and the marketing push. Although still very strong Madden’s sales have dropped over the past few years. I feel that is due in large part to the shift in console generation, but it could also have to do with disappointment in the efforts of the past few years. And because the marketing varies and has such a huge effect that makes it difficult to evaluate the current year. Word of mouth takes a little while to spread. So here we have sales and Metacritic contradicting each other in the case of Madden.

Community Reaction

Ultimately the online community represents only a small percentage of those who purchase the games but it is an important group. Some great feedback and suggestions can come from here and the more hardcore fans of the games have some insight that should be considered.

Again as time is spent with the game a lot of the time reaction starts out positive and turns negative. If all you were to read was the online message boards you’d think Madden 08 was the worst game ever made. Of course it wasn’t nearly that bad but the negativity spreads and takes over. There definitely is the sense that the hardcore group as a whole are just nitpicking whiners but given the right outlet there is the potential to turn the community into a positive. It’s also important to remember that the majority of people who are enjoying the game don’t go out of their way to post about it.

What can really be taken from the community is feedback through improved channels of communication and that is something EA deserves credit for. Whether they act on it is another story, but at least they’re listening.

Miscellaneous

In the case of Madden you also have external events that are looked upon as representing the game. Those include the Madden Challenge and Madden Nation TV show. I’ve always felt that these represent EA, the NFL, and the majority of gamers poorly however they continue to be a success in terms of marketing. Still Madden Challenge numbers have dropped over the past few years. That isn’t likely due to the quality of the game so much as it is the environment and attitudes presented.

The companies conduct polling to get a general idea of thoughts from a wide range of people. This is one of the better ways to gauge reaction and what people want out of the game however a lot depends on how the polls are conducted. The type of people taking it will alter the results. Do it online and you’ll get vastly different responses than doing it at a Madden Challenge for example.

Final thoughts

In the end there is never going to be a single way to determine the final worth of a game or how well it turned out. Opinions will inevitably vary because of its subjective nature. All of the factors I laid out are going to be considered in some shape or form and taking them together should give at least a general picture. But the reliance on reviewers is certainly troubling from the perspective of a fan of sports gaming that just wishes for them to improve in areas such as fun and authenticity.

So what do you think these companies should be doing to get a true feel for the games and the direction they need to go? Leave your thoughts in the comments.

Tags:

Russell Giles

Pasta,

I wanted piggyback off of what you said about the online communities and the feedback that we provide. The next time you conduct an interview with someone at EA, could you ask if the feedback, wishlists, and suggestions really make an impact on the game. And if so, how much?

Typically, I wasn’t someone who often visited forums and the like. But I’ve been keeping up with EA’s Madden forum since 2008 came out and I often see really great ideas/suggestions there. However, most people online are under the impression that EA doesn’t listen seeing as though hardly any of the ideas make it into the game. If EA implemented a portion of the ideas from the community, we would already have an awesome football sim.
Russell Giles

Pasta,

I wanted piggyback off of what you said about the online communities and the feedback that we provide. The next time you conduct an interview with someone at EA, could you ask if the feedback, wishlists, and suggestions really make an impact on the game. And if so, how much?

Typically, I wasn’t someone who often visited forums and the like. But I’ve been keeping up with EA’s Madden forum since 2008 came out and I often see really great ideas/suggestions there. However, most people online are under the impression that EA doesn’t listen seeing as though hardly any of the ideas make it into the game. If EA implemented a portion of the ideas from the community, we would already have an awesome football sim.
Tomba321

metacritic ain’t that bad. it’s a collection of all the reviews. whats bad are the reviewers not being honest or as knowlegable about what should be in the game…
Tomba321

metacritic ain’t that bad. it’s a collection of all the reviews. whats bad are the reviewers not being honest or as knowlegable about what should be in the game…
http://www.nflfans360.com/ XtremeDunkz

I have been in an online madden league (NFLFans) for Xbox 360 since the day Madden 07 came out. We are now midway through our 12th season.

Now jumping to a main issue I can bring up for the 32 of us in the league, that not many people probably know about. EA decides to take steps back every year for online.

Madden 06: EA Locker with spawn game feature and the ability to share your franchise file. (scrapped after one year)

Madden 07: LIVE Opponent, gives you the ability to play games witha friend using your own custom rosters, and Franchise games with friends (scrapped after one year)

Madden 08: ability to bring custom teams online….with big problems….no player stats at all at the end of the game, and you also can not see the names of the opposing players.

This is just unacceptable, on top of the fact that people who want a good online league experiance are already forced to pay a 3rd party site (Leaguedaddy) to run a league successfully.
http://www.nflfans360.com XtremeDunkz

I have been in an online madden league (NFLFans) for Xbox 360 since the day Madden 07 came out. We are now midway through our 12th season.

Now jumping to a main issue I can bring up for the 32 of us in the league, that not many people probably know about. EA decides to take steps back every year for online.

Madden 06: EA Locker with spawn game feature and the ability to share your franchise file. (scrapped after one year)

Madden 07: LIVE Opponent, gives you the ability to play games witha friend using your own custom rosters, and Franchise games with friends (scrapped after one year)

Madden 08: ability to bring custom teams online….with big problems….no player stats at all at the end of the game, and you also can not see the names of the opposing players.

This is just unacceptable, on top of the fact that people who want a good online league experiance are already forced to pay a 3rd party site (Leaguedaddy) to run a league successfully.
Mr James

“If all you were to read was the online message boards you’d think Madden 08 was the worst game ever made. Of course it wasn’t nearly that bad but the negativity spreads and takes over.”

I know that you are not a big fan of the game but i think you are being too kind. When it is nearly impossible to run an offense from under center, i am really not sure what kind of reaction could be expected. When over 50% of the games end in a D/C, how should people react? When an effective strategy is literally closing your eyed and throwing a high pass, can EA really think people would be happy?

Madden 08 may not be the worst game of all time, but comming from a legendary franchise, it sure feels like it.
Mr James

“If all you were to read was the online message boards you’d think Madden 08 was the worst game ever made. Of course it wasn’t nearly that bad but the negativity spreads and takes over.”

I know that you are not a big fan of the game but i think you are being too kind. When it is nearly impossible to run an offense from under center, i am really not sure what kind of reaction could be expected. When over 50% of the games end in a D/C, how should people react? When an effective strategy is literally closing your eyed and throwing a high pass, can EA really think people would be happy?

Madden 08 may not be the worst game of all time, but comming from a legendary franchise, it sure feels like it.
Groundhog

Nice write-up Pasta, but this is all academic. It wouldn’t really matter if review sites gave Madden the poor scores it has deserved – Madden sells like hot-cakes, even if it hasn’t been selling as well as EA have expected. For a publicly-listed company, this is all that matters. Sales matter a billion times more than the Metacritic scores.

The only reason Peter mentions metacritic is because it’s a quick comment to hang his hat on; the game can’t be so bad, because it scores well. If the metacritic rating for Madden 08 was 65, he would have just used sales to make his point, with a line like “the ratings were lower than we expected, but we believe that the strong sales show us that the gamers are enjoying the new relase of Madden as much, if not more, than they ever have”.

As others have said, lack of competition assures EA will continue to move a ton of copies each and every year, regardless of any new features. The key for them is providing enough gimmicks to draw in new gamers, or those people who don’t buy the title every year. The majority of these folks aren’t going to be drawn back to the Madden franchise by increased realism, deeper franchise mode, etc. They are going to be drawn back (or so EA hopes, at least) by comments EA have made regarding making the game easier and more arcade-y feeling.

I mean, the majority of folks with Madden 08 (not to mention reviewers) probably didn’t even notice that the fumbles or ints were out of wack, or that franchise was broken. The kind of people who gripe about these things on forums and websites make up such a small percentage of buyers that EA really couldn’t be too concerned about it.
Groundhog

Nice write-up Pasta, but this is all academic. It wouldn’t really matter if review sites gave Madden the poor scores it has deserved – Madden sells like hot-cakes, even if it hasn’t been selling as well as EA have expected. For a publicly-listed company, this is all that matters. Sales matter a billion times more than the Metacritic scores.

The only reason Peter mentions metacritic is because it’s a quick comment to hang his hat on; the game can’t be so bad, because it scores well. If the metacritic rating for Madden 08 was 65, he would have just used sales to make his point, with a line like “the ratings were lower than we expected, but we believe that the strong sales show us that the gamers are enjoying the new relase of Madden as much, if not more, than they ever have”.

As others have said, lack of competition assures EA will continue to move a ton of copies each and every year, regardless of any new features. The key for them is providing enough gimmicks to draw in new gamers, or those people who don’t buy the title every year. The majority of these folks aren’t going to be drawn back to the Madden franchise by increased realism, deeper franchise mode, etc. They are going to be drawn back (or so EA hopes, at least) by comments EA have made regarding making the game easier and more arcade-y feeling.

I mean, the majority of folks with Madden 08 (not to mention reviewers) probably didn’t even notice that the fumbles or ints were out of wack, or that franchise was broken. The kind of people who gripe about these things on forums and websites make up such a small percentage of buyers that EA really couldn’t be too concerned about it.
G-LOVE

Pasta,
The issue with EA next-gen sports games is that they arent better then current-gen other then the graphics. Go back to the 06 releases of EA current-gen sports games and compare them to 08 next gen releases and ask yourself have the games gotten better or worse:

NBA Live: This is a lost(expletive)cause…

Madden: Solid game, lacks polish/presentation and hurt by bugs and glitches. This is no longer a yearly must have. Sad but true.

NCAA Football: Great game that could be a Super game but again lacks polish/presentation and hurt by glitches.

NHL: Took huge strides in 08. Best EA sports this year.

March Madness: It’s decent. Lacks polish and presentation. Does this sound familier???

Nascar: Took big steps backwards from 06… Why get rid of the 06 features, that was your best effort to date..

FIFA: Solid game every year.

Tiger Woods: Slightly better then last year. Might want to add the rookie tour next year as well as more cources.
G-LOVE

Pasta,
The issue with EA next-gen sports games is that they arent better then current-gen other then the graphics. Go back to the 06 releases of EA current-gen sports games and compare them to 08 next gen releases and ask yourself have the games gotten better or worse:

NBA Live: This is a lost(expletive)cause…

Madden: Solid game, lacks polish/presentation and hurt by bugs and glitches. This is no longer a yearly must have. Sad but true.

NCAA Football: Great game that could be a Super game but again lacks polish/presentation and hurt by glitches.

NHL: Took huge strides in 08. Best EA sports this year.

March Madness: It’s decent. Lacks polish and presentation. Does this sound familier???

Nascar: Took big steps backwards from 06… Why get rid of the 06 features, that was your best effort to date..

FIFA: Solid game every year.

Tiger Woods: Slightly better then last year. Might want to add the rookie tour next year as well as more cources.
smacdady PS3

lol madden 08 is a great game you are all haters and if 60 bucks is such a huge purchase for you
stop playing games go back to school
u welfare mofos lol i spent 60 buck and was disappointed lol
what a bunch of sad mofos you guys are.
EA 4EVER

cant wait to buy madden 09
maybe ill buy 2 just so i can own a mint condition unplayed madden lol then i can cry about 120 dollars ahahahahaha
smacdady PS3

lol madden 08 is a great game you are all haters and if 60 bucks is such a huge purchase for you
stop playing games go back to school
u welfare mofos lol i spent 60 buck and was disappointed lol
what a bunch of sad mofos you guys are.
EA 4EVER

cant wait to buy madden 09
maybe ill buy 2 just so i can own a mint condition unplayed madden lol then i can cry about 120 dollars ahahahahaha
Groundhog

[quote comment="13738"]lol madden 08 is a great game you are all haters and if 60 bucks is such a huge purchase for you
stop playing games go back to school
u welfare mofos lol i spent 60 buck and was disappointed lol
what a bunch of sad mofos you guys are.
EA 4EVER

cant wait to buy madden 09
maybe ill buy 2 just so i can own a mint condition unplayed madden lol then i can cry about 120 dollars ahahahahaha[/quote]

Gentlemen, I present to you exhibit A.

Here we have that very guy/girl who has no clue that there is not 18 fumbles or 10 picks thrown each game during the NFL regular season, and has never played a franchise mode and so isn’t aware of the game-killing bug in that, either.

Thanks for your “valuable” input sir/madam/child, now go back to picking off Peyton Manning 14 times with your Miami Dolphins secondary.
Groundhog

[quote comment="13738"]lol madden 08 is a great game you are all haters and if 60 bucks is such a huge purchase for you
stop playing games go back to school
u welfare mofos lol i spent 60 buck and was disappointed lol
what a bunch of sad mofos you guys are.
EA 4EVER

cant wait to buy madden 09
maybe ill buy 2 just so i can own a mint condition unplayed madden lol then i can cry about 120 dollars ahahahahaha[/quote]

Gentlemen, I present to you exhibit A.

Here we have that very guy/girl who has no clue that there is not 18 fumbles or 10 picks thrown each game during the NFL regular season, and has never played a franchise mode and so isn’t aware of the game-killing bug in that, either.

Thanks for your “valuable” input sir/madam/child, now go back to picking off Peyton Manning 14 times with your Miami Dolphins secondary.
John

Here is the issue for me. When Mr. Moore was pressed or is pressed regarding certain issues he reaches for the metacritic rating constantly. Now Pasta as you said the developers probably don’t play deeply enough into the game to know that there are issues. My question is simply shouldn’t they? I mean they have it from scratch. Is there really any excuse to put out any game with entire modes unplayable? Also is it just me or are we seeing a record amount of glitches in next gen games. I mean these things simply did not happen on older systems when there were no such things as patches. I mean granted maybe its because technology is better and there is more packed into a game that will slip through the cracks. COULD it be however that with the advent of patches developers have gotten perhaps lazy with quality control and maybe realise there are issues but to meet deadline they say “what the hell lets get it out the door and fix it later via a patch.” I mean its a double edged sword.

I just feel when a companies message boards are on fire with angry customers regarding any title its unfair to say well we had our highest metacritic rating. Bottom line, so what 99% of gamers could really give a hoot about that if the games are unplayable and just not good.

Believe me I want to purchase MLB 2k8 for example but I’m wary. I’m wary because there were game freezes. I’m going to wait this time till friends get it and then make a decision.
http://n/a John

Here is the issue for me. When Mr. Moore was pressed or is pressed regarding certain issues he reaches for the metacritic rating constantly. Now Pasta as you said the developers probably don’t play deeply enough into the game to know that there are issues. My question is simply shouldn’t they? I mean they have it from scratch. Is there really any excuse to put out any game with entire modes unplayable? Also is it just me or are we seeing a record amount of glitches in next gen games. I mean these things simply did not happen on older systems when there were no such things as patches. I mean granted maybe its because technology is better and there is more packed into a game that will slip through the cracks. COULD it be however that with the advent of patches developers have gotten perhaps lazy with quality control and maybe realise there are issues but to meet deadline they say “what the hell lets get it out the door and fix it later via a patch.” I mean its a double edged sword.

I just feel when a companies message boards are on fire with angry customers regarding any title its unfair to say well we had our highest metacritic rating. Bottom line, so what 99% of gamers could really give a hoot about that if the games are unplayable and just not good.

Believe me I want to purchase MLB 2k8 for example but I’m wary. I’m wary because there were game freezes. I’m going to wait this time till friends get it and then make a decision.
http://www.zshare.net/download/6518525cdd035a/ shoeprint

(Until Madden steps it up) I am officially a Call of Duty 4 head… Also call of Juarez, until EA step it up I will completly ignore them on the NFL side… College is cool, but the only play is scary. I never played any gun games until 2008, ive had my 360 since 2006… So w/ that being said, the gameplay sux and I will not buy them

Thanks EA (MADDEN), now I like shoot em up games
http://www.zshare.net/download/6518525cdd035a/ shoeprint

(Until Madden steps it up) I am officially a Call of Duty 4 head… Also call of Juarez, until EA step it up I will completly ignore them on the NFL side… College is cool, but the only play is scary. I never played any gun games until 2008, ive had my 360 since 2006… So w/ that being said, the gameplay sux and I will not buy them

Thanks EA (MADDEN), now I like shoot em up games
Craig

The other games you guys speak of have much longer development cycles than one year (yes, a lot of this code can be re-used, I admit). Also, games of long ago weren’t nearly as complicated.

As it stands now, you have the need to add bigger features in a shorter time schedule – you have to pick and choose and take shortcuts (hence the bugs we continually see every year). I wouldn’t say the developers are lazy, I just think that it’s a time vs. money vs. bang for the buck type of thing. A 2 year development cycle would probably create excellent games, but the user base would be screaming for their updated versions and revenue would be cut roughly in half.
Craig

The other games you guys speak of have much longer development cycles than one year (yes, a lot of this code can be re-used, I admit). Also, games of long ago weren’t nearly as complicated.

As it stands now, you have the need to add bigger features in a shorter time schedule – you have to pick and choose and take shortcuts (hence the bugs we continually see every year). I wouldn’t say the developers are lazy, I just think that it’s a time vs. money vs. bang for the buck type of thing. A 2 year development cycle would probably create excellent games, but the user base would be screaming for their updated versions and revenue would be cut roughly in half.
Dave

Someone wrote here Madden is all about online play.
Are you kidding?
This game is mediocre, at best, online … and probably worse.

I’m sorry, but Peter Moore is full of crap on this Metacritic thing.

If I had to guess, 20% of people who own next-gen Madden (hardcore players or not) like it. 30% tolerate it (I’m in that group), and 50% outright hate it.

There really isn’t a lot to like about next-gen Madden.

Features: C
Graphics: C-
Glitches: A+ (game is filled with them)
Online stability/connectivity: D
Online features: F
Offline franchise: F before patch; D after (because it’s too late for many)

This is a very, very broken game no matter how these corporate heads spin it.

I know Pasta is in the uncomfortable position of having to be diplomatic with these putzes, but even he knows what a piece of crap Madden is. How do I know? He’s on my buddylist and I’ve never seen him play Madden once since week one.

FIX THE GAME!

MAKE IT FUN AGAIN!
Dave

Someone wrote here Madden is all about online play.
Are you kidding?
This game is mediocre, at best, online … and probably worse.

I’m sorry, but Peter Moore is full of crap on this Metacritic thing.

If I had to guess, 20% of people who own next-gen Madden (hardcore players or not) like it. 30% tolerate it (I’m in that group), and 50% outright hate it.

There really isn’t a lot to like about next-gen Madden.

Features: C
Graphics: C-
Glitches: A+ (game is filled with them)
Online stability/connectivity: D
Online features: F
Offline franchise: F before patch; D after (because it’s too late for many)

This is a very, very broken game no matter how these corporate heads spin it.

I know Pasta is in the uncomfortable position of having to be diplomatic with these putzes, but even he knows what a piece of crap Madden is. How do I know? He’s on my buddylist and I’ve never seen him play Madden once since week one.

FIX THE GAME!

MAKE IT FUN AGAIN!
smacdady PS3

lol
u life losers are awesome
fumbles are u mofos so poor u cant download the patches lol
serious u losers need to go back to school to be able to get better jobs
smacdady PS3

lol
u life losers are awesome
fumbles are u mofos so poor u cant download the patches lol
serious u losers need to go back to school to be able to get better jobs
Groundhog

[quote comment="13820"]lol
u life losers are awesome
fumbles are u mofos so poor u cant download the patches lol
serious u losers need to go back to school to be able to get better jobs[/quote]

Once again, thanks for your valuable input. The patch did nothing to improve a majority of Madden’s issues, but thanks for stopping by.

It’s not about money for 95% of the folks here I’d imagine. I can’t imagine anyone who has a next-gen console would be all that concerned about shelling out money for a game. The only people that are concerned about money is the folks at EA who see their sales dropping despite being the only show in town.
Groundhog

[quote comment="13820"]lol
u life losers are awesome
fumbles are u mofos so poor u cant download the patches lol
serious u losers need to go back to school to be able to get better jobs[/quote]

Once again, thanks for your valuable input. The patch did nothing to improve a majority of Madden’s issues, but thanks for stopping by.

It’s not about money for 95% of the folks here I’d imagine. I can’t imagine anyone who has a next-gen console would be all that concerned about shelling out money for a game. The only people that are concerned about money is the folks at EA who see their sales dropping despite being the only show in town.
http://www.nflfans360.com/ XtremeDunkz

[quote comment="13820"]lol
u life losers are awesome
fumbles are u mofos so poor u cant download the patches lol
serious u losers need to go back to school to be able to get better jobs[/quote]

you just really show how immature you are and you add nothing to this website.
http://www.nflfans360.com XtremeDunkz

[quote comment="13820"]lol
u life losers are awesome
fumbles are u mofos so poor u cant download the patches lol
serious u losers need to go back to school to be able to get better jobs[/quote]

you just really show how immature you are and you add nothing to this website.