5 Stars rating and natural customer behavior. People more likely to give bad ratings than good ones.

[VictorBana]

I have been working a lot of years in customer relationship programs, customer satisfaction and behavior programs. I know that the most of customers are more likely to rate a company as a punishment for a bad experience more than giving a normal/medium rate for a correct service/product and only in few cases they will take the time to rate for an over the standard or extraordinary service/product. Have Google Maps ever thought about making a good rating / bad rating ratio or algorithm based in the extreme rating vs. quantity of ratings to avoid underqualifing based on basic human behavior in rating operations?.

http://www.nytimes.com/2012/03/24/your-money/why-people-remember-negative-events-more-than-positive-...

[ZachM · Accepted solution]

Hey! I'm Zach, and I'm a Product Manager at Google. I work contribution and personalization features, and I'm also a LG5.

I agree with your instinct that customers who have bad experiences are more likely to leave reviews. We've done quite a bit of research on this topic as I'm sure you can expect.

Turns out when we look at the aggregate behavior of people worldwide, we actually see a surpringly equitable distribution of ratings on places. One of the trends we see in the data is that folks who leave polarizing negative reviews because of a bad experience tend to not be active reviewers -- they leave their one review and they're done. That's then balanced by our power users and our awesome community of Local Guides, who leave hundreds of reviews across all types of places. So while, yes you're definitely correct that people who have bad experiences tend to write about them disproportionately, we still see enough ratings and reviews across many places that the volume still averages out well. So keep rating places and writing those reviews! 😄

But even still, you're 100% correct that there are noticeable trends like this in human behavior. To address that, we currently adjust for a number of factors in the final score that's displayed on a place. If you look at the rating average on places, you'll probably notice that what we currently show isn't an actual strict mathematical average -- a place with 8 5-star reviews might be rated 4.9, and a place with 2 1-star reviews and 2 5-star reviews might be rated 3.8. We use machine learning and a Bayesian model to adjust the final rating given a number of additional factors. Without going into too much detail (company secrets and all ;)), I can say the rating is primarily based on Bayesian analysis and prediction. You can read more about the general prinicples of this type of model here: http://varianceexplained.org/r/empirical_bayes_baseball/

[TacianoM]

Hey guys, I've just created a post to suggest objective criteria for rating places.

Feel free to join the topic and help on this:

- https://www.localguidesconnect.com/t5/Feedback-and-Feature-Requests/Rating-Criteria/m-p/45450#U45450

[AdrianLunsong]

Loving this discussion; great to see the two way communication. It's nice that on top of the feedback that LG gives, we also get explanations from Google how/why things work the way they do

[Sinval]

@ZachM, as a Local Guide and as a psychometrist and psychometrics researcher, I just love to know that you use machine learning and bayesian statistics, Rasch models are also very interesting. Other thing that should be interesting to research about acquiescence and social desirability, bogeymen!

[itsViney]

This is an interesting discussion. I'm familiar with that train of thought that people are more likely to leave a review if they have a bad experience than if they have a good one, but personally I go the opposite way with my reviews. I'm sure if someone looked though my reviews on Google Maps they'd think I was pretty generous and only give out 4 or 5-star reviews. But that's actually because I see my reviews more as recommendations. If I have a bad experience somewhere, I'll tend not to bother leaving a review at all. I see the review itself as a reward, regardless of the star rating, and if I don't like a place I tend to just think they aren't 'worthy' (for lack of a better word) of any more of my time or attention.

Does anyone else take this tack?

[AndréCGurgel]

Excellent (but not surprising) that Google is solving statistically this starry conundrum about ratings. Nonetheless, from the consumer's perspective I feel that my personal standard is odd - like everyone else's - because he-or-she can't understand what is the meaning of 3, 4 or 5 stars given to the same place, unless every LG profile is revealed and seen as a whole.

So, my questions are: 1) why the LGs are forced to rate a place? I may want to opt-out completely to be asked about rating, because, maybe, I have no idea how to rate consistently. 2) As a consequence of (1), why Google should show the LG's individual ratings, since it's a source of confusion to anyone who doesn't understand the very problem we're discussing right here?

Personally I employ only 2,3 and 4 stars, reserving 1 for the absolute trash and 5 for the absolute magnificent. If I give 5 stars all the time, how could I rate a off-the scale place?

As a side note, Facebook folks are used to "rate" in binary terms: like-it or decline-to-like - there is no not-like. Apparently simplistic but more problematic than the 5-stars system.

[Sinval]

Item Response Theory easily solves that. As a psychometrician, I think that the 1 to 5 stars approach it's nice, and in a quantitative way tries to summarize the user global experience. But as in every scale, it's impossible to guarantee that everyone "reads" the scale in the same way. One possible solution, anchoring vignettes.

[AndréCGurgel]

Wow! This is heavy stuff, although understandable. Thanks for pointing me to something I didn't knew.

All in all, the ratings summary that is shown (derived from a secret sauce) is more than enough for 99,99% of the users.

To show individual ratings is a potential source of confusion - and even personal intrigue, since a place owner that is my friend will wonder why I haven't rate him 5 stars. For this very reason I argue that to rate a place at all should be optional for the LGs. I'm sure Google can easily deal with no-star ratings.

[Sinval]

I agree with your concerns, see King et al. (2004):

Respondents in China and Mexico were asked about how much say they had in government affairs. Using only the original answers to the self-assessment question, the result was that respondents in China reported significantly higher levels of political efficacy than respondents in Mexico. Using the anchoring vignette ratings to correct for the differential use of response scales, this result is reversed.

In other words, there is bias, and that bias can change the interpretation of the obtained scores... Inside a country isn't very likely to happen the same, but when we are comparing scores from users of different countries, is expected that google makes some correction for (this kind of) bias.

King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207. https://doi.org/10.1017/S000305540400108X