Fake Wine Reviews: Do They Matter?
Can we trust that the reviews and ratings on the various wine reviewing peer sites are authentic and not paid for?
This question stuck with me over the past few days after reading the New York Times front page story, "In a Race to Out-Rave, 5-Star Web Reviews Go for $5". The expose of paid for good reviews on sites like Amazon.com, Yelp and TripAdvisor began with this:
"In tens of millions of reviews on Web sites like Amazon.com, Citysearch, TripAdvisor and Yelp, new books are better than Tolstoy, restaurants are undiscovered gems and hotels surpass the Ritz. Or so the reviewers say. As online retailers increasingly depend on reviews as a sales tool, an industry of fibbers and promoters has sprung up to buy and sell raves for a pittance."
First, color me unsurprised that hotels, restaurants and publishers are willing to pay for a good review, particularly given the power of some peer review sites. Human nature being what it is, with competition being fierce in every field of consumer goods and services, and given the simplicity of manufacturing inauthentic reviews at peer reviews sites and you can guarantee this problem.
The NY Times story goes on to describe a company that says using an algorithm it can identify fake reviews 90% of the time. They describe how to spot a fake review:
"The fakes tended to be a narrative talking about their experience at the hotel using a lot of superlatives, but they were not very good on description. Naturally: They had never been there. Instead, they talked about why they were in Chicago. They also used words like “I” and “me” more frequently, as if to underline their own credibility."
So, is this happening at the wine peer review sites like Cellar Tracker, Snooth, etc? I'd bet evey bottle of wine I own that it is. The more important questions, however, are does it matter and how much is it happening?
I've listened to a number of people tell me that wine-related peer review sites will be the new "Robert Parker" and "Wine Spectator". The operative verb there is "will be". And maybe they will be, but they are not yet. And this is why fake wine reviews at peer sites, while undoubtedly existing, are likely in small numbers: the payoff for the effort isn't there yet.
The aggregated peer review site is useful in the same way that democracy is useful. Yet I'm aware, particularly now that we are entering into the primary season of presidential politics, that modern American democracy, like peer review sites, is the domain of the committed and not the masses. It's important to recall that the opinion of the committed, of the partisan, is not the same as the opinion of the masses.
I think you’re wrong, and I’ll tell you why. Oh, wait, what does that ad say at the top left about getting some free wine if I leave 20 reviews at wine.com? Got to go…
So what makes this more onerous than the wineries that advertise in the Spectator getting reviews that average 90 pts or better. At least the smaller wineries get some exposure and it may prompt people to try an new wine.
Dosage Dog:
Are you asking what’s worse: A fake review by someone on a peer review site or an ad in the Wine Spectator that touts a high Wine Spectator score?
If that is the question, then I think it’s pretty clear that the former is unethical, while the later is a statement of fact.
If you are suggesting that an ad that notes that a winery’s products produce ratings that “average” 90 points or more, that is still a fact, just not quite as specific as saying, “X Wine got 92 points”. Still, I don’t think touting an “average score” reaches the level of “unethical” the way a fake wine review does.
Fake reviews do create a certain amount of noise, and for new consumers, or those low on time and/or patience, parsing through the good and the bad may inhibit use of the reviews. I seem to be able to scan and make use of what I can find, but not everyone may like the process.
Another issue, is the increasing level of comment spam, and in recent months, I’ve seen robo-comments from two US wine businesses and one in Canada. These folks may have contracted with a PR firm, who are in turn using this tactic. In either case, the content is useful to no one, except as an indicator of how the techologies are being applied.
Tom:
I think Dosage Dog is suggesting that those 90+ point reviews may be, at least in part, related to the winery’s advertising purchases. It seems that, in the end, whether it’s a peer review or a major publisher’s review, the reader has to decide whether he trusts the reviewer to be objective or not. And I would argue that it’s impossible for a human to be 100% objective about anything. So that requires some sort of mechanism by which to judge the degree of a reviewer’s objectivity.
Clearly one could construe at least a potential conflict of interest when the reviewer is also selling a service (advertising) to the reviewee. The fact that money changes hands will always raise at least the ghost of a doubt about objectivity. It’s not too terribly different from the mutual fund ratings companies and their potential conflicts, IMO. So, ultimately, one’s reputation, as a winemaker or as a reviewer, means everything. And full disclosure about any business relationships is vital.
The big problem with peer wine reviews is that, unless you really follow a particular reviewer, it is impossible to know whether your likes and dislikes will mesh with his or hers. Like or hate Robert Parker, his preferences are well known and often commented on, so if he gives a wine a particular score, you have some idea what you are likely to find.
That said, I do look at CellarTracker notes and scores. If there are only one or two of them, I will generally ignore the scores. (I’ll read the notes, since they might give a clue or two as to what is in the bottle, but not always.) Who knows what that taster likes or whether they have a reasonably good pallet?
When there are a lot of scores from a number of different users, I look to see if there is a consensus, both by number and in the description.
Could that be thrown off by fake reviews? I suppose, but seems like a heck of a lot of work for a hit or miss proposition.
Coming from a claims adjuster background, all you have is your integrity.
As a critic or reviewer, it’s the same.
I look at everything with a jaundiced eye.
The slicks that give 90+ plus points to a winery, which coincidently buys a full-page ad. Pay-for-play? Or separate unrelated acts?
And quite frankly, in the wine business, aren’t we all whores? To me, it just a matter of degrees. There isn’t anyone who doesn’t accept samples – not Parker, nor Spectator, nor the Enthusiast or the spate of new media writers. We all accept samples, but then draw distinctions of purity – the people who won’t accept swag, or junkets, or advertising or sponsorship. To paraphrase Oscar Wilde, “we all know what you are madam, now we are just negotiating on price.”
Best Wishes,
Nannette Eaton
P.S. – Hey Tom, what’s going to cost to get a link in your blogroll?!
Harlot:
I’m not clear. Are you saying that a 90+ point score in a magazine gets the magazine and full page ad?
Tom…
The “test of purity” for reviewers is their willingness to hand out negative reviews, whether the wine (or book or CD or whatever) is a sample or not. I increasingly read, in the posts of young wine bloggers, that they don’t want to write anything negative because that would be “aggressive” or “rude” — I’m quoting — and that no reviews should be written that aren’t positive. This sort of smiley-face attitude does not help consumers, who are looking for guidance, nor does it help the wine industry, which must be held accountable for flawed, bland, innocuous products. As Nannette says, “All you have is your integrity,” to which I would add experience and knowledge.
Wine Spectator reviews all new releases in blind tastings; the reviewer does not know the producer or the price. Consequently, advertisers cannot buy preferential treatment. Our loyalty is to our readers, and we value our integrity above all.
Thomas Matthews
Executive editor
Wine Spectator
Thomas:
Well put.
As a critic for Better Wine Guide, I tasted everything blind and just as Thomas Mathews stated, there really is no way to be duplicitous with scores. Believe me when I say there were more than a few occasions that surprised me, given my preconceived notions about about certain wineries. It really makes a strong case for blind tasting in critical circumstances and I believe in the honesty of that approach.
Better Wine Guide also has community rating functionality but the program flags abusers that try to rate the same wine a specified number of times. Ultimately for me, community scores eventually cancel the effect of having a useable score because they typically end up in the middle ground, which isn’t really helpful. However, people seem to like community scores because it empowers them somehow.
Michael got it right: unless you follow someone in particular, the mass ratings are going to average out to become not useable information, except for extremes on either end.
David Boyer
classof1855.com
I don’t know Tom, chicken or egg.
Did they get a high score (or an editorial write-up) because they are an advertiser? Or are they placing ads in the same issue because they got a high score or an article written about them? Or is it sheer coincidence? In the smaller magazines and small local papers it’s so glaring. And it’s not wine, it’s media in general.
But as a consumer when I see the glowing editorial (or rating) right next to their ad, it looks like they negotiated editorial coverage with their ad buy.
Harlot:
I’m thinking about the Spectator right now and the fact is, there are relatively few ads in the Spectator from wineries, and even fewer that tout scores.
I once did an analysis of Wine Spectator scores (years ago in the 1990s. I looked at whether or not advertisers got higher scores. They got 1.3 points higher—insignificant in every way.
I’ve never seen any evidence that editorial coverage or high scores come to advertisers. And keep in mind, I’ve advertised on behalf of clients in the Wine Spectator.
Consider something else: What’s the main thing that would put the Wine Spectator out of business? The one thing: Selling scores or editorial. If they don’t need to do this, and they clearly don’t by the looks of the many lifestyle ads they get, why would they expose themselves to that kind of trouble.
Makes no sense and I see no evidence of it.
ALSO: to Fredric,
When I first started to rate wine, I actually felt guilty about writing negative reviews. My soliloquy, “could I actually have done any better if I had been the winemaker?” really challenged me. To me, winemakers are the rock stars of the world so I had to come to terms with this question.
As I began to think about it, I came to the conclusion that perhaps I could not do any better at winemaking, however, if I made wine this bad I certainly wouldn’t try to sell it and profit from it or propagate such swill onto the pubic’s palate, – I’d be embarrassed to be associated with a wine that essentially rips people off. That ended Mr Nice-Guy and opened the door for poor reviews. A number of people have told me they are amused by such low rated wine reviews but I really do think that it’s a disservice to not reveal low quality winemaking. Besides I think some housecleaning on the massive wine wall that consumers are faced with these days wouldn’t hurt.
Thanks too, for creating a way for me use the word ‘soliloquy’ – I’ve waited years for that.
David Boyer
classof1855.com
Tom,
Your comments have that certain “je ne se quois” with overtones of cigar box and heady spice. The underlying “flavor” if you will, for lack of a better word, is indicative of a full bore 29+ brix expose of the asinine, synthetic, and highly over rated world of the wine review pubs. Their initial bouquet of protest reeks of barnyard and brett, followed by a mid-palate of “they do protest too much” and a finish of “who me?”
Thanks Tom, great comments. As a very small wine producer who submitted wines for review for many years believing it was all “cricket” and nothing was rigged, I can tell you that you are completely on the mark. I actually had a fairly famous wine reviewer tell me that “I thought your wine was a 98, but ranked it an 89 because my readers aren’t sophisticated enough to like the type of wines I do.” This is so wrong in so many ways – if he was simply trying to be nice, this wasn’t the way; if he was telling the truth, then he should be flogged; and if it was something in between… well, you get the point.
Another prominent wine comp in Southern Cali never entered my wines in the comp despite the fact that I entered them way prior to the deadline – I received a note way after the event from their so called “reviewer” who said “Oh, I tried your wines and loved them, unfortunately, we overlooked them in the warehouse during the competition…” No refund, no wine back, nothing but that note!
In any event, thanks for the comments!
In this “best of all possible worlds” it would be a splash of bracing citric-like refreshment coupled with a slightly subdued but strong middle if consumers would learn to think for themselves.
I’ve asked this question for years, and it is the reason for the absence of wine rating in my wine writing: why should anyone care what I think about a wine? I certainly never cared what others think about a wine–especially about a wine that I like.
I know, I know: consumers can’t taste everything. That’s true before and after they read reviews. For me, the true test of a wine consumer is his or her willingness to at least give the “gotta taste everything myself” attempt a try. Unless of course, most are lazy, and if so, I suspect that’s why fake reviews have a place in our world.
Me, I trust so little that i read online that I’m convinced I didn’t even post this comment.
It’s pretty clear which wine publications are simply selling high scores, and which ones, such as Wine Enthusiast, for whom I write and review, keep an impermeable barrier between reviews and ad sales. After the review is done, are ads sold? Of course. But there is absolutely NO pressure at any time to give anyone a preferential review. In fact, quite the opposite. The specific commitment to blind tasting in peer groups has been a policy of the magazine since reviews were brought in-house years ago. Is there value in Cellartracker type community reviews? The value is there when there are myriad reviews that are clearly unique and individual. Then it’s possible to collate them and look for consensus. The other end of the spectrum is to follow an individual reviewer, with a track record of expertise and knowledgeable commentary. You may not always agree with that person, but you will know where they are coming from.
Tom: I do not endorse fake reviews in any form just as I have been skeptical many of the high scores in the well known wine publications. I had the occasion some years ago to meet an ex-employee of the Spectator who used to set up the tastings and he told me that there was a definite advantage in being and advertiser as compared to a non-advertiser. All of our palates are different and what may be nectar to one may be swill to another. It doesn’t hurt to have peer reviews out there. Let’s hope that most of the folks writing them have enough integrity to be honest.
Dosage Dog
There are fake reviews, and then there are fake reviews.
I can tell you for sure that at Snooth I’ve seen glowing reviews posted by ITBs for their own products as well as legions of fans (some of whom might very well be called shills).
On the flip side, and with deep sadness, I have also come across ITB reviewer who have posted shockingly and consistently negative reviews for competitor or previous employers products.
As far as paid product reviews, we haven’t created any inducement to do so, like the Wine.com example, so I don’t see why we would have any. Which is not to say that some other entity has not. It would be a way to game the system, and systems always get gamed.
The best we can do is police as best we can, which for me means erring on the side of caution when trying to control this phenomenon, be transparent with attribution, and be honest with the wines.
One of issues with crowd sourced reviews that is going to be more important in the future is the tendency of large numbers of reviews to verge to the mean. there are already a lot of wines with what might be considered an average score out there, 3.5 stars, 90pts, and people are going to have to rely more on the written word to distinguish between them.
As Michael said, it’s going to come down to following reviewers whom you trust, and those who review only one brand are not going to garner much trust. As with the wines, we will need to begin to rate the reviewers to allow the most trustworthy to rise to the top. But I see what’s coming. Give me a great rating and I’ll slip you $20…
Great post.
It’s going to happen in all industries, shouldn’t be a surprise it is happening within ours. I think the points mentioned already are good ones – trusted reviewers will be key. But not just the reviewers need to be credible the publishers not only need to have policies in place (no pay for play as was mentioned) but also be transparent about those policies so that buyers can get a sense of the trustworthiness of the publisher. I for one think that publishers have the most to lose by not doing this as the money derived from advertisers receiving great reviews will surely not outweigh the cost of being labeled “fake” or “untrustworthy” – at least in the long run. Just my two cents.
-HWC
I’d rather put cigars out on my arm daily than trust or submit my wines to any corporate wine pub. It’ a sinking ship and new small wineries should avoid this avenue to market their wines at nearly all costs.
I just have to question a magazine dependant on advertising dollars not giving large advertisers a little special treatment. It is sad to say but bussiness ethics is an oxymoron like jumbo shrimp. As for people being paid for reviews I find it dipicable. Sadly it is a very tough thing to police.
Randy I love the passion, fight the power baby!
I once spotted a review of a wine on CellarTracker that came almost verbatim from the winery press release. I told the CT webmaster about it, and he removed it.