The Review-Volume Trap: Why a 4.8 With 40 Reviews Isn’t a 4.8
A 4.8 on 40 reviews isn’t the same evidence as a 4.8 on 600, yet most benchmarks treat them as equals. This article shows three places where sample size distorts a hotel benchmark: the way a market rating is averaged, how concentrated the review volume is, and aspect scores that rest on a handful of mentions. It closes with five questions to ask of any benchmark before you trust it.

SENTEEZ
Guest Experience Insights

Two hotels sit side by side in a benchmark table. Both show a 4.8. One has 40 reviews behind it. The other has 600.
Most benchmarks treat them as equals. They aren’t. The first number could be moved by a handful of guests in either direction. The second has been tested by hundreds of them.
Published hotel benchmarks are good at answering “what is the number?” They’re much weaker at answering “how much should I trust it?” That second question is what this article is about. A strong average is a hiding place, not a finding, and a thin sample is one of the easiest places for an average to hide.
Why small samples distort a score
A hotel rating is a summary of what guests chose to write. When there are only a few dozen reviews, that summary is fragile. One or two very happy guests can lift the score. One or two very unhappy guests can drag it down. A single busy weekend, a wedding party or a short run of staff changes can define how a hotel looks for months.
With hundreds of reviews, those individual voices are balanced by many others. The score settles, and it begins to describe the hotel rather than a few of its guests.
There’s a second effect that’s easy to miss. Hotels with very few reviews are more likely to sit at the extremes, both high and low, simply because there’s less to average out. A table sorted by rating will often put the thinnest hotels at the very top and the very bottom. That’s a feature of small numbers, not necessarily of the hotels.

The practical consequence is simple. A rating on its own is incomplete. It needs the review count beside it, every time. For a wider look at what a benchmark can and can’t tell you, see our earlier article, What a Guest Sentiment Benchmark Can’t Tell You.
Evidence 1: The rating moves depending on how you average
A market’s headline rating can be calculated two ways. The simple average treats every hotel equally. The review-weighted average gives each hotel influence in proportion to its review count. When review volumes are very uneven, the two can disagree, and the direction of the gap tells you where the small samples sit.
In our public dataset, three markets show three different pictures:

In the UK sample, 8 of the 20 hotels have fewer than 100 reviews, including two rated 5.0 on just 21 and 30 reviews. Those two hotels look perfect, but they rest on very little evidence. In the Bali sample, 6 of the 84 hotels are under 100 reviews, including one rated 4.78 on only 27 reviews.
The point is that small samples can bias an average in either direction. Whenever you see a market or portfolio rating, ask which average it is, and how many of the hotels behind it are thin. Market types differ (coastal town, resort town, village), as do hotel mix and traveler base, so compare the pattern, not the levels. The Bali sample is also resort and villa-heavy, with 79% of its 84 hotels rated 4.5 or higher, so these figures describe those 84 hotels, not the destination as a whole.
Evidence 2: Concentration and the ranking problem
The second trap is concentration. If a few hotels hold most of a market’s reviews, the market rating mostly describes those few hotels, and a ranking of the rest rests on thin ground.
The US coastal village in our dataset shows this clearly. Three hotels hold 50.1% of all reviews, and only 10 of its 15 hotels have 100 or more reviews. A top five in that market would have to reach down to a hotel far below the others on guest sentiment. Because that ranking would mislead rather than inform, we don’t publish one.
The Bali sample is the opposite case. Its top three hotels hold just 9.8% of reviews, the lowest concentration in our dataset, and 78 of its 84 hotels qualify for ranking. The same rule that suppresses a ranking in one market supports a robust one in another.

The lesson for anyone reading a benchmark is to ask how the volume is distributed. A market where a handful of hotels dominate the reviews is telling you about those hotels. A market where volume is spread widely is telling you about the market.
Evidence 3: The same trap at aspect level
Sample size matters just as much inside a hotel’s score. We break guest feedback into aspects such as Front Desk & Staff, Room Quality and Maintenance & Upkeep. Each aspect score is built from Review Mentions, the number of times guests spoke about that aspect. A single review can mention several aspects, so mentions aren’t unique reviews.
Some aspect scores rest on very little:
- In one Swiss alpine market of 45 hotels, Maintenance & Upkeep scores 56.3%, but from only 32 Review Mentions across 2 hotels. It looks like a strong result. It’s too thin to mean anything.
- In one Swiss lake and mountain market of 42 hotels, Housekeeping & Cleanliness scores 16.7%, from 66 Review Mentions across just 2 hotels. A dramatic number, with very little behind it.
- In another Swiss market, a Safety & Security aspect shows 0.0%, resting on 3 Review Mentions at a single hotel.

This is why we apply a thin-data rule. We exclude aspects with fewer than 50 Review Mentions or fewer than three hotels reporting, and we show coverage (how many hotels report the aspect) next to every aspect score. Each hotel reports its top eight aspects, so coverage measures how prominent a topic is, not whether it’s absent elsewhere.
An aspect score without its coverage beside it is a number without context.
Why we require 100+ reviews for rankings
On our platform, a hotel needs at least 100 reviews to be ranked. This is a practical minimum rather than a statistical law. Below it, we still show the hotel and its aspects, but we flag the data as thin and leave the hotel out of rankings.
The effect is visible in the numbers above. Eight hotels in the UK sample and six in the Bali sample sit under the threshold, including some with excellent ratings. They’re unranked, not dismissed. An unranked hotel with a 4.8 on 27 reviews may well be a very good hotel. We simply can’t say it’s better than a hotel with a 4.7 on 500 reviews, and a ranking would imply that we can.
The rule also protects the hotels that do qualify. A ranking built on thin data can crown or bury a property on the strength of a few guests. Holding the line at 100 reviews keeps the list closer to what many guests actually said.
Five questions to ask of any benchmark
Before you act on a hotel rating, a market comparison or a ranking, run it through these five questions:
- How many reviews sit behind the number? Under 100, treat it as a signal to watch, not a result.
- Weighted or simple average? If the two differ noticeably, small hotels are pulling one of them.
- How many hotels report this aspect, and how many Review Mentions? Fewer than three hotels or 50 mentions is too thin to draw a conclusion from.
- How concentrated is the volume? If a few hotels hold half the reviews, the market figure mostly describes them.
- Is everything on the same time window? We use a 12-month window, refreshed quarterly, so that every hotel is compared over the same period.
None of these questions needs special software. They need the review count, the aspect coverage and a habit of asking. A benchmark that can’t answer them is a number you should hold loosely.
The number matters less than the evidence behind it
A rating tells you how a hotel is doing on average. It doesn’t tell you how much evidence supports that average. Two hotels with the same 4.8 can be very different stories, and a market’s headline rating can be pulled up or down by a few small hotels.
The fix isn’t complicated. Read the review count next to every score, know which average you’re looking at, and respect the coverage behind every aspect.
See it across eleven regions
You can explore how this plays out across eleven regions on our free Guest Experience Insights platform, where thin data is flagged rather than hidden. If you operate several hotels and want to see how the same discipline applies to your own properties, we’re happy to walk you through a demo.
Guest Experience Insights platform / demo page
Frequently Asked Questions
Key answers regarding hospitality sentiment diligence and data methodology.
No. It can still show useful signals, especially at aspect level. It should simply be flagged as thin and kept out of rankings, and its score read with extra care.
They answer different questions. The simple average describes the typical hotel. The review-weighted average describes the typical guest experience. When they differ, the gap itself is information about where the small samples sit.
Our methodology uses a 12-month window, refreshed quarterly, so figures reflect recent guest experience rather than years-old feedback.
Publicly available guest reviews only. SENTEEZ calculates its own rating from review content rather than copying one from a booking platform. Figures describe what guests say, not what happens inside a hotel.
See what your guests are really saying about your hotel.
A free, analyst-built snapshot of your guest sentiment and how you compare with one competing hotel, delivered within 48 hours.
- Your hotel's rating, with a clear summary
- One-competitor benchmark: where you win and lose
- Your top recurring guest themes
- Delivered within 48 hours
- Free, with no commitment

Written by SENTEEZ
Guest Experience Insights
Related Insights

Your Hotels Against Each Other: Portfolio Benchmarking for Independent Groups
A portfolio-wide guest score blends strong and weak hotels into one number that describes none of them. For a group of 3 to 20 independent hotels, the first benchmark isn’t a competitor. It’s your own other hotels. This article explains why the average misleads, how a property-by-aspect grid shows where a group is uneven, and how to tell a property-specific weak spot from a group-wide one.


One Benchmark, Four Readers: What Owners, GMs, Department Heads and Revenue Managers Each Need from the Same Comparison
A guest experience benchmark is usually written for no one in particular, so it ends up used by no one in particular. This article reads one anonymized Lucerne benchmark through four sets of eyes: the owner, the general manager, the department head and the revenue manager. It ends with a one-page meeting format that gives each reader one question to take to the next meeting.


What a Guest Sentiment Benchmark Can’t Tell You (and What to Pair It With)
A guest sentiment benchmark describes what public reviewers say about a hotel. It doesn’t measure revenue, operations or the opinions of guests who didn’t write a review. This article covers the five questions a review-based benchmark can answer, the five it can’t, why refreshed datasets can mislead, and which sources to pair it with, including STR and PMS data.
