Beyond the Stars: Mendoza Researchers Automate Granular Customer Sentiment Analysis

These days, a five-star rating can make or break a restaurant, hotel,or handyman's business. But that single number, averaged across dozens or hundreds of reviews, often hides more than it reveals.

Consider a diner who writes, "The food is great, but the server was not friendly at all." Averaged out, that review might register as a somewhat harsh two-star review. In reality, the customer loved the food and disliked the service; these are two very different pieces of information that a business owner would want to act on separately (and other diners may want to weigh individually).

New research from the University of Notre Dame's Mendoza College of Business proposes a more detailed way to read and assess reviews, breaking them into the specific categories that matter most to customers (such as food quality, cleanliness, price or service) without the cost of hand-labeling thousands of reviews to teach an AI model how to do this distillation work. 

The paper, co-authored by John Lalor, assistant professor of IT, Analytics, and Operations, and Vamsi Kanuri, the Viola D. Hank Associate Professor of Marketing, combines a new conceptual approach to assessing written reviews with an AI-fueled technical innovation to oversee the assessment. Yixing Chen, who led the research while at Mendoza, has since joined Singapore Management University.

Lalor and Kanuri said the resulting tool stands to provide fresh insight to both business owners and would-be customers alike.

“It’s always bothered me that the average rating is so uninformative,” Kanuri said. “It doesn't take into account the fine-grained aspects within a review.” One result of this, he points out, is that reams of helpful, category-level feedback are left buried and unnoticed in unread reviews.

“From a managerial standpoint, it would be very informative for, say, a restaurant to know which specific aspects people are talking about in their reviews,” Kanuri added. “This is a solid example of how academics can solve a substantive problem that industry faces.”

A Pyramid, Not an Average

Kanuri and Lalor said that, ideally, review platforms would ditch the overall star rating altogether in favor of letting customers sort reviews by the aspects that matter most to them. Someone choosing a handyman from the home services website Angi, for instance, might not care about punctuality, but does care a great deal about cleanup afterward.

For sure, ratings aren't going away. Customers like the shorthand of a single number, and businesses like sorting by it. The researchers built a tool that could deliver both: an overall score and the fine-grained category ratings behind it. 

Their model, called the Distantly Supervised Pyramid Network, or DSPN, treats a review as a three-tier pyramid. At the base are individual words and the sentiment (good or bad) that they carry. Those words roll up into general categories, or “aspects,” such as a restaurant’s quality of food, service and atmosphere, which in turn roll up into the review's overall star rating.

Traditionally, teaching an AI model to identify aspects and score sentiment for each one requires reviews to be painstakingly labeled by hand — first tagging which topics are mentioned, then scoring sentiment toward each one. That kind of annotation is slow, costly and rarely available at scale.

DSPN sidesteps that problem. The model needs only the data that review sites already collect by default: the review text and its overall star rating. From that data, it learns to identify the aspects present in a review, gauge the sentiment toward each one and predict the overall rating.

"Providing feedback to the model so it can improve its predictions and then ultimately output a better classifier is often expensive," Lalor said. “We managed to reduce the amount of data required to do that training."

Finding the right data to test the model proved to be one of the project's biggest challenges. To evaluate DSPN's aspect-level predictions, the researchers needed reviews with both an overall star rating and written reviews that included sentiment on various sub-categories of the service being rated — a combination that turned out to be surprisingly rare. The team ultimately settled on an English-language hotel review dataset from TripAdvisor and a Chinese-language restaurant review dataset.

Comparable Performance, Lower Cost

The payoff was a model that performed close to fully human-supervised alternatives at a fraction of the labeling cost. DSPN matched or beat specialized models at identifying which aspects were mentioned in a review and at predictingthe overall star rating, while trailing more modestly on aspect-level sentiment, the one task for which it received no direct supervision.

Lalor noted that the aspect-level results, in particular, exceeded the team's expectations, given that DSPN never saw a single labeled example of aspect sentiment during training. 

"The ability of our tool to do as well as it did with the low cost of not requiring labels at all — just those distantly supervised, overall document labels — that was actually pretty surprising, in a good way," Lalor said.

The researchers have also made the model's code publicly available, inviting others to build on it.

"We've put the starting point out there for folks to say, 'Here's what we did — go ahead and improve it,'" Lalor said.

Beyond Restaurant Reviews

Eventually, Lalor and Kanuri said, the DSPN tool could offer real-time feedback to people even as they’re leaving reviews. The researchers discussed a current project in progress with an online school review platform that invites caregivers to leave reviews about children’s school experiences to post for the benefit of other interested families — and the schools themselves.

The platform has learned which sub-categories of feedback schools are most interested in hearing about, but it has struggled to capture reviews that cover those categories.

The DSPN tool could eventually read reviews in real time, and offer advice about other areas to weigh in on. “The tool could automatically check reviews when they're submitted or as they're being generated and say something like, ‘You've written a lot about topic A, B, and C, but you don't have a whole lot about topic D, and we know topic D is important. Maybe you want to add a line or two,’” Lalor says. “At this point, it’s just something we’re discussing: real-time, almost interactive feedback for reviews.”

While the paper focuses on hotel and restaurant reviews, the researchers see the framework applying anywhere organizations gather feedback on a multidimensional experience: employee engagement surveys, patient satisfaction scores at hospitals or social media chatter about a brand.

Kanuri pointed to Coca-Cola's real-time social media monitoring operations as one example of the insight the model could deliver more efficiently. Lalor added that smaller organizations without a "Coke-sized" social-media budget stand to benefit most, since they typically can't afford to hand-label large volumes of feedback, but still want aspect-level insight from the data available to them in customer chatter online.