How Big Data can get hotel rate predictions wrong in a big way

So much for predicting hotel rates using Big Data and other analytics.

Three months ago DealAngel, a startup that tries to separate hotel hype from legitimate deals, introduced what it termed a “first” for the travel industry: hotel-rate predictions for 200 destinations in the U.S., Canada, Mexico, and Europe.

The idea behind DealAngel’s predictions is that when you look for hotels on the site, you can mouse over a chart and view historical trends in the market, and DealAngel gets into fortune-telling and predicts whether rates will rise, fall or be stable.

You can also set up alerts and DealAngel will email you periodically before the date of your stay to update you on whether rates are stable, dropping or rising so you know whether to “book now” or take your time and wait.

Are DealAngel’s predictions accurate?

Skift decided to test DealAngel’s predictions and on May 29 started tracking rates at 50 hotels in 10 U.S. cities, for a Wednesday evening stay on June 26. The properties are located in Atlanta, New Orleans, Dallas, Oklahoma City, Seattle, New York City, Las Vegas, San Francisco, Detroit, and Washington, D.C.

By DealAngel’s own accounting, as measured by the alerts it sent as of June 5, its predictions on hotel-rate fluctuations were wrong in 50% of the cities, namely Atlanta, Dallas, Washington, D.C., Oklahoma City, and New York City.

DealAngel initially predicted rates in Atlanta and Dallas would drop in the short term, but a week later the site sent notifications that rates had stopped dropping.

The startup forecast that rates across Washington, D.C., would be stable, but a week later it sent a notification that rates had dropped 0.46%, “and prices are dropping further.”

For Oklahoma City and New York City, DealAngel predicted at first that rates would be stable, but on June 5 DealAngel emailed notifications that rates were rising, 1.43% in Oklhahoma City, and 11.51% in New York City.

Didn’t help as predictor for specific properties

DealAngel’s predictions cover a city as a whole, and not individual properties. But, if you are interested in how DealAngel’s market predictions might relate to specific properties in the 10 markets Skift tracked, then take note that DealAngel’s predictions ran counter to the actual rates at 29 of the 50 properties, and that’s an error rate of 58%.

We considered a prediction “wrong” if the rate did or didn’t move $5 per night or more.

For example, DealAngel initially predicted that the pricing trend in Atlanta for a June 26 stay was dropping: “Price trend no rush, rates for this arrival date are likely to drop in the short term.”

However, we considered the prediction unhelpful as it related to the rates at the Twelve Centennial Park Hotel in Atlanta, the InterContinental Buckhead, and Stonehurst Place where the base rates did not change from May 29 to June 5 for a June 26 stay, remaining at $167, $205, and $169, respectively.

For some cities, DealAngel’s predictions were way wrong.

For example, on May 29 DealAngel predicted that rates in New York City would be stable for a June 26 stay. However, on June 5, DealAngel sent an “urgent” email notification stating that rates in New York City had already risen 11.51% across the market, and would rise further.

Consider if you were browsing DealAngel on May 29, and looking to stay on June 26 at the Hyatt 48 Lex hotel. The base rate at that time was $369, and DealAngel predicted that rates in New York City would be stable.

But, by June 5 the base rate for a stay at Hyatt 48 Lex on June 26 had soared $59 to $428 per night.

The base rate at the Howard Johnson Manhattan Soho similarly rose $50 to $229 per night despite DealAngel’s predictions that rates in New York City would be stable.

New Orleans

Even in some cities, such as New Orleans, where DealAngel sort of got it right, predicting rates would “drop significantly in the short term,” the prediction ran counter to rates at four of the five hotels in the city that Skift tracked.

The hotel rates in New Orleans actually dropped 1.27% over the week from May 29 to June 5 for a June 26 stay, according to DealAngel, and whether that is a “significant” drop is open for debate.

Contrary to DealAngel’s predictions, the base rate at the Ritz Carlton New Orleans stayed the same at $199; the rate at the Renaissance New Orleans remained at $101, the nightly rate at La Pavillon Hotel rose $1 to $104, and the base rate at the JW Marriott New Orleans dropped just $4 to $121 per night.

The base rate at Country Inn & Suites in New Orleans met our arbitrary $5 “right” or “wrong” threshold, and fell $5 to $101 per night so we considered that a correct DealAngel prediction.

Staying in Las Vegas

DealAngel got half of the 10 cities right as a whole. For example, DealAngel predicted that rates in Las Vegas would be stable for the June 26 stay, and the rates in the city indeed were relatively unchanged through June 5.

Rates at the Wynn Las Vegas, Wyndham Grand Deese, Encore at Wynn Las Vegas, and Mirage Resort & Casino didn’t budge at all. The base rate that the Desert Rose Resort, however, fell $13 to $58 per night for the June 26 stay.

In its trends and alerts launch announcement in March, DealAngel noted that “even in medium-sized cities the sheer number of hotels all pricing independently makes hotel prices typically more unpredictable than flight prices.”

Big Data and crystal balls

Hotel rates obviously can be very volatile and difficult to predict.

DealAngel may provide some valuable services, including its core businesses of showing whether rates at individual properties are legitimate bargains based on historical and market trends, as well as its city-based rate-trend notifications, but DealAngel hasn’t nailed the predictions’ business.

Even with all of its Big Data, and crystal ball.

Bob Rogers, DealAngel co-founder and COO, commented on Skift’s findings:

Five out of 10 is not particularly good, that’s for sure. The algorithm has a learning component, so the predictions will improve over time, but it’s only a few months old, so we’re still getting this fine-tuned. The functionality is primarily meant to show price trending, and let the consumer decide for themselves whether to buy or wait. Hence the spark line when you mouse over that lets you graphically see what’s going on in the market during the last six weeks for that arrival date.

“What we have seen so far is that the predictions are less accurate the further out you look, which is expected given the market is still making corrections. We are thinking of moving the updates to every two weeks for one month before arrival and more, and set a higher threshold for days with consecutive price changes to reduce the noise.”

Overall results by market

City	Prediction	Actual	Verdict
Atlanta	Drop	Fell 0.94% and Stopped dropping	Incorrect
New Orleans	Drop Significantly	Fell 1.27%	Correct
Dallas	Drop	Fell 0.28% and Stopped dropping	Incorrect
Oklahoma City	Stable	Rose 1.43%	Incorrect
Seattle	Stable	Continue Stable	Correct
New York City	Stable	Rose 11.51%	Incorrect
Las Vegas	Stable	Continue Stable	Correct
San Francisco	Drop	Fell 7.66%	Correct
Detroit	Drop	Fell 0.58%	Correct
Washington, D.C.	Stable	Fell 0.46% and Dropping Further	Incorrect