AI Is Changing the Short-Term Rental Business at Plum Guide: Full Video

The cost of listing a new home on the luxury short-term rental platform Plum Guide has decreased 85 percent in the past year because of implementing new AI tech.

Doron Meyassed, CEO of Plum Guide, shared details about what's been happening and what’s next for the company at the Skift Short-Term Rental Summit on June 7. 

After his presentation, Skift senior research analyst Seth Borko joined him onstage to ask a few questions. 

View the full video of the presentation and the transcript below. 

Doron Meyassed: Hi, my name is Doron. I'm the founder and CEO of Plum Guide. For those that don't know us, Plum Guide is a short-term rental platform specializing in the luxury segment. We're a seven-year-old company. We have 30,000 homes. We're about just shy of a $100 million revenue and there's three things that make Plum different. The first one is that we vet every single home that we accept onto our platform. Roughly, we accept one in every 15 homes that we vet. Two, we are big on radical transparency. If you go onto any one of our listings, you will see we have a long list of reasons on every listing why you should not book the home. And three, we provide service to the touch of a button. So pre your stay, during your stay, you can press a button and speak to someone, usually within a couple of minutes to help you.

That's a little bit about us. Hello, this is us. But what I'm really here to tell you about is a little adventure that we've been on moving from, let's call us a traditional tech company, to a more of an AI driven company. Despite being obsessed with a topic, I am not going to lie to you. I am already tired of the words AI and machine learning and all of that. So I hope I'm able to give you a more down-to-earth under the hood view of the travails we've been on and the mistakes we've made and the mess ups along the way. Before I do that, I just want to give you a little bit of context. When we started the business, and we still do in some cases, we used to send hospitality experts into every home.

These are people who used to work in the hotel industry. They would show up at the house with an iPad app. They would test the bedding, the toiletries, the Wi-Fi speed, the shower pressure. They were experts in the field. The results from having homes vetted in that way were exceptional. So our [net promoter score] was in the 70s. Our effective repeat rate — so, if a customer went back to a destination we are in — was, in a four-year period, 600 percent. So they booked once and rebooked six times. So we were massively onto something powerful in terms of identifying quality. The problem arrived when we started to — we were in about 10 locations around the world — we started to think about how we scale the business, and it became clear that we would need thousands of millions, not hundreds of millions, alone doing that to get this business to be in every town, city, and village around the world, which was a journey we did not want to take.

So we began, we went on a mission and the mission was really to say, "Can we figure out a way of digitizing this in-person test and all the elements that go around with that. Can we build an AI that is as effective as a human being who's worked in the hospitality industry, who shows up at the home and can touch it and can smell it? Can we build a robot to do something like that?" And the answer is not yet, but the results are pretty impressive, especially in the last year as things have taken off. I'm going to show you a couple of charts before I tell you what we've been doing, but our cost of acquiring a home has dropped 85 percent over the last 12 months. Our revenue per head in the business is 4x, and we've done that while maintaining NPS and improving conversion.

That's key for us because for us, just adding a lot more homes is not necessarily the difficult bit. We are rejecting most of the homes that we look at. So that's always the key measure for us here. And so what I want to share with you is how we made that happen. I will be here for five hours if I try to talk about every bit of it, but I've chosen three parts of our process where we have, which I think are particularly interesting, where we have managed to use AI or machine learning to do what a human used to do. The first and my absolute favorite is design curation. So for those of you who know Plum have been on our site, we are an extremely design obsessed business. We are really into aesthetics. Those home critics that I told you about, the people who would go test these homes were not only hired for their hospitality skills but also for their design and aesthetic eye.

I'm giving you a vibe of some of the people that go out and do these tests in the home, which choose the design. We'd hire these individuals and then we'd upscale them in this book called our Design Manifesto. It's like a 200-page document. It details what is Plum art and what is not Plum art, what is a Plum decor and not Plum decor. It is the most OCD document you have ever seen and it's got in it case studies and live exams that you need to do. And we would ask these home critics to go through this example. And then they would sit down in front of a screen, they would get a sample of homes and they would just start flicking through them and go, pass, fail, pass, fail. This is good design, bad design. This is on brand, not on good brand.

They would get together once a week. They would have a session when they would debate the homes, they couldn't agree it would be feisty and argumentative. And then these dudes came along. These are two people from our tech team and two data scientists in our organization and they said, "We'd like to have a go at building a model that can curate like our creatives in the business." They built a model, it's built with the ResNet Neural Network for the geeks around us. In effect, what we've done is we took the 1 million homes or so that were manually graded and passed and the 4 million homes or pictures that were manually graded and failed. And we used them to train the model so that the model could supposedly go and replicate that decision making process. And then we tested it, we took 100 homes, you put them through the model. The model decides pass or fail and we give the same to our creative director and we say pass or fail and we compare the results.

The first results were rubbish. There was about a 30 percent match rate between what the robot would say, if you like, what the grading model would say and what our home critics would say. Today, there is a 91 percent match rate between what our grading model, his name's Robo Jack, named after the creative director at the time, what the creative model says and what our creative directors would say. I have to say that definitely exceeded my expectations. This is really digitizing what is, in effect, a very creative process. It's still an A-B decision, but a lot of what a creative director will do is to say yes, no decisions. We are today messing around with it, by the way. So we do stuff like we're playing around with scraping the so house website, using the imagery from that to retrain the model and then getting it to regrade our own collection and select the homes that look at that style.

I don't know if this is helpful for people who are interested in it, but a few of the things that helped the scoring go up. So the match rate from the 30s to the 90s, the number one thing was a great feedback loop. So more important than the original data set, the train, the model is the feedback that you give it so it spits out pass fail and then you correct it and you say, "These ones, you were wrong. These ones, you were right." And that really accelerates how quickly it's learning. And the second big thing was tagging the rooms. So the second we started to tag the rooms and we had to build a model for that too, by the way. But this is a bedroom, this is a toilet. It became much more accurate and slowly but surely, we tagged more data into the imagery which made it far more accurate. So price points, locations, and so on and so forth.

The second area where AI has been a big disruptor for us is in listing production. So one of the things on Plum is that we write our own listings. We have these copywriters. The listings have a very unique tone of voice. Our personality is in it, on it, bold and cheeky. It is highly critical written in the third person and the way it used to work is that you had a copywriter, they would sit down for about 60 minutes, and they would write up the listing. Today, this is actually built on ChatGPT. The first version of the listing is written by that AI. We've trained it to do it pretty accurately in our tone of voice. And then the copywriter instead spends 15 minutes reviewing the listing and editing it. The time for writing a listing has gone from 60 minutes to 15 minutes and the conversion rate of the listing. So the conversion from the viewed listing to checkout page has gone up as it's done that. So they're becoming the cyborgs, if you like, the copywriters and doing a much better job at editing and curating the listing.

I think ChatGPT has definitely made it possible to do, as everyone knows, far more complex stuff. It is far from being perfect. We are not ready to let this thing run on its own. I'll give you an example. One of the things that the AI does is choose the names of the homes, so every home has a name. The listing writer, when they are in the CMS, they will see a name generated by the AI and they can say, "I don't like it, give me another name, I don't like it. Give it another name." I went into the database and selected names of rejected names. You can imagine that not many people would stay in the “I'm a Basement Dweller,” “Two Girls, Two Graves.” So we are not quite ready to let the AI run on its own. 

By the way, it's disturbing actually that there are some common themes here. And for example, there's a lot of sexual names that were rejected and there's no coincidence. I suspect that the actual manually named homes that the AI was trained on have a lot of sexually suggestive stuff that's just a bit more subtle than that. And the AI has picked on that and is using it that way. 

And the final one is home testing. That's sending the person to the home. So as I was reminding you guys, we send these people over, they're counting the plates and they're checking the light bulbs and they're collecting 500 data points about the home. We are pretty close to digitizing this with as effectively as someone being in the home. We're not quite there. I'm not going to go through it in detail, but the premise is the same.

We have tens of thousands of data sets of homes we have sent someone to test. Each one of those tests has a large bank of data points and then we are training the model with that data set to decipher from digital signals whether we think the home passes or not. The output in effect is all our criteria that we care about, and the model gives it a confidence score and we decide definite pass, definite fail or we're not confident enough, send a tester over and retrain the model.

Really, I want to share with you guys three big takeouts for me from this experiment or this journey that we've been on. The first one is this I really believe AI, whether it's now or in the next two years or five years, is going to kill the mediocre experience in short-term rentals. Our view of reviews is reviews, I guess went mainstream 20 years ago let's say. I think reviews are exceptional at getting rid of bad actors, at getting rid of the crap; they are terrible at surfacing to the top, the exceptional. I think reviews were very effective at getting rid of anyone who delivered a disappointing experience. When I see how effective the AI we are building is at vetting these homes. I am convinced that it's only a matter of time before AI is going to get rid of the mediocre experience.

We are at the foothills of this journey and we're getting pretty precise results from digital signals replicating what a hospitality expert in the house is managing to do. And I think the bar is going to go way up and providers that have spaces homes that are okay, but the maintenance is not always great and the designs not too soulful I think are really going to struggle.

Second big takeout for me is, I think the homes that will do exceptionally well in a future that is where the search at least is driven by AI are going to be those that are hyper targeted to a specific target segment, consumer segment. I think a lot of businesses, we started to play around with chat-based search and when you do you see that that's a much more intuitive way of searching. I think we've talked about that already today. But you don't just say, "I want a three bedroom at this price point at this time," but you also say, "I want a communal space because it's a multi-generational holiday and we want to be able to book babysitters and we are child safety obsessed so we want to make sure the house is child safe." On chat-based searches, you can write this kind of stuff, you can search this kind of stuff.

If your data is very well-indexed, it's pretty successful at surfacing it. If that becomes the mainstream, which I believe it will, I think we will be in a world where it's the homes that are very, very clear on their segment that win, that come to the top of the search. And those that are kind of sat in that middle of the positioning spectrum and trying to own a lot struggle more. 

And last, but by no means least, a few principles from our experiments that I wish I understood from day one. Number one, I think you need almost no technical skills to start messing about. If you have someone young in your business and they know a bit of coding and they are hungry, let them play. You are unlikely to be able to build the thing that differentiates your business on it, but you can get pretty far. Having said that, spending money on what are extremely expensive machine learning engineers, if you know you want to do something serious, is really worthwhile.

The feedback loop is more important than your original model. I wish I understood that early on. What that means is when we started doing it, for example, grading the photography and the quality of the home and we would give the feedback to the model. So we test the model, we'd give it to the most junior person to do. The model learns off that more than the original data set. So actually, now we give that to the most senior person to do so that it is learning much more quickly. We have automated things with AI that cost more in processing power than having someone manually do it. Don't know if the naming is one of those, but it won't be far off that.

And the final one is guard your golden asset. I'm still trying to figure out what is the moat of businesses in a world that's driven by AI. I'm not 100 percent sure I'm clear on that yet, but the one thing that's particularly valuable for us is that data set of tested homes and all that we have around that. And there are endless suppliers and companies out there that will do stuff for you for almost free in return for that data, and it's very tempting to give it away, but it’s that manual bit that will train something that's unique to you. Thank you very much.

Seth Borko: Hey, everyone. That was a great presentation. That was awesome.

Meyassed: Thanks, Seth.

Borko: I just wanted to ask you a couple questions on it. Just talk a little bit more and I think we maybe can take some audience questions, but I've got some for you anyways in case. So you talked a lot about artificial intelligence, machine learning, there's this new tech ChatGPT, it's super exciting, but we've had machine learning and other types of AI around for a while now. So how much of your breakthroughs have been the new stuff, the generative AI stuff, and how much of your breakthroughs have been the old stuff that's kind of been around for a while now?

Meyassed: There is really a mix of the two, I don't know. In my mind, I often think of the old and the new, whether it's using neural networks or whether it's using large language models. It really is a mix of the two. There is no doubt that the new stuff that's come out that you can talk about standing on the shoulders of giants, I mean the pace at which you can do stuff there is unreal. We've built a tool now that analyzes — you get a house, it has 60 reviews across 30 websites. We analyze all those reviews and tag against 150 criteria the likelihood that it meets those criteria. We'd never be able to do that without ChatGPT.

Borko: Yeah. And as I was asking the question, it's very funny to refer to a neural network as old technology, 'cause it's quite modern, quite impressive. But that's just the pace that things are changing. I wanted to ask you about your grading, your first point about these creative directors grading on taste and style. So you take all of their taste, you grade it, yes or no, you train the model against it. But won't tastes kind of drift over time. Doesn't a creative director now think differently than a creative director 10 years ago? So how do you guard your model against that?

Meyassed: Honestly, I think that's a really good question and it's a risk with all these AI models because it learns, at least the way we've built ours, it's not learning by itself. And so the hope is that as long as we keep a subsample of what we call testing, so you keep testing homes that the model has done and a creative director will say, "No, I don't agree with that. This is now cool." The model should be learning and adapting, but the temptation to stop doing that when there's cost on pressures is big. So you can easily get stuck in a specific style and the robot doesn't learn outside of it.

Borko: So you got to keep investing your robot to keep it fresh as new information comes into the ecosystem.

Meyassed: Definitely.

Borko: You mentioned a couple times, let's talk about costs. I mean profits, dollars and cents. Do you actually make money on your AI investment? You spend a lot of money, does it pay back?

Meyassed: Yeah, so for sure. I mean for us, it's an existential thing to cut the cost of acquiring a home. The way we did it is we drew a map of every human step that we have along the way. We put a dollar number against each step, and we just started going one by one by one and automating the bigger ones. But for us, it was either do that or have a very, very small business. So I'd say there's definitely a big ROI on it. And by the way, this stuff, we're not building stuff from complete scratch. So there's probably about 15 engineers, data scientists and product managers working on all of this machine learning stuff. It's not a huge team.

Borko: And you can physically see the cost of your operations have gone down.

Meyassed: Massively. So it was that previous charge, the cost just in the last 12 months cost of acquiring a home has gone down 85 percent. It's the kind of numbers you only see in a business. It's like a year old, not a business doing $100 million. It surprises me actually, and I have to say when we started that wasn't the goal. The goal was not nearly as high as that.

Borko: That's awesome. I'm really excited to hear more about what's to come from Plum Guide and AI.