How I used a Node.js web scraper, scheduler, and Twilio SMS to book a room through untraditional methods
When my wife saw this post on Instagram, she was immediately hooked:
With our honeymoon in Kenya on the horizon, we set out to book a room. Consulting my aunt who had been to Kenya years ago, she stayed here and had no difficulties booking. It came to our surprise when we heard that this place was fully booked a year or two in advance.
The sudden popularity had to stem from something. A little researched showed this place being recently Ellen’ed.
Damn it, Ellen.
Initially, we checked their website to see if the dates we would be in Kenya were available, no luck. We then emailed the manor and again, no beauno, we were told we were put on their “waitlist”. Likely competing with other people on the waitlist, and our trip only being a few months away, me and my wife’s hopes drew thin.
The search for solutions
The website they were using to show availability was read-only, no functionality to book rooms.
Calling and email were the only way to reach them, a slow and arduous process. I assumed that when a date became free, their website would update first and then they would start contacting waitlist members. This way, they would still get bookings if people fell through.
What I assumed next is that if we were to contact them the day the room became available, likely we would bypass the waitlist. But checking the website every hour was not going to fun.
I put my programmer pants on and thought that this would be a good use case for a good-ol web-scrapper, jazz hands. Hit the site every 30 min and SMS both my phone and my wife’s so that we could give them a call. Unlikely that this 1990’s Kenyan website had protection against bots.
What looked like a simple table turned out to be a simple table:
This is what I needed to find, if it the node text was
1, it was available.
After investigating the simple html structure, I started writing the Node.js service to scrap it. I stumbled upon an NPM module, crawler, which gave me all I needed out of the box.
This took a bit of debugging but now I had the HTML from Giraffe Manors website to play around with.
Next up, I went searching through the results with an NPM package called cheerio.
Now comes the interesting part, SMS text my wife when the room show as available. I used Twilio for this but many other services exist. This required setting up a free account, I know I wouldn’t be sending more than a few SMS messages.
After testing with a few dates that were unbooked, it worked! Now to schedule it to run every 5 minutes (because why not?).
To host and run the code, I chose Heroku as I have experience with it and knew the free tier would work for what I needed. I have no idea how their free tier supports background service jobs but anyways.
A couple of weeks later, (I actually forgot it was running), my wife received the text to her phone! We called them immediately and got it! Seemingly bypassing the waitlist, just like we had hoped. She got a barrage of texts and used up my free tier on Twilio as I didn’t write a stop method when it found an available room 🤣
I particularly liked doing this because it’s not often I code to solve a problem in my life but I thought it would be worth it for pictures like this:
This was one example of how I used my programming skills for a “real” world problem. I would love to hear a problem you may have solved, comment here.