Is Web Scraping the Answer?

Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

Recently I’ve heard many people talking about the wonders of scraping websites and how it will solve all of our data gathering problems. And unfortunately, I must disagree. But before delving into why I believe web scraping is a flawed practice, it is important to truly understand what it is.


While web scraping can be done manually by a software user, the term typically refers to automated processes implemented by websites, data aggregators, and others using a bot or web crawler. Essentially it is a form of copying, in which, specific data is gathered and copied from the web – typically into a central local database or spreadsheet – for later retrieval or analysis.


I could go into further detail about all of the technical complications of web scraping and how it is a behemoth task for software developers. Still, those details are not imperative to the implications it has. Let’s talk about why web scraping is not the panacea it’s positioned to be.


Limited unit availability


When listing apartment homes on an ILS, operators may deliberately withhold the number of units that appear online. This may be due to a limited number of units available at the property or, in the case of new development properties, the operator may want to create the appearance of low supply. Either way, if you are relying on web scraping, you are only getting the data that properties are making available.


Leasing Statistics


Leasing statistics, such as occupancy, leased, and availability percentage is not available on any website and therefore cannot be scraped or copied. Moreover, you can forget about finding out how much traffic or leases a property had the prior week. These are property-level details that play a significant role in property performance. Without this knowledge, decisions are being made based on inaccuracies and limited visibility. 


Pricing Clarity


Likewise, it’s nearly impossible to tell if rents on an ILS or property website are market rents or lease rents. Some operators even have a call to action that suggests the visitor “Call for Pricing,” meaning their pricing on the website will not give the whole picture. The only way to get actual pricing from web scraping is to know that the property has a daily Revenue Management pricing feed. Otherwise, you may miss out on concessions offered.


But this is a small window into the world of web scraping. And while it seems like an easy, automated way to procure your data, the gaps in information cannot be ignored. The longer they are ignored, the more negative impact it will have on your performance.

More To Explore


Automating workflows for property management teams

The idea of automation can feel daunting, especially in an industry that has been experiencing unprecedented growth for the past several years. Property management teams lead this charge, notoriously carrying heavy workloads with little time left for planning automation. As markets grow and workloads increase, the pressure to simplify internal processes…


Q&A with Ashley Hatfield, Client Success Manager at Radix

Like many of us at Radix I come from the multifamily apartment industry. Prior to starting in multifamily, I worked on the services side of the industry, offering personal training and group fitness classes to residents. After getting to know one…