Is Web Scraping the Answer?



Article 3

Share This Post

Recently the multifamily industry has discussed the wonders of scraping websites and how it will solve inefficiencies in gathering data. While most believe this is true, web scraping could be a potentially flawed practice, and it is crucial to understand what it truly is.


While web scraping can be done manually by a software user, the term typically refers to automated processes implemented by websites, data aggregators, and others using a bot or web crawler. It is a form of copying in which specific data is gathered and copied from the web – typically into a central local database or spreadsheet – for later retrieval or analysis.


One could delve further into the technical complications of web scraping and how it poses a significant challenge for software developers. However, it is argued that such details are optional in understanding the impact of web scraping. Instead, the focus should be on why web scraping is not the universal solution it is perceived to be.


Limited unit availability


When listing apartment homes on an ILS, operators may deliberately withhold the number of units that appear online. Because of the potentially limited number of units available at the property or, in the case of new development properties, the operator may want to create the appearance of low supply. Either way, if you rely on web scraping, you only get the data that properties are making available.


Leasing Statistics


Leasing statistics, such as occupancy, leased, and availability percentage, are unavailable on any website and cannot be scraped or copied. Moreover, you must remember how much traffic or leases a property had the prior week. These are property-level details that play a significant role in property performance. This knowledge is necessary for decisions based on inaccuracies and limited visibility. 


Pricing Clarity


Likewise, it’s nearly impossible to tell if rents on an ILS or property website are market or lease rents. Some operators even have a call to action that suggests the visitor “Call for Pricing,” meaning their pricing on the website will not give the whole picture. The only way to get actual pricing from web scraping is to know that the property has a daily Revenue Management pricing feed. Otherwise, you may miss out on the concessions offered.


But this is a small window into the world of web scraping. And while it seems like an easy, automated way to procure your data, the information gaps must be addressed. The longer they are ignored, the more negative impact they will have on your performance.



Take advantage of Radix, the cloud-based online multifamily market research and analytics platform, to drive performance and make informed decisions. Access built-in tools that help you understand ambiguous data and effortlessly find information on your competitors and submarkets. With Radix, you can streamline your data analysis process and minimize the risk of human error. Eliminate the need for manual effort and instead focus on making data-driven decisions backed by reliable insights.

Request a demo today to learn more:

About the author

More To Explore

RAOT Blog Thumbnail Week of November th

Rent and Operating Trends – Week of November 26th 2023

It was a fairly quiet week in the U.S. economy as the Thanksgiving holiday limited data releases last week. Existing trends continued from the prior week as the 10-year treasury continued to drift lower. The yield on the 10-year is now 4.42%, nearly 60 basis points below its recent peak in mid-October.

Chart of the Week Blog November th

Chart of The Week – November 27th 2023

This week we examine the top and bottom performing markets on a net effective rent basis. We have talked a lot about the markets that are both outperforming and underperforming in the previous charts of the week and rent and operating trends report, yet a key distinction that continues to hold true is the severity of the declines in the worst performing markets compared to the modest gains in the best performing markets.