ePhi:
Processing enormous sets of data from Google Maps and automatically detecting targets with AWS
Location:
Paris, France
Technology:
Amazon Web Services
Client
ePhi is an environmental services company from France that provides cleaning services for the industrial use
The Challenge
The business challenge was simple but interesting: the ePhi team wanted to process satellite imagery of large tracts of industrialized areas in numerous cities. They needed to review every square meter of land and index all the buildings that could be considered “commercial development targets.” Due to the enormous data sets, automated data processing solutions had to be implemented.
Initially, they desired to implement custom AI-based solutions in-house, but it wasn’t cost-effective enough to proceed. Time was also an issue.
Luckily, the Flobotics team was ready to step in!
Solution:
After collecting all the project requirements, we had to decide on the right tools and technology.
We decided to initiate the map scraping process using Google Maps. The reason? After doing the cost calculations, we found that it perfectly balanced the satellite images' cost and quality.
In perspective, with Google Maps scraping entire metropolitan areas, costs are usually less than $90 (depending on the metro area size). Using API assures that resulting images will be up to spec in terms of resolution and size. The low development complexity was also a factor, as it brought the costs of the whole project down.
We had to deal with only one issue: Google Maps Static API is not meant to provide consistent footage for Machine Learning purposes. To solve it, we have written a custom algorithm that refines and enriches the data returned from the API on the go.
For map scraping automation technology, we went with Amazon Web Services. Since the API has rate limits imposed, we had to run it for many hours and even days at a time. AWS enabled us to launch the process in a fire-and-forget fashion. Furthermore, we opted to use AWS S3 to store the captured images. That enabled easy integration with Amazon’s Machine Learning services like SageMaker.
The Outcome
We developed an algorithm to capture, organize, and effortlessly share images from Google Maps with consistent scale, definition, and deterministic running costs. Images were then automatically scraped and transformed into readable data for future needs.