The construction of a building or public facility is considered over the long term: beyond the requirements associated with urban planning, it is necessary to integrate the way it will evolve over time. Inclement weather, aging materials and, more and more clearly, climate change, will indeed have an impact on the sustainability of the construction and how it functions. For example, the evolution of forest fires or landslides must be taken into account well in advance of a development and public or private actors must therefore best plan for the risks associated with these external factors. But to do so, they depend on these specialized actors who are in particular able to accurately assess the environment in which their territory will evolve over several decades.

The Climate Data Factory, a start-up created in 2015 by a scientific training entrepreneur in response to the many needs and complexity in meeting them, specializes in this climate and associated environmental risks project capacity. “The climate models data is today published and distributed worldwide by about twenty-some research centers, notes Harilaos Loukos, the company’s founder and CEO. But the raw data is most often unsuitable for local operational risk assessment needs as it is available on a very large scale, of about 100 kilometers. Our work thus consists of compiling this information in very high volumes and applying it to sophisticated statistical models to get projections over about ten or so kilometers. Finally, we add a risk modeling layer (heat waves, extreme precipitation or winds, forest fires, landslides, etc.) to extract the full value and make them usable by consultants or engineering firms who assess the climate change physical risks to which their clients' assets are exposed.

Solution | Le Cloud, a performance accelerator

To achieve such data detail, The Climate Data Factory’s teams use climate model simulations which serve as a scientific basis for international negotiations to reduce greenhouse gas emissions which, when applied to the considerable volumes processed (basic calculations require several terabytes of RAM and hundreds of CPUs), involve quite a lot of data manipulation and calculations. “Upon launching our business activity, we decided to rely on an HPC cluster, recalls Thomas Noël, a data engineer for the company. Which offered and still offers acceptable performance, but since the resource is shared, we face problems of availability. What we gain detail in calculations, we encounter bottlenecks. For example, to go from a 25-km to a 10-km scale on our data, we need six times the resources! So we decided to call upon Amazon Web Services, whose storage plan we have already been using, and which seemed to offer an appropriate response, both in terms of processing capacity and HPC infrastructure availability in the cloud”.

The gains were experienced right away: The Climate Data Factory’s team proceeded successively, integrating increasingly complex datasets, gradually validating AWS infrastructure capabilities. "Each phase of the POC validated the previous one, and we now know how to make the most of the tools at our disposal, Thomas Noël adds. But the main gain of this migration is elsewhere: we can now launch our simulations on the fly, whereas previously we had to wait several months and sometimes even a year to have our results due to lack of sufficient availability on the shared cluster. There are therefore many benefits: we not only have a scalable solution to allocate resources adapted to the dimensions of our projects, but we are also able to meet customer demands that we previously could not meet due to lack of availability. This does not complicate our IT, in fact just the opposite: I can manage the entire processing infrastructure”.

Flexibility and availability at meetings

This responsible now makes The Climate Data Factory stand out against its competitors, as it continues to surprise its customers by how quickly it delivers. And their cost: “Beyond the performance of the instances, AWS offers us an full suite of applications which help us to make the most of our resources, Harilaos Loukos boasts. By adapting the type of instances and file system to the type of processing, urgency and criticality of the calculations that we have conducted, we have successfully optimized our stack of resources and our final cost is 10 times lower than what we had originally estimated. This allows us to develop new offers better meeting our client’s needs and time frames, at competitive prices”.

So AWS and The Climate Data Factory have forged a true partnership: beyond the availability of cloud data production instances, the start-up can now access through the ASDI (Amazon Sustainability Data Initiative), a free and open environmental database on sustainable development, climate simulations of the main research laboratories worldwide that it needs. It can also sell datasets directly on the Amazon Web Services marketplace, which opens up new commercial opportunities and is a valuable basis for planning its many future projects.