Technical Blog Vol. 2
Orchestrating Deep Learning Across Europe's HPC Landscape
Running EU-wide Deep Learning projects for Weather The WeatherGenerator project is an ambitious Europe-wide project to build a “Large Language Model for the weather.” From the outset, this leads to a very practical challenge: how to coordinate the development of a state-of-the-art model across 6 High Performance Computing (HPC) systems and 10 research groups? Each HPC system is custom built with the purpose of maximizing processing power at the expense of ease of use or convenience. This experience is very different compared to using a public cloud such as Amazon Web Services, Azure, Google Cloud. Public clouds are meant to be cost effective to build and operate. They are the Ford T’s of large computing: easy to use, simple, adapted to many scenarios. Public cloud operators put a large focus on standardization and automation as well as ease of use for the final customers.
Timothee Hunter,
Christian Lessig