Numerical Weather Prediction (NWP) is a field that is highly data-intensive. Meteorological data is produced and processed in enormous amounts with very tight, time-critical, constraints. To produce a global forecast, millions of world-wide measurements and satellite images must be analysed and then fed into a model of the atmosphere to compute raw solution fields that are subsequently post-processed to generate millions of specialised user-driven products.
Historically, the tools and models that make up this complex chain have been hand-tuned and parallelised with MPI/OpenMP and run on special, purpose-built supercomputers. Written in monolithic Fortran, the code base for these models has become hard to maintain and difficult to adapt to rapidly evolving concurrent hardware like GPGPUs and many-core CPUs. The NWP community has started to recognise the need to embrace modern software techniques for the many-core era, with special emphasis on high-performance languages like C++ and the use of DSL's.
In a way, H. Sutter's "free-lunch" has been paid for in advance by NWP: its software models have been concurrent and distributed since the ‘90s. While there is still much work to do in model scalability, the challenges ahead also include: electrical power limits, data I/O rates, exponential volume growth, etc.
We will show how we are addressing these challenges, and present some solutions to modernise our software stack. In particular we will present a C++ DSEL inspired in functional programming, for building pipelines for parallel processing of meteorological data. Scientists with little or no knowledge of C++, can extend this DSEL to pre-process meteorological observations and post-process model output to generate meteorological products.
Slides
