HLRN Brings Advanced Performance to HPC

HLRN chose Intel® Xeon® Platinum 9200 processors to meet their increasingly diverse needs for HPC workloads.

Executive Summary
HLRN supercomputers are used by over 100 universities and over 120 research institutions enabling exploration of the many frontiers of scientific research to help unlock a better future. The selection of Intel’s latest processor technology to power the newest HLRN supercomputer came after detailed testing to find the best solution. Prof. Dr. Ramin Yahyapour of Göttingen University explains “the expectation for HLRN’s supercomputer acquisition was to have a significant step up in computer power for new experiments”.

“Science in general is getting more compute and data intensive. This means that having larger systems available translates into an ability for the scientists to do better work. That’s why HLRN is crucial for scientific research,” says Prof. Dr. Ramin Yahyapour.

HLRN lays claim to being a very demanding client—HLRN has substantial expertise from their prior deployments of three supercomputer systems. Prof. Alexander Reinefeld from Zuse Institute Berlin emphasizes that “We are expecting the highest performance for all benchmark applications. Our benchmark suite was carefully chosen so that each code challenges specific parts of the system: CPU, communication network, and parallel I/O. We are not looking for peak theoretical performance—we demand real system performance, which makes it more complicated for vendors to optimize their infrastructure for our applications. That meant that our selection of the right processor and the right interconnect are all crucial for the overall performance”.

As with most research today, the need for more real-world computer capacity stems from the reality that simulations of many kinds are critical to the researchers. Faster computers are primarily used to increase the simulation in size and resolution—with the expectation of finding new discoveries.

“We demand real system performance… that meant that our selection of the right processor and the right interconnect are all crucial for the overall performance.” — Prof. Reinefeld

HLRN procured a new supercomputer with just under a quarter of a million cores. The Intel® Xeon® Platinum 9200 processors (from the 2nd Generation Intel® Xeon® Scalable processor family) will serve as the “right processors” for HLRN. For the “right interconnect,” HLRN chose Intel® Omni-Path Architecture (Intel® OPA). The system is produced by Atos (formerly Bull Computing) and will be physically split between the Zuse-Institute Berlin (ZIB) and the Georg-August-Universität Göttingen (University of Göttingen). These sites have previously used this split system model, and already have in place a dedicated, redundant, 10 gigabit, fiber optic cable spanning the more than 170 miles between Berlin and Göttingen.

Researchers at ZIB will use HLRN-IV for fluid dynamics, including developing turbulence models for aircraft wings.

HLRN has announced that the new system, HLRN-IV, will be approximately six times as fast as the prior systems—offering 16 PetaFLOP/s performance.1 The excitement among researchers is palpable, and the list of research being done is mind-boggling. Prof. Reinefeld summed up his excitement saying, “It’s a great system. Our users will benefit right away from the more powerful system without needing to change their code. The homogeneous architecture of the 2nd Gen Intel® Xeon® Scalable processors will provide true performance portability, which is a crucial aspect for our researchers in order to quickly benefit from the new, more powerful system”.

Key research areas within HLRN include:

  • Earth System Sciences - Which includes work on climate change. Subjects include the dynamics of oceans, rain forests, glaciers, Antarctic phytoplankton (microalgae), mineral dust cycles, and the stratosphere.
  • Fluid Dynamics - Which includes turbulence models for ship turbines, wind turbines, and aircraft wings. These models are notorious for needing enormous compute power—the acquisition of HLRN-IV will enable the running of more fine-grained turbulent simulations on large systems such as wind flow through a city, or across a blade on a turbine. Modeling complete cities will allow studies in how new buildings would change wind flow, and other factors that impact various microclimates within the city. This may lead to new design aspects to enhance city life. Other researchers hope to gain understanding that will pave the way for future high-lift commercial aircraft. Other researchers are hoping to save lives and ships by studying liquefaction of solid bulk cargo (such as iron ore or nickel ore). Failure to properly deal with this issue has led to the complete loss of at least seven vessels around the world in the past decade.
  • Healthcare - Is a broad area of research, and HLRN researchers hope to help in many ways including improving medical care at home. Gaining a better understanding of illness and treatment of diseases stands to impact us all. Research includes simulations of drug efficacy, interactions, and side-effects. Enormous compute power allows leading researchers in these fields to start exploring the “personal medicine” aspects of these simulations, not just the average effects on a general population.

At the University of Göttingen, research areas include collaborative projects on cellular and molecular machines.

High Performance Across Diverse Research
In terms of science communities, HLRN has to support all types of workloads for their many researchers. Therefore, HLRN systems need to have the characteristics of a general purpose system but still be of the highest performance. Their final choice had no accelerators.

“Although we looked at accelerators, including GPUs, as part of the procurement process, there was no advantage with regards to obtaining the highest performance in using GPUs or other accelerators in the system.”— Dr. Thomas Steinke, Head of ZIB Supercomputing

HLRN’s benchmarks are open and include benchmarks that can take advantage of GPUs. HLRN found that any advantage in performance on some workloads are insufficient, when considering the reduction in general purpose compute capacity, or additional costs involved. A homogeneous system based on the 2nd Gen Intel® Xeon® Scalable processors proved itself to be the best choice for the diverse needs of the HLRN scientists and researchers.

Beating Back Amdahl’s Law
Ever mindful of Amdahl’s Law, Dr. Thomas Steinke is fond of emphasizing the use of fast algorithms for fast computers. He shared that “The pressure of optimizing code for scaling on a node is less because of the high real-world performance of the 2nd Gen Intel® Xeon® Scalable processors compared to previous many-core architectures”.

The 2nd Gen Intel® Xeon® Scalable processor family offers an outstanding choice for high performance computing (HPC) and helps programmers cope with Amdahl’s Law.

“Our users will benefit right away from the more powerful system without needing to change their code.”— Prof. Reinefeld

Future of AI in HPC
AI and Machine Learning stand to impact all areas of HLRN research. A hot area of interest is the blending of machine learning and AI techniques with traditional simulation capabilities. While promising results have been reported, there is much work to be done. The exploration of algorithms is likely to take researchers in many directions, and this need for flexibility is one reason HLRN chose 2nd Gen Intel® Xeon® Scalable processors to support their next generation of research.

Avoid Data Movement
Prof. Yahyapour emphasized that “the CPU is quite good for artificial intelligence and machine learning. That’s an area where we see more need from our researchers. Traditionally they were not so much into data intensive work but that’s something we see as a new trend for the new system that will also be of particular interest”.

Intel® Advanced Vector Extensions 512 (Intel® AVX-512) proved to be the logical choice to help increase HLRN’s compute power, and with the addition of Intel® Deep Learning Boost (Intel® DL Boost) to augment AVX-512, offered outstanding performance for the new frontier of HPC applications.

The ability to compute data where it is, for all types of algorithms, saves data movement. That represents a boost for compute capacity, and less wasted energy. A double win!

When exploring new algorithms, and new application techniques, nothing is more important than the flexibility of a system. The 2nd Gen Intel® Xeon® Scalable processor delivers high performance coupled with the flexibility needed to meet future challenges.

Explore Related Intel® Products

Intel® Xeon® Scalable Processors

Drive actionable insight, count on hardware-based security, and deploy dynamic service delivery with Intel® Xeon® Scalable processors.

Learn more

Intel® Deep Learning Boost (Intel® DL Boost)

Intel® Xeon® Scalable processors take embedded AI performance to the next level with Intel® Deep Learning Boost (Intel® DL Boost).

Learn more

Intel® Omni-Path Architecture (Intel® OPA)

Intel® Omni-Path Architecture (Intel® OPA) lowers system TCO while providing reliability, high performance, and extreme scalability.

Learn more


インテル® テクノロジーの機能と利点はシステム構成によって異なり、対応するハードウェアやソフトウェア、またはサービスの有効化が必要となる場合があります。実際の性能はシステム構成によって異なります。絶対的なセキュリティーを提供できるコンピューター・システムはありません。詳細については、各システムメーカーまたは販売店にお問い合わせいただくか、http://www.intel.co.jp を参照してください。// 性能に関するテストに使用されるソフトウェアとワークロードは、性能がインテル® マイクロプロセッサーだけに最適化されていることがあります。SYSmark* や MobileMark* などの性能テストは、特定のコンピューター・システム、コンポーネント、ソフトウェア、操作、機能を使用して測定したものです。結果はこれらの要因によって異なります。製品の購入を検討される場合は、他の製品と組み合わせた場合の本製品の性能など、ほかの情報や性能テストも参考にして、パフォーマンスを総合的に評価することをお勧めします。詳細については、https://www.intel.co.jp/benchmarks (英語) を参照してください。// 性能の測定結果はシステム構成の詳細に記載された日付時点のテストに基づいています。また、現在公開中のすべてのセキュリティー・アップデートが適用されているとは限りません。詳細については、公開されている構成情報を参照してください。絶対的なセキュリティーを提供できる製品やコンポーネントはありません。// 記載されているコスト削減シナリオは、指定の状況と構成で、特定のインテル® プロセッサー搭載製品が将来のコストに及ぼす影響と実現されるコスト削減の例を示すためのものです。状況によって異なる可能性があります。インテルは、いかなるコストもコスト削減も保証いたしません。// インテルは、本資料で参照しているサードパーティーのベンチマーク・データまたはウェブサイトについて管理や監査を行っていません。本資料で参照しているウェブサイトにアクセスし、本資料で参照しているデータが正確かどうかを確認してください。// いくつかのテスト結果は、インテル社内での分析またはアーキテクチャーのシミュレーションあるいはモデリングで推定 / シュミレートされており、情報提供を目的として提供されています。システム・ハードウェア、ソフトウェア、構成などの違いにより、実際の性能は掲載された性能テストや評価とは異なる場合があります。



以前のシステムである HLRN-III は、ベルリンのZIB およびハノーバーのライプニッツ大学 IT サービス (LUIS) にある 2 基の複合システムから構成され、HLRN 専用の 10GigE 光ファイバー接続で連結されていわゆる単一システムビューを提供します。2 フェーズで提供されるコンピュート・ノードの詳細は以下の通りです。フェーズ 1 は、それぞれ 744 のコンピュート・ノードで構成され、合計 1488 のデュアルソケット のインテル® ジーオン® プロセッサー E5-2695v2 と合計 93 TB のメインメモリーを搭載した 2 台の Cray XC30 コンピューターを含み、これは Dragonfly トポロジの高速 Cray Aries ネットワーク経由で接続されました。フェーズ 2 では、85248 のコンピュート・コアを持つ 2064 のインテル® ジーオン® プロセッサー E5-2680 v3 コンピュート・ノードが、ベルリンの 1872 のコンピュート・ノードおよび ハノーバーの 1680 のコンピュート・ノードに追加され、合計 2.7 ペタフロップス / 秒のピーク・パフォーマンスと 222 TB のメインメモリー拡張となりました。