StyM
11-15-2016, 05:16 AM
http://www.tomshardware.com/news/seagate-clusterstor-nytro-hpc-supercomputer,33018.html
The supercomputing world is full of multi-million, and sometimes multi-billion, dollar compute clusters that feature the most powerful compute capabilities on the planet. Often, orchestrating tens of thousands of CPUs, GPUs, Knights Landing or FPGAs working in concert across a bleeding-edge networking infrastructure all comes to a crashing halt due to the ever-present nemesis of primary storage.
Seagate announced the ClusterStor 300N at Supercomputing 2016 as its solution to the storage bottleneck. The first step to jumping the storage hurdle is to employ the correct high-performance file system, so Seagate offers the Lustre parallel distributed file system and Spectrum Scale (licensed GPFS Global Parallel File System from IBM), which are two of the most popular HPC filesystems.
However, the hardware is the big change to the ClusterStor 300N equation. Supercomputers require massive bursts of data either to or from the compute cluster during normal operations (up to and beyond 100GB/s), such as during a checkpointing operation, which easily overwhelms the primary storage system. The data bursts are transitory in nature, and building the entire storage system to handle the speed of the bursts increases costs dramatically, which isn't the best solution when data bursts only occur for a small fraction of the operating time.
Burst buffers are the most common solution. A burst buffer is a dedicated intermediary appliance packed to the gills with RAM and flash technology that sets between the compute cluster and primary storage. The burst buffer absorbs the overwhelming and short-lived bursts and then flushes the data to the HDD layer at its leisure. The buffer removes any performance penalties from the slow disk-based storage and negates the need to upgrade the entire storage subsystem, and while it adds cost, it's always more cost-effective than upgrading the entire system. Many burst buffers start in the $500,000 range and move up from there, and they also require proprietary agents running on the compute cluster, which steals cycles. Some also require specialized non-POSIX compliant file systems.
http://media.bestofmicro.com/ext/aHR0cDovL21lZGlhLmJlc3RvZm1pY3JvLmNvbS8zL0EvNjI4Nj c4L29yaWdpbmFsLzAxLkpQRw==/r_600x450.JPG
http://media.bestofmicro.com/3/B/628679/original/02.JPG
The supercomputing world is full of multi-million, and sometimes multi-billion, dollar compute clusters that feature the most powerful compute capabilities on the planet. Often, orchestrating tens of thousands of CPUs, GPUs, Knights Landing or FPGAs working in concert across a bleeding-edge networking infrastructure all comes to a crashing halt due to the ever-present nemesis of primary storage.
Seagate announced the ClusterStor 300N at Supercomputing 2016 as its solution to the storage bottleneck. The first step to jumping the storage hurdle is to employ the correct high-performance file system, so Seagate offers the Lustre parallel distributed file system and Spectrum Scale (licensed GPFS Global Parallel File System from IBM), which are two of the most popular HPC filesystems.
However, the hardware is the big change to the ClusterStor 300N equation. Supercomputers require massive bursts of data either to or from the compute cluster during normal operations (up to and beyond 100GB/s), such as during a checkpointing operation, which easily overwhelms the primary storage system. The data bursts are transitory in nature, and building the entire storage system to handle the speed of the bursts increases costs dramatically, which isn't the best solution when data bursts only occur for a small fraction of the operating time.
Burst buffers are the most common solution. A burst buffer is a dedicated intermediary appliance packed to the gills with RAM and flash technology that sets between the compute cluster and primary storage. The burst buffer absorbs the overwhelming and short-lived bursts and then flushes the data to the HDD layer at its leisure. The buffer removes any performance penalties from the slow disk-based storage and negates the need to upgrade the entire storage subsystem, and while it adds cost, it's always more cost-effective than upgrading the entire system. Many burst buffers start in the $500,000 range and move up from there, and they also require proprietary agents running on the compute cluster, which steals cycles. Some also require specialized non-POSIX compliant file systems.
http://media.bestofmicro.com/ext/aHR0cDovL21lZGlhLmJlc3RvZm1pY3JvLmNvbS8zL0EvNjI4Nj c4L29yaWdpbmFsLzAxLkpQRw==/r_600x450.JPG
http://media.bestofmicro.com/3/B/628679/original/02.JPG