Centralized Mass Storage System

Long-term experience of ACC CYFRONET AGH in supporting scientific communities indicates that one of the most important and recurring problems is the provision of a sufficient quantity of efficient mass storage space.

Therefore, in 2007 a new, advanced storage system was deployed at the Centre. This system, based on the cutting-edge HP EVA8000, HP EVA8100 and HP XP12000 disk arrays, is attached to the Centre's computing infrastructure through a SAN (Storage Area Network). It is highly flexible, resilient and efficient. The SAN infrastructure is characterized by high availability, in terms of redundancy of access paths and power supplies. This applies both to SAN switches and the disk arrays. The solution deployed at ACC CYFRONET AGH is resistant to the failure of any single switch, any single RAID controller in the disk array, any single disk loop and any single power supply. The installed software enables automatic switching of access paths in case of a failure (the so-called path failover mechanism). Load balancing technology enables simultaneous use of redundant paths under normal operating conditions in order to increase data transfer rates.

In 2008, the Mass Storage System was extended with a cluster of Sun X4540 servers, which make the basis of a new object-oriented distributed file system utilizing Lustre software. This solution has already established its strong position in High Performance Computing (HPC) Centres, as it offers extremely high efficiency, flexibility and scalability. The Lustre architecture allows for utilizing economy servers equipped with commonly used SATA disks, as well as resources of advanced disk arrays working in SAN environment, to create physical storage space for data.

Active participation of ACC CYFRONET AGH in research projects, such as PL-Grid and PLATON, resulted in dynamic growth of potential of the Mass Storage System. New FC switches made it possible to increase the interconnection bandwidth of the Storage Area Network to 8 Gb/s per transmission path. Additional storage servers with HP MDS600 high density disk enclosures as well as efficient SGI InfiniteStorage 4600 disk arrays provide the necessary storage capacity for storing our key users' data. One of activities realized under the PLATON project is the Universal Archiving Service, which is intended to allow backup of user data in a distributed network of data centers. In case of primary site failure, recovery will be possible from replicas located in other sites. To ensure sufficient capacity, Mass Storage System resources are increased by modern tape library IBM TS3500 with a capacity of 2.5 PB plus dedicated disk arrays with high density drawers and servers for installation of specialized software, including IBM Tivoli Storage Manager.

In 2011, a new, high-performance and highly available NAS file-sharing system has been implemented. It is based on hardware clustered file servers (called filers) HNAS 3080 Hitachi Data Systems, combined with Hitachi's AMS2500 array, equipped with 175 x 600 GB SAS drives with 15 000 rpm speed. The solution provides very high efficiency, and is able to reach up to 80 000 IOPS (I/O operations per second) in real production environment. Performance achieved by the new system is far greater than the one obtained from classic NFS servers. The implementation of the HNAS system allowed to remove performance bottlenecks in access to specialized scientific software installed on Zeus cluster and also to significantly reduce the load of conventional center NFS servers used in Cyfronet's environment. An additional benefit of the new solution is the ability to connect high-performance disk resources of AMS2500 array to servers which do not have a dedicated communication interface to the SAN network, by using 10Gb iSCSI ports. In 2012, two new SGI IS5500 disk arrays have been purchased to expand the capacity and performance of the scratch file system for Zeus supercomputer. Currently, this space is based on two SGI IS4600 and two SGI IS5500 disk arrays, which gives it a 720 TB of highly available storage and about 10 GB/s of sustained performance.

 

Resources of Mass Storage System

  • HP StorageWorks XP12000 disk array – 6.4 TB (FC)
  • HP StorageWorks EVA8000 disk array – 120 TB (FATA)
  • HP StorageWorks EVA8100 disk array – 91 TB (FATA)
  • 2 x SGI InfiniteStorage 4600 disk arrays – 480 TB (SATA)
  • SGI InfiniteStorage 5000 disk array – 7.2 TB (SAS)
  • 2 x SGI InfiniteStorage 5500 disk arrays – 240 TB (SAS)
  • HDS AMS2500 disk array – 105 TB (SAS)
  • Sun Fire X4500 server – 36 TB (SATA)
  • 6 x Sun Fire X4540 servers – 288 TB (SATA)
  • HP Blade storage servers – 840 TB (SAS)
  • 2 x HDS HNAS 3080 filers
  • HP StorageWorks ESL712e tape library:
    • 6 x Ultrium LTO-3 tape drives
    • 6 x Ultrium LTO-4 tape drives
    • 636 slots for LTO tapes.

 Total physical storage space 2.468 PB, including:

  • 13.6 TB on high-performance FC disks
  • 211 TB on economy FATA disks
  • 1192 TB on SAS disks
  • 1052 TB on economy SATA disks.

 Resources of PLATON - Universal Archiving Services

  • IBM System Storage DS5300 disk array – 200 TB (SATA)
  • IBM System Storage DS5020 disk array – 6.75 TB (FC), 48 TB (SATA)
  • IBM System Storage TS3500 tape library
    • 16 x Ultrium LTO-5 tape drives
    • 2003 slots for LTO tapes.