Strumenti Utente

Strumenti Sito


oph:cluster:storage

Questa è una vecchia versione del documento!


Storage types

The DIFA-OPH cluster offers different storage areas with very different characteristics and limitations.

$HOME

  • This is a storage area available from every node. Should be used for source files, compiling code, jobs that do not need a lot of space.
  • NFS-mount via Ethernet (1Gbps): not very fast but quite responsive.
  • Quota limit: 50GB (soft) / 100GB (hard)

/home/public

/home/work

  • Used as work area for jobs that do not need very big datasets or need to have lots of files in a single directory (not recommended, might degrade performances for all users!).

/scratch

  • Main archive area. To be used for large files, big datasets and archives.
  • IMPORTANT: DO NOT USE FOR LOTS OF SMALL FILES!!!: /scratch is a clustered filesystem that on one hand allows to store big files and to scale out capacity by adding more servers, but on the other hand introduces quite a lot of overhead (it must contact 3 servers to determine where to get a file from, for example) that won't get amortized with small files. Way worse are directory lookups, that nearly crawl to a halt when a directory contains more that about a thousand files. If you need some big files (say a dataset) and your codes do not allow for specifying a different path, just use symlinks from $HOME to the files in /scratch.

Quota: per-sector, with extra space allocated for specific projects or bought by individuals.

oph/cluster/storage.1681285951.txt.gz · Ultima modifica: 2023/04/12 07:52 da marco.baldi5@unibo.it

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki