Files and storage on Unix

Your home directory on the Unix cluster has a usage quota, to avoid too much space being taken by individual users.

To find your quota and current usage, you can use this command:

quota

This tells you your usage in 1 Kbyte disk blocks. To find out more about the quota command, type man quota.

The quota has both a soft limit, which you are able to exceed for a limited grace period, and a hard limit, beyond which the system won't let you use any more space. If you exceed your soft limit, the system should send you an email message periodically telling you of your current quota and usage. If you need more space, you can reply to the email to request a change from us.

The home directories are backed up nightly. You can retrieve recently-deleted files from backup yourself.

Other filesystems are provided for research or project-specific data, under the /data hierarchy. These filesystems are usually named either after the research group group, or with the name of the computer which hosts it, and contain further directories organized by user or by project. These areas should be used for large data sets and storage for local processes. Warning these areas are generally not backed up, other than by special request. If you have a large amount of data which requires backup, you should talk to us about the available options.

Filesystems under /scratch may be used for scratch space for local processes. Don't place any files you may want to keep long-term here - files which have not been accessed for 30 days or more may be purged from this area, or when a workstation is updated.

Your scratch directory on the local system is always named /scratch/local/username, and can be accessed using the environment variable SCRATCHDIR. You can access non-local scratch directories using the path /scratch/hostname/username.

If running jobs under Condor, the environment variable CONDOR_SCRATCH_DIR gives the name of a directory where the job may place temporary data files.

Don't use /tmp for creating temporary files, if you can help it. This area is very limited in size, and filling it will cause problems for the system.

We define an environment variable TMPDIR, which points to a suitable (and larger) area for temporary files. Though you should consider whether SCRATCHDIR might be more suitable for your purpose…

New with scientific linux 5.x you can mount any filesystem you have ssh access to as a filesystem using fuse (the userspace filesystem driver).

mkdir ~/mnt  #create a place to put it, can be called anything you want.
sshfs username@remotesystem:/path/to/mount ~/mnt

later, to unmount it:

fusermount -u ~/mnt