Skip to main content

Access your files

The cluster has a parallel file system which provides fast and reliable access to your data. You should have at least 3 file sets:

  1. User home directory
    • For storing your persistent data
    • Located at /pfss/home/$USER
    • Environment variable: $HOME
    • The default quota is 10GB
  2. User scratch directory
    • To be used for I/O during your job
    • Located at /pfss/scratch01/$USER
    • Environment variable: $SCRATCH
    • The default quota is 100GB
    • Inactive files may be purged every 30 days
  3. Group scratch directory
    • For storing your files with your group mate
    • Located at/pfss/scratch02/$GROUP
    • Environment variable: $SCRATCH_<GROUP NAME>
    • The default quota is 1TB

There are three different ways to download or upload your files.

Web portal

We built a file browser in the portal. You may open it up by clicking files in the top menu bar.

files.png

You may switch between different file sets by clicking the hard disk icon. On top of that is the quota and consumption of the current file set. In the top right corner, there is a button for you to upload files. You may upload multiple files at once under a limit of 10GB. If you are uploading 1000+ files, we recommend you zip it before uploading, then unzip it in the web terminal.

Click on the item name to enter a directory or download a file. If it is a .sbatch file, the job launcher will be fired for enqueuing a new job. If it is a plain text file or any format that your browser can handle, it will be opened in a new tab. This comes in handy when viewing your job result without downloading it. There is a dropdown menu on the right-hand side to copy the path, rename, or delete an item.

SFTP

Another way is to use your favorite SFTP client. Download your access key from the home page. Below is an example using the Linux SFTP client.

# a strict file permission is required
chmod 400 loki-pri.pem

# remember to change loki to your own ID
sftp -i loki-pri.pem [email protected]

Mount to your local computer

When troubleshooting or profiling your application, you may check output files more often. So we provided a shortcut to mount the file set to your local computer using our CLI client.

It depends on SSHFS and is supported only in Linux and Mac OSX. Please make sure the command line sshfs is accessible.

First, you need to download and set up the CLI client on your computer. You should find everything on the web portal home page. Please make sure "Never send private key" is set to no to allow establishing an SSH connection through the client.

cliclient.png

Then you may list and mount file sets as follows:

$ hc filesystem-ls
ID                     | Type        | Usage (GB)  | Limit (GB)  | Usage
home                   | USR         | 2.631       | 10          | 27%
scratch                | USR         | 0           | 100         | 0%
scratch_appcara        | GRP         | 0           | 1000        | 0%

$ hc filesystem-mount -t home -m ~/hpc-home
Mounted successfully.
Please use 'fusermount -u /home/loki/hpc-home' to unmount.

$ fusermount -u /home/loki/hpc-home
Back to top