File transfer from and to Linux and Mac OS

Before you can transfer files you need access to the ALICE file transfer node. For details on setting up ssh login, see Login to cluster. Once you have successfully set up the ssh configuration, you can use two different methods to transfer data: SCP and SFTP.

SCP file transfer

You need to know to which file systems on ALICE you want to (and are allowed to) copy your files. See the Summary of available file systems for detail on which directory structures are available to you. Once you know where to/from you want to copy data use the Unix SCP command to perform the copy operation. Note: you have successfully made an ssh configuration hostname hpc1:

 scp <local_file_name> hpc1:data/<some_directory>

where <username> is your account name. If you do not have setup key-based authentication, scp will ask you for your password for the ssh-gateway and the login node. This copies a local file to ALICE assuming you have already created the directory <some_directory> in your directory /data/<username> on the shared scratch file system. If the directory does not exists yet, you have to create it first by connecting to ALICE with ssh

 ssh hpc1
 mkdir ./data/<some_directory>
 logout

The example here makes use of the fact that ssh will land you in your home-directory by default and that the home-directory contains a symbolic link to your directory on the shared scratch file system.

To copy a file on an ALICE file system back to your local desktop or storage medium use:
 scp hpc1:data/<username>/<remote_directory>/<remote_file_name> ./

where the . is representing the current directory.

You can also copy an entire directory (including its sub-directories) to ALICE. This only requires adding the -r option to scp. For example, copying from your local computer to ALICE can look something like this:

 scp -r <local_directory> hpc1:data/<username>/

or to copy from ALICE to your local computer while using your local computer

 scp -r hpc1:data/<username>/<remote_directory> ./

For more details on how to use scp te:

 man scp

SFTP file transfer

Data can also be transferred using the sftp copy program. Again we presume you have successfully configured the ssh login with key based authentication. In that case you can simply type:

 $ sftp hpc1:
 #############################################################
 
                Welkom bij de SSH Gateway van de
                      Universiteit Leiden
 
        Deze gateway dient slechts als SSH toegang tot
        achterliggende systemen. Oneigenlijk gebruik van 
        deze server kan leiden tot het ontzeggen van toegang.
            
                Welcome to the SSH Gateway of the 
                    University of Leiden
        The only purpose of this gateway is SSH access to other
        systems. Improper use can lead to denial of access.
 
                              N.B.
         Public Key authentication and more is possible,
         see for this SSH-README.txt
         Use screen at the destination.
 
         Voor vragen/For questions: issc.leidenuniv.nl

 #############################################################
 Connected to hpc1.
 Changing to: /home/deuler/.
 sftp>

We can then use various commands to traverse and manipulate both file systems. A list of commands are listed below:

Command	Function	Example
cd	Changes the directory of the remote computer	cd remote_directory
lcd	Changes the directory of the local computer	lcd local_directory
ls	Lists the contents of the remote directory	ls
lls	Lists the contents of the local directory	lls
pwd	Prints working directory of the remote computer	pwd
lpwd	Prints working directory of the local computer	lpwd
get	Copies a file from the remote directory to the local directory	get remote_file
put	Copies a file from the local directory to the remote directory	put local_file
exit	Closes the connection to the remote computer and exits the program	exit
help	Displays application information on using commands	help

File transfer from and to Linux and Mac OS

From ALICE Documentation