View source for File transfer

== Best Practices ==
ALICE has a dedicated data transfer node, sftp-gw.alice.leidenuniv.nl, that is described on the [[ALICE Data Transfer Server]] page. For the fastest possible data transfer rates use sftp-gw.alice.leidenuniv.nl, and transfer files directly to your /data directory. Keep in mind that data on /data is not backed up, so it is good practice to save a copy of the data to your project directory (assuming its file size is smaller than your project’s disk quota), where it will be backed up nightly.

== Options for Data Transfer ==
ALICE supports large file transfer directly–through command-line based Linux utilities such as rsync and sftp. Additionally, you are welcome to install your file transfer applications in your project directory.

=== Between your computer and ALICE ===
For many ALICE users, the most convenient way to transfer files between their computer and ALICE is to use a GUI (graphical user interface)-based SFTP client. The SFTP client is installed and run on your computer. It works by connecting to the SFTP server running on sftp-gw.alice.universiteitleiden.nl, enabling you to transfer files back and forth.

ALICE login credentials are required for access. Most third-party SSH clients can maintain an open connection so as to minimize the number of authentication requests during a transfer session. Refer to the documentation for how to do this for your preferred client.

=== From the Internet to ALICE ===
You can transfer a file from the Internet directly to your project directory on ALICE (without first downloading to your computer). For example, if you want to transfer a repository from github, use the command '''git clone ''REPOSITORY_URL''''', where ''REPOSITORY_URL'' is the link you copied from github. If you want to transfer a file from a web page, you can use the command '''wget ''URL'''''. If you need to transfer data from a private location (i.e., one that requires logging in), the site may or may not allow you to use '''wget''' for the transfer.

If you need to frequently transfer files, plan to move large amounts of data, or need assistance transferring data from a private location, feel free to contact us at helpdesk@alice.universiteitleiden.nl for advice on how to

=== Creating and Editing Files on HPC ===
You can always create files on your personal computer and transfer them to ALICE but sometimes it is easiest to create them directly on ALICE.

HPC supports the vi/vim, gedit, nano and emacs text editors. Nano is used in HPC training sessions because it is an easy editor to learn. Gedit is a good option if you log in with “X11 forwarding” enabled which is pre-configured on USC’s version of X-Win32 and enabled by XQuartz’s X11.app on Mac OS. Vi/vim, which comes standard on all UNIX/Linux machines, and emacs, which is a popular coding environment, both have steeper learning curves.

To edit a file, simply type the editors name, e.g., ''nano'' or ''gedit'', at the command line and then type in your file’s text.

===File transfer from and to Linux and Mac OS===
Before you can transfer files you need access to the ALICE file tranfer node. For details on setting up ssh login, see [[Login to cluster]]. Once you have successfully set up the ssh configuration, you can use two different methods to transfer data: SCP and SFTP.

====SCP file transfer====
You need to know to which file systems on ALICE you want to (and are allowed to) copy your files. See the [[File storage|File systems]] section for detail on which directory structures are available to you. Once you know where to/from you want to copy data use the Unix SCP command to perform the copy operation. Note: you have successfully setup key-based authentication and made an ssh configuration hostname hpc1:
  scp <local_file_name> hpc1:/data/<username>/
where you have previously created a directory on the /data file system
  mkdir /data/<username>
where <username> is your account name. This copies a local file to ALICE. To copy a file on an ALICE file system back to your local desktop or storage medium use:
  scp hpc1:/data<username>/<remote-file-name> .
where the . is representing the current directory. For more details on how to use scp te:
  man scp

====SFTP file transfer====
Data can also be transferred using the sftp copy program. Again we presume you have successfully configured the ssh login with key based authentication. In that case you can simply type:
  $ sftp hpc1:
  #############################################################
  
                 Welkom bij de SSH Gateway van de
                       Universiteit Leiden
  
         Deze gateway dient slechts als SSH toegang tot
         achterliggende systemen. Oneigenlijk gebruik van 
         deze server kan leiden tot het ontzeggen van toegang.
             
                 Welcome to the SSH Gateway of the 
                     University of Leiden
         The only purpose of this gateway is SSH access to other
         systems. Improper use can lead to denial of access.
  
                               N.B.
          Public Key authentication and more is possible,
          see for this SSH-README.txt
          Use screen at the destination.
  
          Voor vragen/For questions: issc.leidenuniv.nl

  #############################################################
  Connected to hpc1.
  Changing to: /home/deuler/.
  sftp>
We can then use various commands to traverse and manipulate both file systems. A list of commands are listed below:

{| class="wikitable"
!Command
!Function
!Example
|-
|cd
|Changes the directory of the remote computer
|cd remote_directory
|-
|lcd
|Changes the directory of the local computer
|lcd local_directory
|-
|ls
|Lists the contents of the remote directory
|ls
|-
|lls
|Lists the contents of the local directory
|lls
|-
|pwd
|Prints working directory of the remote computer
|pwd
|-
|lpwd
|Prints working directory of the local computer
|lpwd
|-
|get
|Copies a file from the remote directory to the local directory
|get remote_file
|-
|put
|Copies a file from the local directory to the remote directory
|put local_file
|-
|exit
|Closes the connection to the remote computer and exits the program
|exit
|-
|help
|Displays application information on using commands
|help
|-
|}

===File transfer from and to Windows===
You are welcome to use any SFTP client you wish. We frequently receive requests for help setting up FileZilla, Cyberduck and WinSCP. For convenience, these instructions are below.

====FileZilla====
You can download FileZilla from the vendor website at <nowiki>https://filezilla-project.org</nowiki>.

Make sure to download FileZilla '''client'''. Once the installation is complete you will need to create an HPC profile. You can do so by going to the '''Site-Manager''':

Create a new site and enable these settings:
*'''Protocol''': SFTP – SSH File Transfer Protocol
*'''Host''': sftp-gw.alice.universiteitleiden.nl
*'''Logon Type''': Interactive
*'''User''': Your ALICE username
After the '''General''' tab settings have been filled out, select the T'''ransfer Settings''' tab:
*Check '''Limit number of simultaneous connections'''
*Maximum number of connections: '''1'''
These settings will keep a single connection open so you will not have to re-authenticate.
====Cyberduck====
You can download Cyberduck from the vendor website at <nowiki>https://cyberduck.io/</nowiki>.

Save your HPC login setting by creating a new “bookmark” with these settings:
*Server: sftp-gw.alice.universiteitleiden.nl
*Username: Your ALICE username

==== WINSCP ====
To transfer files between your desktop and the ALICE file systems you first need to setup your ssh configurations to allow key-based login to the sshgw machine, including the ssh login tunnels as described in [[Login to cluster]]. After this you can startup WinSCP and configure a session to ALICE login node 1:[[File:Winscp1.png|none|thumb]]
Select the 'New Site' and specify as Host Name: localhost and as Port number: 2222 (which are the settings for the tunnel you have running, since you have started putty with the ALICE_GW config). You can now setup the key-based authentication by clicking the Advanced button. In the new window, select SSH and then Authentication. In the  Private key file box, you browse to the private key file you have previously created (see [[Login to cluster]] for details). You may choose from the File protocol drop-down menu either SCP or SFTP. It does not matter for the main window of WinSCP which protocol is used.
[[File:Winscp4.png|none|thumb]]
Once this has been done, save this configuration into a session config by clicking Ok and then clicking Save. Give this configuration a useful name, e.g. ALICE login1, and click Ok.
[[File:Winscp2.png|none|thumb]]
You can now click the Login button. On first access, you may be presented a confirmation window for key exchange. Click Ok.
[[File:Winscp3.png|none|thumb]]
Now you will be presented the WinSCP main window. On the left, you have the local file system and on the right, you have the remote file system on ALICE login node 1, your home directory. 
[[File:Winscp5.png|none|thumb]]
Now you can transfer files to and from ALICE.<br />