Difference between revisions of "File transfer"
From ALICE Documentation
(→SCP file transfer)
(→SFTP file transfer)
|Line 35:||Line 35:|
== SFTP file transfer ==
$ sftp hpc1:
$ sftp hpc1:
Revision as of 09:49, 3 April 2020
- 1 Best Practices
- 2 Options for Data Transfer
- 3 File transfer from and to Linux and Mac OS
- 4 File transfer from and to Windows
Options for Data Transfer
Between your computer and ALICE
ALICE login credentials are required for access. Most third-party SSH clients can maintain an open connection so as to minimize the number of authentication requests during a transfer session. Refer to the documentation for how to do this for your preferred client.
From the Internet to ALICE
You can transfer a file from the Internet directly to your project directory on ALICE (without first downloading to your computer). For example, if you want to transfer a repository from github, use the command git clone REPOSITORY_URL, where REPOSITORY_URL is the link you copied from github. If you want to transfer a file from a web page, you can use the command wget URL. If you need to transfer data from a private location (i.e., one that requires logging in), the site may or may not allow you to use wget for the transfer.
Creating and Editing Files on ALICE
HPC supports the vi/vim, gedit, nano and emacs text editors. Nano is used in HPC training sessions because it is an easy editor to learn. Gedit is a good option if you log in with “X11 forwarding” enabled which is pre-configured on USC’s version of X-Win32 and enabled by XQuartz’s X11.app on Mac OS. Vi/vim, which comes standard on all UNIX/Linux machines, and emacs, which is a popular coding environment, both have steeper learning curves.
File transfer from and to Linux and Mac OS
Before you can transfer files you need access to the ALICE file transfer node. For details on setting up ssh login, see Login to cluster. Once you have successfully set up the ssh configuration, you can use two different methods to transfer data: SCP and SFTP.
SCP file transfer
scp <local_file_name> hpc1:/data/<username>/
where you have previously created a directory on the /data file system
scp hpc1:/data<username>/<remote-file-name> .
where the . is representing the current directory. For more details on how to use scp te:
SFTP file transfer
|cd||Changes the directory of the remote computer||cd remote_directory|
|lcd||Changes the directory of the local computer||lcd local_directory|
|ls||Lists the contents of the remote directory||ls|
|lls||Lists the contents of the local directory||lls|
|pwd||Prints working directory of the remote computer||pwd|
|lpwd||Prints working directory of the local computer||lpwd|
|get||Copies a file from the remote directory to the local directory||get remote_file|
|put||Copies a file from the local directory to the remote directory||put local_file|
|exit||Closes the connection to the remote computer and exits the program||exit|
|help||Displays application information on using commands||help|
File transfer from and to Windows
You are welcome to use any SFTP client you wish. We frequently receive requests for help setting up FileZilla, Cyberduck and WinSCP. For convenience, these instructions are below.
You can download FileZilla from the vendor website at https://filezilla-project.org.
Make sure to download FileZilla client. Once the installation is complete you will need to create an HPC profile. You can do so by going to the Site-Manager:
Create a new site and enable these settings:
- Protocol: SFTP – SSH File Transfer Protocol
- Host: sftp-gw.alice.universiteitleiden.nl
- Logon Type: Interactive
- User: Your ALICE username
After the General tab settings have been filled out, select the Transfer Settings tab:
- Check Limit number of simultaneous connections
- Maximum number of connections: 1
These settings will keep a single connection open so you will not have to re-authenticate.
You can download Cyberduck from the vendor website at https://cyberduck.io/.
Save your HPC login setting by creating a new “bookmark” with these settings:
- Server: sftp-gw.alice.universiteitleiden.nl
- Username: Your ALICE username
Now you can transfer files to and from ALICE.