File transfer

File transfer

There are several possible ways to transfer files and data to and from HPC2N's systems.

FTP

FTP (File Transfer Protocol) is a simple data transfer mechanism. FTP is the original program for data transfer, but  it was not designed for secure communications. FTP exists on the systems, but HPC2N does not permit connections using FTP because of the security problems. There are several modern FTP clients which support either SFTP or SCP which are similar, secure protocols for file transfer. Use one of those methods instead of FTP.

SCP

SCP (Secure CoPy) is a simple way of transferring files between two machines that use the SSH (Secure SHell) protocol. You may use SCP to connect to any system where you have SSH (log-in) access. There are some graphical file transfer programs which offers SCP as protocol and it is also a command line program on most Linux, Unix, and Mac OS X systems. SCP can copy single files, but will also recursively copy directory contents if given a directory name.

Command-line usage:

(to a remote system from local)

$ scp sourcefilename user@hostname:somedir/destfilename

example:

$ scp irf.png user@machine.umu.se:C/irf.png
Password: 
irf.png                   100% 2100     2.1KB/s   00:00    

(from a remote system to local)

$ scp user@hostname:somedir/sourcefilename destfilename

example:

$ scp user@machine.umu.se:irf.png irfpic.png
Password: 
irf.png                   100% 2100     2.1KB/s   00:00    

(recursive directory copy to a remote system from local)

$ scp sourcedirectory/ user@hostname:somedir/

Installation

Linux / Solaris / AIX / HP-UX / Unix:

  • The "scp" command line program should already be installed.

Microsoft Windows:

  • WinSCP is a full-featured and free graphical SCP and SFTP client.
  • PuTTY also offers "pscp.exe", which is an extremely small program and a basic SCP client.
  • Secure FX is a commercial SCP and SFTP client.

Mac OS X:

  • The "scp" command line program should already be installed. You may start a local terminal window from "Applications->Utilities".

SFTP

SFTP (SSH File Transfer Protocol or sometimes called Secure File Transfer Protocol) is a rnetwork protocol that provides file transfer over a reliable data stream. You may use SFTP to connect to most of HPC2N's systems. SFTP is a command -line program  on most Unix, Linux, and Mac OS X systems. It is also available as a protocol choice in some graphical file transfer programs. SFTP has more features than SCP and allows for other operations on remote files, such as remote directory listing, and it is also possible to resume interrupted transfers. Note, however, that command-line SFTP cannot recursively copy directory contents. If you need to do so, you must either use SCP or a graphical SFTP client.

Command-line usage:

$ sftp -B buffersize user@hostname

examples:

(to a remote system from local)

enterprise-d [~]$ sftp user@abisko.hpc2n.umu.se
Connecting to abisko.hpc2n.umu.se...
user@abisko.hpc2n.umu.se's password:
sftp> put file.c C/file.c
Uploading file.c to /afs/hpc2n.umu.se/home/u/user/C/file.c
file.c                          100%    1    0.0KB/s   00:00   
sftp> put -P irf.png pic/
Uploading irf.png to /afs/hpc2n.umu.se/home/u/user/pic/irf.png
irf.png                         100% 2100    2.1KB/s   00:00   
sftp>

(from a remote system to local)

sftp> get file2.c C/file2.c
Fetching /afs/hpc2n.umu.se/home/u/user/file2.c to C/file2.c
/afs/hpc2n.umu.se/home/u/user/file.txt  100%  1  0.1KB/s 00:00    
sftp> get -P file3.c C/
Fetching /afs/hpc2n.umu.se/home/u/user/file3.c to C/file3.c
/afs/hpc2n.umu.se/home/u/user/file.txt  100%  1  0.4KB/s 00:00    
sftp> exit
enterprise-d [~]$ 

The following two flags can be useful

  • -B: optional, specify buffer size for transfer; larger may increase speed, but costs memory
  • -P: optional, preserve file attributes and permissions

Regarding buffer size; in order to find a optimal buffer size, use the following formula:

optimal buffer size = 2 x bandwidth x delay

or

optimal buffer size = bandwidth x RTT

RTT = round trip time, which you get from ping, since delay is about ping average time/2

Example:

ping average = 2.653 ms

the data link's capacity = 1 GB 

So optimal buffer size should be 2.653 ms x (1 GB/8 bit) = 331625 bytes.

See http://fasterdata.es.net/TCP-tuning/ for more information.

Installation:

Linux / Solaris / AIX / HP-UX / Unix:

  • The "sftp" command line program should already be installed.

Microsoft Windows:

  • WinSCP is a full-featured and free graphical SFTP and SCP client.
  • PuTTY also offers "psftp.exe", which is an extremely small program and a basic SFTP client.
  • Secure FX is a commercial SFTP and SCP client which is available free to Purdue students, faculty, and staff with a Purdue career account.

Mac OS X:

  • The "sftp" command-line program should already be installed. You may start a local terminal window from "Applications->Utilities".
  • MacSFTP

LFTP

LFTP is a command-line file-transfer program for Linux and Unix systems. FTP, HTTP, FISH, SFTP, HTTPS and FTPS protocols. LFTP has additional features not provided by SFTP such as bandwidth throttling, transfer queues, and parallel transfers. It may be used interactively or scripted. Every operation in LFTP is reliable, that is any non-fatal error is handled and the operation is retried automatically. So if downloading breaks, it will be restarted from the point automatically.

LFTP has shell-like command syntax allowing you to launch several commands in parallel in background (&). It is also possible to group commands within () and execute them in background. All background jobs are executed in the same single process. You can bring a foreground job to background with ^Z (c-z) and back with command `wait' (or `fg' which is alias to `wait'). To list running jobs, use command `jobs'. With parallel transfers LFTP can be much faster than SCP or SFTP, so its use is encouraged when possible.

LFTP is simply a client, so it is not needed on the remote machine involved in a transfer (the remote system need only support SFTP).

Examples:

lftp> cat file | gzip > file.gz
lftp> get file &

lftp> (cd /path && get file) &

The first command retrieves the file from the ftp server and passes its contents to gzip which in turn stores compressed data to file.gz. Other commands show how to start commands or command groups in the background.

LFTP has a built in mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server.

More interactive examples:

$ lftp user@hostname

(transfer all ".dat" files from remote system to local)

lftp :~> mget *.dat

(transfer "filename.dat" file from local system to remote)

 lftp :~> put filename.dat

(transfer a directory and all contents from remote system to local, using 5 connections in parallel)

lftp :~> mirror --parallel=5 remotedir localdir/

(transfer a directory and all contents from local system to remote, using 8 connections in parallel)

lftp :~> mirror -R --parallel=8 localdir remotedir/

Batch usage:

(specify all actions on command line)

$ lftp user@hostname -e "mget *.dat"

(specify all actions in the script file "transfer.lftp")

$ lftp user@hostname -f transfer.lftp
Updated: 2017-11-21, 20:49