Copying a lot of files between two servers

Let’s say you have 5 billion files in one folder that needs to be transferred.

This is simple and works.

tar c some/dir | gzip – | ssh host2 ‘cd /target/dir; tar xz’

you may also want to try (not tested)

# requires netcat on both servers
nc -l -p 2342 | tar -C /target/dir -xzf -   # destination box
tar -cz /source/dir | nc Target_Box 2342    # source box

Credits:
http://superuser.com/questions/291803/best-way-to-copy-millions-of-files-between-2-servers