Resume recursive scp transfer (with rsync?)

Question

I was transferring several thousand files each ~1MB via scp and my connection was broken after the first 2k files or so. I wanted to know if there was a way to resume the recursive transfer w/o starting over. Something like

$ scp -r [email protected]:/datafiles/ ./
... Happy Transfer ...
...     BREAK!     ...
$ rsync -P [email protected]:/datafiles/ ./
... Continue transf...

The problem is I can't seem to get the syntax correct if it is possible. Can anyone shed some light on if/how it can be done?

PS. If you specify the slash after "datafiles" in the rsync line, does that transfer the directory or its contents? I saw conflicting comments when I googled.

score 12 · Answer 1 · answered Jun 01 '10 at 16:01

12

The following line should do the trick for that:

rsync --partial --progress --rsh=ssh -r [email protected]:/datafiles/ ./

I've never used this for recursive directories before, but when I texted it just now it seemed to work as expected.

answered Jun 01 '10 at 16:01

Tyler

221
1
4

1

This works too, as long as you include the -r =) – physicsmichael Jun 01 '10 at 17:07
Most of my rsync transfers are of the type "rsync -ahP something/ user@host:/somewhere/". The --rsh=ssh is completely unnecessary. Then, also -P drags in --partial and --progress. – killermist Jun 05 '13 at 01:56

score 10 · Accepted Answer · answered Jun 01 '10 at 16:01

10

if you are rsyncing from a local machine to a remote host, this would work:

rsync -avzl -e ssh /directory/with/files/ [email protected]:/new/directory/

answered Jun 01 '10 at 16:01

kibitzer

216
1
4

Thanks. I didn't think this was working because it would begin listing all my files but I didn't realize that it was showing the "file transfer" which just mean noticing the file was up to date. Right were my transfer left off it picked up with real (slower) transfers. – physicsmichael Jun 01 '10 at 16:23
Hmm, for me `ssh -i akey.pem` is required and I can't get the `-i` part to work with these commands. – isomorphismes Sep 24 '12 at 03:26
2

If you are trying to resume a large file, this options may be useful: `--partial --inplace --no-whole-file`. – Rafael Xavier Mar 11 '13 at 18:59
@kibitzer - once it showed you the file (on remote server), how did you start the transfer again? Mine just build the list file and shows me the file and how much was uploaded. But it just hangs with no option to continue or progress output. Has the transfer started again or do I need to do something to start it? – masterninja01 Mar 23 '14 at 23:11
3

`-l` is not necessary. It's implied by `-a` – Guido May 17 '14 at 15:27

score 2 · Answer 3 · answered Jun 05 '13 at 01:30

2

Merging kibitzer's and Tyler's answers worked best for me:

rsync --partial --progress -avzl -e ssh /directory/with/files/ [email protected]:/new/directory/

(rsyncing recursively and showing progress of each file)

answered Jun 05 '13 at 01:30

Tarc

171
1
7

score 1 · Answer 4 · answered Apr 02 '11 at 22:19

The rsync command you listed would work, if you only added "-r". but you would also most likely want "-a" and "-v".

And about the trailing slash, [email protected]:/data/ is equivalent to /data/*, in other words, if you add a slash, it copies all the contents. but [email protected]:/data would be the directory itself [and naturally, its subfolders]

Resume recursive scp transfer (with rsync?)

4 Answers4