Home > Blockchain >  Copying multiple files in multiple directories from server to local, with files going into directori
Copying multiple files in multiple directories from server to local, with files going into directori

Time:06-08

Sorry if the title is confusing

Basically, on my server, I have multiple files for multiple runs of an analysis for multiple traits, e.g.:

/Weight/run1/file1.txt
/Weight/run1/file2.txt
/Weight/run2/file1.txt
/Weight/run2/file2.txt
/LegLength/run1/file1.txt
/LegLength/run1/file2.txt
/LegLength/run2/file1.txt
/LegLength/run2/file2.txt

In total I have 11 traits with two runs each, with each run generating 26 files.

I want to copy them across onto my local machine, but keeping the file hierarchy the same. I can only find code that copies from multiple directories into one directory, which is not what I want.

So far I've managed to hack together the following code from somewhat related stackoverflow questions, though obviously doesn't work and just tries to find ~/Documents/path/to/local/directories on the server. I was wondering if anyone had any advice on how to tweak it?

`ssh user@server 'for f in /path/to/server/directories/*/*/*txt ; do newpath="${f%/*}"; newpath=${newpath:41}  ; scp user@server:${f} ~/Documents/path/to/local/directories/${newpath};  done'`

I also have multiple other files in each end directory on the server with different extensions, but I am not interested in copying them across.

(Edit to add: I'm using newpath=${newpath:41} because it removes /path/to/server/directories/ from the variable)

CodePudding user response:

It's a little complex to get it right but a combination of --include and --exclude in the rsync command should be able to do what you want:

rsync -av \
      --include '/Weight/' \
      --include '/Weight/run[12]/' \
      --include '/Weight/run[12]/file[12].txt' \
      --include '/LegLength/' \
      --include '/LegLength/run[12]/' \
      --include '/LegLength/run[12]/file[12].txt' \
      --exclude '/*' \
      --exclude '/*/*' \
      --exclude '/*/*/*' \
      user@server:'/path/to/server/directories/' ./here/

note: in this case, the first / in the includes & excludes means /path/to/server/directories/

  • Related