In general, I can scp files from the remote server to my localhost one by one; however, I want to extract 200 files from a large remote folder, and it is not efficient to scp one by one. I wonder if there is a more elegant way to achieve this. This is what I did normally one by one: scp remote/Users/folder/file.txt local/Desktop

I have created a .txt file including filenames I wanted to scp.

Any comment would be appreciated!

CodePudding user response：

Provided that your txt file containing the files to transfer, does not contain filenames with spaces inside, this will transfer several files in a single invocation:

sed 's:^:remote/Users/folder/:' filelist.txt | xargs -i -r scp {} target_directory

where filelist.txt is your text file. If you do have files with spaces, there are two work-arounds, depending on how your file list is beeing produced:

• If you create the filelist manually, prepend each space with a backslash, so that xargs takes the whole line as a single file.

• If you use a program to create the list, ensure that each file name is terminated by a null byte (for instance find -print0 is doing this), and add the -0 option to xargs.

You can speed up things by asking xargs to use several processes in parallel. See the option -P.

CodePudding user response：

Do not use scp. Instead use ssh to run tar on the remote system to create an archive. Run another tar on the local system to extract the data.

ssh user@remote 'tar cf - folder' | tar xf -

Or with a list of files:

ssh user@remote 'tar cf - -T mylist.txt' | tar xf -

Add compression, if you like.

ssh user@remote 'tar cJf - -T mylist.txt' | tar xJf -

CodePudding user response：

You could try grabbing them in parallel with GNU Parallel like this:

parallel -a FILELIST.TXT scp remote:{} $HOME/Desktop

Performance may be better or worse depending on the sizes of your files, your local and remote disk subsystems and your network.

In general, if the files are large in size and largely constant in content, you'll do better with rsync. If the files are very small, you'll probably do better with tar. If your network is "laggy", you may do better with parallel. If performance is important, it's best to measure and analyse.