How to use xargs in parallel

A good start is to use -P 4 -n 1 to run 4 processes in parallel (-P 4), but give each instance of the command to be run just one argument (-n 1)

These are the xargs options for parallel use from the xargs manpage:

xargs-options.txt
 -P, --max-procs=MAX-PROCS    run at most MAX-PROCS processes at a time

 -n, --max-args=MAX-ARGS      use at most MAX-ARGS arguments per command line

Example:

xargs-example.sh
cat urls.txt | xargs -P 4 -n 1 wget

This command will run up to 4 wget processes in parallel until each of the URLs in urls.txt has been downloaded. These processes would be run in parallel

xargs-parallel-1.txt
wget [URL #1]
wget [URL #2]
wget [URL #3]
wget [URL #4]

If you would use -P 4 -n 2 these processes would be run in parallel:

xargs-parallel-2.txt
wget [URL #1] [URL #2]
wget [URL #3] [URL #4]
wget [URL #5] [URL #6]
wget [URL #7] [URL #8]

Using a higher value for -n might slightly increase the efficiency since fewer processes need to be initialized, but it might not work with some commands if you pass multiple arguments.


Check out similar posts by category: Linux