04 · Parallelism¶

Level 2 introduced backgrounding jobs with & and waiting with wait. That works for a handful of jobs, but running hundreds of tasks that way would overwhelm the system — you need to control how many run at once. This module covers xargs -P and GNU parallel for controlled concurrency.

The problem with naive backgrounding¶

# BAD at scale — launches ALL 500 downloads simultaneously,
# exhausting file descriptors / bandwidth / memory
for url in $(cat urls.txt); do
    curl -s -O "$url" &
done
wait

`xargs -P` — a controlled worker pool¶

# run up to 4 downloads at a time, from a file of URLs
xargs -P 4 -I {} curl -s -O {} < urls.txt

# -n 1 : pass one argument per invocation
# -P 4 : run at most 4 processes concurrently
cat hostnames.txt | xargs -n 1 -P 4 ping -c 1

# combine with find to process many files concurrently
find . -name "*.log" -print0 | xargs -0 -P 4 -I {} gzip {}

-0/-print0 use NUL-separated names instead of newlines — the safe way to handle filenames that might contain spaces or even newlines.

Checking each job's result with `xargs`¶

xargs -P 4 -I {} bash -c 'curl -sf -o /dev/null "{}" && echo "OK: {}" || echo "FAIL: {}"' < urls.txt

Wrapping the command in bash -c '...' lets you use shell logic (&&, ||) per-item, since xargs itself only runs a single command with arguments appended.

GNU `parallel` — a more powerful alternative¶

# macOS
brew install parallel

parallel -j 4 curl -s -O {} :::: urls.txt

# apply to a range of numbers
parallel -j 8 'echo "processing item {}"; sleep 1' ::: {1..20}

# combine multiple input lists — parallel computes the cartesian product
parallel echo {1}-{2} ::: a b c ::: 1 2
# a-1
# a-2
# b-1
# b-2
# c-1
# c-2

Tool	Strength
`xargs -P`	ubiquitous (no install needed), good for simple fan-out
GNU `parallel`	richer templating, built-in progress bar (`--bar`), result logging (`--joblog`)

Tracking progress and failures with `parallel`¶

parallel --bar -j 4 --joblog job.log 'curl -sf -o /dev/null {} && echo ok || echo fail' :::: urls.txt

# job.log records exit status, runtime, and command for every job — useful
# for retrying just the failures afterward
awk '$7 != 0 { print $9 }' job.log     # print the commands that failed (column 7 = exit code)

A manual worker-pool pattern (no external tools)¶

#!/usr/bin/env bash
set -euo pipefail

max_jobs=4

run_job() {
    local item="$1"
    echo "processing $item"
    sleep 1
}

for item in "${@}"; do
    run_job "$item" &

    # throttle: if we've hit the limit, wait for ANY one job to finish
    while (( $(jobs -r -p | wc -l) >= max_jobs )); do
        wait -n
    done
done

wait     # wait for any stragglers
echo "all jobs complete"

wait -n (bash 4.3+) waits for the next background job to finish (rather than all of them) — combined with counting jobs -r -p (running job PIDs), this throttles concurrency without any external dependency.

When NOT to parallelize¶

# writes to a SHARED file from multiple parallel workers — a race condition!
parallel 'echo "{}" >> shared_output.txt' ::: a b c d

# safer: each worker writes its OWN file, merge afterward
parallel 'echo "{}" > "output_{#}.txt"' ::: a b c d
cat output_*.txt > merged.txt
rm output_*.txt

Concurrent writes to the same file (or the same lock, the same counter variable in the parent shell) are a classic source of subtle, hard-to- reproduce bugs — give each parallel worker its own output, and merge afterward.

How It Actually Works¶

Backgrounding several jobs with & and collecting them with wait relies on the kernel's process scheduler doing real preemptive multitasking across however many CPU cores are available — each & forks an independent process with its own PID, and the kernel is free to run them simultaneously on separate cores (true parallelism) or time-slice them on one core (concurrency without parallelism); bash itself doesn't schedule anything, it just tracks the PIDs in its job table and calls waitpid() on each in turn (or all outstanding children, for a bare wait).

xargs -P N and GNU parallel maintain a fixed-size pool of forked worker slots: each reads one chunk of input, forks a child to process it, and as soon as any child exits (detected via waitpid() in a loop), immediately forks a replacement to keep exactly N processes in flight — this is a manual implementation of a worker-pool pattern that many higher-level languages provide as a library primitive.

A key limitation this reveals: shell variables set inside a backgrounded subshell (or inside a pipeline stage, which also runs in a subshell) never propagate back to the parent shell, because each &'d command is a completely separate forked process with copy-on-write private memory — "copy-on-write" means the child conceptually gets its own copy of the parent's memory pages at fork time (physically shared and copied lazily only if written to), so mutations inside it are invisible to the parent no matter how quickly they happen.

Cheat sheet¶

Command	Purpose
`xargs -P N`	run up to N processes concurrently
`xargs -0`	NUL-delimited input (safe with `find -print0`)
`parallel -j N`	GNU parallel, N concurrent jobs
`parallel --joblog file`	record exit status/runtime per job
`jobs -r -p \\| wc -l`	count currently running background jobs
`wait -n`	wait for the next background job (not all) to finish

Exercise¶

Write fetch_all.sh that reads a list of URLs from a file, downloads each with curl using xargs -P 4, and writes an OK/FAIL line per URL to a results file. Then rewrite it a second way using GNU parallel with --joblog, and compare the two approaches for a list of 20 URLs.