A photographer who shot 1,800 frames at a wedding does not have an image-processing problem. They have a pipeline problem. The same 1,800 frames need to become a curated 400-frame web gallery at 2048 pixels, a 60-frame Instagram set in square 1:1 crops, a print-ready set of 30 high-resolution TIFFs with sRGB to ProPhoto conversions, and a backup archive of all 1,800 originals in lossless DNG. Doing that one frame at a time is a week of mouse clicks. Doing it as a batch is forty minutes of CPU time and a coffee.
This article walks through where batch image conversion actually saves time, what the throughput limits are on common hardware, and the tooling choices that separate a hobbyist macro from a production pipeline. The focus is on tools that scale: ImageMagick, libvips, ffmpeg's image2 demuxer, and the few commercial options that genuinely outperform open source.
The economic case for batching
The naive view of batch conversion is "it saves time." The accurate view is that batch conversion changes which costs dominate the workflow, which in turn changes what is worth optimizing. Manual one-by-one conversion is dominated by human labor at roughly 30 to 90 seconds per image including click-throughs. A scripted batch is dominated by I/O and CPU time at roughly 50 to 500 milliseconds per image. The crossover at which a scripted batch breaks even on engineering time is somewhere between 40 and 200 images, depending on how reusable the script is.
| Workflow | Throughput per hour | Bottleneck | Setup cost |
|---|---|---|---|
| Manual GUI conversion | 60 to 120 images | Human attention | Zero |
| Scripted ImageMagick batch | 800 to 4,000 images | CPU | One to two hours |
| Scripted libvips batch | 4,000 to 30,000 images | Disk I/O | Two to four hours |
| Multi-machine distributed batch | 100,000+ images | Network and orchestration | One to two days |
| GPU-accelerated AVIF batch | 8,000 to 25,000 images | GPU memory bandwidth | Half a day |
"Programs must be written for people to read, and only incidentally for machines to execute." Harold Abelson and Gerald Jay Sussman, Structure and Interpretation of Computer Programs
A batch script that nobody on the team can read is worse than no batch script, because when it fails on a Friday afternoon nobody can fix it before the Monday delivery. Write batches with the same care you write production code.
ImageMagick: the universal default
ImageMagick is installed on every Linux server and most macOS dev machines, and it understands roughly 200 image formats. Its mogrify and convert commands are the lingua franca of batch image processing.
# Convert all TIFFs in a folder to high-quality WebP at 2048px wide,
# preserving aspect ratio and color profile
mogrify -path ./out -format webp \
-resize "2048x>" \
-quality 82 \
-define webp:method=6 \
./in/*.tif
A few details that matter. The -resize "2048x>" syntax means "resize to 2048 wide only if the source is wider"; the trailing > prevents upscaling small inputs. The -quality 82 is well within the perceptually transparent range for WebP. The -define webp:method=6 enables the slowest, best-quality compression mode, which is acceptable for unattended batches.
ImageMagick's weakness is memory. By default it loads each image fully into RAM, and large 16-bit TIFFs can exceed 1 GB per image. For batches that mix small and very large files, set per-process resource limits to prevent OOM kills:
export MAGICK_MEMORY_LIMIT=4GiB
export MAGICK_MAP_LIMIT=8GiB
export MAGICK_DISK_LIMIT=20GiB
These caps tell ImageMagick to spill to disk-backed mmap rather than crash when an image exceeds available RAM.
Libvips: the throughput champion
When ImageMagick is too slow, libvips is the next stop. Its core insight is that most image operations can be expressed as a streaming graph that pulls scanlines on demand, so memory consumption is proportional to image width rather than image area. This makes it dramatically faster on large images and on multi-core machines.
# Same WebP batch with vips, typically 4-8x faster than ImageMagick
ls in/*.tif | parallel -j 8 \
'vips webpsave_buffer {} out/{/.}.webp \
--Q 82 --effort 6 --strip false'
For Python pipelines, pyvips offers the same speed in a more programmable form:
import pyvips
import pathlib
for src in pathlib.Path("in").glob("*.tif"):
image = pyvips.Image.new_from_file(str(src), access="sequential")
if image.width > 2048:
image = image.resize(2048 / image.width, kernel="lanczos3")
image.webpsave(f"out/{src.stem}.webp", Q=82, effort=6)
The access="sequential" hint tells libvips that the script will process scanlines in order, which enables streaming and dramatically reduces memory use on large TIFFs. Without that hint, libvips falls back to a random-access mode that buffers more aggressively.
Format selection by destination
A batch that outputs one format for every destination is leaving 30 to 60 percent of file size on the table. Match the format to where the file will live.
| Destination | Best format | Quality target | Why |
|---|---|---|---|
| Modern web gallery | AVIF with WebP fallback | Q 75 AVIF, Q 80 WebP | 30-50% smaller than JPEG at equivalent quality |
| Email and CMS uploads | Progressive JPEG | Q 82 | Universal compatibility, progressive load |
| Print delivery | TIFF with embedded ICC | Lossless | Required by print shops |
| Archival master | DNG or 16-bit TIFF | Lossless | Survives format obsolescence |
| Animation or screen recording | APNG or animated WebP | Lossless | Better than GIF in size and color |
| Vector source | SVG | N/A | Resolution-independent |
| Social media uploads | JPEG, sRGB only | Q 85 | Platforms strip non-sRGB profiles |
Color management without surprises
Color management is where image batches silently destroy work. Three failure modes recur.
The first is stripping the ICC profile during conversion. An Adobe RGB or ProPhoto image that loses its profile becomes a desaturated mess on browsers, which assume sRGB when no profile is present. Always carry the profile, or explicitly convert to sRGB before stripping.
# Convert any input color space to sRGB, then embed the standard sRGB profile
convert input.tif \
-profile /usr/share/color/icc/sRGB.icc \
-strip \
-profile /usr/share/color/icc/sRGB.icc \
output.jpg
The double -profile is intentional: the first call converts the pixel values, -strip removes everything, and the second call re-attaches a clean sRGB profile.
The second failure mode is forgetting EXIF orientation. A phone photo taken in portrait often has landscape pixel data with an orientation flag. Resizing the pixel data without honoring the flag produces a sideways image. Always normalize orientation early:
mogrify -auto-orient -strip input.jpg
The third is mixing 8-bit and 16-bit pipelines. A batch that resizes a 16-bit TIFF down to 8-bit JPEG without dithering can produce visible banding in skies and skin tones. Use Floyd-Steinberg or blue-noise dithering during the bit-depth reduction step, especially for HDR-derived content.
Resampling kernels and what they cost
Downscaling a 50-megapixel image to 2,048 pixels wide is a sampling problem with a clear answer. The wrong kernel produces aliasing, moire on textiles, and the dreaded "AI-upscaled" look on the way back up.
| Kernel | Best use | Speed | Quality |
|---|---|---|---|
| Nearest neighbor | Pixel art, masks | Fastest | Aliased on photos |
| Bilinear | Thumbnails, previews | Fast | Soft, acceptable |
| Bicubic | General upscaling | Medium | Good |
| Lanczos3 | Photographic downscaling | Slower | Excellent |
| Mitchell-Netravali | Print downscaling | Slower | Sharp without ringing |
"Beauty of style and harmony and grace and good rhythm depends on simplicity." Plato, The Republic, quoted in Edsger Dijkstra's Turing Award lecture
The simplest pipeline that meets the quality bar is the one that survives revisions. Resist the urge to chain seven sharpening filters; one well-tuned unsharp mask after Lanczos3 is almost always sufficient.
A reproducible product-photo pipeline
Here is the kind of pipeline an e-commerce team actually runs, written so the same script handles the daily 200-image batch and the quarterly 8,000-image catalog refresh.
#!/usr/bin/env bash
set -euo pipefail
INPUT_DIR="${1:-./raw}"
OUTPUT_DIR="${2:-./web}"
PARALLEL="${3:-8}"
mkdir -p "$OUTPUT_DIR"/{thumb,large,print}
export -f convert_one
convert_one() {
local src="$1"
local base
base=$(basename "$src" | sed 's/\.[^.]*$//')
# 400px square thumbnail, sRGB, JPEG Q80
vips thumbnail "$src" \
"$OUTPUT_DIR/thumb/$base.jpg[Q=80,strip,optimize_coding,interlace]" \
400 --crop centre
# 2048px large for product page, AVIF
vips thumbnail "$src" \
"$OUTPUT_DIR/large/$base.avif[Q=72,effort=6]" \
2048
# Print-ready 4096px sRGB JPEG Q92
vips thumbnail "$src" \
"$OUTPUT_DIR/print/$base.jpg[Q=92,strip=false]" \
4096
}
find "$INPUT_DIR" -type f \( -iname "*.tif" -o -iname "*.cr2" -o -iname "*.nef" \) \
| parallel -j "$PARALLEL" convert_one
The script is idempotent (re-running it overwrites outputs), parallel (one job per CPU core by default), and explicit about every quality setting. It also separates the three output sizes into their own folders so a CDN sync job can pick them up without a glob mistake.
When the batch is part of a larger system
Image batches rarely run in isolation. They are usually one stage in a pipeline that also handles uploads, metadata extraction, and downstream syndication. The pattern that works at scale is to treat each conversion as a job message in a queue, with separate workers for ingest, conversion, and delivery. The same architectural ideas show up in content-distribution workflows and in the asset pipelines feeding sites such as identity-and-photo content at Strange Animals or business-formation document galleries at Corpy.
For multi-tenant systems where each tenant has different output specs, store the recipe per tenant rather than hard-coding it in the script. A small JSON config keyed by tenant ID lets the same worker process serve dozens of brands without code changes.
Metadata: the part that costs nothing to preserve and everything to lose
Image metadata is small. EXIF, IPTC, and XMP together rarely exceed 50 KB. Stripping it saves nothing meaningful in file size but loses copyright, geolocation, camera settings, and creator attribution. The default for any production batch should be to preserve metadata, with explicit stripping only on the final social-media-ready output where platforms strip it anyway.
# Preserve all metadata
exiftool -tagsFromFile source.cr2 -all:all output.jpg
# Strip only specific fields (keep copyright, drop GPS)
exiftool -gps:all= -overwrite_original output.jpg
For a privacy-sensitive batch (real-estate photos, for instance), strip GPS while keeping everything else. The IPTC standard reserves specific fields for copyright and creator that should always survive into deliverables.
Resilience: handling corrupt and weird inputs
A 5,000-image batch will encounter at least one corrupt JPEG, one zero-byte file, one image that opens fine but has a damaged ICC profile that crashes the encoder, and one CR3 raw from a brand-new camera that the toolchain does not yet support. Plan for failure.
ls in/*.* | parallel --joblog batch.log --retries 1 \
'vips thumbnail {} out/{/.}.webp 2048 --Q 82 \
|| echo "FAILED: {}" >> failures.log'
After the run, failures.log contains the inputs that need manual triage. This is far better than discovering on Monday that 17 images silently never made it to the web gallery.
"Errors should never pass silently. Unless explicitly silenced." Tim Peters, The Zen of Python
A batch script that swallows errors is worse than one that crashes loudly. Loud failures get fixed; silent ones become production incidents months later.
RAW input handling: where most batches break
Camera RAW files are the most common cause of batch failures because every camera model produces a slightly different format, and the toolchain must keep up. A 2026 batch script must handle CR3, NEF, ARW, RAF, RW2, ORF, DNG, and the increasingly common compressed-RAW variants.
The robust approach is to standardize on DNG as the input format, using Adobe DNG Converter or libraw's dnghdr to convert proprietary RAWs to DNG once, then run the rest of the pipeline against DNG only. This insulates the pipeline from camera firmware updates that introduce new RAW dialects.
# One-time normalization to DNG
for f in raw/*.cr3 raw/*.nef raw/*.arw; do
dngconverter -e -p1 -fl -d normalized "$f"
done
# Now the conversion pipeline only ever sees DNG
ls normalized/*.dng | parallel -j 8 \
'dcraw_emu -T -W -o 1 -q 3 {} \
&& vips webpsave {/.}.tiff out/{/.}.webp --Q 82'
The -q 3 flag selects AHD demosaicing, which is slower than the default but produces noticeably cleaner color in fine detail.
A note on AI upscalers in the batch
Tools like Topaz Gigapixel and Real-ESRGAN are tempting for batch upscaling. They genuinely produce better results than Lanczos for very small inputs, particularly faces. But they are slow (2 to 30 seconds per image on consumer GPUs) and they hallucinate. A 200-image batch through Real-ESRGAN takes the better part of an hour and produces outputs that may have invented details that were not in the source.
For archival or evidentiary work, do not use AI upscalers. For a marketing campaign where the goal is "make this look good," they are a fine tool, but verify each output rather than trusting the batch wholesale.
The same principle applies in adjacent domains: tools that draft content quickly need human verification before they become deliverables. The pattern shows up in everything from image upscaling to test-prep practice question generation where speed without verification produces confidence-undermining errors.
Common mistakes that survive years of practice
Three errors recur. First, batches that hard-code paths break when the source folder moves. Always parameterize. Second, batches that run without a dry-run mode produce surprises. Always support a --dry-run flag that lists what would be written without writing it. Third, batches that do not log their inputs and settings cannot be reproduced. Always write a manifest alongside the output folder recording the script version, command line, and timestamp.
A pipeline that meets these three rules will outlast every individual script in it.
References
- ISO/IEC 10918-1:1994, "Information technology - Digital compression and coding of continuous-tone still images: Requirements and guidelines." International Organization for Standardization (JPEG specification).
- ISO/IEC 15444-1:2019, "Information technology - JPEG 2000 image coding system." International Organization for Standardization.
- AOMedia, "AV1 Image File Format (AVIF) Specification, v1.0.0." Alliance for Open Media, 2019. Available: https://aomediacodec.github.io/av1-avif/
- Google, "WebP Compression Study." Available: https://developers.google.com/speed/webp/docs/webp_study
- International Color Consortium, "Specification ICC.1:2010 (Profile version 4.3.0.0)." Available: https://www.color.org/specification/ICC1v43_2010-12.pdf
- Cupitt, J., and Martinez, K., "VIPS: An image processing system for large images." Proceedings of SPIE, vol. 2663, 1996. doi:10.1117/12.230327
- Adobe Systems, "Digital Negative (DNG) Specification, version 1.7.1.0." Adobe, 2023.
- Lanczos, C., "Applied Analysis." Prentice Hall, 1956.
Frequently Asked Questions
When the batch is part of a larger system?
Image batches rarely run in isolation. They are usually one stage in a pipeline that also handles uploads, metadata extraction, and downstream syndication. The pattern that works at scale is to treat each conversion as a job message in a queue, with separate workers for ingest, conversion, and delivery. The same architectural ideas show up in content-distribution workflows and in the asset pipelines feeding sites such as identity-and-photo content at Strange Animals or business-formation document galleries at Corpy.
Related Articles
Ready to Convert Your Files?
Use our free online file converter supporting 240+ formats. No signup required, fast processing, and secure handling of your files.
Convert Files