Five Patterns for Scalable Earth Observation Pipelines
EO data is large, irregular, and expensive to process. These five architectural patterns make the difference between a pipeline that works in a demo and one that holds up in production.
Earth observation data has a way of humbling people who underestimate it.
A Sentinel-2 scene is around 700MB compressed. A single satellite pass might generate dozens of them. Processing a full country’s worth of optical data at 10m resolution involves terabytes of imagery, hundreds of compute hours, and enough coordinate system edge cases to make experienced engineers anxious. And that’s before you add SAR, meteorological overlays, or temporal analysis.
Most EO pipelines start as research code. They’re written by scientists who know their data and their domain deeply, but weren’t necessarily optimising for operational reliability or scale. The journey from a working prototype to a production system is where most of the hard problems live.
Here are five patterns that consistently separate pipelines that survive contact with production from those that don’t.
1. Separate ingestion, processing, and serving
The single most common mistake in early EO pipelines is treating acquisition, transformation, and delivery as one continuous process. They aren’t.
Ingestion deals with external APIs, partial downloads, retry logic, and raw data storage. Processing is compute-intensive, stateless, and often parallelisable. Serving needs to be fast, filtered, and queryable. Mixing them creates systems that are hard to scale, harder to debug, and brittle in ways that only become visible under load.
Separating these layers gives you independent scaling, independent failure modes, and the ability to reprocess historical data without re-acquiring it.
2. Design for reprocessing from the start
New algorithms, corrected calibration coefficients, updated cloud masks — in EO, you will reprocess data. Plan for it.
This means keeping raw data intact and treating processed outputs as derived artefacts. It means building processing steps as idempotent functions that produce the same output for the same input. It means designing your storage layout so you can run a new pipeline version against historical acquisitions without touching the serving layer.
The teams that didn’t design for reprocessing are the ones spending weeks on data migrations when a new sensor calibration update arrives.
3. Handle irregular acquisition geometry explicitly
EO data arrives in swaths, tiles, and orbital geometries that don’t align neatly with administrative boundaries or analysis grids. Failing to account for this early means either resampling everything immediately (expensive, sometimes lossy) or accumulating a set of coordinate system assumptions that silently corrupt analysis downstream.
Pick a target projection and resolution for each output product and be explicit about it. Document what resampling algorithm is used and why. Treat CRS transformations as first-class operations with explicit validation, not implicit side effects.
4. Use tile-based processing, not scene-based
Scenes are acquisition units. Analysis areas rarely align with them. Tiled processing — splitting data into fixed-size, indexed chunks before running analysis — makes parallelisation straightforward, enables incremental updates when new imagery arrives, and makes caching practical.
Well-established formats like Cloud-Optimised GeoTIFF (COG) and ZARR are built around this assumption. Using them as your intermediate format rather than an afterthought makes the rest of the pipeline significantly simpler.
5. Treat cloud masking as a pipeline stage, not an afterthought
Cloud contamination is the most common source of silent errors in optical EO analysis. A change detection algorithm that doesn’t distinguish between land cover change and cloud shadow is producing noise, not signal — and it might not be obvious from the outputs alone.
Robust cloud and cloud-shadow masking should be an explicit, monitored stage with quality metrics attached. Track the fraction of valid observations per area over time. Alert when cloud coverage is anomalously high. Downstream products should carry metadata about the masking approach used and the percentage of pixels that were masked.
None of these patterns are surprising to anyone who has built data systems before. What’s different about EO is the scale, the domain-specific failure modes, and the fact that many pipelines start in a research context where operational concerns aren’t the priority.
Getting these foundations right early is significantly cheaper than retrofitting them later.