My satellite's infrared was made of RGB

A satellite can photograph far more than it can send home. A ground pass might give you 32 kbps for five minutes, about 1.2 MB, and a single lossless image tile costs half of that. So I built a preprocessor that runs object detection in orbit and sends down what it found instead of what it saw: a few hundred bytes per tile saying what, where and how confident.

The detector is a YOLOv8n with its first layer widened from three input channels to six, to take Sentinel-2's visible bands (B2, B3, B4) plus near- and short-wave infrared (B8, B11, B12). This post is mostly about how those three extra channels turned out to be fake, and what the network did with them anyway.

Where the infrared came from

I couldn't download training data with real short-wave infrared at a resolution where ships and planes are visible, and there's a physical reason it barely exists. SWIR wavelengths are roughly three times longer than visible light, so the same aperture resolves roughly three times less detail, and silicon detectors can't see SWIR at all. Sentinel-2 has 10 m visible and 20 m SWIR. WorldView-3 gets 31 cm panchromatic and about 3.7 m in SWIR.

So I trained on DOTA, an aerial RGB dataset, and made the other bands up:

# data/synthetic_bands.py
b8      = np.clip(0.25 * r + 0.45 * g + 0.30 * b, 0.0, 1.0)
raw_b11 = np.clip(0.80 * r + 0.30 * g - 0.20 * b, 0.0, 1.0)
raw_b12 = np.clip(0.70 * r + 0.20 * g - 0.10 * b, 0.0, 1.0)

B11 and B12 are then downsampled by two and upsampled back, to imitate SWIR's coarser 20 m grid.

The arithmetic that undermines them

Nothing in those three lines wasn't already in R, G and B. A convolution's first job is computing weighted mixes of its input channels, so the network could form any of these itself. Handing them over pre-mixed adds no information.

You can see it in the singular values of a derived six-channel tile (mean-centred, normalised to the largest):

tile                 σ1      σ2      σ3      σ4      σ5      σ6
P0007_00000_01497   1.0000  0.0976  0.0550  0.0461  0.0003  0.0000   cliff 147x
P0961_01920_00471   1.0000  0.1728  0.0837  0.0543  0.0006  0.0000   cliff  93x
P1854_01440_01920   1.0000  0.1318  0.0549  0.0202  0.0003  0.0000   cliff  61x
P2794_00000_03360   1.0000  0.1764  0.0538  0.0297  0.0003  0.0000   cliff  88x

The core of it is four lines (tools/band_svd.py):

x = read_tile(path).reshape(-1, 6).astype(np.float64)   # pixels x bands
x -= x.mean(axis=0)
s = np.linalg.svd(x, compute_uv=False)
s / s[0]

Three components are the original RGB. Then two smaller ones, then floating-point noise. I wanted to know where the middle two came from, so I rebuilt one tile's bands with each step switched off:

Without clipping or resampling, six channels are exactly rank three, as the arithmetic says they must be. Clipping barely registers, because fewer than 0.7% of pixels go negative. Both extra directions come from the resampling: a blurred copy of R, G and B isn't a linear mix of the sharp ones. σ5 is small because B11 and B12 are nearly the same mix. So my "short-wave infrared" is, precisely, a blurry red-ish image, twice.

And yet the network cares

So I expected dropping the fake bands to cost nothing. An earlier version of my README said exactly that, and said it "reproduces on DOTA". I wrote it before I'd measured anything on DOTA, and it was wrong.

Zeroing bands at inference, over 96 held-out tiles (differences of about 0.01 are noise at this sample size):

A confession while we're here. Sentinel-2 calls blue B2 and red B4, but my pipeline maps the RGB array straight onto B2, B3, B4, so my "B2" holds the red channel and my "B4" holds blue. The code comment says it was "for simplicity". It doesn't change any result, but it does mean every earlier write-up of mine that said "B2 (blue)" was wrong about the colour.

Either fake band alone barely matters, but both together cost 5.4% relative. The two bands the network leans on hardest are the plain red and blue channels, not the infrared the whole design was built around.

These two results don't contradict each other. The extra channels carry no new information, but the network still learned to use them. I initialised the new input weights from the mean of the pretrained RGB weights (so from step zero they were live edge detectors), and 32 epochs of training pulled B11 and B12 into carrying part of the load, however redundantly. Zeroing a channel the network learned to rely on is a distribution shift, not a loss of information. Information-redundant and safe to remove turn out to be different claims.

The honest summary: the six-channel stem is correct engineering for a sensor this project doesn't have.

The fake bands also cost time

Deriving the bands takes two cv2.resize calls per tile. I benchmarked the pipeline in Docker held to two cores and 4 GB, over 95 held-out tiles. That's x86 under cgroups, a stand-in for a small onboard computer and not a claim about ARM.

Making up the bands costs about as much as running the detector, and more at the tail. On a real six-band sensor that work wouldn't exist. The p99 of 307.7 ms is inside the 400 ms budget I set, so the margin holds even in the slowest 1% of tiles.

Quantization wasn't what got it under budget. FP32 already ran at 94 ms a tile on an unconstrained machine. I used static INT8 post-training quantization for two other reasons. It makes the model 3.43x smaller (12.67 MB to 3.69 MB). And static quantization, unlike dynamic, produces bit-identical output on every run, and dynamic also makes latency depend on the data. It cost 0.9 points of mAP@0.5 (0.889 to 0.880, over 3,677 tiles). One caveat: the 32 calibration tiles came from the same split I score on, so about 0.9% of the test set was seen by the calibrator.

Flipping bits

Radiation in orbit flips bits in memory, so I flipped bits uniformly at random across the 25 million bits of quantised weights:

Below about 0.13% the model barely notices. Between 0.13% and 0.52% it collapses. Past 1% it goes silent and then loud: 19 times the normal number of detections, every one of them wrong, and nothing anywhere in the stack raises an error.

Which bit flips matters far more than how many. Flipping 65,536 bits, all in the same bit position of each weight byte:

The bottom four bits are close to free, and nearly all the damage comes from the top two. This is a conditional measurement, not a radiation model: it says what survives a given number of flips, not how often that many happen.

A CRC-32 per weight tensor catches it from outside: 252 bytes of state for a 3.69 MB model, checked in 11 ms. In the test, 65,536 flips spread across all 63 weight tensors were detected 63 out of 63 times, and restoring the tensors took mAP from 0.359 back to exactly 0.836.

What comes down

Over 20 held-out tiles, raw lossless imagery (CCSDS 123) is 12,234,137 bytes, about six ground contacts. The briefs for the same 20 tiles, carrying the same 21 detections, are 8,246 bytes, 0.42% of one pass. That's 1,484x fewer bytes, as the JSON the scheduler accounts for (protobuf gets it to 3,087).

The confidence threshold is the real dial. Across 1,000 tiles, briefs at 0.35 match raw lossless recall and precision exactly (0.862 and 0.920) at 894 bytes a tile. Above about 0.7, the detector goes silent: zero detections across all 9,472 objects, and each tile still costs 273 bytes of envelope.

What doesn't transfer

DOTA is aerial imagery at 0.1 to 1 m per pixel, and the orbital part of this project models 10 m. The 0.880 mAP is a real measurement on real photographs of the right four classes, and it says nothing about orbital imagery. Storage tanks are the weak class: recall 0.653 at precision 0.936, so the model isn't confusing tanks with anything, it's missing them, mostly in cluttered refineries.

If you know where to find SWIR imagery sharp enough to see ships, or you've tested a detector on real multispectral data, I'd love to know whether the infrared ever starts pulling its weight: email me. The code, the raw latency output and every measurement script are in orbital-preprocessor.

References

• Gui-Song Xia et al., DOTA: A Large-scale Dataset for Object Detection in Aerial Images, CVPR 2018.
• ESA, Sentinel-2 mission: band definitions and their 10 m / 20 m resolutions.
• ONNX Runtime, Quantize ONNX models: static vs dynamic, QDQ format, calibration.
• Ultralytics, YOLOv8.
• Every measurement script and raw artifact: orbital-preprocessor.