← All work

Compressing a Real-Time Object Detector

Shrinking an RT-DETR object detection model to run on edge devices. A team project; scope covered the baseline and full-model quantization, taking it from 161 MB to 22 MB.

R&D · IN PROGRESS2026Machine LearningQuantizationEdge AI
Compressing a Real-Time Object Detector

The project

RT-DETR R50 detects objects in real time, but at 161 MB in full precision it is too heavy for the small devices it would need to run on. This project measures what happens when the trained model is quantised to lower precision (post-training quantization): how much smaller it gets, how much memory it saves, and how much accuracy it gives up.

This is a team project. The scope below is the part handled here.

Scope

  • The shared test setup and the FP32 baseline model, exported to ONNX
  • A quantization pipeline for the whole model, at INT8 (W8A8) and INT4 (W4A16)
  • Measurements on GPU: accuracy, model size, memory and latency

Results so far

Precision Model size mAP GPU memory
FP32 161 MB 55.2 638 MiB
INT8 42.5 MB 39.7 380 MiB
INT4 22.3 MB 29.7 380 MiB

What made the difference

  • Activation quantization. Working out how the activations were being quantised raised INT8 accuracy from 15.9 to 39.7 mAP.
  • INT4 for convolution layers. Standard tools do not convert Conv layers to INT4, so a custom module was written to do it.
  • A calibration bug. A defect in ONNX Runtime’s calibration step was silently dropping data; it was traced and worked around.

Details

  • Team project, in progress. Figures will be updated as the work continues
  • PyTorch · ONNX · ONNX Runtime · CUDA

Thinking about building something similar?

Start a Conversation →