Insights / Drones & UAV
AI in Drone Defence: From Object Detection to Autonomous Action
How drones detect and classify objects on board: sensing methods, training data, the edge ML pipeline, compute choices and integration with flight control.
The short answer
AI object detection on a drone runs a trained model on an onboard companion computer that processes camera, thermal, radar or RF data in real time, classifies what it sees and passes the result to the flight controller and the ground station. The hard parts are not the model name. They are training data that matches real conditions (smoke, fog, clutter, jamming), a model small enough for the power and weight budget, a latency budget the flight behaviour can rely on, and a clear rule for what the system may do on its own and what a human operator decides.
Key takeaways
- Detection is a sensor-fusion problem: visual, thermal, radar or LiDAR, acoustic and RF each fail in different conditions.
- Data quality decides model quality; simulation and synthetic data fill gaps that field footage cannot.
- Size the model to the compute, power and latency budget, then optimise it for the target chip.
- Close the loop: field data returns to retraining, and updates go back to the fleet through a signed channel.
- Define where autonomy stops and operator judgement starts before writing control code.
Why detection has to run on the drone
Small, inexpensive drones are now common in both civil and defence settings, which makes timely detection harder. Fast-moving objects leave little time for a person to notice, identify and react, and a video link to a ground station adds delay and can be lost or jammed.
Running detection on board removes the round trip. The drone analyses size, shape, motion and heat signature locally, raises an alert within its latency budget and keeps working when the link degrades. This is why lightweight models on edge hardware sit at the centre of modern drone defence engineering.
Object detection: the foundation of any autonomous behaviour
Before a drone can avoid, follow or report anything, it has to recognise it and decide what it is: a bird, a balloon, a friendly aircraft or an unknown drone. No single sensor does this reliably in every condition, so systems combine several.
| Method | What it does | Strong in | Weak in |
|---|---|---|---|
| Visual recognition (CNN-based) | Classifies objects in camera video | Daylight, good visibility | Fog, smoke, glare, darkness |
| Thermal imaging | Detects heat signatures | Low light, obscured scenes | Low thermal contrast, fine classification |
| Radar and LiDAR fusion | Adds range and speed | Fast or small targets, depth | Weight and power on small airframes |
| Acoustic and RF analysis | Identifies audio or radio signatures | Detecting drones before they are visible | Noisy environments, silent or non-emitting objects |
Fog, cluttered urban scenes and electronic jamming all degrade detection. That is the reason to favour lightweight models optimised for small companion computers: they keep processing locally when the environment and the link are at their worst.
From detection to autonomous behaviour
Once a drone can see and classify, it can act without waiting for a command: alert the operator, adjust its route, keep an object in view, or hand information to other systems. In surveillance of a secured site, for example, a patrol drone that detects an unidentified aircraft can classify it, alert the command post and keep it in view while operators decide what happens next.
- Autonomous surveillance: patrol drones map terrain, navigate around obstacles and report anomalies in real time.
- Ground platforms: unmanned ground vehicles use the same detection and tracking stack to navigate difficult terrain and support crews without putting people at risk.
- Swarm coordination: several drones share detections so that one spots, another confirms and the group covers more area with fewer operators.
Data: what the model learns from
A detection model is only as good as its training data, and clean, labelled field data is scarce. Teams usually combine four sources.
- Flight logs: GPS tracks, altitude, acceleration and mission patterns, used for behaviour modelling and anomaly detection.
- Camera and infrared footage: visual and thermal recordings that teach the model to recognise moving objects and heat signatures in poor visibility.
- RF data: recordings that help recognise jamming attempts and signal anomalies.
- Simulated environments: 3D simulators and digital twins that generate scenarios real footage cannot cover, such as swarms or rare events.
AI-assisted labelling tools speed up bounding boxes, segmentation and time-based event tags. Temporal tags that link events in sequence let a model learn behaviour, such as an object approaching a restricted area, and not only appearance. Because smoke, dust and signal loss make field data messy, domain adaptation and blending synthetic with real data help the model generalise beyond the simulator.
The ML pipeline: from raw data to onboard inference
- Prepare dataDenoise, normalise resolution, merge thermal and visual inputs, augment with rotation, mirroring and occlusion, and time-align image, sensor and position data.
- Select modelsFast single-stage detectors (for example the YOLO family) on board; more precise two-stage detectors such as Faster R-CNN on the ground; trackers such as DeepSORT or ByteTrack for multiple targets.
- TrainGPU training in the cloud or on premises, with transfer learning, synthetic footage and fine-tuning for the terrain and mission.
- Optimise and deployConvert to ONNX or optimise with a vendor toolchain such as TensorRT, quantise, and deploy to the companion computer within the latency budget.
- Learn from the fieldReturn telemetry and edge-recorded footage for retraining, using real error patterns to raise confidence and cut false positives.
Each deployment produces the data for the next training round; updates reach the fleet through the same signed update path as firmware.
Hardware: AI on compact, power-efficient processors
On a drone, every gram and every watt comes out of flight time. The compute has to run detection and tracking without overloading the airframe or draining the battery, and usually works alongside a separate flight controller.
- GPU edge modules (NVIDIA Jetson Orin class): CUDA, GPU acceleration and mature deep-learning libraries for onboard detection, tracking and navigation.
- Application processors (Arm Cortex-A class), optionally with an NPU: a balance of cost, power and Linux software support for video processing and lighter inference.
- Microcontrollers (Arm Cortex-M class): sensor processing, flight control and very small models where power is tight.
- FPGAs: deterministic, low-latency image processing where timing matters more than flexibility.
System integration: making the AI part of the drone
A detection model adds value only when its output reaches the systems around it, quickly and in a form they can use.
- Flight control (PX4, ArduPilot): detections become flight behaviour, such as obstacle avoidance, through the autopilot's companion-computer interface.
- Telemetry and radio: LoRa, 4G/5G or UHF links carry compact status and alerts, including in bandwidth-constrained or contested environments.
- Cloud dashboards (for example AWS IoT Core or Azure): fleet telemetry, mission data and analytics for planning and review.
- Operator interfaces: mobile apps, tablets or rugged HMI panels that show alerts and predictions clearly enough to act on.
Our own work in this area includes the Tykho Zir computing module, built on an Arm Cortex-A72 platform for real-time video and image analysis, multi-camera input (thermal, digital and analog) and AI inference on the device, with a modular design that allows later upgrades.
Engineering trade-offs
Onboard inference vs ground-station inference
- You gain
- Onboard inference works when the link is lost or jammed and avoids video round-trip delay.
- You pay
- It adds weight and power draw and limits model size.
- Choose it when
- On board for anything time-critical; on the ground for second-opinion analysis and heavier models.
Lightweight model vs more accurate model
- You gain
- Smaller models fit the latency and power budget of a companion computer.
- You pay
- They miss more small or partly hidden objects and produce more false positives.
- Choose it when
- Use the smallest model that meets the mission's detection requirement, measured on realistic footage.
Synthetic data vs field data
- You gain
- Synthetic data is cheap, labelled and covers rare scenarios.
- You pay
- Models trained mainly on simulation can fail on real noise, weather and sensors.
- Choose it when
- Blend both, and always test on field data the model has never seen.
Decision matrix: choosing onboard compute
Start from what must run on board and the power budget; the compute class follows.
| Device | Best at | Power draw | Flexibility | Choose when |
|---|---|---|---|---|
| GPU edge module | Multi-model detection and tracking on video | Highest of the four | High: standard frameworks and toolchains | Detection quality matters most and the airframe can carry the power |
| Application processor with NPU | Video processing and quantised models | Moderate | High on Linux; NPU toolchains vary | You need a balance of cost, power and capability |
| Microcontroller | Flight control, sensor processing, tiny models | Lowest | Limited for vision | The task is control or very light inference |
| FPGA | Fixed, low-latency image pipelines | Low to moderate | Low: changes need hardware design work | Timing must be deterministic and the pipeline is stable |
Decision rule: Prototype on a GPU module to prove the model, then decide whether an application processor with an NPU can carry the optimised version within the flight-time budget.
Common mistakes
Training on clean, well-lit footage only
The model performs in the lab and fails in smoke, fog, dust or at dusk.
Instead: Collect and synthesise data for the worst conditions the drone will meet.
Treating detection confidence as a decision
False positives trigger actions nobody intended.
Instead: Define thresholds, confirmation steps and operator roles for every behaviour.
Choosing the chip before the model
The model does not fit the latency or power budget, or the toolchain does not support it.
Instead: Benchmark the optimised model on candidate hardware early.
No update path for models
Model drift goes uncorrected and every fix needs a unit back at base.
Instead: Ship models through the same signed over-the-air channel as firmware.
Where we've built this
Case studyFlight-controller and ESC electronics for a 7-inch drone flight stackA custom flight stack for 7-inch drones, with flight controller and ESC boards compatible with popular open-source firmware: the flight-control side an onboard AI module has to integrate with.Questions engineers ask
How does AI detection and tracking work on a drone?
A companion computer runs a trained detection model on camera or thermal video, a tracker follows each detected object between frames, and the results go to the flight controller and the ground station. The model is trained on field and synthetic data and optimised for the onboard chip.
Can an NVIDIA Jetson Orin Nano run AI on an autonomous drone?
Jetson Orin-class modules are widely used as onboard companion computers for detection and tracking. Whether one fits depends on your model's latency target, the airframe's power and weight budget, and cooling.
Was this useful?
Where this shows up in our work
ExpertiseAI for IoT & Edge AIChoosing models for onboard detection, optimising them for edge hardware and closing the retraining loop is the edge AI work we do with our firmware and hardware teams.For the wider drone stack this AI has to fit into, from flight controllers to fleet software, see UAV and drone systems engineering.
Apply this to your product