Frequency Vision: drone confirmation by propeller hum
Frequency Vision confirms drones by their propeller hum. A normal 30 fps camera proposes objects. Inside each box, the event camera stream is checked for a blade-pass frequency peak at hundreds of Hz, which a 30 fps camera cannot see. An agent decides what to look at next: re-check a box with a longer or tighter event window, track it, raise an alert, or ask a human. Everything runs in OpenCV 5 on AWS Graviton.
Logo: an eye whose iris is a ring of spectrum bars; one bar flares at the selected clip's measured propeller frequency, a sine wave runs faster for higher frequencies, and a counter shows the measured frequency in Hz.
Slide 1: the 30 fps RGB video with the annotated drone box and a zoomed view where no rotor is visible, next to a bar strip of brighter-minus-darker event counts inside the drone box, one bar per 0.5 ms over 40 ms, with orange ticks at the measured blade-pass period.
Slide 2: an orbitable x-y-time point cloud of the events in 40 ms on the drone, with x, y and time axes labelled. Black points: a pixel got brighter (ON event). Grey points: darker (OFF event). The orange lines are one per measured blade pass. The "vs empty" view stacks a same-size patch from the no-drone clip (FRED sequence 138) under it. The strip below is the same brighter-minus-darker count per 0.5 ms.
Slide 3: the event spectrum of a box in the drone clip next to a box in a clip with no drone, computed live on the server by box_spectrum (OpenCV 5 calcHist and dft) for each click. The shaded band is the propeller band, 150 to 1000 Hz. Power is shown relative to the band median on a log scale. A box hums when its band peak stands at least 98 times above the local spectral floor and it holds at least 200 events. Window buttons set the event window length. Thumbnails pick the frame and box.
Slide 4, the math, in four panels (tabs on narrow screens). (1) Sampling: a 30 fps camera only sees frequencies up to its Nyquist limit of 15 Hz; a propeller blade-pass rate of several hundred Hz folds (aliases) to a slow apparent rate, the wagon-wheel effect: f_alias = |f - k*30|. Events binned at 0.1 ms (10 kHz) see up to 5 kHz. (2) Fourier transform: the brighter-minus-darker event count inside the box over time, multiplied by a Hanning window, goes through cv.dft; the power spectrum shows the blade-pass peak. X(f) = sum over t of x(t) w(t) exp(-i 2 pi f t). (3) Score: the peak power divided by the median power within plus or minus 100 Hz of it. Thresholds chosen on FRED training data: ambiguous above 7.9, hum above 98. (4) Frequency resolution is 1/T for a window of T seconds, so a longer window sharpens the peak; this is why the agent asks for a longer window (request_window) when a spectrum is ambiguous. (5) OpenCV 5 calls in order: calcHist (event binning), dft (spectrum), connectedComponentsWithStats (event blobs and masks), dnn (RGB detector).
Slide 5: the agent's recorded steps from the evaluation traces (eval_v3), grouped into three rows: look, listen and decide. Play shows one step about every 0.6 seconds and stops at the approval step until you press Approve or Reject; the default episode is the drone that appears late (FRED 142), where the camera detector finds nothing and the frequency map finds the drone. Each step is a card: look (detect_frame, track), listen (box_spectrum, request_window, frequency_map) with the measured peak in Hz, coloured orange when it passes the hum test, outlined when ambiguous, grey when there is no hum; decide (ask_human, raise_alert, finish). Episodes: a drone (FRED 210), a drone that appears late (FRED 142), no drone (FRED 138), each planned by Claude Sonnet 4.6 on Amazon Bedrock, and a fixed-rule baseline on FRED 210. Nothing plays until you press Play. "Run agent live" starts the fixed-rule planner on the server against the real tools (detector, box spectrum, tracker) on FRED sequence 210; it stops before the critical alert and waits up to 60 seconds for your Approve or Reject. The Approve and Reject buttons record a decision on the server only; nothing external is triggered. The approval inside the recorded run came from an automatic stand-in for the operator, marked SIMULATED in the trace.
Slide 6: one card per drone clip with the measured median blade-pass frequency and the share of 100 ms windows that pass the hum test; false alarms over 84.3 minutes of FRED test video with the RGB detector alone versus RGB plus the hum check; and the OpenCV 5 benchmark on AWS Graviton4 c8g.xlarge versus x86 c7i.xlarge, p50 milliseconds at 4 threads, from bench/results.
Data: FRED (real event + RGB recording, Apache-2.0 per its Hugging Face card), test split, huggingface.co/datasets/GabrieleMagrini/FRED. Nothing on this page is simulated except the operator stand-in noted above.
Machine-readable summary: /api/summary. Server build: /version.