{"name":"Frequency Vision","claim":"Frequency Vision confirms drones by their propeller hum: an RGB detector proposes boxes, the event-camera stream inside each box is checked for a blade-pass frequency peak (150-1000 Hz) that a 30 fps camera cannot see, and an agent chooses the next look, alert, or human approval. OpenCV 5 on AWS Graviton.","measured_hum":{"clip":"clear_mini2_seq202","fred_sequence":"202","drone":["DJI Mini 2"],"n_windows":88,"n_clear":88,"median_hz":316.0,"p25_hz":313.7,"p75_hz":319.5,"window_ms":100,"model":"DJI Mini 2","by_class":{"DJI Mini 2":{"n_windows":88,"n_clear":88,"median_hz":316.0,"p25_hz":313.7,"p75_hz":319.5,"window_ms":100}},"method":"median peak frequency over the 100 ms event windows inside the FRED drone box that pass the hum test","source":"app/static/data/clips.json, written by tools/export_web.py"},"hum_test":{"band_hz":[150.0,1000.0],"clear_score":98.0,"ambiguous_score":7.9,"local_hz":100.0,"fs_hz":10000,"min_events":200,"module":"hertz/freq.py, the event-window frequency check","score":"peak power divided by the median power within local_hz of the peak; thresholds chosen on FRED training sequences"},"math":["30 fps sampling sees at most 15 Hz; a blade-pass f aliases to |f - k*30|","event counts binned at 1/fs_hz s see up to fs_hz/2","X(f) = sum_t x(t) w(t) exp(-i 2 pi f t), x = ON minus OFF events in the box, w = Hanning, computed with cv.dft","frequency resolution 1/T for a window of T s: the agent asks for a longer window T when a spectrum is ambiguous","OpenCV 5 calls: calcHist, dft, connectedComponentsWithStats, dnn"],"clips":[{"id":"clear_210","title":"DJI Mini 2 in front of trees and parked cars","fred_sequence":"210","drone":true,"role":"agent","provenance":"FRED (real event + RGB recording, Apache-2.0)","source":"FRED test split, Apache-2.0, huggingface.co/datasets/GabrieleMagrini/FRED","hum":{"n_windows":133,"n_clear":99,"median_hz":320.7,"p25_hz":314.9,"p75_hz":327.9,"window_ms":100,"model":"DJI Mini 2"}},{"id":"background_138","title":"No drone, building wall and a sapling","fred_sequence":"138","drone":false,"role":null,"provenance":"FRED (real event + RGB recording, Apache-2.0)","source":"FRED test split, Apache-2.0, huggingface.co/datasets/GabrieleMagrini/FRED","hum":null},{"id":"clear_mini3_seq93","title":"DJI Mini 3, far, steady hover","fred_sequence":"93","drone":true,"role":"demo","provenance":"FRED (real event + RGB recording, Apache-2.0)","source":"FRED test split, Apache-2.0 per its Hugging Face card (huggingface.co/datasets/GabrieleMagrini/FRED)","hum":{"n_windows":87,"n_clear":87,"median_hz":247.4,"p25_hz":246.2,"p75_hz":249.3,"window_ms":100,"model":"DJI Mini 3"}},{"id":"clear_mini2_seq202","title":"DJI Mini 2, close","fred_sequence":"202","drone":true,"role":"demo","provenance":"FRED (real event + RGB recording, Apache-2.0)","source":"FRED test split, Apache-2.0 per its Hugging Face card (huggingface.co/datasets/GabrieleMagrini/FRED)","hum":{"n_windows":88,"n_clear":88,"median_hz":316.0,"p25_hz":313.7,"p75_hz":319.5,"window_ms":100,"model":"DJI Mini 2"}},{"id":"clear_cineape20ger_seq64","title":"Two DarwinFPV drones, one cineape20ger humming","fred_sequence":"64","drone":true,"role":"demo","provenance":"FRED (real event + RGB recording, Apache-2.0)","source":"FRED test split, Apache-2.0 per its Hugging Face card (huggingface.co/datasets/GabrieleMagrini/FRED)","hum":{"n_windows":87,"n_clear":87,"median_hz":225.2,"p25_hz":222.6,"p75_hz":228.1,"window_ms":100,"model":"DarwinFPV cineape20ger"}},{"id":"clear_tello_seq230","title":"DJI Tello EDU, close, in front of a wall","fred_sequence":"230","drone":true,"role":"demo","provenance":"FRED (real event + RGB recording, Apache-2.0)","source":"FRED test split, Apache-2.0 per its Hugging Face card (huggingface.co/datasets/GabrieleMagrini/FRED)","hum":{"n_windows":88,"n_clear":85,"median_hz":584.8,"p25_hz":535.6,"p75_hz":592.0,"window_ms":100,"model":"DJI Tello EDU"}}],"clip_roles":{"demo":"clip picker: FRED test segments with the drone in view and at least 96% of the 100 ms windows of its box passing the hum test","agent":"used by the recorded agent episode only","null":"no-drone reference"},"default_window_ms":{"value":100,"why":"best hum-score AUC on the FRED test split against the busiest no-drone patch: 0.758 at 100 ms, 0.753 at 50 ms, 0.702 at 20 ms, 0.727 at 200 ms","source":"results/engine/summary.json, variant signed_mask_notch"},"agent_episodes":[{"episode":"clear_210","planner":"bedrock","model":"us.anthropic.claude-sonnet-4-6","score_vs_fred":{"success":true,"true_alerts":1,"false_alerts":0,"steps":14,"wall_s":19.835},"trace_file":"bedrock_clear_210.jsonl"},{"episode":"late_142","planner":"bedrock","model":"us.anthropic.claude-sonnet-4-6","score_vs_fred":{"success":true,"true_alerts":1,"false_alerts":0,"steps":11,"wall_s":17.985},"trace_file":"bedrock_late_142.jsonl"},{"episode":"background_138","planner":"bedrock","model":"us.anthropic.claude-sonnet-4-6","score_vs_fred":{"success":true,"true_alerts":0,"false_alerts":0,"steps":10,"wall_s":13.45},"trace_file":"bedrock_background_138.jsonl"},{"episode":"clear_210","planner":"rule","model":null,"score_vs_fred":{"success":true,"true_alerts":1,"false_alerts":0,"steps":6,"wall_s":0.181},"trace_file":"rule_clear_210.jsonl"}],"approval":"POST /api/approve records approve or reject in a demo log on the server only; nothing external is triggered. The approval inside recorded runs came from an automatic operator stand-in marked SIMULATED.","approval_log":{"kind":"demo log","rows":4,"rate_limit":"10 per minute per IP","note":"clicks from the public demo page; no sign-in, so anyone can add a row; nothing external is triggered"},"results":{"false_alarms":{"rgb_only":{"false_alarms":34796,"false_alarms_per_min":412.69,"recall":0.544,"precision":0.67,"map50":0.505,"note":"OpenCV 5 DNN drone detector alone, score >= 0.125, false detections over 84.3 min of FRED test video"},"rgb_event":{"false_alarms":26006,"false_alarms_per_min":308.44,"recall":0.544,"precision":0.731,"map50":0.516,"note":"same detector, a box is kept only if its events show a propeller hum (hum_train_chosen: {'snr': 3, 'min_events': 0}), rule chosen on train"},"source":"eval/run_eval.py, FRED test, all 47 sequences, every frame; full metrics in results_test.json"},"bench":{"status":"ok","graviton4":{"label":"pip-graviton4-c8g.xlarge","instance":"c8g.xlarge","machine":"aarch64","opencv":"5.0.0","threads":4,"file":"bench/results/pip-graviton4-c8g.xlarge.json","time_utc":"2026-10-26T14:21:57Z"},"x86":{"label":"pip-x86-c7i.xlarge","instance":"c7i.xlarge","machine":"x86_64","opencv":"5.0.0","threads":4,"file":"bench/results/pip-x86-c7i.xlarge.json","time_utc":"2026-10-26T14:21:46Z"},"ops":[{"op":"resize","graviton4_ms":0.392,"x86_ms":0.423,"x86_over_graviton4":1.08,"graviton4_usd_per_m":0.017441933416668002,"x86_usd_per_m":0.02098418158958285},{"op":"cvtColor","graviton4_ms":0.08,"x86_ms":0.165,"x86_over_graviton4":2.07,"graviton4_usd_per_m":0.0035704003464461782,"x86_usd_per_m":0.008193122729166382},{"op":"GaussianBlur","graviton4_ms":0.398,"x86_ms":0.512,"x86_over_graviton4":1.29,"graviton4_usd_per_m":0.018847965295331847,"x86_usd_per_m":0.025419414227083454},{"op":"dft","graviton4_ms":1.431,"x86_ms":2.079,"x86_over_graviton4":1.45,"graviton4_usd_per_m":0.06341280648133282,"x86_usd_per_m":0.10317721359166597},{"op":"connectedComponentsWithStats","graviton4_ms":0.897,"x86_ms":1.834,"x86_over_graviton4":2.04,"graviton4_usd_per_m":0.039955728132000205,"x86_usd_per_m":0.09110889106666707},{"op":"dnn_forward","graviton4_ms":233.11,"x86_ms":198.154,"x86_over_graviton4":0.85,"graviton4_usd_per_m":10.342597448631107,"x86_usd_per_m":9.821137539937485}]},"end_to_end":[{"label":"e2e-cool-graviton4-c8g.xlarge-r1","opencv":"5.1.0-dev","file":"bench/results/e2e-cool-graviton4-c8g.xlarge-r1.json","windows":210,"timed_windows":630,"p50_ms":106.20965449999176,"p95_ms":131.21853094998812,"windows_per_s":8.83018083947276,"cpu_pct_of_all_vcpus":93.76811081916311,"usd_per_1000_windows":0.005018143106767548},{"label":"e2e-cool-graviton4-c8g.xlarge-r2","opencv":"5.1.0-dev","file":"bench/results/e2e-cool-graviton4-c8g.xlarge-r2.json","windows":210,"timed_windows":630,"p50_ms":106.5036589999977,"p95_ms":131.75677790002283,"windows_per_s":8.799587525821282,"cpu_pct_of_all_vcpus":93.73655854886768,"usd_per_1000_windows":0.005035589563838729},{"label":"e2e-pip-graviton4-c8g.xlarge-r1","opencv":"5.0.0","file":"bench/results/e2e-pip-graviton4-c8g.xlarge-r1.json","windows":210,"timed_windows":630,"p50_ms":98.92419450000034,"p95_ms":129.16733134998566,"windows_per_s":9.319157659296451,"cpu_pct_of_all_vcpus":91.82328757155987,"usd_per_1000_windows":0.004754840805478591},{"label":"e2e-pip-graviton4-c8g.xlarge-r2","opencv":"5.0.0","file":"bench/results/e2e-pip-graviton4-c8g.xlarge-r2.json","windows":210,"timed_windows":630,"p50_ms":98.93400299998234,"p95_ms":129.1926897500133,"windows_per_s":9.314431944207314,"cpu_pct_of_all_vcpus":91.80999010403391,"usd_per_1000_windows":0.0047572531933810945}],"live_spectrum":true,"live_detector":true},"slides":[{"title":"the hum","shows":"30 fps RGB video played at half speed, with a zoom on the drone (blades are a blur) next to the brighter-minus-darker event count in the drone box, one bar per 0.5 ms over 40 ms, with orange ticks one blade pass apart; the big number is the clip's median blade-pass frequency; 'run detector live' sends the shown frame to POST /api/detect, which runs the OpenCV 5 dnn drone detector on the server and draws its boxes"},{"title":"events in 3D","shows":"orbitable x-y-time point cloud of 40 ms of events on the drone; black = pixel got brighter (ON), grey = darker (OFF); orange lines one blade pass apart; 'vs empty' stacks a same-size patch from the no-drone clip"},{"title":"spectrum","shows":"live spectrum of a drone box next to a no-drone box for a chosen event window; shaded band 150-1000 Hz is the propeller band; power relative to the band median, log scale"},{"title":"the math","shows":"aliasing of the blade-pass rate at 30 fps (f_alias = |f - k*30|, Nyquist 15 Hz vs 5 kHz for 0.1 ms event bins); x(t) * Hanning window -> cv.dft; score = peak / median within +-100 Hz; resolution 1/T vs window length"},{"title":"agent","shows":"recorded agent steps as cards in three rows, look, listen and decide; listen cards are orange when the hum test passes, dashed when ambiguous; Play shows a step about every 0.6 s and stops at the approval card until Approve or Reject; default episode: the late drone (FRED 142) that the camera detector misses and the frequency map finds; 'Run agent live' runs the fixed-rule planner on FRED 210 against the live tools and waits for the visitor's approval before the critical alert"},{"title":"results","shows":"one card per drone clip, its thumbnail framed on the drone, the measured median blade-pass frequency and the share of 100 ms windows passing; false alarms of the RGB detector alone vs with the hum check over the FRED test split; OpenCV 5 p50 benchmark Graviton4 vs x86"}],"judging":{"overall":[{"criterion":"Technical execution","weight":30,"evidence":"OpenCV 5 hum engine (calcHist, dft, connectedComponentsWithStats, dnn ENGINE_NEW, KalmanFilter) evaluated on all 47 FRED test sequences: false alarms 412.7 to 308.4 per minute (-25%) at the same recall 0.544; hum-score AUC 0.917 vs random patch, 0.758 vs busiest patch; an OpenCV 5.0.0 int64 defect found and written up for upstream","where":["results/detector/results_test.json","results/engine/summary.json","docs/REPORT.md section 8","docs/UPSTREAM_ISSUE_int64.md"]},{"criterion":"Innovation","weight":20,"evidence":"the RGB detector proposes, the event camera confirms by the propeller's blade-pass frequency inside each box, and an agent picks the next measurement","where":["docs/REPORT.md section 13","live page slides 1 to 3"]},{"criterion":"Real-world impact","weight":20,"evidence":"8,790 of 34,796 false detections removed while 26 of 70,794 true detections are lost; the high-precision setting reaches precision 0.815","where":["results/detector/results_test.json","docs/REPORT.md sections 1 and 2"]},{"criterion":"User experience","weight":10,"evidence":"live six-slide page with a clip picker, live detector, live spectrum, 3D event cloud and a live agent run that waits for the visitor's Approve or Reject; keyboard navigation, screen-reader text, phone layout","where":["/","app/static/deck.html"]},{"criterion":"Documentation and presentation","weight":10,"evidence":"README with a results table and figures redrawn from result files, technical report, architecture and agent diagrams, video","where":["README.md","docs/REPORT.md","docs/architecture.svg","docs/agent_workflow.svg"]},{"criterion":"Cloud delivery, reproducibility, responsible operation","weight":10,"evidence":"always-on COOL endpoint on Graviton4 behind HTTPS, deployed by one script, /health and /version, rate limits, no SSH, human approval enforced in code","where":["deploy/aws.py","deploy/README.md","/health","/version","docs/REPORT.md sections 5 and 11"]}],"best_use_of_cool":[{"criterion":"Verified COOL integration on Graviton","weight":30,"evidence":"the live endpoint loads cv2 from /opt/cool (OpenCV 5.1.0-dev, aarch64, KleidiCV HAL); proof file with instance type, AMI and Marketplace product code","where":["/version","bench/results/cool-proof-c8g.xlarge.json"]},{"criterion":"Architecture and technical quality","weight":25,"evidence":"COOL serves the page, the hum engine, the detector and the agent tools on one Graviton4 host; benchmark hosts are separate","where":["docs/architecture.svg","docs/REPORT.md section 6"]},{"criterion":"Measured value","weight":20,"evidence":"same Graviton4 host, COOL vs pip OpenCV 5: GaussianBlur 1.89x, dft 1.54x, connectedComponentsWithStats 1.17x faster; frequency map 1.13x faster end to end; outputs identical","where":["bench/results/cool-graviton4-c8g.xlarge.json","bench/results/pip-graviton4-c8g.xlarge-samehost.json","bench/results/E2E.md"]},{"criterion":"Innovation","weight":15,"evidence":"COOL runs signal processing on event-camera time series: a 10 kHz dft per box and a per-pixel frequency map with dft DFT_ROWS","where":["hertz/freq.py"]},{"criterion":"Reproducibility and demonstration","weight":10,"evidence":"python deploy/aws.py cool deploys the endpoint; python deploy/aws.py cool-bench reruns the same-host benchmark","where":["deploy/README.md","bench/"]}],"agentic_vision":[{"criterion":"OpenCV 5 and agent integration","weight":30,"evidence":"every agent tool is an OpenCV 5 call (detect_frame, box_spectrum, request_window, frequency_map, track); in trace late_142 the detector finds nothing, the frequency map finds a 545 Hz spot and the agent confirms the drone","where":["hertz/agent/","results/agent/eval_v3/traces/bedrock_late_142.jsonl"]},{"criterion":"Orchestration and autonomy","weight":25,"evidence":"after an ambiguous spectrum the Bedrock planner asked for a new window 6 of 9 times; a guard checks each call against schema, clock and budgets","where":["results/agent/eval_v3/results.json","hertz/agent/guard.py"]},{"criterion":"Task effectiveness and evaluation","weight":20,"evidence":"8 fixed FRED scenarios: Bedrock 4 of 8 with 0 false alerts, the same score as the rule planner with 12.25 vs 17.38 tool calls per episode","where":["results/agent/eval_v3/results.json"]},{"criterion":"Failure handling, observability, security, human control","weight":15,"evidence":"critical alerts need human approval, enforced by the guard; injected planner and tool faults are tested; every step is logged as JSON lines","where":["tests/","hertz/agent/guard.py","POST /api/agent/live"]},{"criterion":"User experience, documentation, demonstration","weight":10,"evidence":"slide 5 replays real traces and runs the agent live on the server with the visitor's Approve or Reject","where":["live page slide 5","docs/agent_workflow.svg"]}],"requirements":[{"requirement":"OpenCV 5 for substantive analysis","where":["hertz/freq.py","hertz/detect.py","/version"]},{"requirement":"Meaningful AWS component","where":["COOL on Graviton4 c8g.medium serves this page","Amazon Bedrock planner","docs/architecture.svg"]},{"requirement":"Working web endpoint","where":["https://100-27-6-111.sslip.io"]},{"requirement":"Video of at most five minutes","where":["README.md, Video link"]},{"requirement":"Technical report","where":["docs/REPORT.md"]},{"requirement":"Code repository with pinned dependencies and build, deploy and test steps","where":["requirements.txt","requirements-dev.txt","README.md, Reproduce","deploy/README.md"]},{"requirement":"Architecture diagram with OpenCV 5, AWS, COOL and agent parts","where":["docs/architecture.svg","docs/agent_workflow.svg"]},{"requirement":"Evaluation with failure cases","where":["docs/REPORT.md sections 8 and 9","results/"]},{"requirement":"Rights to data, models and media","where":["THIRD_PARTY.md","FRED under Apache-2.0","detector weights not distributed"]}]},"routes":{"page":"/","summary":"/api/summary","spectrum":"POST /api/spectrum {clip, box[x0,y0,x1,y1], t, window_ms}","approve":"POST /api/approve {episode, step, decision}","agent_live":"POST /api/agent/live {scenario} starts the fixed-rule planner on the server against the live tools; GET /api/agent/live/{id} returns its steps; POST /api/agent/live/{id}/decide {decision} answers its approval question","detect":"POST /api/detect {clip, frame}","results":"/api/results","health":"/health","version":"/version"}}