ONNX Object Tracker

In this article:

• Introduction

• Key Features

• Properties

• Practical Use

• Technique 1: ONNX-Based Detection

• Technique 2: Hybrid Tracking with Template Stabilization

• Technique 3: Facial Tracking from Webcam Stream

• Notes

The ONNX Object Tracker element performs automatic tracking of a selected visual object using fast ONNX Runtime models. It supports multiple object detection architectures and GPU acceleration via DirectML. During runtime, the element continuously returns the tracked object’s position, size, detection confidence, and processing time.

Introduction

The ONNX Object Tracker element enables real-time tracking of visual objects within a stimulus event. It analyzes the visual content rendered inside its viewport and detects the selected target object using a specified ONNX detection model. The element can operate in three modes:

• ONNX Detection – deep learning–based object detection using an ONNX model.

• Template Matching – tracking based on a captured template image.

• Hybrid – combined ONNX detection and template matching for increased robustness.

The tracker returns object position (X, Y), size (Width, Height), detection confidence (0.0–1.0), and processing time for each detection pass. These values can be used for logging, flow control, adaptive stimulus presentation, or closed-loop experimental paradigms.

GPU acceleration via DirectML can be enabled to improve inference performance on compatible hardware.

Key Features

• Real-time object tracking using ONNX Runtime

• Support for multiple detection models (e.g., COCO-based architectures)

• Optional DirectML GPU acceleration

• Hybrid tracking mode (ONNX + template matching)

• Adjustable confidence threshold

• Configurable detection pace

• Runtime access to detection confidence and processing time

• Trajectory recording and export

• Integration with event structure and flow routes

• Viewport-based tracking area definition

Practical Use

The ONNX Object Tracker element can be used in experiments requiring:

• Tracking of hands, faces, or specific objects in video stimuli

• Real-time facial tracking directly from a live camera stream

• Spatial trajectory recording and analysis

• Interactive human–computer interaction tasks

• Attention and visual search experiments

The element can operate continuously during stimulus presentation and can trigger flow routes based on object location, size, or confidence level.

Technique 1: ONNX-Based Detection

1. Add the ONNX Object Tracker element to a stimulus event.

2. Set Model Path to the desired ONNX model (ensure the corresponding XML descriptor file is present in the same directory).

3. Optionally enable GPU Acceleration for improved performance.

4. Set Target Class ID (e.g., -1 for any detected class or a specific class ID).

5. Adjust Confidence Threshold to control detection sensitivity.

6. Set appropriate Detection Pace (in ms) based on expected processing time.

7. Use the runtime outputs (Position, Size, Detection Confidence) for:

   - Data logging

   - Flow route conditions

   - Adaptive stimulus control

Technique 2: Hybrid Tracking with Template Stabilization

1. Add the element to the event and set Tracking Mode to Hybrid.

2. Select the event and use Grab Template Now to capture the target object.

3. Adjust Template-Onnx Weight:

   1. 0.0 → pure ONNX detection,

   2. 1.0 → pure template matching,

   3. intermediate values → combined tracking.

4. Monitor Detection Confidence to ensure stable tracking.

5. Use trajectory controls:

   1. Reset Trajectory Now to clear recorded path,

   2. Save Trajectory Now to export trajectory to file,

   3. Preview Trajectory for visual inspection.

Hybrid mode is recommended in scenarios with lighting variability, partial occlusion, or moderate visual noise.

Technique 3: Facial Tracking from Webcam Stream

The ONNX Object Tracker element can be paired with a Webcam OpenCV element to implement real-time facial tracking directly from a live camera stream.

1. Add a Webcam OpenCV element to a stimulus event and configure the desired camera device and resolution.

2. Ensure the webcam stream is rendered on the event surface (e.g., full screen or defined viewport).

3. Add the ONNX Object Tracker element to the same event and position its viewport over the webcam stream area.

4. Set Model Path to a face detection ONNX model (e.g., a lightweight face detector compatible with ONNX Runtime).

5. Set Target Class ID according to the model specification (or -1 if the model detects faces only).

6. Adjust Confidence Threshold (e.g., 0.5–0.7 for stable facial detection).

7. Enable GPU Acceleration if available to reduce inference latency.

8. Tune Detection Pace to match the webcam frame rate and processing time.

At runtime, the tracker will analyze the live webcam frames and continuously return:

• Face position (X, Y),

• Face bounding box size (Width, Height),

• Detection confidence.

These outputs can be used to:

• Log head movement trajectories,

• Trigger events when the participant looks toward predefined regions,

• Implement gaze- or face-position–dependent stimulus adaptation,

• Monitor participant presence and compliance.

Example Use Case

In an attention experiment, a trial may proceed only when the participant’s face is detected within a predefined central region of the screen. If detection confidence drops below a threshold (e.g., participant moves away), the experiment can pause automatically until stable facial tracking resumes.

Notes

• Detection Pace should not be shorter than the average processing time to avoid performance degradation.

• Larger viewport sizes increase computational load.

• GPU acceleration requires a DirectML-compatible graphics card.

• Template matching performs best with visually stable and well-defined objects.

• Always verify that the ONNX model and its XML descriptor file are correctly paired.

• For high-frame-rate video stimuli, carefully tune detection pace to maintain real-time performance.