CosmoEdge Test Scope and Test Cases
This document defines the current test baseline for QA, development, and integration teams. Every conclusion must identify the software version, target platform, model, input, configuration, and duration. A pass on x86, in a build container, or on another chip is not target-device acceptance.
1. Test Scope and Evidence Boundaries
Testing covers these layers:
- Static and unit testing: formatting, static analysis, DTO, route, service, and component tests.
- Container integration testing: the login, upload, configuration, inference, and event loop on x86 Docker.
- Target-platform testing: target-specific models, media paths, and resource behavior on real BM1688, CV186X, or RK3576 devices.
- Business acceptance: false positives, misses, event semantics, and sustained operation on fixed positive, negative, and difficult samples.
- Release evidence: a frozen candidate commit, package digest, target marker, running version, and complete results.
Preserve these boundaries:
- Each build selects exactly one inference backend. ONNX, Sophon
.nn/.bmodel, and RKNN.rknnartifacts are not cross-backend artifacts. - BM1688, CV186X, RK3576, and RV1126B artifacts are not interchangeable based on file names.
- Open packages use plaintext models and do not require a device-binding certificate. Protected model distribution and its device certificate are separate capabilities.
COSMO_DEV_MODEcontrols development logging, watchdog, and related production behavior. It is not an HTTP-login or device-SN authentication bypass.- Live boxes and labels are rendered from Pipeline metadata. The current action catalog has no standalone OSD node.
- Performance gates must come from measurements of the current candidate, not fixed channel or latency values from another release, model, or platform.
2. Test Environments and Materials
2.1 Platform Matrix
| Platform | Backend / artifact | Applicable tests | Evidence boundary |
|---|---|---|---|
| x86 Linux / Windows | ONNX Runtime / .onnx | UI, API, workflow, and CPU integration | Does not establish NPU behavior or capacity |
| Apple Silicon macOS Preview | Emulated linux/amd64 / .onnx | One local-video evaluation path | Not native arm64, NPU, or production performance evidence |
| Sophon BM1688 | BMRT / target-specific model package | Installation, inference, media, capacity, and stability | Requires a BM1688 package and device |
| Sophon CV186X | BMRT / target-specific model package | Installation, inference, media, capacity, and stability | Requires a CV186X package and device |
| Rockchip RK3576 | RKNN Runtime / .rknn | RKNN, MPP/RGA, capacity, and stability | Requires an RK3576 package and device |
| Rockchip RV1126B | RKNN Runtime / .rknn | Build and conditional device validation | Deployable acceptance also needs the target model overlay |
Record the actual OS, driver, runtime, model, and package identity in the report. This table is not a version lock.
2.2 Test Materials
- Fixed videos and images with SHA-256 values; inputs must be legally usable and free of unsanitized customer data.
- Model source, version, digest, input/output contract, preprocessing, postprocessing, and label order.
- Positive, negative, and difficult samples with an expected result for every sample.
- Target chip, software version, package digest, runtime, and driver identity.
- An isolated MQTT broker or HTTP webhook receiver for push testing.
- A versioned ScenarioBench package and explicit gates for performance tests.
3. Functional and Integration Test Cases
3.1 Authentication, Sessions, and Onboarding (TC-AUTH)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-AUTH-001 | First login | Sign in to a new data directory with admin / admin | Login returns an mtk; the response requires changing the factory-default password |
TC-AUTH-002 | Invalid session | Call /gtw/cwai/System/QueryDeviceInfo without an mtk or with an invalid one | HTTP 401 or an explicit authentication failure; it must not be confused with a missing route |
TC-AUTH-003 | Password change and session revocation | Change the password, then reuse every token issued before the change | The password persists; all old tokens for that user are revoked; the new password can log in |
TC-AUTH-004 | Onboarding boundary | Call Status and Complete anonymously; call Reset anonymously and after login | Status/Complete allow anonymous access; Reset requires a valid mtk; state persists |
TC-AUTH-005 | Open model-distribution boundary | Query model-authorization status in an Open package and run a bundled plaintext model | An unavailable provisioning tool does not block Open models; application features are not restricted by device SN |
TC-AUTH-006 | Protected certificate status | Test no certificate, an invalid certificate, and a valid certificate in a Protected package containing the provisioner | States are actionable; invalid or foreign-device certificates are not valid; a valid certificate binds to this device |
3.2 Video Input and Media Lifecycle (TC-CAM)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-CAM-001 | Local-video upload | Query upload capabilities, upload a fixed MP4 in chunks, and create a channel | Progress, completion, and channel data are correct; no server temporary path is disclosed |
TC-CAM-002 | Play count | Set play count to 0, 1, and a value greater than 1 | 0 loops indefinitely; 1 plays once; a positive number is total plays; values outside 0–100 cannot persist |
TC-CAM-003 | RTSP input | Add a fixed RTSP source and observe first frame, sustained playback, and cleanup | It connects and decodes; stop or delete leaves no task, decoder, or preview state behind |
TC-CAM-004 | RTSP reconnection | Interrupt a running RTSP source and then restore it | Disconnection is visible; reconnect is automatic; no crash, duplicate task, or permanent stuck state |
TC-CAM-005 | Invalid input | Submit an unsupported protocol, invalid address, corrupt video, and over-budget file | Each input is safely rejected with an actionable error; other channels remain available |
3.3 Model Repository and Target Compatibility (TC-MOD)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-MOD-001 | x86 model import | Import a supported fixed ONNX with complete metadata | The model is usable; positive and negative image results match the baseline |
TC-MOD-002 | Sophon model import | Import a chip-matched .bmodel/model package on the target device | Loading, tensors, and postprocessing pass; upload success alone is not acceptance |
TC-MOD-003 | RKNN model import | Load the corresponding .rknn on RK3576 or RV1126B | Runtime, input format, and output parsing match the target profile |
TC-MOD-004 | Wrong target artifact | Import a target-specific artifact on an incompatible chip or backend | Initialization or preflight fails explicitly; no silent fallback is reported as success |
TC-MOD-005 | Export policy | Export a user-managed model and a preset/encrypted/device-bound model | Exportable models return attachments; protected models return an explicit non-exportable error |
3.4 Pipeline Orchestration (TC-FLOW)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-FLOW-001 | Minimal detection chain | Create “Video Decode → Object Detection → optional Tracking/Filter → Event Report” | Nodes, edges, and parameters persist; the running chain produces explainable targets and events |
TC-FLOW-002 | Invalid connection | Save a missing upstream input, incompatible type, or broken edge | Frontend or backend rejects it and identifies an actionable node or field |
TC-FLOW-003 | Import/export | Export a Pipeline and import it into an isolated environment on the same version | Node IDs, model references, parameters, and edges persist; missing models are reported as dependencies |
TC-FLOW-004 | Current action catalog | Inspect the action list and configure live overlays | No standalone OSD action exists; Live Display renders Pipeline output metadata |
3.5 Scenario Tasks, Regions, and Strategies (TC-TASK)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-TASK-001 | ROI behavior | Place the target inside and outside a fixed region under identical settings | The positive sample inside creates the expected event; the outside negative sample creates no matching region event |
TC-TASK-002 | Parameter persistence | Change thresholds, alarm interval, deduplication, and time rules; save and restart | UI, runtime configuration, and post-restart values agree; one-variable changes remain explainable |
TC-TASK-003 | Schedule | Configure an active and an inactive time window | Tasks run only in the active window; boundary transitions do not duplicate tasks or leak resources |
TC-TASK-004 | Batch switching | Enable, disable, and repeat requests across multiple channels | Operations are idempotent; failures identify their item; inference and media resources release after stop |
3.6 Live Display and Rendering (TC-LIVE)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-LIVE-001 | Raw/algorithm preview | Switch one channel between raw and algorithm streams | First-frame, loading, no-signal, and switching states are correct; stale players and connections are cleaned up |
TC-LIVE-002 | Metadata overlay | Enable an algorithm that supports visualization and select its overlay | Boxes, labels, regions, and tracks match target metadata; there is no standalone OSD-node dependency |
TC-LIVE-003 | Sustained playback | Preview for the task-defined duration while switching channels and overlays | Player, connection, and memory counts do not grow without bound; no persistent black frame or stale overlay |
The task must define latency gates and record network, codec, and playback mode. This page does not set a universal millisecond target.
3.7 VLM and DINO (TC-LLM)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-LLM-001 | VLM image baseline | Run one prompt on fixed positive, negative, and difficult samples | Output matches the task definition; false positives, misses, latency, and resources are recorded and repeatable |
TC-LLM-002 | VLM video task | Bind a fixed video after the image baseline passes and set analysis frequency | Events, captures, and timing match results; no repeated initialization or sustained timeout |
TC-LLM-003 | DINO open vocabulary | Test clear English target words on positive and negative samples | Boxes match the target meaning and location; ambiguous terms and similar negatives are recorded |
VLM/DINO gates must bind the model, prompt, input, target FPS, and device. Do not use a universal inference-time value.
3.8 Events and External Push (TC-ALARM)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-ALARM-001 | Event query/export | Query and export fixed events by channel, algorithm, and time | List and export counts, times, and fields agree; encoding is correct; sensitive paths are not disclosed |
TC-ALARM-002 | MQTT | Configure an isolated broker, trigger one fixed event, and listen on the device-report topic | Registration, heartbeat, and event structures match the MQTT reference; disconnect and reconnect are observable |
TC-ALARM-003 | HTTP webhook | Configure an isolated receiver that returns success, timeout, and failure | Success payload fields are correct; failure does not block the main task; retry/status matches the implementation |
3.9 System, Upgrade, and Recovery (TC-SYS)
| Case | Goal | Steps | Expected result |
|---|---|---|---|
TC-SYS-001 | Resource-aware upload | Query capabilities and upload with ample space and at the safety reserve | Available/required space and refusal reason agree; the disk reserve is enforced |
TC-SYS-002 | Software upgrade | Record bootId and version, upload a valid package, and upgrade | Name, MD5, archive, and space validation precede reboot; a new bootId plus version proves recovery |
TC-SYS-003 | Invalid upgrade | Use a wrong name, digest, target, or dangerous archive structure | Upgrade fails before replacing the active application; the old version remains usable or follows the recovery plan |
TC-SYS-004 | Watchdog | Simulate a process failure only on an authorized test device | A non-development build follows its service recovery policy; a COSMO_DEV_MODE build records that watchdog is disabled |
4. Performance and Stability Tests
| Case | Goal | Method | Pass condition |
|---|---|---|---|
TC-PERF-001 | Capacity staircase | Increase load channel by channel with a versioned ScenarioBench package | Report the last passing point and first failure; do not turn the highest configured point into a recommendation automatically |
TC-PERF-002 | Preview overhead | Test no preview, one preview, every-channel preview, and multiple clients on one stream | Record media, inference, discard, and resources separately for every mode; conclusions are not interchangeable |
TC-PERF-003 | Sustained run | Run the fixed full-load point for the required 12/24/72 hours | Full-load coverage reaches the gate; sampling is continuous; no crash, counter rollback, or unbounded resource growth |
TC-PERF-004 | Business accuracy | Compare reference and candidate output on a versioned offline dataset | Use task-approved class, IoU, confidence, and matching gates; humans review false positives and misses |
A public benchmark describes only its recorded platform, model, input, and version. A new candidate needs its own result or an explicit unverified status.
5. Minimum Validation by Change Type
| Change | Minimum validation |
|---|---|
| Documentation | Complete documentation checks and site build |
| Frontend | i18n, resource i18n, production build, and affected-page acceptance |
| C++ | Formatting, CPU test build, cosmo-tests, and focused unit tests |
| API / network | Authentication, invalid input, error response, and lifecycle tests |
| Model / inference / media | Target package, target device, fixed real input, and initialization logs |
| Build / package / resources | Target marker, package content, SHA-256, and deployment compatibility |
Use CI and Quality Checks and the Build Guide for commands; this page does not maintain a second command list.
6. Test Report Requirements
Every report must include:
- candidate commit and source tree;
- platform, OS, runtime, driver, and media path;
- package name, target marker, and SHA-256;
- model, input, configuration, and ScenarioBench package identities;
- exact commands, start/end time, and full-load duration;
- pass, fail, blocked, and waived results;
- first failure, logs, and recovery status;
- unverified boundaries, especially missing target-device, performance, stability, or production acceptance;
- confirmation that customer data, credentials, device identities, and private addresses were sanitized.
A successful build, x86 result, model upload, or returning web page proves only that layer. None alone is target-device or production acceptance.