Test plan¶
Regarding the release process, several test should be performed to ensure quality and reliability of the software.
Test Goals¶
The tests focus on the End-User satisfaction, and help to reduce the amount of error handling required during release and development. The application behavior should be correct, deterministic, repeatable and represented by meaningful output with an according documentation.
Quality Gates¶
The following quality gates are defined to be passed by a software release before use in production:
- Alpha
- Beta
- RC (Release Candidate)
- Gold (not directly a quality gate, represent the final state of the release; ready for production)
Test Execution¶
Twister helps to automate test with use of HITL and pure software.
CI: pure Software, automated by Github CI runner execution
HITL: hardware dependent execution, e.g. on a BMS board, possible to automate (in the future) by a Github CI runner
manual: manual execution (stimuli generation) and evaluation
Test Definitions¶
[🏛️high level] Main Application Tests¶
| Test Name | Description | Automation Level | Quality Gate |
|---|---|---|---|
| Compile of bms_selftest | Successful compile of the app for bms.d2, bms.d1 | CI | Alpha |
| Compile of bms_app | Successful compile of the app for bms.d2, bms.d1 | CI | Alpha |
[🏗️intermediate level] System Module Tests¶
| Test Name | Description | Automation Level | Quality Gate |
|---|---|---|---|
| Unit test: - analog |
Cell Balancing Class; voltages, timing Analog Measurement Class; mean calculation, std-dev |
CI | Alpha |
| State machine test | State machine transitions, timing, in/output behavior | CI | Alpha |
| DFU: device firmware update | correct and functioning software update via cyphal (CAN). Verifies: - new software running after update - new software still running after reboot (cyphal + hard reset) - verification of update only after running for a specific timeout ( boot_write_img_confirmed()) |
HITL | Beta |
| DFU: broken image detection | update should be avoided for: - truncated image (header correct but len too short) - extended image - wrong files - version mismatch (e.g. software for D.1 on D.2 board) |
HITL | Beta |
| DFU: interrupt | update should recover: - File server and uavcan.file.Read() no longer serves correctly (stop / loop segment)- new updates should be possible after recovering |
HITL | Beta |
[⚙️low level] Unit / Specific Tests¶
| Test Name | Description | Automation Level | Quality Gate |
|---|---|---|---|
| Drivers | - BMSLed driver: native_sim - Latch: native_sim |
CI | Alpha |
| Drivers | - Comp: specific analog - SCP: specific analog - bq76925: specific analog - HS Controller: _ok |
HITL | Beta |
Missed Tests (possible failures)¶
- compatibility: interaction of multiple BMS in different versions (e.g. partly updated)
- interaction of multiple BMS in different states (FSM states)
- interaction of multiple BMS of different sets (detection of fault usage)
- error handling (scp/over.../detectability, missing analog values, exception handling e.g. stack overflow)
- release process (script execution on windows)
- fail during update
- performance bottleneck (thread analyzer, maybe memory stack full when new thread; e.g. DFU is spawned dynamic)
- documentation update when input/output/fsm changes
Brainstorm for Tests¶
Unordered list of brainstormed test cases for the BMS. Some of these may be already implemented, some may be duplicates, some may be irrelevant.
Software Update Process¶
- Perform regular software update through Cyphal
- Make sure the update process itself works.
- Verify that the new software is running afterwards.
- Reboot the device via Cyphal and make sure the correct image is running.
- Reboot the device via pyocd and make sure the correct image is running.
- Attempt an update to an invalid (broken) image. Make sure the device is still reachable afterwards, e.g. not bricked.
- Truncated image (header correct, but overall length too short)
- Extended image (header correct, but overall length too long)
- Some other file (picture of a cat?)
- Perform an update to an image which does not call
boot_write_img_confirmed(). Make sure the previous image is swapped back after a reset. - Interrupt (or stall out) the update process: respond correctly to the first few (10?)
uavcan.file.Read()requests, but stop responding to later requests. Make sure the device can recover. Make sure the device can start a new update. - Perform an update to an incompatible image (e.g. a D.1 image for a D.2 device). Make sure the device rejects the image and recovers.