Skip to main content

Writing tests

Tests for a topology do what a test engineer does with a vehicle: they set the inputs of the system, by controlling mocks of the ECUs that send them, and check the outputs on the bus, in the namespaces of the ECUs that receive them. Since tests only use the signals on the channels, the same tests work whether the device under test is a behavioral model, a virtual ECU, or hardware.

Tests reach the topology through the TopologyBroker, which has a gRPC API. You can write tests in any language and framework, but the examples on this page use Python and the RemotiveTopology framework, which has helpers for testing in remotivelabs.topology.testing.

The examples use the sample vehicle, where SCCM is a mock, BCM runs a behavioral model, and the tests check what FLCM receives.

Connect to the topology​

Connect a BrokerClient to the TopologyBroker. Its URL depends on where the tests run, so make it a parameter:

  • http://localhost:50051 when you run tests on your computer.
  • http://topology-broker.com:50051 when tests run in a container inside the topology.

With pytest, add an option in conftest.py that defaults to the TopologyBroker on your computer, and create the client in a fixture, as in the next section:

import pytest


def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption(
"--broker_url",
action="store",
default="http://localhost:50051",
help="URL of the TopologyBroker",
)

In a test container, pass the other URL on the command line: pytest --broker_url=http://topology-broker.com:50051.

Wait for the topology​

Containers start in parallel, so a test can start before the mocks and models are ready. Send a PingRequest to the ECU, with retries, before the test starts. Every mock and behavioral model answers it:

from typing import AsyncIterator

import pytest
import pytest_asyncio
from remotivelabs.broker import BrokerClient
from remotivelabs.topology.behavioral_model import PingRequest, RebootRequest
from remotivelabs.topology.control import ControlClient


@pytest_asyncio.fixture()
async def broker_client(request: pytest.FixtureRequest) -> AsyncIterator[BrokerClient]:
url = request.config.getoption("broker_url")
async with BrokerClient(url=url) as broker_client, ControlClient(broker_client) as cc:
# Wait until the SCCM mock has started
await cc.send(target_ecu="SCCM", request=PingRequest(), timeout=1, retries=10)
# Start every test with the BCM model in its initial state
await cc.send(target_ecu="BCM", request=RebootRequest(), timeout=1, retries=10)
yield broker_client

RebootRequest resets the restbuses of a model to their default values, so each test starts from a known state. Ping and reboot are control requests: they don't exist in the vehicle. When the ECU is hardware or a virtual ECU, replace them with what the hardware needs, for example a power cycle.

Controlling mocks​

A mock sends its cyclic frames with default values until a test changes them. Update the values of its restbus in the namespace of the mock, <ECU>-<Channel>:

from remotivelabs.broker import BrokerClient, RestbusSignalConfig


async def press_hazard_button(broker_client: BrokerClient) -> None:
await broker_client.restbus.update_signals(
("SCCM-DriverCan", [RestbusSignalConfig.set(name="HazardLightButton.HazardLightButton", value=1)])
)

The mock then sends the new value in every cycle, until it's changed again. A RestbusSignalConfig can also send a sequence of values: initial values are sent once, in order, then the loop values repeat. For example, to press the button for one cycle and then release it:

RestbusSignalConfig(name="HazardLightButton.HazardLightButton", initial=[1], loop=[0])

To put signals back to their default values, use broker_client.restbus.reset_signals(("SCCM-DriverCan", ["HazardLightButton.HazardLightButton"])), or broker_client.restbus.reset_namespaces("SCCM-DriverCan") for all signals in the namespace.

See remotivelabs.broker.restbus for the full API.

Subscribing to frames​

capture_frames subscribes to frames in a namespace and collects them in the background while the test runs. The wait_for_* methods then wait until the expected frames have arrived, or fail with a CaptureTimeoutError that lists what was captured:

from remotivelabs.topology.testing.frames import capture_frames


async def test_hazard_button_turns_on_both_turn_lights(broker_client: BrokerClient):
await press_hazard_button(broker_client)

async with capture_frames((broker_client, "FLCM-BodyCan"), ["TurnLightControl"]) as cap:
await cap.wait_for_frame(
"TurnLightControl",
{"TurnLightControl.LeftTurnLightRequest": 1, "TurnLightControl.RightTurnLightRequest": 1},
timeout=2,
)
MethodWaits for
wait_for_frame(frame, signals, timeout)A frame with the given signal values.
wait_for_signal_values(frame, signal, values, timeout)A signal that takes each of the values, in order.
wait_for_frames(sequence, timeout)A sequence of (frame, signals) pairs, in order, across different frames.
next(timeout)The next frame.

The capture keeps its position: each wait_for_* continues after the frame that the previous one matched. This lets you check a sequence of events step by step in one capture. Frames that don't match are skipped, not treated as failures. To subscribe to all frames in a namespace, leave out the list of frames.

For SOME/IP, capture_events from remotivelabs.topology.testing.some_ip works the same way for events. See the testing reference.

Timing assertions​

Every wait_for_* method has a timeout, so the test also checks that the frame arrives in time. Choose timeouts from the requirements of the function, with some margin: a test that's run in CI on a busy machine is slower than on your computer.

For assertions that the capture methods don't cover, the optional hamcrest extra has await_at_most. It calls a function repeatedly until its result matches a PyHamcrest matcher, or fails when the time is up:

pip install remotivelabs-topology[hamcrest]
from remotivelabs.broker import FrameSubscription, SignalValue
from remotivelabs.topology.namespaces.can import CanNamespace
from remotivelabs.topology.testing.hamcrest import await_at_most, contains_signal_value_sequence


async def test_left_turn_light_turns_on(broker_client: BrokerClient):
async with CanNamespace("FLCM-BodyCan", broker_client) as ns:
stream = await ns.subscribe_frames(FrameSubscription(name="TurnLightControl"))
values: list[SignalValue] = []
await press_hazard_button(broker_client)

async def consume_next() -> list[SignalValue]:
frame = await anext(stream)
values.append(frame.signals["TurnLightControl.LeftTurnLightRequest"])
return values

# The light is off, then turns on within 5 seconds
await await_at_most(5).until(consume_next, contains_signal_value_sequence(0, 1))

The remotivelabs.topology.testing.hamcrest module also has the matchers ends_with_signal_value_sequence, contains_signal_value_sequences, and has_last_element. See the hamcrest reference.

Test frameworks​

Since tests are ordinary programs that connect to the TopologyBroker, you choose the framework:

  • pytest with pytest-asyncio, since the framework is asynchronous. The examples on this page use pytest.
  • Behave lets you write the scenarios in Gherkin, as Given, When, and Then steps, so that domain experts can read and review them. The step implementations use the same framework calls as the pytest examples.
  • Jupyter notebooks are useful to explore a running topology, to control mocks by hand, and to build simple user interfaces, before you turn what you learned into tests.

The examples repository has tests written with pytest and Behave, and a Jupyter notebook.

For other languages, use the RemotiveBroker API directly.

Running tests​

You typically run tests in two ways:

  • Interactively, while you write tests: start the topology and keep it running, then run the tests from your editor or terminal against http://localhost:50051 as often as you like. Only the tests rerun, so it's fast. See Interactive mode.
  • In a test container, in CI or to check everything from scratch: add the tests as a container with a profile, and start the topology with that profile. The topology stops when the tests finish, and the exit code tells whether they passed. See Running tests and CI.

The same tests run both ways, with only the broker URL changed.

In this section​