Testing the state machine
Just like software, testing our code is vital. Unlike software however, we don’t have (easy) access to fancy tools like debuggers, printfs or error messages when we run on hardware. Therefore, we usually simulate FPGA designs to make sure they work in simulation in order to avoid painful debugging in hardware.
Currently, writing simulation code directly in Spade is not possible, as the things you want to do in a simulator are quite different to describing hardware. Instead, tests are written in python using the cocotb testing framework.
If you haven’t already, refer to the [installation instructions](../installation.md) to see how to install cocotb.
Setting up tests
To do our testing, we need to do a tiny bit more setup in swim.
To do testing, we need to tell swim where we put our test benches. To do so, create a directory called test
Then edit swim.toml adding a simulation section like so:
[simulation]
testbench_dir = "test"Inside the test folder we put our test benches in python files. Let’s create our first one by creating test/state_gen.py. Each Spade test file must start with a comment telling swim which unit is to be tested, the “top module”, like so. The simulator already knows that we’re in the project ws2812b, so we can simply put it as state_gen
# top=state_genWe’ll also add an empty test to that file like this:
# top=state_gen
import cocotb
from spade import SpadeExt
@cocotb.test()
async def normal_operation(dut):
passEach test is annotated by @cocotb.test() and is an async python function which takes a single parameter dut, the Design Under Test.
Running swim test (or swim t for the lazy :)) presents us with the following error1
Error:
0: In test/test.py
1: state_gen is generic which is currently unsupported in test benchesWhich is a limitation of the Spade python interface. To test our module, we’ll need to create a dummy entity without any generic parameters, for now we’ll use one with 10 LEDs. Typically, we’ll name this <unit_name>_th for “test harness”.
entity state_gen_th(
clk: clock,
rst: bool,
num_leds: uint<5>,
timing: Timing,
) -> OutputControl<uint<5>> {
inst state_gen$(clk, rst, num_leds, timing)
}
After updating the top to # top=state_gen_th we can swim test again and we should see a nice PASS (along with some other output which we’ll ignore for now)
HEAD is now at f1c17dc0 feat!: Use Rust syntax for exclusive ranges
[INFO] /home/frans/Documents/spade/ws2812-spade/build/spade.sv is up to date
[INFO] Checking if spade-python needs rebuilding. (This may print an error, it is expected)
⚠️ Warning: `project.version` field is required in pyproject.toml unless it is present in the `project.dynamic` list
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Not using a specific python interpreter
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
Finished `release` profile [optimized] target(s) in 0.15s
📦 Built wheel for abi3 Python ≥ 3.8 to /home/frans/Documents/spade/ws2812-spade/build/dist/spade-0.13.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
[INFO] Finished: Checking if spade-python needs rebuilding. (This may print an error, it is expected)
HEAD is now at f1c17dc0 feat!: Use Rust syntax for exclusive ranges
PASS test/state_gen.py [normal_operation]
ok test/state_gen.py 0/1 failed
🭼 normal_operation ok [build/state_gen_normal_operation/state_gen.vcd]Writing Some Tests
Now we are ready to actually test our module. All Spade test functions start with s = SpadeExt(dut) which creates a nice Spade interface around the cocotb functions.
Since our entity is clocked, we need to generate a clock for it. This is done by starting a cocotb clock task like this. The exact clock frequency is not really important here, it only decides the mapping between simulation time and real time.
import cocotb
from spade import SpadeExt
from cocotb.clock import Clock
The design also takes a reset signal which we need to set to get our initial state defined. If we forget to do this, most of the signals will be undefined.
We can access the input ports of our design using s.i.<input_name> and give them values. The following code sets the reset signal to true:
await FallingEdge(clk)
s.i.rst = True
Thes.iinterface does not work well with cocotb built in functions likeClockYou can access the raw verilog input ports on the dut viadut.<name>_ias above which is nice if you want to pass them to special cocotb functions likeClock. However, most of the time you should use the Spade interface since that doesn’t require you to know the Spade internal representation of types.
For our design to start running, we need to take it out of reset again, you might think that we can just add another line s.i.rst = false. However, this would give the design no time to see the change in reset. Instead, we need to let the simulation step forward a bit. The easiest way to do that is to let it step forward one clock cycle, which we do by waiting until the next time the clock goes from 1 to 0. In order to be sure that the reset sees at least one clock edge, we’ll also wait for a falling edge before setting it
# top = state_gen_th
from cocotb.triggers import FallingEdge
import cocotb
from spade import SpadeExt
from cocotb.clock import Clock
@cocotb.test()
async def simple_test(dut):
s = SpadeExt(dut)
clk = dut.clk_i # Access the raw clock signal via the dut
await cocotb.start(Clock(clk, 1, units="ns").start())
await FallingEdge(clk)
s.i.rst = True
await FallingEdge(clk)
s.i.rst = False
await FallingEdge(clk)
s.i.rst = True
await FallingEdge(clk)
s.i.rst = False
This will create a waveform that looks like this
You can find more things to wait for in the cocotb documentation for triggers.
Now we can do our first test, ensuring that the initial output of the circuit is RET. We can access the output of our design with s.o, and run assertions on it like this:
s.o.assert_eq("OutputControl::Ret()")
Note how we’re writing a Spade expression as a string here. For simple values like integers bools and arrays, you can write the corresponding python value, but for more complex values you generally want to write a Spade expression as a string like this.
If you return a struct from a unit, you can access them as normal python fields on thes.ofield. For examples.o.x.y.assert_eq(...).
Our test file now looks like this:
# top = state_gen_th
from cocotb.triggers import FallingEdge
import cocotb
from spade import SpadeExt
from cocotb.clock import Clock
@cocotb.test()
async def simple_test(dut):
s = SpadeExt(dut)
clk = dut.clk_i # Access the raw clock signal via the dut
await cocotb.start(Clock(clk, 1, units="ns").start())
await FallingEdge(clk)
s.i.rst = True
await FallingEdge(clk)
s.i.rst = False
s.o.assert_eq("OutputControl::Ret()")
s.i.timing = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
and calling swim test should tell us that all our assertions passed.
It is always good practice to make sure an assertion can fail, try changing the asserted value to something else and see if it fails now.
A failing test
Next, we may want to ensure that we output Ret in the next clock cycle as well. So, we’ll advance the clock and assert that the value is still correct.
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
Calling swim test results in the following output:
...
AssertionError:
Assertion failed
expected: OutputControl::Ret()
got: UNDEF
verilog ('0XXXXXXXXXXXXXXXXXXXXXXX' != 'xxxxxxxxxxxxxxxxxxxxxxxx')
assert False
**************************************************************************************
** TEST STATUS SIM TIME (ns) REAL TIME (s) RATIO (ns/s) **
**************************************************************************************
** state_gen.normal_operation FAIL 0.50 0.03 14.56 **
**************************************************************************************
** TESTS=1 PASS=0 FAIL=1 SKIP=0 0.50 0.06 8.67 **
**************************************************************************************
VCD info: dumpfile /home/frans/Documents/spade/ws2812-spade/build/state_gen_normal_operation/state_gen.vcd opened for output.
FAIL test/state_gen.py [normal_operation]
FAIL test/state_gen.py 1/1 failed
🭼 normal_operation FAILED [build/state_gen_normal_operation/state_gen.vcd]
Error:
0: 1 test case failedOh no, something went wrong, why? To debug our tests, the best method by far is to look at the wave dump. It contains the value of all the signals in the design over time and can give plenty of debug information. To see it, we need to install a waveform viewer, and for this guide we will use Surfer which was built for Spade and understands complex Spade types.
Swim translates Verilog values in the wave dump back into Spade files and stores the result in a new vcd file which is printed along with the failing tests:
normal_operation FAILED [build/state_gen_normal_operation/state_gen.vcd]If you ranswim install-toolsyou can runswim run-command surferto launch Surfer, or you can install it from the website or via your package manager of choice.
Let’s open build/state_gen_normal_operation/state_gen.vcd in surfer:
surfer build/state_gen_normal_operation/state_gen.vcdThis should open a window that looks something like this:
The black portion shows the value of the signals we select over time. The left pane contains a list of the units in our design, in this case ws2812b::state_gen_th. If you select it, the signal list below will be populated by all the values in that module. In this case, it is just a wrapper around the actual design state_gen, so expand the module and select the submodule. This should give you a lot more signals
In the variables list, you can now click the signals you’re interested in seeing to view their values over time. Or you can press space and type variable_add_from_scope <signal name>
Let’s add clk, rst, state and timing which are a good starting point for debugging this problem.
This shows the value of the signals in your design over time. Single bit signals like clocks are shown as “graphs”, while multi-bit signals like the state and timing are shown as bubbles. You can expand enums and structs to show their inner values.
Here we get quite a bit of information. We see that our state starts out undefined, then becomes defined as Ret(0) when the reset signal is high. However, it goes back to being undefined in the next clock cycle when rst has been unset. The timing signal gives a hint of what is going on, z values are so-called “high impedance” values, for now, you can think of it as “not given a value yet”.
Going back to the Spade code, to compute the next state in Ret, we check if the duration we’ve been in Ret so far is greater than t_ret. However, we haven’t set t, so we are essentially comparing duration to t_ret, which is undefined, resulting in another undefined value.
Spade tries to do its best to avoid undefined values, it is certainly harder to write undefined values in Spade than in verilog. However, when forgetting to specify inputs, and when working with memories, they can pop up.
Let’s specify the timings to fix this issue. Again, exact timing here isn’t important, we’ll set some small values that make testing easy:
s.i.timing = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
With that change, our assertions should pass pass.
More tests
Now we can get to testing the rest of the design. Since our state space is quite small in this case, we can ensure that all state transitions happen as they should. Since this is python, we can write things like loop, helper functions etc.
First, let’s ensure that we stay in Ret for the specified amount of time, i.e. us280 clock cycles:
s.i.timing = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
for i in range(0, 28):
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
After that, we should be emitting the value of the first LED. Here we can write a function to check a whole LED output, since we’ll do that quite a few times
async def check_led(clk, s, index):
# Each bit of the LED should be emitted
for b in range(0, 24):
# And each duration from 0 to us1_25 in each bit
# For simulation performance, we'll just check the first and last bit explicitly
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 0)")
for d in range(0, 5):
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 5)")
We can now test all our LEDs by calling it in a loop, and finally ensure that we go back to the ret state at the right time. The final test bench looks like this:
# top = state_gen_th
from cocotb.triggers import FallingEdge
import cocotb
from spade import SpadeExt
from cocotb.clock import Clock
async def check_led(clk, s, index):
# Each bit of the LED should be emitted
for b in range(0, 24):
# And each duration from 0 to us1_25 in each bit
# For simulation performance, we'll just check the first and last bit explicitly
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 0)")
for d in range(0, 5):
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 5)")
@cocotb.test()
async def normal_operation(dut):
s = SpadeExt(dut)
clk = dut.clk_i
await cocotb.start(Clock(clk, 1, units='ns').start())
s.i.rst = "true"
await FallingEdge(clk)
s.i.rst = "false"
s.o.assert_eq("OutputControl::Ret()")
s.i.timing = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
for i in range(0, 28):
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
# Check all 10 leds
for i in range(0, 10):
await check_led(clk, s, i)
# Ensure we get back to the ret state
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
Running it gives us another assertion error:
AssertionError:
Assertion failed
expected: OutputControl::Led$(payload: 0, bit: 1, duration: 0)
got: Led(0,0,6)
verilog ('100000000001000000000000' != '100000000000000000000110')
assert False
**************************************************************************************
** TEST STATUS SIM TIME (ns) REAL TIME (s) RATIO (ns/s) **
**************************************************************************************
** state_gen.normal_operation FAIL 287.50 0.08 3516.24 **
**************************************************************************************
** TESTS=1 PASS=0 FAIL=1 SKIP=0 287.50 0.12 2393.34 **
**************************************************************************************
VCD info: dumpfile /home/frans/Documents/spade/ws2812-spade/build/state_gen_normal_operation/state_gen.vcd opened for output.
FAIL test/state_gen.py [normal_operation]
FAIL test/state_gen.py 1/1 failed
🭼 normal_operation FAILED [build/state_gen_normal_operation/state_gen.vcd]
Error:
0: 1 test case failedTry to see if you can figure out what happened. Looking at the waves can be helpful, but in this case it might be enough to look at what states it transitioned to.
If you can’t figure it out, jump to the next section for the answer
- 1 The first time you run
swim test, it will set up a python environment with the required libraries which requires compiling a separate part of the Spade compiler. Don’t be alarmed at the timeswimtakes, or the amount of output the first time you run it in a new project.
test