Testing the state machine
Just like software, testing our code is vital. Unlike software however, we don't have (easy) access to fancy tools like debuggers, printfs or error messages when we run on hardware. Therefore, we usually simulate FPGA designs to make sure they work in simulation in order to avoid painful debugging in hardware.
Currently, writing simulation code in spade is not possible, as the things you want to do in a simulator are quite different to describing hardware. Instead, tests are written in python using the cocotb testing framework.
If you haven't already, refer to the installation instructions to see how to install cocotb.
Setting up tests
To do our testing, we need to do a tiny bit more setup in swim.
To do testing, we need to tell swim where we put our test benches. To do so,
create a directory called test
Then edit swim.toml adding a simulation section like so:
[simulation]
testbench_dir = "test"
Inside the test folder we put our test benches in python files. Let's create
our first one by creating test/state_gen.py. Each spade test file must start
with a comment telling swim which unit is to be tested, the "top module", like
so. The path must be a fully namespaced name, and since our module resides in
main.spade, it will be main::state_gen
# top=main::state_gen
We'll also add an empty test to that file like this:
# top=main::state_gen
import cocotb
from spade import SpadeExt
@cocotb.test()
async def normal_operation(dut):
s = SpadeExt(dut)
Each test is annotated by @cocotb.test() and is an async python function
which takes a single parameter dut, the Design Under Test.
Running swim test (or swim t for the lazy :)) presents us with the following error1
Error:
0: In {tb}
1: main::state_gen is generic which is currently unsupported in test benches
Which is a limitation of the spade python interface. To test our module, we'll need to create a dummy entity without any generic parameters, for now we'll use one with 10 LEDs.
entity state_gen_10(clk: clock, rst: bool, t: Timing) -> OutputControl<int<5>> {
inst state_gen(clk, rst, 10, t)
}
After updating the top to # top=main::state_gen_10 we can swim test again and we
should see a nice PASS (along with some other output which we'll ignore for now)
[INFO] Building spade compiler
Finished release [optimized] target(s) in 0.04s
[INFO] Built spade compiler
[INFO] build/spade.sv is up to date
[INFO] Building spade-python
Finished release [optimized] target(s) in 0.04s
[INFO] Built spade-python
-.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:109 in set_program_name_in_venv Using Python virtual environment interpreter at /home/frans/Documents/spade/ws2812-spade/build/.env/bin/python
-.--ns INFO cocotb.gpi ../gpi/GpiCommon.cpp:99 in gpi_print_registered_impl VPI registered
/home/frans/Documents/spade/ws2812-spade/build/spade.sv:177: Warning: Calling system function $value$plusargs() as a task.
/home/frans/Documents/spade/ws2812-spade/build/spade.sv:177: The functions return value will be ignored.
/home/frans/Documents/spade/ws2812-spade/build/spade.sv:111: Warning: Calling system function $value$plusargs() as a task.
/home/frans/Documents/spade/ws2812-spade/build/spade.sv:111: The functions return value will be ignored.
0.00ns INFO Running on Icarus Verilog version 11.0 (stable)
0.00ns INFO Running tests with cocotb v1.6.2 from /home/frans/Documents/spade/ws2812-spade/build/.env/lib/python3.10/site-packages/cocotb
0.00ns INFO Seeding Python random module with 1657993983
0.00ns WARNING Pytest not found, assertion rewriting will not occur
0.00ns INFO Found test state_gen.normal_operation
0.00ns INFO running normal_operation (1/1)
state_gen.py.vcd
VCD info: dumpfile state_gen.py.vcd opened for output.
0.00ns INFO normal_operation passed
0.00ns INFO **************************************************************************************
** TEST STATUS SIM TIME (ns) REAL TIME (s) RATIO (ns/s) **
**************************************************************************************
** state_gen.normal_operation PASS 0.00 0.00 4.45 **
**************************************************************************************
** TESTS=1 PASS=1 FAIL=0 SKIP=0 0.00 0.00 0.26 **
**************************************************************************************
[INFO] Building vcd translator
Finished release [optimized] target(s) in 0.04s
[INFO] Built vcd translator
[INFO] Translating types in "./build/state_gen/state_gen.py.vcd"
[INFO] Translated VCD: ./build/state_gen/state_gen.py.translated.vcd
test/state_gen.py: PASS
The first time you run swim test, it will set up a
python environment with the required libraries which requires compiling a
separate part of the spade compiler. Don't be alarmed at the time swim test takes, or the amount of output the first time you run it in a new
project.
Writing Some Tests
Now we are ready to actually test our module. All spade test functions start
with s = SpadeExt(dut) which creates a nice spade interface around the cocotb
functions.
Since our entity is clocked, we need to generate a clock for it. This is done by starting a cocotb clock task like this. The exact clock frequency is not really important here, it only decides the mapping between simulation time and real time.
# import the clock generator
from cocotb.clock import Clock
# ...
clk = dut.clk_i
await cocotb.start(Clock(clk, 1, units='ns').start())
The design also takes a reset signal which we need to set to get our initial state defined. If we forget to do this, most of the signals will be undefined.
We can access the input ports of our design using s.i.<input_name> and give
them values by assigning strings containing spade expressions to them. The following code sets the reset signal to true:
s.i.rst = "true"
The
s.iinterface does not work well with cocotb built in functions likeClockYou can access the raw verilog input ports on the dut viadut.<name>_ias above which is nice if you want to pass them to special cocotb functions likeClock. However, most of the time you should use the spade interface since that doesn't require you to know the spade internal representation of types.
For our design to start running, we need to take it out of reset again, you might
think that we can just add another line s.i.rst = false. However, this would
give the design no time to see the change in reset. Instead, we need to let the
simulation step forward a bit. The easiest way to do that is to let it step forward
one clock cycle, which we do by waiting until the next time the clock goes from 1 to 0
# import trigger
from cocotb.triggers import FallingEdge
# ...
await FallingEdge(clk)
s.i.rst = "false"
This will create a waveform that looks like this
---+ +---+
clk: | | |
+---+ +---...
---+
rst: |
+-----------...
In order to let the circuit catch up to the fact that the reset has been turned off, we'll advance the simulation another tiny time step (1 picosecond):
# import timer
from cocotb.triggers import Timer
# ...
await Timer(1, units='ps')
You can find more things to wait for in the cocotb documentation for triggers.
Now we can do our first test, ensuring that the initial output of the circuit is RET. We can access the output of our dut with s.o, and run assertions on it like this:
s.o.assert_eq("OutputControl::Ret()")
If you return a struct from a unit, you can access them as normal python fields on the
s.ofield. For examples.o.x.y.assert_eq(...)
Our test file now looks like this:
# top=main::state_gen_10
from spade import SpadeExt
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import FallingEdge, Timer
@cocotb.test()
async def normal_operation(dut):
s = SpadeExt(dut)
clk = dut.clk_i
await cocotb.start(Clock(clk, 1, units='ns').start())
s.i.rst = "true"
await FallingEdge(clk)
s.i.rst = "false"
await Timer(1, units='ps')
s.o.assert_eq("OutputControl::Ret()")
and calling swim test should tell us that all our assertions passed.
A failing test
Next, we may want to ensure that we output Ret in the next clock cycle as well. So, we'll advance the clock and assert that.
Changes to our state happens on the rising edge of clocks, so I prefer to do my assertions on the falling edge. That way I don't have to worry about if values have or have not changed right at the RisingEdge.
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
Calling swim test results in the following output:
...
VCD info: dumpfile state_gen.py.vcd opened for output.
1.50ns INFO normal_operation failed
Traceback (most recent call last):
File "/home/frans/Documents/spade/ws2812-spade/build/state_gen/state_gen.py", line 22, in normal_operation
s.o.assert_eq("OutputControl::Ret()")
File "/home/frans/Documents/spade/ws2812-spade/spade/spade-python/spade/__init__.py", line 75, in assert_eq
assert False, message
AssertionError:
Assertion failed
expected: OutputControl::Ret()
got: UNDEF
verilog (0XXXXXXXXXXXXXXXXXXXXXXX != xxxxxxxxxxxxxxxxxxxxxxxx)
...
[INFO] Building vcd translator
Finished release [optimized] target(s) in 0.04s
[INFO] Built vcd translator
[INFO] Translating types in "build/state_gen/state_gen.vcd"
[INFO] Translated VCD: build/state_gen/state_gen.translated.vcd
test/state_gen.py: FAIL [build/state_gen/state_gen.translated.vcd]
Failed test cases:
normal_operation
Oh no, something went wrong, why? To debug our tests, the best method by far is to look at the wave dump. It contains the value of all the signals in the design over time and can give plenty of debug information. To see it, we need to install a vcd viewer, and the defacto standard is gtkwave.
Swim translates Verilog values in the wave dump back into spade files and stores the result in a new vcd file which is printed along with the failing tests:
test/state_gen.py: FAIL [build/state_gen/state_gen.translated.vcd]
Let's open build/state_gen/state_gen.translated.vcd in gtkwave:
gtkwave build/state_gen/state_gen.translated.vcd
This should open a window that looks something like this:
The black portion shows the value of the signals we select over time. The left
pane contains a list of the units in our design, in this case
e_proj_main_state_gen_10. This is the verilog name of our module
main::state_gen_10. If you select it, the signal list below will be populated
by all the values in that module. In this case, it is just a wrapper around the
actual design state_gen, so expand the module and select the submodule. This should give you a lot more signals
Now we have lots of signals to play with! Broadly, we can group them into
several categories. Some signals start with p_. These contain the spade value
of the corresponding signal, for example p_clk_n... is a spade value, and
clk_n... is the raw verilog bits.
Names on the form _e_<numbers> and p_e_<numbers> are subexpressions that
are not named in the spade program. Unless you're debugging the compiler, you
can ignore those.
Names on the form <name>_n<numbers> and p_<name>_n<numbers> are values which
are named in your spade code. These are the values you will actually want to look at
Finally, there are some signals called <name>_i. These are input input
values. The spade translation does not translate those, so it is better to look
at the corresponding <name>_n<numbers> signals.
To add a signal to the waveform window, double click it. To debug this value,
we'll want to look at a few signals clk, rst, state and t,
so go ahead and add those to the wave view
Here we get quite a bit of information. We see that our state is defined until
the first clock cycle after reset. We also see that all fields of t, our
timing struct is "HIGHIMP". The name is a bit confusing, but this is caused by
us forgetting to set that parameter.
Going back to the spade code, to compute the next state in Ret, we check if
the duration we've been in Ret so far is greater than t_ret. However, we
haven't set t, so we are essentially comparing duration to t_ret, which
is undefined, resulting in another undefined value.
Spade tries to do its best to avoid undefined values, it is certainly harder to write undefined values in spade than in verilog. However, when forgetting to specify inputs, and when working with memories, they can pop up.
Let's specify the timings to fix this issue. Again, exact timing here isn't important, we'll set some values that make testing possible:
s.i.t = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
With that change, our assertions pass.
More tests
Now we can get to testing the rest of the design. Since our state space is quite small in this case, we can ensure that all state transitions happen as they should. Since this is python, we can write things like loop, helper functions etc.
First, let's ensure that we stay in Ret for the specified amount of time,
i.e. us280 clock cycles:
s.i.t = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
for i in range(0, 28):
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
After that, we should be emitting the value of the first LED. Here we can write a function to check a whole LED output, since we'll do that quite a few times
async def check_led(clk, s, index):
# Each bit of the LED should be emitted
for b in range(0, 24):
# And each duration from 0 to us1_25 in each bit
# For simulation performance, we'll just check the first and last bit explicitly
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 0)")
for d in range(0, 5):
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 5)")
We can now test all our LEDs by calling it in a loop, and finally ensure that we go back to the ret state at the right time. The final test bench looks like this:
# top=main::state_gen_10
from spade import SpadeExt
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import FallingEdge, Timer
async def check_led(clk, s, index):
# Each bit of the LED should be emitted
for b in range(0, 24):
# And each duration from 0 to us1_25 in each bit
# For simulation performance, we'll just check the first and last bit explicitly
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 0)")
for d in range(0, 5):
await FallingEdge(clk)
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: 5)")
@cocotb.test()
async def normal_operation(dut):
s = SpadeExt(dut)
clk = dut.clk_i
await cocotb.start(Clock(clk, 1, units='ns').start())
s.i.rst = "true"
await FallingEdge(clk)
s.i.rst = "false"
await Timer(1, units='ps')
s.o.assert_eq("OutputControl::Ret()")
s.i.t = """Timing$(
us280: 28,
us0_4: 4,
us0_8: 8,
us0_45: 4,
us0_85: 8,
us1_25: 12,
)"""
for i in range(0, 28):
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
# Check all 10 leds
for i in range(0, 10):
await check_led(clk, s, i)
# Ensure we get back to the ret state
await FallingEdge(clk)
s.o.assert_eq("OutputControl::Ret()")
Running it gives us another assertion error:
VCD info: dumpfile state_gen.py.vcd opened for output.
28.50ns INFO normal_operation failed
Traceback (most recent call last):
File "/home/frans/Documents/spade/ws2812-spade/build/state_gen/state_gen.py", line 44, in normal_operation
await check_led(clk, s, i)
File "/home/frans/Documents/spade/ws2812-spade/build/state_gen/state_gen.py", line 13, in check_led
s.o.assert_eq(f"OutputControl::Led$(payload: {index}, bit: {b}, duration: {d})")
File "/home/frans/Documents/spade/ws2812-spade/spade/spade-python/spade/__init__.py", line 75, in assert_eq
assert False, message
AssertionError:
Assertion failed
expected: OutputControl::Led$(payload: 0, bit: 1, duration: 0)
got: proj::main::OutputControl::Led(0,0,12)
verilog (100000000001000000000000 != 100000000000000000001100)
28.50ns INFO **************************************************************************************
** TEST STATUS SIM TIME (ns) REAL TIME (s) RATIO (ns/s) **
**************************************************************************************
** state_gen.normal_operation FAIL 28.50 0.06 461.47 **
**************************************************************************************
** TESTS=1 PASS=0 FAIL=1 SKIP=0 28.50 0.07 432.88 **
**************************************************************************************
[INFO] Building vcd translator
Finished release [optimized] target(s) in 0.04s
[INFO] Built vcd translator
[INFO] Translating types in "./build/state_gen/state_gen.py.vcd"
[INFO] Translated VCD: ./build/state_gen/state_gen.py.translated.vcd
test/state_gen.py: FAIL [./build/state_gen/state_gen.py.translated.vcd]
Failed test cases:
normal_operation
Try to see if you can figure out what happened. Looking at the waves can be helpful, but in this case it might be enough to look at what states it transitioned to.
If you can't figure it out, jump to the next section for the answer