VUnit
VUnit “features the functionality needed to realize continuous and automated testing” of HDL code. It takes care of building, running tests (via a simulator like GHDL) and evaluating the results. It also offers VHDL convenience procedures, functions and components as well as Verification Components. This post will cover the build/run functionality, the convenience library will be covered in a separate post.
Table of Contents
Basics
You can install VUnit with pip instal vunit_hdl and documentation is available here
For a simple, one file design and a single testbench the following python script will build and run the simulation:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/usr/bin/env python3
"""!
@file run.py
Based on https://vunit.github.io/user_guide.html#introduction
"""
from vunit import VUnit
# Create VUnit instance by parsing command line arguments
vu = VUnit.from_argv()
# Create library 'lib'
lib = vu.add_library("lib")
# Add basic.vhd from the current working directory to library
lib.add_source_file("basic.vhd")
# Run vunit function
vu.main()
Note that VUnit needs to get access to your VHDL testbench to do its magic so this is the minimum you need to add for VUnit to run your tests:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
library vunit_lib;
context vunit_lib.vunit_context;
entity tb is
generic (runner_cfg : string);
end entity;
architecture test of tb is
begin
main : process is
begin
test_runner_setup(runner, runner_cfg);
-- Your tests here.
test_runner_cleanup(runner);
end process;
end architecture;
After making run.py executable with chmod +x run.py, the output from running ./run.py is:
Re-compile not needed
Starting lib.tb.all
Output file: _work/test_output/lib.tb.all_97ae318fe83495e631bb3dda02fd2de620f21025/output.txt
pass (P=1 S=0 F=0 T=1) lib.tb.all (0.3 seconds)
==== Summary ======================
pass lib.tb.all (0.3 seconds)
===================================
pass 1 of 1
===================================
Total time was 0.3 seconds
Elapsed time was 0.3 seconds
===================================
All passed!
Configuring Compilation and Tests
I’ll demonstrate configuration of tests with the same half adder design from nandland which I used in the VHDL Style Guide and GHDL posts. To show the configuration of compilation, we’ll use the VHDL-2008 standard by setting GHDL’s --std option to 08. The test configuration will be to inject an error into the test by setting a generic.
I have isolated the code that configures the VUnit library from the CLI code in the Library class, this way I can import the configuration in another python script such as in an Invoke script, example here.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
#!/usr/bin/env python3
"""!
@file: run_vunit.py
Based on https://vunit.github.io/user_guide.html#introduction
"""
import vunit
class Library:
def __init__(self, vu):
"""
Create the VUnit library.
"""
# Note that the library name 'work' is not allowed.
self.lib = vu.add_library("lib")
self.ghdl_common_flags = ["--std=08"]
self.sources = ["half_adder.vhd", "half_adder_tb.vhd"]
def setup(self):
"""
Add sources and configure library.
"""
self.add_sources()
self.configure_compile()
self.configure_run()
def add_sources(self):
"""
Point VUnit to the source files.
"""
for s in self.sources:
self.lib.add_source_file(s)
def configure_compile(self):
"""
Configure how VUnit builds the design and tests.
"""
self.lib.add_compile_option("ghdl.a_flags", self.ghdl_common_flags)
def configure_run(self):
"""
Configure how VUnit runs the tests.
"""
self.lib.set_sim_option("ghdl.elab_flags", self.ghdl_common_flags)
# Find test bench in lib
tb = self.lib.test_bench("half_adder_tb")
# Find test in test bench
test = tb.test("logic")
# We'll be overwriting the default configuration,
# so need to add it explicitly.
test.add_config("default")
# Add a configuration with generic "fail" set to true
# to demonstrate test failure.
test.add_config("fail", generics={"fail": True})
def main():
# Create VUnit instance from command line arguments
vu = vunit.VUnit.from_argv()
Library(vu).setup()
# Run VUnit
vu.main()
if __name__ == "__main__":
# Runs when this file is executed, not when it is imported.
main()
On the last line of configure_run I’ve used a generic fail by calling add_config on the logic test to demonstrate how to configure tests and as a byproduct: to show the appearance of failing tests in VUnit. I could have used lib.set_generic, but that sets the generic for all tests and I would have to run the python script several times with different arguments, perhaps leveraging VUnit’s custom CLI arguments recipe. However, by adding a test configuration VUnit runs the test twice, once for each configuration I’ve told it about.
Expand the “VHDL tesbench code” below for a look at the VHDL code that is being tested. Here I’ve made use of VUnit’s multiple test case recipe as well.
VHDL testbench code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
--! \file: half_adder_tb.vhd
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
library vunit_lib;
context vunit_lib.vunit_context;
entity half_adder_tb is
generic (
runner_cfg : string;
fail : boolean := false
);
signal r_BIT1 : std_logic := '0';
signal r_BIT2 : std_logic := '0';
signal w_SUM : std_logic;
signal w_CARRY : std_logic;
end entity half_adder_tb;
architecture test of half_adder_tb is
begin
UUT : entity work.half_adder
port map (
i_bit1 => r_BIT1,
i_bit2 => r_BIT2,
o_sum => w_SUM,
o_carry => w_CARRY
);
main : process is
--! \test Output port defaults
procedure test_output_port_defaults is
begin
assert w_SUM = '0'
report "sum should be zero with no input"
severity error;
assert w_CARRY = '0'
report "carry should be zero with no input"
severity error;
wait for 10 ns;
end procedure test_output_port_defaults;
--! \test Logic
procedure test_logic is
begin
r_BIT1 <= '0';
r_BIT2 <= '0';
wait for 10 ns;
assert w_SUM = '0'
report "sum should be '0' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
assert w_CARRY = '0'
report "carry should be '0' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
r_BIT1 <= '0';
r_BIT2 <= '1';
wait for 10 ns;
assert w_SUM = '1'
report "sum should be '1' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
assert w_CARRY = '0'
report "carry should be '0' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
r_BIT1 <= '1';
r_BIT2 <= '0';
wait for 10 ns;
assert w_SUM = '1'
report "sum should be '1' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
assert w_CARRY = '0'
report "carry should be '0' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
r_BIT1 <= '1';
r_BIT2 <= '1';
wait for 10 ns;
assert w_SUM = '0'
report "sum should be '0' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
if not fail then
assert w_CARRY = '1'
report "carry should be '1' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
else
-- Note(sl): Intentionally wrong check for demonstration purposes.
assert w_CARRY = '0'
report "carry should be '1' with inputs " &
std_logic'image(r_BIT1) & " and " &
std_logic'image(r_BIT2)
severity error;
end if;
end procedure test_logic;
begin
test_runner_setup(runner, runner_cfg);
while test_suite loop
if run("output_port_defaults") then
test_output_port_defaults;
end if;
if run("logic") then
test_logic;
end if;
end loop;
test_runner_cleanup(runner);
end process main;
end architecture test;
Running the python script with ./run_vunit.py results in
Re-compile not needed
Starting lib.half_adder_tb.output_port_defaults
Output file: _work/test_output/lib.half_adder_tb.output_port_defaults_71fd68b0f08bfd958330c6ab9385261009020fcd/output.txt
pass (P=1 S=0 F=0 T=3) lib.half_adder_tb.output_port_defaults (0.4 seconds)
Starting lib.half_adder_tb.default.logic
Output file: _work/test_output/lib.half_adder_tb.default.logic_7362646f53747f514f8519c7c0ef9e207eb7940f/output.txt
pass (P=2 S=0 F=0 T=3) lib.half_adder_tb.default.logic (0.4 seconds)
Starting lib.half_adder_tb.fail.logic
Output file: _work/test_output/lib.half_adder_tb.fail.logic_50ea30667ba22f8479e89f8a9eb166b63ba2e489/output.txt
/home/sturlalange/Dev/sturla22.github.io/_posts/includes/vunit/half_adder_tb.vhd:106:9:@40ns:(assertion error): carry should be '1' with inputs '1' and '1'
/usr/local/bin/ghdl:error: assertion failed
in process .half_adder_tb(test).main
from: lib.half_adder_tb(test).main.test_logic at half_adder_tb.vhd:106
/usr/local/bin/ghdl:error: simulation failed
fail (P=2 S=0 F=1 T=3) lib.half_adder_tb.fail.logic (0.4 seconds)
==== Summary ==================================================
pass lib.half_adder_tb.output_port_defaults (0.4 seconds)
pass lib.half_adder_tb.default.logic (0.4 seconds)
fail lib.half_adder_tb.fail.logic (0.4 seconds)
===============================================================
pass 2 of 3
fail 1 of 3
===============================================================
Total time was 1.1 seconds
Elapsed time was 1.1 seconds
===============================================================
Some failed!
As you can see from the summary section of the output, the logic test runs twice and fails in the second attempt as expected since the fail generic has been set to True. You can also see that VUnit is running this with GHDL since the error message comes from /usr/local/bin/ghdl. Running the python script with the arguments --clean -v will show you the compilation and run commands that VUnit issues on your behalf.
Invoke example
With Invoke I can automate numerous tasks that depend on CLI or python interfaces. In the following script I set VUnits output directory to _work.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
"""!
@file tasks.py
@brief Invoke tasks.
"""
import invoke
import vunit
import run_vunit
@invoke.task
def run(ctx):
cli = vunit.VUnitCLI()
vu = vunit.VUnit.from_args(cli.parse_args(["-o=_work"]))
l = run_vunit.Library(vu)
# Manipulate library if needed.
l.setup()
vu.main()
Having installed Invoke with pip install invoke I can run inv run to execute this task.
CI example
VUnit offers XUnit output in xml format which can be easily integrated with CI tools like Gitlab CI or Jenkins. To instruct VUnit to create an xml file we provide the -x argument and a filename: ./run.py -x results.xml
Gitlab
Gitlab offers CI on shared runners for free accounts, it requires a yaml specification in a file called .gitlab-ci.yml
1
2
3
4
5
6
7
8
9
10
# file: .gitlab-ci.yml
image: ghdl/vunit:gcc
test:
script:
- ./run_vunit.py -o build -x build/test_results/results.xml
artifacts:
reports:
junit: build/test_results/results.xml
Note: Don’t forget to set your VUnit python script to be executable with chmod +x run_vunit.py, even on windows, where you might need to do something like git update-index --chmod=+x run_vunit.py before pushing to gitlab.
Gitlab even provides an overview of the number of tests and their results based on the XUnit output.
Github
According to VUnit’s documentation the following yaml should work for github:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# file: github_action.yml
name: VUnit CI Tests
on:
push:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: VUnit/vunit_action@v0.1.0
with:
run_file: run_vunit.py
As far as I can see, there is no official support for XUnit result parsing on Github.
XUnit-Viewer
Xunit Viewer can generate HTML from XUnit results files.
See example output here.
Conclusion
The build/run functionality of VUnit is a great abstraction of this otherwise tedious process, the results parsing helps you focus on the important output and to discard the uninteresting parts, such as long command lines which executed successfully. Getting a CI process up and running in 7 lines of yaml is pretty impressive too!
Comments? You are welcome to start a discussion on Github.