This post is an overview of testbench utility libraries, verification components will be covered in a separate post. The intention is to help with selecting which library to use, since I haven’t found a neutral comparison of them anywhere.


Table of Contents


The libraries I’ll be exploring in this post are:

  • OSVVM
  • PlTbUtils
  • UVVM
  • VUnit

I’ll be deferring the discussion of cocotb to another post, since it is not implemented in VHDL.

High-Level Comparison

Let’s take a look at some of the most common features offered by the libraries.

  OSVVM PlTbUtils UVVM VUnit
Assert Affirm check check_* check[_*]
Wait WaitFor(Clock,Level) wait(sig,clks) await_*  
Logging Log print[,v,2] log log
Signal Generators Create(Clock,Reset) clkgen [adjustable_]clock_generator  
Watchdog WaitForBarrier waitsig watchdog set_timeout
Testbench Control   Yes   Yes
Text Utils Yes Yes Yes  
Random Yes   Yes  
License Apache 2.0 LGPL Apache 2.0 Mozilla Public License, v.2.0
Table 1: Library comparison

Here I’ve grouped together similar functionality and tried to indicate how to access it in the libraries that implement it, even though the features within each functionality may be vastly different between libraries. This gives a birds-eye view of the common functionality. Empty cells indicate that the functionality is not implemented.

  • Assert are procedures that act like assert statements, checking a condition.
  • Wait functionality is an extension of wait statements.
  • Logging comes in many flavors in these libraries, the procedures mentioned in the table can all print to the transcript. Some offer much more functionality.
  • Testbench control indicates that the libraries provide procedures that are meant to be used when initializing a testbench or individual tests.
  • Text utils indicates that the library provides some form of functions to handle text, common features are replacing substrings, changing case and adding/removing whitespace.
  • Signal generators are primarily used to generate clock signals for simulations.
  • The watchdog functionality stops the simulation in case of an abnormally long-running test.
  • OSVVM and UVVM offer random number generation.
  • Finally the open-source licenses used by the libraries are mentioned.

This list of features is by no means complete with regards to the features of each library. The following sections will describe some more details, but for a full understanding, readers are referred to the documentation of each library.

OSVVM Utility Library

OSVVM, or Open Source VHDL Verification Methodology, “provides a methodology and library to simplify the entire verification effort.” Among other features, it supports transaction level modeling, functional coverage, randomized test generation, data structures, and basic utilities. The packages that I have deemed to belong to the utility category (rather than the verification components category) are:

  • AlertLog
  • Coverage
  • Random
  • TbUtil (Wait, Signal Generators)
  • TextUtil
  • Transcript (File IO for logging)

Since the Wait functionality offers timeout, a watchdog functionality is easily implemented.

These are documented in OSVVM’s documentation repository: https://github.com/OSVVM/Documentation

Example testbench code

half_adder_osvvm_tb.vhd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

library osvvm;
context osvvm.OsvvmContext;

-- Test case entity
entity half_adder_osvvm_tb is
end entity half_adder_osvvm_tb;


-- Test case architecture
architecture hierarchy of half_adder_osvvm_tb is
  constant TBID : AlertLogIDType := GetAlertLogID("HA_TB", ALERTLOG_BASE_ID);

  -- DUT stimuli and response signals
  signal r_BIT1  : std_logic := '0';
  signal r_BIT2  : std_logic := '0';
  signal w_SUM   : std_logic;
  signal w_CARRY : std_logic;
begin

  -- Instantiate DUT
  i_half_adder: entity work.half_adder
    port map (
        i_bit1  => r_BIT1,
        i_bit2  => r_BIT2,
        o_sum   => w_SUM,
        o_carry => w_CARRY
        );
    
  Testbench_1 : block 
  begin
    p_main: process
    begin
      SetAlertLogName("half_adder_tb") ; 
      wait for 0 ns ;   -- make sure all processes have elaborated
      SetLogEnable(DEBUG, TRUE) ;  -- Enable DEBUG Messages for all levels of the hierarchy

      Log("Check defaults on output ports", INFO);
      ------------------------------------------------------------
      AffirmIfEqual(TBID, w_SUM, '0');
      AffirmIfEqual(TBID, w_CARRY, '0');
      wait for 1 ns;

      Log("Check logic", INFO);
      ------------------------------------------------------------
      r_BIT1 <= '0';
      r_BIT2 <= '0';
      wait for 10 ns;
      AffirmIfEqual(TBID, w_SUM, '0');
      AffirmIfEqual(TBID, w_CARRY, '0');
      r_BIT1 <= '0';
      r_BIT2 <= '1';
      wait for 10 ns;
      AffirmIfEqual(TBID, w_SUM, '1');
      AffirmIfEqual(TBID, w_CARRY, '0');
      r_BIT1 <= '1';
      r_BIT2 <= '0';
      wait for 10 ns;
      AffirmIfEqual(TBID, w_SUM, '1');
      AffirmIfEqual(TBID, w_CARRY, '0');
      r_BIT1 <= '1';
      r_BIT2 <= '1';
      wait for 10 ns;
      AffirmIfEqual(TBID, w_SUM, '0');
      AffirmIfEqual(TBID, w_CARRY, '1');

      wait for 1 ns;
      ReportAlerts;   
      std.env.stop; 
      wait; -- to stop completely

    end process p_main;
  end block Testbench_1;

end hierarchy;
Listing 1: Half Adder Testbench Using OSVVM

Simulator Support

OSVVM’s documentation states which simulators are supported in their documentation: Aldec (Active-HDL/RivieraPRO), Mentor (Questa/ModelSim) and GHDL. GHDL officially supports OSVVM and runs scheduled testsi to check that it works.

PlTbUtils

PlTbUtils is described as “a collection of functions, procedures and testbench components that simplifies creation of stimuli and checking results of a device under test.” PlTbUtils consists of three packages:

  • Components (Generate signals)
  • Functions (Assert, Wait, Text Utilities, Testbench Control, Logging,
  • Text Util

Since the Wait functionality offers timeout, a watchdog functionality is easily implemented.

I have created Doxygen documentation for PlTbUtils, which is hosted here: https://sturla22.github.io/pltbutils

Example testbench code

half_adder_pltb_tb.vhd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

--! \file half_adder_pltb_tb.vhd

library ieee;
    use ieee.std_logic_1164.all;
    use ieee.numeric_std.all;

library pltbutils;
    use pltbutils.txt_util.all;
    use pltbutils.pltbutils_func_pkg.all;
    use pltbutils.pltbutils_comp_pkg.all;

-- Test case entity
entity half_adder_pltb_tb is
end entity;

-- Test case architecture
architecture behave of half_adder_pltb_tb is
  -- Simulation status- and control signals
  -- for accessing .stop_sim and for viewing in waveform window
  signal pltbs          : pltbs_t := C_PLTBS_INIT;

  -- DUT stimuli and response signals
  signal r_BIT1  : std_logic := '0';
  signal r_BIT2  : std_logic := '0';
  signal w_SUM   : std_logic;
  signal w_CARRY : std_logic;
begin

  -- Instantiate DUT
  i_half_adder: entity work.half_adder
    port map (
        i_bit1  => r_BIT1,
        i_bit2  => r_BIT2,
        o_sum   => w_SUM,
        o_carry => w_CARRY);

  p_main: process
    variable pltbv  : pltbv_t := C_PLTBV_INIT;
  begin
    startsim("Half Adder Testbench", "", pltbv, pltbs);

    starttest(1, "Check defaults on output ports", pltbv, pltbs);
    check("sum should be zero with no input", w_SUM, '0', pltbv, pltbs);
    check("carry should be zero with no input", w_CARRY, '0', pltbv, pltbs);
    endtest(pltbv, pltbs);

    starttest(2, "Check logic", pltbv, pltbs);
    r_BIT1 <= '0';
    r_BIT2 <= '0';
    wait for 10 ns;
    check("sum should be 0 with 00 input", w_SUM, '0', pltbv, pltbs);
    check("carry should be 0 with 00 input", w_CARRY, '0', pltbv, pltbs);
    r_BIT1 <= '0';
    r_BIT2 <= '1';
    wait for 10 ns;
    check("sum should be 1 with 01 input", w_SUM, '1', pltbv, pltbs);
    check("carry should be 0 with 01 input", w_CARRY, '0', pltbv, pltbs);
    r_BIT1 <= '1';
    r_BIT2 <= '0';
    wait for 10 ns;
    check("sum should be 1 with 10 input", w_SUM, '1', pltbv, pltbs);
    check("carry should be 0 with 10 input", w_CARRY, '0', pltbv, pltbs);
    r_BIT1 <= '1';
    r_BIT2 <= '1';
    wait for 10 ns;
    check("sum should be 0 with 11 input", w_SUM, '0', pltbv, pltbs);
    check("carry should be 1 with 11 input", w_CARRY, '1', pltbv, pltbs);
    wait for 10 ns;
    endtest(pltbv, pltbs);

    endsim(pltbv, pltbs);

  end process p_main;

end architecture;
Listing 2: Half Adder Testbench Using PlTbUtils

Simulator Support

The officially supported simulators (according to the docs) are ModelSim, and ISim/XSim. I’ve also had good results with GHDL. PlTbUtils does not rely on any language features in VHDL-2008+ which makes it likely to work with many simulators.

UVVM Utility Library

UVVM, which stands for Universal VHDL Verification Methodology is “a free and Open Source Methodology and Library for making very structured VHDL-based testbenches.”

The Utility Library Quick Reference documentation is helpful with getting an overview of UVVM Utility Library’s capabilities. The broad categories presented there are

  • Checks and awaits
  • Logging and verbosity control
  • Alert handling
  • Reporting
  • Randomization
  • String handling
  • Signal generators
  • Synchronization
  • BFM Common Package
  • Watchdog
Example testbench code

half_adder_uvvm_tb.vhd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

library IEEE;
use IEEE.std_logic_1164.all;
use IEEE.numeric_std.all;

library STD;
use std.env.all;

library uvvm_util;
context uvvm_util.uvvm_util_context;

-- Test case entity
entity half_adder_uvvm_tb is
    generic (
        G_DEBUG : integer := 0);
end entity half_adder_uvvm_tb;

-- Test case architecture
architecture func of half_adder_uvvm_tb is
  signal r_BIT1  : std_logic := '0';
  signal r_BIT2  : std_logic := '0';
  signal w_SUM   : std_logic;
  signal w_CARRY : std_logic;
begin

  -- Instantiate DUT
  i_half_adder: entity work.half_adder
    port map (
        i_bit1  => r_BIT1,
        i_bit2  => r_BIT2,
        o_sum   => w_SUM,
        o_carry => w_CARRY
        );
    
  p_main: process
    constant C_SCOPE     : string  := C_TB_SCOPE_DEFAULT;
  begin
    -- Print the configuration to the log
    if G_DEBUG > 0 then
        report_global_ctrl(VOID);
        report_msg_id_panel(VOID);
        enable_log_msg(ALL_MESSAGES);
    end if;
    log(ID_LOG_HDR, "Start Simulation of TB for half adder", C_SCOPE);
    ------------------------------------------------------------

    log(ID_LOG_HDR, "Check defaults on output ports", C_SCOPE);
    ------------------------------------------------------------
    check_value(w_SUM, '0', ERROR, "sum should be zero with no input", C_SCOPE);
    check_value(w_CARRY, '0', ERROR, "carry should be zero with no input", C_SCOPE);

    log(ID_LOG_HDR, "Check logic", C_SCOPE);
    ------------------------------------------------------------
    r_BIT1 <= '0';
    r_BIT2 <= '0';
    wait for 10 ns;
    check_value(w_SUM, '0', ERROR, "sum should be 0 with 00 input", C_SCOPE);
    check_value(w_CARRY, '0', ERROR, "carry should be 0 with 00 input", C_SCOPE);
    r_BIT1 <= '0';
    r_BIT2 <= '1';
    wait for 10 ns;
    check_value(w_SUM, '1', ERROR, "sum should be 1 with 01 input", C_SCOPE);
    check_value(w_CARRY, '0', ERROR, "carry should be 0 with 01 input", C_SCOPE);
    r_BIT1 <= '1';
    r_BIT2 <= '0';
    wait for 10 ns;
    check_value(w_SUM, '1', ERROR, "sum should be 1 with 10 input", C_SCOPE);
    check_value(w_CARRY, '0', ERROR, "carry should be 0 with 10 input", C_SCOPE);
    r_BIT1 <= '1';
    r_BIT2 <= '1';
    wait for 10 ns;
    check_value(w_SUM, '0', ERROR, "sum should be 0 with 11 input", C_SCOPE);
    check_value(w_CARRY, '1', ERROR, "carry should be 1 with 11 input", C_SCOPE);

    -- Ending the simulation
    ------------------------------------------------------------
    wait for 1000 ns; -- to allow some time for completion
    -- Report final counters and print conclusion for simulation (Success/Fail)
    report_alert_counters(FINAL);

    log(ID_LOG_HDR, "SIMULATION COMPLETED", C_SCOPE);

    -- Finish the simulation
    std.env.stop;
    wait; -- to stop completely

  end process p_main;

end func;
Listing 3: Half Adder Testbench Using UVVM

Simulator Support

GHDL officially supports UVVM and runs scheduled tests to check that it works.

According to the invitation to the Bitvis course, “Advanced VHDL Verification - Made Simple”, at the very least the following simulators are supported: Questa/ModelSim, and Active-HDL/Riviera-PRO.

VUnit VHDL Libraries

VUnit “features the functionality needed to realize continuous and automated testing” of HDL code and includes several VHDL libraries for convenience but also for integration with their python based run/check system. VUnit provides four utility libraries:

  • Logging
  • Check
  • Run (Watchdog, Testbench Control)
  • Communication

VUnit also provides two datastructures:

  • Queue
  • Integer Array
Example testbench code

half_adder_vunit_tb.vhd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

-- file: half_adder_vunit_tb.vhd

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

library vunit_lib;
context vunit_lib.vunit_context;

entity half_adder_vunit_tb is
  generic (runner_cfg : string);
end entity;

architecture tb of half_adder_vunit_tb is
  signal r_BIT1  : std_logic := '0';
  signal r_BIT2  : std_logic := '0';
  signal w_SUM   : std_logic;
  signal w_CARRY : std_logic;
begin
  UUT : entity work.half_adder
    port map (
      i_bit1  => r_BIT1,
      i_bit2  => r_BIT2,
      o_sum   => w_SUM,
      o_carry => w_CARRY
    );

  main : process
  begin
    test_runner_setup(runner, runner_cfg);

    while test_suite loop

        if run("Check defaults on output ports") then
            check(w_SUM = '0', "sum should be zero with no input");
            check(w_CARRY = '0', "carry should be zero with no input");
            wait for 10 ns;
        end if;

        if run("Check logic") then
            r_BIT1 <= '0';
            r_BIT2 <= '0';
            wait for 10 ns;
            check(w_SUM = '0', "sum is '0' with inputs " &
                std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));
            check(w_CARRY = '0', "carry is '0' with inputs " &
                std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));

            r_BIT1 <= '0';
            r_BIT2 <= '1';
            wait for 10 ns;
        check(w_SUM = '1', "sum is '1' with inputs " &
            std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));
        check(w_CARRY = '0', "carry is '0' with inputs " &
            std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));

            r_BIT1 <= '1';
            r_BIT2 <= '0';
            wait for 10 ns;
            check(w_SUM = '1', "sum is '1' with inputs " &
                std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));
            check(w_CARRY = '0', "carry is '0' with inputs " &
                std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));

            r_BIT1 <= '1';
            r_BIT2 <= '1';
            wait for 10 ns;
            check(w_SUM = '0', "sum is '0' with inputs " &
                std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));
            check(w_CARRY = '1', "carry is '1' with inputs " &
                std_logic'image(r_BIT1) & " and " & std_logic'image(r_BIT2));
      end if;
    end loop;

    test_runner_cleanup(runner);
  end process;
end architecture;
Listing 4: Half Adder Testbench Using VUnit

Simulator Support

VUnit supports several simulators for their build/run environment, it is safe to assume that their utility libraries work in those as well.

The officially supported simulators are Active-HDL/Riviera-PRO, GHDL, and ModelSim.

Simulator Support Matrix

The difference in simulator support comes down to the language features used by the libraries vs. the language features implemented by the simulator.

I’ve based the following matrix on the documentation I’ve found, issues on github and this list of VHDL simulators on Wikipedia.

  OSVVM PlTbUtils UVVM VUnit
Active-HDL/Riviera-PRO Yes   Yes Yes
GHDL Yes Unofficial Yes Yes
Incisive No     In Progress?
ModelSim/Questa Yes Yes Yes Yes
NVC       In Progress?
SynaptiCAD       No
VCS       In Progress?
ISim/XSim   Yes   In Progress?
Xcelium In Progress?     In Progress?
NCSim       Experimental?
Table 2: Simulator Support Matrix

It’s hard to keep a list like this up to date, so when this has aged like milk let me know and I’ll update it.

Conclusion

It is left up to the reader to pick the libraries that best suit their needs, indeed there is nothing stopping us from using several (or all) of these libraries together. When doing so, be prepared for some naming clashes, for example, VUnit’s and PlTbUtils’ check procedures.

Personally, I believe the best approach for me will be to rely on VUnit as a base and then add the other libraries as their functionalities are needed, VUnit makes adding OSVVM especially easy as I showed in my post on the Test Controller design pattern.

I’ll be diving into the Verification Components soon, which is a big part of some of the frameworks mentioned here.

Changelog

2021-04-10

  • Mention cocotb
  • Add information on simulator support after request from u/threespeedlogic

Comments? You are welcome to start a discussion on Github.