Limerick(9) - Cocotb Tutorial: Test Bench Structure
4/21/2023
Hello, everyone. Welcome to the nineth episode of FPGA limerick. In this episode, we will take a deep look into the test bench structure of cocotb.
As we know, cocotb stands for Coroutine Cosimulation Testbench. So before we start, we might ask ourselves: What is coroutine?
In multi-task scenario, coroutine can be viewed as a collaborative non-preemptive thread. The following shows two coroutines calling each other, with yield at every step. And the execution order will always be 1, 2, 3, 4, 5
In cocotb, the coroutine is marked by putting “async” in front of the function name. And the yield is done with “await” command. We will soon see the details when it comes to the test bench next.
The Classic Way
If you read the official document of cocotb, it uses Makefile to organize the test bench. And for each coroutine, it will be decorated @cocotb.test(), like the following example:
@cocotb.test()
async def test_xxx(dut):
…
However, what we will recommend is something different by
taking advantage of the cocotb-test framework.
Using cocotb-test Framework
The cocotb-test is a framework based on pytest. Pytest is a Python test framework that supports features like parameterized testing, assertion, which makes unit test and regression test much easier.
Also, as the test gets more and more complicated, the test bench will import more third party packages with various versions, and the environment setup gets complicated as well. To deal with that and make the test portable across different platform, we will use tox to setup the virtual environment.
So to put them together, here is what we will use as our test architecture:
With the above architecture, we will need:
1. a file called tox.ini to setup the virtual environment
2. Python files with the file names staring as “test”. This is a requirement from pytest to look for unit tests. And we will put our test bench in those python files.
This file is largely boilerplate code, with the following part that can be customized:
1. The required packages and version, such as cocotb 1.7.2
2. The environment variable. For questa sim, the LM_LICENSE_FILE needs to be setup properly.
After you type in tox in the WSL command line, the first thing it will do is to download those required packages into a local folder called .tox.
Once the packages are all in place, the pytest will look for the test_xxx.py files under its sub directories. And as a convention, we will put all of our python test code in the tb folder. Now let’s take a look at the tb/test_nco.py
1. Test Parameters
For each DUT, it usually has some parameters (or Generics in VHDL’s lingo). And the test_param will list all the possible parameter combinations that we would like to test. For example, in the Hello World example, we would like to test 3 sets of parameters:
G_CLK_RATE = 100MHz, G_OUTPUT_RATE = 24MHz
G_CLK_RATE = 100MHz, G_OUTPUT_RATE = 10MHz
G_CLK_RATE = 50MHz, G_OUTPUT_RATE = 99KHz
2. Test Prepare
Please note in this step, we will provide a function name starting with “test”. And it will also be decorated by the @pytest.mark.paramtrize to sweep all the possible parameters set in step 1.
Please also note that this function is not marked by “async”. So it is just not a coroutine. What it does is the following:
a) Optional: If SpinalHDL is used, SBT will be invoked to regenerate the Verilog/VHDL based on the test parameter
b) Compile the Verilog/VHDL source code
c) Start the simulator
In this step, Pytest will spawn a new process to communicate with the simulator.
3. Test Bench / Test Factory
There are mainly two sub-steps in step:
a) Prepare a class for the test bench, such as the class TB() in the Hello World example. This class is basically an implementation of the functional simulation approach we mentioned in previous episodes:
Please note that all the key pieces in the above diagram are implemented as coroutine in class TB() (with async mark in front of the function name)
b) Test Factory
The Test Factory will feed the Parameters into the current test run instance. Please note that the parameters from Step 2 can to be passed to this step currently, as they are run in separate processes. The way we deal with this is to save the parameters in step 2 into a local file, and read it out in this step.
And the test factory will work with a coroutine that launches the test bench, as shown in the run_test_nco function.
Posted by
FPGA Limerick
at
April 21, 2023
|
0 Comments
Limerick(8) - Cocotb Tutorial: Introduction
4/07/2023
Hello everyone, welcome to the eighth episode of FPGA Limerick. In this episode, we will talk about the cocotb to make Python based test bench.
And to show why we choose cocotb for verification, let’s visit other possible options:
Overview
1. Verification, the old-fashioned way:
Use Verilog/VHDL to make both testbench and DUT:
As of today, a lot of companies still do things this way. However, because the Verilog/VHDL lacks the features offered by high level programming language, this method becomes very inflexible, verbose and unproductive.
2. Verification: HVL + HDL
3. Verification / Design with System Verilog
Although System Verilog did a good job introducing high level programming language features, such as Object Oriented syntax, it is still considered too verbose and low level comparing to newer programming languages like Python. And thus comes the cocotb.
Cocotb stands for Coroutine Cosimulation Testbench. It is a Python based verification framework. And practically it can be a good supplement to SpinalHDL.
As shown above, the cocotb runs Python over VPI or FLI interface to communicate with the simulator and exercise the DUT. If the DUT is written in Verilog, the VPI interface is used. If the DUT is written in VHDL, FLI interface is used. (FLI is short for Foreign Language Interface, which is a standard developed by Mentor Graphics. So you will need Questa Sim or ModelSim as the simulator.)
Cocotb Setup
Ok, now let’s get hands on to see how it works. We will run the cocotb on WSL (Windows Subsystems for Linux). The setup for WSL can be found at the end of Episode (3). Basically, we will use Ubuntu 20.04 as our Linux distro. And we will run the scripts/wsl_setup.sh to install all the necessary packages and the simulators.
As for the simulator, we will install two simulators: Verilator and Questa Sim.
The Verilator is an open-source simulator that supports Verilog / System Verilog. It will convert the Verilog / System Verilog into C++ code and compile the C++ using GCC (which is kind similar to VCS). Because this RTL-to-C++ approach, the Verilator runs probably 5x faster than Questa Sim. However, the drawback of Verilator is that:
1.It does not support VHDL at this point
2.It does not support gate level simulation
3.It does not support third party library
Because of the above, we also need a commercial simulator if third party IPs (such as DDR controller) are involved. Fortunately, we can get a poor man’s version of Questa Sim from Intel website (See the end of wsl_setup.sh). To use this version of Questa, we also need to apply for a fixed seat license (with no cost) from Intel Self Service License Center:
A fixed seat license is usually tied to a MAC address. However, under WSL, the MAC address changes every time it reboots. Fortunately, we can fix this by setting up a dummy MAC address to match the one used by the license file in WSL. And scripts/dummy_mac_addr.sh is an example of setting up such dummy MAC address.
And let’s see how we can run the cocotb with our Hello World example. Please don’t worry about all the tech details yet, as we will soon discuss them in the next episode.
In fact, all we have to do is to open the WSL, setup the dummy MAC address if we want to use Questa. And enter the HelloWorld/test/cocotb/verilator or HelloWorld/test/cocotb/questa and run tox.
Posted by
FPGA Limerick
at
April 07, 2023
|
0 Comments
