This fork aims to create a more flexible test bench that supports easy exploration of multiple parameters.

Within this folder:

• Required SystemVerilog libraries
• conv_1d, conv_2d, gemms, gemmt: benchmark modules

Within this folder:

• arch: contains the ArchFactory class that must be implemented to generate a corresponding architecture file.
• design: contains the Design class that must be implemented to generate a corresponding SystemVerilog wrapper, as well as TCL and SDC files (for Quartus):
• exp contains:
  • Experiment: instantiated with an ArchFactory and a Design, and then runs on a tool (e.g., VTR, Quartus) for generated architecture and design based on given parameters:
  • ExperimentFactory: parameters can be given as a list instead of a value (described under Usage/Parameters). This class can then generate a new Experiment for each unique combination of parameters.
• run: contains Runner, which takes in an Experiment subclass, as well as an ArchFactory and Design, and runs all possible combinations of experiments as generated by the ExperimentFactory.
• consts: contains system-wide constants:
  • keys: recognized top-level parameter keys
  • shared_defaults: default values for certain parameters
  • shared_requirements: parameters that must be provided by users
  • translation: parameter keys to human label matching
• util provides several useful base classes:
  • Abstract provides a convenience function raise_unimplemented() to simulate interface behaviour of standard OOP languages, e.g., Java.
  • ParamsChecker: Abstract provides methods to check for required parameters and/or simultaneously fill in default values.
  • DynamicallyNamed: Abstract is an interface for objects that should have their names (i.e., a label) dynamically generated at runtime.

This folder contains concrete implementations of the interfaces found in structure:

• Under arch:
  • base contains BaseArchFactory, the baseline FPGA architecture implementation from the Kratos paper.
• Under design:
  • conv_1d, conv_2d, gemms, gemmt: implementations for the various benchmarks
• Under exp, concrete implementations of Experiment:
  • vtr contains VtrExperiment, which will run the experiment on VTR.
  • quartus contains QuartusExperiment (TO-DO)

Sample usage can be found in sample.py. Some key points to note:

• A run returns a pandas.DataFrame, which then can be used to plot graphs, perform data analysis etc.
• You can provide filter_params and filter_results to the run_all_threaded() method to obtain a smaller DataFrame.

This test bench uses one dictionary passed into the runner. Class-specific parameters are then split using keys under structure.consts.keys (which we will shorten to keys here). A sample one (adapted from sample.py) is presented here with explanations:

{
    keys.KEY_EXP: {
        # Experiment parameters
        'root_dir': 'experiments/conv_2d_pw', # Base folder to create the experiment in
        'verilog_search_dir': os.path.join(os.path.dirname(os.path.realpath(__file__)), 'verilog') # Search path for SystemVerilog files
    },
    keys.KEY_ARCH: {
        # Architecture parameters
        'lut_size': 3 # LUT size used
    },
    keys.KEY_DESIGN: {
        # Design parameters
        'data_width': 4, # Data width
        'sparsity': 0.5, # Sparsity
        # ...and any other design-specific parameters
    }
}

Provide all parameters that should be explored simultaneously as a list. The test bench will then automatically create and run all possible permutations.

If you would like to simultaneously explore LUT size and data width for example, then provide:

{
    # ...other parameters
    keys.KEY_ARCH: {
        # Architecture parameters
        'lut_size': [3, 4, 5] # Exploring sizes 3-5
        # ...other parameters
    },
    keys.KEY_DESIGN: {
        # Design parameters
        'data_width': [4, 8] # Exploring 4-bit, 8-bit
    }
}

In this example, the test bench will run experiments for:

• 3-LUT, 4-bit
• 4-LUT, 4-bit
• 5-LUT, 4-bit
• 3-LUT, 8-bit
• ...

In general, the resultant final number of experiments run will be a * b * c * ..., where a, b, c, ... are the lengths of each provided list.

run_bg.sh is provided to facilitate running of the Python scripts as background processes, so it can continue even when the terminal is closed, e.g., SSH connection terminated:

1. Ensure that run_bg.sh is executable, i.e., chmod +x run_bg.sh
2. Edit the parameters at the top of run_bg.sh:
   • BASH_RUN_SCRIPT: directs the script to run another bash script in the background.
   • A sample bash script to use, run_with_venv.sh, has been included.
   • Note the known limitation in terminating the script, as detailed at the top of this file.
3. To use the script:
   • Start execution: ./run_bg.sh start
   • Stop execution: ./run_bg.sh stop
   • Retart execution: ./run_bg.sh start (may not work as intended due to the limitation as detailed)

When the script is running, there will be a few files generated:

• <script_name>.out: This serves as the stdout of the script.
• <script_name>.pid: This logs the PID of the parent bash script.