Intel Accelerator Functional Unit (AFU) Simulation Environment (ASE) Quick Start User Guide
The Intel Accelerator Functional Unit (AFU) Simulation Environment
(ASE) is a hardware and software co-simulation environment for the
Xeon® Processor with Integrated FPGA.
The ASE provides a transactional model
for the Core Cache Interface protocol and a memory model for the FPGA-attached local memory.
The local memory model provides a simulation model for the dual-memory banks on the Intel®
Programmable Acceleration Card with Intel® Arria® 10 GX FPGA.
The ASE also validates Accelerator
Functional Unit (AFU) compliance to the following protocols and APIs:
Intel® Quartus® Prime Pro Edition (17.1.1 version)
Setting Up the Environment
You must set up your simulation
environment and install the OPAE software before
running the ASE.
Set the following environment variables for your simulation
$ export VCS_HOME=<path to VCS installation directory>
$ export PATH=$VCS_HOME/bin:$PATH
The VCS installation directory structure is as
Make sure your system has a valid VCS
$ export MTI_HOME=<path to Modelsim installation directory>
$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH
The Modelsim/Questa installation directory
structure is as follows:
Make sure your system has a valid Modelsim
For Intel® Quartus® Prime Pro
$ export QUARTUS_HOME=<path to Intel Quartus Prime Pro Edition installation directory>
The Intel® Quartus® Prime installation directory
structure is as follows:
$ export LM_LICENSE_FILE=<Quartus Prime License>
Extract the runtime archive file, and install OPAE libraries, binaries, include files, and
ASE libraries as described in the Installing the OPAE
Software chapter of the
IntelAcceleration Stack1.2 Quick Start Guide for
Intel® Programmable Acceleration Card with Intel® Arria® 10 GX
Your environment must be set up correctly to configure and build an
AFU. In particular, you must ensure that the OPAE SDK is properly installed. OPAE
SDK scripts must be on PATH and include files and
libraries must be available to the C compiler. In addition, you must ensure that the
OPAE_PLATFORM_ROOT environment variable is
To ensure that the OPAE SDK and ASE are properly installed, in a
shell, confirm that the afu_sim_setup program
is found on the PATH.
Migrating Simulation from Acceleration Stack Version 1.0 to Version 1.1 and Greater
The Acceleration Stack v. 1.1 and greater provide the afu_sim_setup tool, which supports the inclusion of network interfaces, and
better connects the files used for simulation and synthesis. Wherever possible, Intel
recommends using afu_sim_setup rather than the setup_sim.sh and run_app.sh
Simulating hello_afu in Client-Server Mode
The hello_afu example is a simple AFU template that demonstrates the primary
CCI-P interface. The RTL satisfies the minimum requirements of an AFU, responding to
memory-mapped I/O reads to return the device feature header and the AFU's UUID.
図 1. hello_afu Directory Tree
注: This document uses <AFU example> to refer to an example design directory, such
as hello_afu in the figure above.
The software demonstrates the minimum requirements to attach to an FPGA using
OPAE. The RTL demonstrates the minimum requirements to satisfy the OPAE driver and the hello_afu example software.
RTL simulation and synthesis are controlled by filelist.txt.
To successfully configure and build the AFU samples, your environment must be
set up correctly, as described in Setting Up the
The following example flow introduces the basic
examples can be simulated with the ASE, except eth-e2e_e10 and
Simulation requires two software processes: one for RTL simulation and the
other to run the connected software. To construct an RTL simulation environment, execute the
following in $OPAE_PLATFORM_ROOT/hw/samples/hello_afu:
This command constructs an ASE environment in the build_sim subdirectory.
To build and execute the simulator:
$ cd build_sim
$ make sim
The simulator prints a message that it is ready for simulation. It also
prints a message prompting you to set the ASE_WORKDIR
Open another shell for software simulation. To build and run the software in
the new shell:
$ cd $OPAE_PLATFORM_ROOT
$ export ASE_WORKDIR=$OPAE_PLATFORM_ROOT/hw/samples/hello_afu/build_sim/work
$ cd $OPAE_PLATFORM_ROOT/hw/samples/hello_afu/sw
$ make clean
$ make USE_ASE=1
注: The specific pathname for ASE_WORKDIR may vary. Use the pathname provided by the simulator
The software and simulator run, log transactions and exit.
Simulation Log Files
The waveform, CCI-P transactions, and simulation log files are
stored in the simulation work directory.
To view the waveform
Go to the directory in which you executed the make sim command.
$ make wave
A listing of CCI-P transactions is provided in ./work/ccip_transactions.tsv.
Log files are available in the directory specified in
RTL sources are specified in $OPAE_PLATFORM_ROOT/hw/samples/<AFU
<AFU example> is the example directory as shown in
(.json) file is also declared there. The AFU .json
describes the interfaces required by the AFU. It also holds a UUID to identify the AFU when it
is loaded on an FPGA.
The AFU declares that it expects the ccip_std_afu top-level interface by setting afu-top-interface to ccip_std_afu in hw/rtl/hello_afu.json. This is the base CCI-P interface with
clocks, reset and CCI-P Rx/Tx structures. Other interface options are described in more
The AFU UUID is declared only once: in the .json file.
The RTL loads the UUID from afu_json_info.vh, which is
generated automatically by the OPAE scripts. The software loads the UUID from afu_json_info.h, which is generated by sw/Makefile.
Troubleshooting Client-Server Simulation
If the afu_sim_setup command fails, confirm
afu_sim_setup is on your PATH. It
should be in /usr/bin, or in <opae install
you built OPAE from source files.
You have Python version 2.7 or higher
If you are unable to build and execute the simulator, it is likely that your
RTL simulation tool is not installed properly.
When you attempt to build and run the software, if you see an "Error
enumerating AFCs" message, you omitted setting USE_ASE=1 on the make command line. The
software is searching for a physical FPGA device. To recover, repeat the steps from the
make clean command.
Simulating an AFU in Regression Mode
For most AFUs, the afu_sim_setup
flow and the regression flow are functionally equivalent. You might choose to use
the regression flow for one of the following reasons:
The regression flow triggers both the simulator process and
the software process using a single command.
The regression flow compiles Platform Designer
components from Platform Designer source files
and includes the generated RTL in the RTL
Use the regress.sh script,
located in $OPAE_PLATFORM_ROOT/hw/common/scripts, to execute the
simulator and the application exactly once.
VCS if installed;
otherwise, Modelsim or QuestaSim
Target platform name
containing an OPAE installation
Board specified by
AFU configuration variant
Depends on AFU
selected (-a flag).
Depends on AFU
selected (-a flag).
Path to OPAE installation
Local memory model
Path to OPAE source
No (except for
Target Platform Name -p
The target platform name is used by the Platform Interface Manager
-p defaults to discrete if OPAE_PLATFORM_ROOT is
undefined or does not specify a valid OPAE release. discrete models a generic PCIe card with two banks of local
recommends setting OPAE_PLATFORM_ROOT as
described in Setting Up the
AFU Configuration Variant -v
regress.sh looks in the
hw/rtl/filelist*.txt file to determine the
configuration variant. When you specify the variant as -v
<variant>, regress.sh looks in hw/rtl/filelist_<variant>.txt for the AFU RTL source files.
The default value of -v depends on
AFU selected (-a flag):
Native loopback (NLB) AFU: RTL files listed in hw/rtl/filelist_mode_3.txt
Other AFUs: RTL files listed in hw/rtl/filelist.txt
Path to OPAE Installation -i
You must specify the install path if you do not use the RPM flow. If you are
using the RPM flow, the install path is not required. 2
Local Memory Model -m
-m selects the simulation model
for FPGA private memory.
EMIF_MODEL_BASIC uses a simple System
Verilog array to model dual banks of DRAM.
注:Intel recommends using
EMIF_MODEL_BASIC for a faster
EMIF_MODEL_ADVANCED uses an
advanced cycle-accurate model of the EMIF memory controller.
Path to OPAE Source -b
注: This flag is required when simulating the NLB AFU in the
regression flow. -b is used to locate the
application source relative to the OPAE source for NLB.
2 If you
built OPAE from the source as instructed in Building the
OPAE Software section of IntelAcceleration
Stack 1.2 Quick Start Guide for
Intel Programmable Acceleration Card with Intel® Arria® 10 GX FPGA,
provide the installation path. For example:/home/john/opaeinstall
表 3. AFU ExamplesEach AFU example includes a detailed README file, providing
an operational description and notes on how to simulate the design. For a fuller
understanding of the simulation process, review the README file in each AFU
hello_mem_afu demonstrates an AFU that
builds a simple state machine to access memory. The state machine is
capable of several access patterns to local memory directly attached
to FPGA pins, such as DDR4 DIMMs. This memory is distinct from the
host memory accessed over CCI-P. The hello_mem_afu controller state machine is managed by
the host through MMIO requests to CSRs.
hello_intr_afu demonstrates the user
interrupt feature in ASE.
dma_afu demonstrates a DMA Basic
Building Block for host to FPGA, FPGA to host and FPGA to FPGA
memory transfers. When simulating this AFU, the buffer size used for DMA transfer is
intentionally kept small to keep the simulation time
nlb_mode_0 is a CCI-P system
demonstrating the memory copy test. The software application is
located at $OPAE_PLATFORM_ROOT/sw/opae-<release_number>/sample/hello_fpga.c
To simulate the NLB AFU in the regression
flow, you must specify the path to the OPAE source, using
the -b flag, as
$ sh regress.sh -a <afu dir> -r rtl_sim
-s < vcs|modelsim|questa > [-i <opae install path>]
-b <path to opae source dir>
If the following error appears during simulation, correct it by
following the steps below.
# [SIM] An ASE instance is probably still running in current directory !
# [SIM] Check for PID 28816
# [SIM] Simulation will exit... you may use a SIGKILL to kill the simulation process.
# [SIM] Also check if .ase_ready.pid file is removed before proceeding.
Type pkill ase_simv to kill
zombie simulation processes and remove any temporary files (left behind by
failed simulation processes or crashes).
Delete the .ase_ready.pid file, found in
Document Revision History for Intel Accelerator Functional Unit (AFU) Simulation Environment (ASE) Quick Start User Guide
IntelAcceleration Stack Version
1.1 Production (supported with Intel® Quartus® Prime Pro Edition