This User Guide is applicable to H-Tile and L-Tile variants of the Intel^® Stratix^® 10 devices.

The Intel L-/H-Tile Avalon-MM+ for PCI Express IP supports the following features:

• Complete protocol stack including the Transaction, Data Link, and Physical Layers implemented as a hard IP.
• Support for Gen3 x16 for Endpoints.
• Support for 512-bit Avalon-MM interface to the Application Layer at the Gen3 x16 data rate for Intel^® Stratix^® 10 devices.
• Support for address widths ranging from 10-bit to 64-bit for the Avalon-MM interface to the Application Layer.
• Platform Designer design example demonstrating parameterization, design modules, and connectivity.
• Standard Avalon^® -MM interfaces:
  • High-throughput bursting Avalon^® -MM slave with byte enable support.
  • High-throughput bursting Avalon^® -MM master with byte enable support associated with 1 - 7 Base Address Registers (BARs).
• High-throughput data movers for DMA support:
  • Moves data from local memory in Avalon^® -MM space to system memory in PCIe* space using PCIe* Memory Write (MWr) Transaction Layer Packets (TLPs).
  • Moves data from system memory in PCIe* space to local memory in Avalon^® -MM space using PCIe* Memory Read (MRd) TLPs.
• Support for legacy interrupts (INTx), Message Signaled Interrupts (MSI) and MSI-X.
• Advanced Error Reporting (AER): In Intel^® Stratix^® 10 devices, Advanced Error Reporting is always enabled in the PCIe Hard IP for both the L and H transceiver tiles.
• Completion timeout checking.
• Modular implementation to select the required features for a specific application:
  • Simultaneous support for data movers and high-throughput Avalon^® -MM slaves and masters.
  • Avalon^® -MM slave to easily access the entire PCIe* address space without requiring any PCIe* specific knowledge.
• Autonomous Hard IP mode, allowing the PCIe IP core to begin operation before the FPGA fabric is programmed. This mode is enabled by default. It cannot be disabled.
  Note: Unless Readiness Notifications mechanisms are used (see Section 6.23 of the PCIe Base Specification), the Root Complex and/or system software must allow at least 1.0 s after a Conventional Reset of a device before it may determine that a device which fails to return a Successful Completion status for a valid Configuration Request is a broken device. This period is independent of how quickly Link training completes.
• Available in Intel^® Quartus^® Prime Pro Edition, in both Platform Designer and IP Catalog.
• Operates at 250 MHz in -1 or -2 speed grade Intel^® Stratix^® 10 devices.
• Easy to use:
  • No license requirement.

Using Intel^® Quartus^® Prime Pro Edition, you can generate a simple DMA design example for the Intel L-/H-Tile Avalon-MM+ for PCI Express IP. The generated design example reflects the parameters that you specify. It automatically creates the files necessary to simulate and compile in the Intel^® Quartus^® Prime Pro Edition software. You can download the compiled design to the Intel^® Stratix^® 10 Development Board. To download to custom hardware, update the Intel^® Quartus^® Prime Pro Edition Settings File (.qsf) with the correct pin assignments .

1. Change to the testbench simulation directory, pcie_example_design_tb.
2. Run the simulation script for the simulator of your choice. Refer to the table below.
3. Analyze the results.

The simulation reports, "Simulation stopped due to successful completion" if no errors occur.

Figure 9. Partial Transcript from Successful Simulation Testbench

The PCIe Hard IP implements the Transaction, Data Link, and Physical layers stack required by the PCI Express* protocol. This stack allows the user application logic in the Intel^® Stratix^® 10 FPGA to interface with another device via a PCI Express* link.

While the PCIe* Hard IP uses a 500 MHz clock, other modules in the Bridge use a 250 MHz clock. The Width and Rate Adapter handles the necessary conversion between these two clock domains. An additional adapter converts between the embedded header format of the PCIe Hard IP and the separate header and data format used by the other modules in the Avalon^® -MM Bridge.

The Avalon^® -MM Bridge mainly consists of four leaf modules that interact with an Avalon^® -ST Scheduler. The four leaf modules are:

• Bursting Master: This module converts memory read and write TLPs received over the PCIe link into Avalon^® -MM burst read and write transactions, and sends back CplD TLPs for read requests it receives.
• Bursting Slave: This module converts Avalon-MM read and write transactions into PCIe memory read and write TLPs to be transmitted over the PCIe link. This module also processes the CplD TLPs received for the read requests it sent.
• Write Data Mover: This module uses PCIe memory write TLPs and Avalon^® -MM read transactions to move large amounts of data from your application logic in the Avalon^® -MM space to the system memory in the PCIe space.
• Read Data Mover: This module uses PCIe memory read TLPs and Avalon^® -MM write transactions to move large amounts of data from the system memory in the PCIe space to the FPGA memory in the Avalon^® -MM space.

The Avalon^® -ST Scheduler serves two main functions. In the receive direction, this module distributes the received TLPs to the appropriate leaf module based on the TLP’s type, address and tag. In the transmit direction, this module takes TLPs ready for transmission from the leaf modules and schedules them for transfer to the Width and Rate Adapter to be forwarded to the PCIe Hard IP, based on the available credits. These data transfers follow the PCI Express* Specifications' ordering rules to avoid deadlocks.

Descriptors provided to the Data Movers through one of their Avalon^® -ST sink interfaces control the data transfers. The Data Movers report the transfers’ status through their Avalon^® -ST source interfaces.

The Config Slave converts single-cycle, 32-bit Avalon^® -MM read and write transactions into PCIe configuration read and write TLPs (CfgRd0, CfgRd1, CfgWr0 and CfgWr1) to be sent over the PCIe link. This module also processes the completion TLPs (Cpl and CplD) it receives in return.

The Response Reordering module assembles and reorders completion TLPs received over the PCIe link for the Bursting Slave and the Read Data Mover. It routes the completions based on their tag (see Tags for tag allocation information).

No re-ordering is necessary for the completions sent to the Config module as it only issues one request TLP at a time.

You can individually enable or disable the Bursting Master and Slave, and the Read and Write Data Mover modules by making GUI selections in any combination in the Parameters Editor inside the Intel^® Quartus^® Prime Pro Edition.

Endpoint applications typically need the Bursting Master to enable the host to provide information for the other modules.

The modules that issue non-posted requests need tags to associate the completions with the requests.

The remaining 128 tags are not used.

The tag field of outgoing posted requests is a "don't care" and is set to 0.

Read Data Mover (RDDM) interface: This interface transfers DMA data from the PCIe* system memory to the memory in Avalon^® -MM address space.

Write Data Mover (WRDM) interface: This interface transfers DMA data from the memory in Avalon^® -MM address space to the PCIe* system memory.

Bursting Master (BAM) interface: This interface provides host access to the registers and memory in Avalon^® -MM address space. The Busting Master module converts PCIe Memory Reads and Writes to Avalon^® -MM Reads and Writes.

Bursting Slave (BAS) interface: This interface allows the user application in the FPGA to access the PCIe* system memory. The Bursting Slave module converts Avalon^® -MM Reads and Writes to PCIe Memory Reads and Writes.

The modular design of the Intel L-/H-Tile Avalon-MM+ for PCI Express IP lets you enable just the interfaces required for your application.

When the single destination bit is set, the transfers use the same destination address. If the bit is not set, the address increments for each transfer.

The Read Data Mover internally keeps two queues of descriptors. The priority queue has absolute priority over the normal queue. Use it carefully to avoid starving the normal queue.

If the Read Data Mover receives a descriptor on the priority interface while processing a descriptor from the normal queue, it switches to processing descriptors from the priority queue as soon as it has completed the current descriptor. The Read Data Mover resumes processing the descriptors from the normal queue once the priority queue is empty. Do not use the same descriptor ID simultaneously in the two queues as there would be no way to distinguish them on the Status Avalon-ST source interface.

This interface does not have a ready input. The application logic must always be ready to receive status information for any descriptor that it has sent to the Read Data Mover.

The Read Data Mover copies over the application specific bits in the rddm_tx_data_o bus from the corresponding descriptor. A set priority bit indicates that the descriptor is from the priority descriptor sink.

A status word is output on this interface when the processing of a descriptor has completed, including the reception of all completions for all memory read requests.

The waitrequestAllowance of this interface is four.

When the single source bit is set, the source address is used for all the transfers unchanged. If the bit is not set, the address increments for each transfer

When set, the immediate bit indicates immediate writes. The Write Data Mover supports immediate writes of one or two dwords. For immediate transfers, bits [31:0] or [63:0] contain the payload for one or two dwords transfers respectively. The two dwords immediate writes cannot cross a four KB boundary.

The Write Data Mover internally keeps two queues of descriptors. The priority queue has absolute priority over the normal queue. Use it carefully to avoid starving the normal queue.

If the Write Data Mover receives a descriptor on the priority interface while processing a descriptor from the normal queue, it switches to processing descriptors from the priority queue after it has completed processing the current descriptor from the normal queue. The Write Data Mover resumes processing descriptors from the normal queue once the priority queue is empty. Do not use the same descriptor ID simultaneously in the two queues as there would be no way to distinguish them on the Status Avalon-ST source interface.

This interface does not have a ready input. The application logic must always be ready to receive status information for any descriptor that it has sent to the Write Data Mover.

The ready latency does not matter because there is no ready input.

The Write Data Mover copies over the application specific bits in the wrdm_tx_data_o bus from the corresponding descriptor. A set priority bit indicates that the descriptor was from the priority descriptor sink.

For more details on these interface signals, refer to the Avalon^® Interface Specifications.

The bam_bar_o bus contains the BAR address for a particular TLP. This bus is as an extension of the standard address bus.

For more details on these interface signals, refer to the Avalon^® Interface Specifications.

The Bursting Master module transforms PCIe memory read (MRd) and memory write (MWr) request packets received from the PCIe system into Avalon^® -MM read and write transactions.

The Bursting Master module compares the address of each incoming read or write TLP with the BARs defined in the PCIe Hard IP. For TLPs that match with a BAR, the offset from the BAR is put out on the address bus and the number of the matching BAR comes out on the bam_bar_o bus.

Although these signals are in conduits separate from the Avalon^® -MM master interface, they are synchronous to it and can be treated as extensions of the address bus. The user logic can decode the Avalon^® -MM address and physical/virtual function numbers to translate them into the Avalon^® -MM address space.

Because the additional signals are in a separate conduit, they must be routed along with the signals from the Avalon^® -MM master interface to a single module with an Avalon^® -MM slave interface and matching conduits for the BAR and function numbers. This module decodes the BAR information contained in the bam_bar_o signals in the conduit, and routes the data to one or more standard Avalon^® -MM slaves. This decoder (referred to as the BAR Interpreter in the available design example) is necessary because Platform Designer cannot route the conduit to multiple points.

The following figure shows an example of translating the PCIe address into an Avalon^® -MM address. This example shows that PCIe address regions grouped with BAR number are translated into Avalon^® -MM address regions grouped with virtual function number.

Figure 12. Example PCIe to Avalon^® -MM Address Mapping

The Intel L-/H-Tile Avalon-MM+ for PCI Express IP generates the 250 MHz Application clock, coreclkout_hip and the reset signal. This IP core also provides a synchronized version of the reset signal, app_nreset_status, to the Application. This is an active low reset.

MSI interrupts are signaled on the PCI Express link using Memory Write (MWr) TLPs. You can obtain the necessary MSI information (such as the message address and data) from the configuration output interface (tl_cfg_*) to create the MWr TLPs.

Information on the tl_cfg_ctl bus is time-division multiplexed (TDM). Examples of information multiplexed onto the tl_cfg_ctl bus include device number, bus number, MSI information (address, data, mask) and AER information. For more details, refer to the Transaction Layer Configuration Space Interface section of the Intel^® Stratix^® 10 Avalon^® streaming and Single Root I/O Virtualization (SR-IOV) Interface for PCI Express Solutions User Guide.

In the PCI Express* protocol, a memory read request from the host with a length of 1 dword and byte enables being all 0’s translate to a flush request for the Completer, which in this case is the Intel L-/H-Tile Avalon-MM+ for PCI Express IP. However, this flush request feature is not supported by the Intel L-/H-Tile Avalon-MM+ for PCI Express IP.

For more details on the signals in this interface, refer to the Intel^® Stratix^® 10 Avalon^® -MM Interface for PCI Express* Solutions User Guide.

Unused inputs must be tied low.

The Intel^® Quartus^® Prime Pro Edition software optionally generates a functional simulation model, a testbench or design example, and vendor-specific simulator setup scripts when you generate your parameterized PCI Express* IP core. For Endpoints, the generation creates a Root Port BFM. There is no support for Root Ports in this release of the Intel^® Quartus^® Prime Pro Edition.

The Intel^® Quartus^® Prime Pro Edition supports the following simulators.

This DMA example design includes a DMA Controller and an on-chip memory to exercise the Data Movers, and a Traffic Generator and Checker to exercise the Bursting Slave.

The example design also connects the Bursting Master to the on-chip memory to allow high-throughput transfers should the host or some other component of the PCIe system be capable of initiating such transfers (e.g. a Root Complex with a DMA engine).

An address translation module is inserted between the Traffic Generator and the Bursting Slave to show an example of how address translation can be performed between the Avalon^® and PCIe address spaces.

The on-chip memory that the Data Movers and the Bursting Master connect to is a dual-port memory to allow full-duplex data movement.

The Bursting Master connects to a BAR Interpreter module, which combines the address and BAR number and allows the Bursting Master to control the DMA Controller, the Traffic Generator and the Address Mapper. The BAR Interpreter also connects the Bursting Master to the dual-port memory.

The example design is generated dynamically based on the selected variation of the Intel L-/H-Tile Avalon-MM+ for PCI Express IP. However, some of the user’s parameter selections may need to be overwritten to ensure proper functionality. A warning appears when such a need arises.

The DMA Controller is instantiated for variations that implement at least one of the Data Movers and the Bursting Master. The Traffic Generator and Checker module is instantiated for variations that implement at least the Bursting Slave and the Bursting Master.

You cannot generate an example design for variations that do not implement the Bursting Master as it is necessary for all flavors of the example design. A warning appears to explain that the Bursting Master with appropriate BAR settings must be added in order to generate the example design.

The following figure shows the system-level view in Platform Designer of the DMA example design.

Figure 27. Stratix 10 Intel L-/H-Tile Avalon-MM+ for PCI Express IP DMA Example Design in Platform Designer

The DMA Controller in this example design consists of six addressable queues: two write-only queues and one read-only queue each for the Read Data Mover and the Write Data Mover. In addition, the DMA Controller has two MSI control registers for each Data Mover module.

The write-only queues directly feed into the Data Movers’ normal and priority descriptor queues. The read-only queues read directly from the Data Movers’ status queues.

The MSI control registers control whether MSI generation is enabled and defines the address and data to be used for the MSI.

The entire example design is in the coreclkout_hip clock domain.

The registers in the DMA Controller are 512-bit wide to match the data path width of the Bursting Master's and Read Data Mover's Avalon^® -MM Master. This allows the Read Data Mover to write a descriptor in a single cycle if desired.

For the data written to the descriptor queue registers, use the same format and content as the data on the corresponding Avalon^® -ST interfaces of the Data Movers. The least significant of the application specific bits indicates whether an interrupt should be issued when processing of that descriptor completes.

The DMA Controller double buffers the write-only queues so that the descriptors can be built one DWORD at a time if required, for example by a 32-bit host controller. The content of the register is transferred to the Data Movers' Avalon^® -ST input when the most significant DWORD is written.

Attempting to write to a descriptor queue when the corresponding Data Mover's ready signal is not asserted causes the DMA Controller to assert its waitrequest signal until ready is asserted. You must make sure the Read Data Mover does not attempt to write to the same queue that it is processing while the queue is full, as that would lead to a deadlock. For more details on deadlocks, refer to the section Deadlock Risk and Avoidance.

You can find the status of the ready signal of a descriptor queue interface by checking the ready bit (bit [31]) of the queue registers. In addition, bits [7:0] of the queue registers indicate the approximate fill level of the queues. The other bits of the queue registers are set to 0.

Only the least significant DWORD of the WS and RS registers contains significant information. The other bits are set to 0.

The format and content of the status queues are identical to the corresponding Avalon^® -ST interfaces of the Data Movers with the addition of bit 31 indicating that the queue is empty. Reading from one of the status queues when it is empty returns 512'h8000_0000.

The format of the WI and RI interrupt control registers is as follows: {enable, priority, reserved[414:0], msi_msg_data[15:0], reserved[15:0], msi_address[63:0]}.

The enable bit controls whether or not an MSI is sent. The priority bit specifies whether to use the priority queue to send the MSI. The MSI memory write TLP also uses the contents of the msi_msg_data and msi_address fields.

Under certain circumstances, it is possible for the DMA engine in the example design hardware to get into a deadlock. This section describes the conditions that may lead to a deadlock, and how to avoid them.

When you program the DMA Controller to use the Read Data Mover to fetch too many descriptors for the Read Data Mover descriptor queue, the following loop of back-pressure that leads to a deadlock can occur.

Once the Read Data Mover has transferred enough descriptors through the DMA Controller to its own descriptor queue to fill up the queue, it deasserts its ready output. The DMA Controller in turn asserts its waitrequest output, thus preventing the Read Data Mover from writing any remaining descriptor to its own queue. After this situation occurs, the Read Data Mover continues to issue MRd read requests, but because the completions can no longer be written to the DMA Controller, the tags associated with these MRd TLPs are not released. The Read Data Mover eventually runs out of tags and stops, having gotten into a deadlock situation.

To avoid this deadlock situation, you can limit the number of descriptors that are fetched at a time. Doing so ensures that the Read Data Mover's descriptor queue never fills up when it is trying to write to its own descriptor queue.

Two application specific bits (bits [13:12]) of the status words from the Write Data Mover and Read Data Mover Status Avalon^® -ST Source interfaces control when interrupts are generated.

The DMA Controller makes the decision whether to drop the status word and whether to generate an interrupt as soon as it receives the status word from the Data Mover. When generation of an interrupt is requested, and the corresponding RI or WI register does enable interrupts, the DMA Controller generates the interrupt. It does so by queuing an immediate write to the Write Data Mover's descriptor queue specified in the corresponding interrupt control register using the MSI address and message data provided in that register.You need to make sure that space is always available in the targeted Write Data Mover descriptor queue at any time when an interrupt may get generated. You can do so most easily by using the priority queue only for MSIs.

Setting the interrupt control bits in the immediate write descriptors that the DMA Controller creates to generate MSI interrupts to "No interrupt and drop status word" can avoid an infinite loop of interrupts.

To initiate a single DMA transfer, you only need to write a well-formed descriptor to one of the DMA Controller's descriptor queues (WDN, WDP, RDN or RDP).

To initiate a series of DMA transfers, you can prepare a table of descriptors padded to 512 bits each in a memory location accessible to the Read Data Mover. You can then write a single descriptor to the DMA Controller's priority descriptor queue (RDP) register to initiate the DMA transfers. These transfers move the descriptors from the source location in PCIe memory to the desired descriptor queue register.

To transmit an MSI interrupt upon completion of the processing of a descriptor, you must program the DMA Controller's WI or RI register with the desired MSI address and message before writing the descriptor.

This module creates read and write transactions to exercise the Bursting Slave module. You program it by writing to its control registers through its Control and Status Avalon^® -MM slave interface.

The traffic that it generates and expects is a sequence of incrementing dwords.

In the checking process, the first dword is accepted as-is, and the following dwords are checked to be incrementing.

The registers in the Traffic Generator are 32-bit wide.

The RAdd and WAdd registers contain the base addresses that the Traffic Generator reads from and writes to respectively. These are byte addresses and must be Avalon^® -MM word aligned (i.e. bits [5:0] are assumed to be set to 0).

Write to the RCnt and WCnt registers to specify the number of transfers to execute. Writing a 0 to these registers means non-stop transfers. Reading from one of these registers returns the number of transfers that have occurred since it was last read.

Reading the RErr register returns the number of errors detected since the register was last read. A maximum of one error is counted per clock cycle. Because the write error detection feature is not available yet, you cannot get a valid number of errors by reading the WErr register.

The RCtl and WCtl registers contain fields that define various aspects of the transfers, and start and stop the transfers.

All addresses are read or written sequentially. For example, if the transfer size is one byte and the first target byte address is N, the second transfer targets byte address N+1 (using the appropriate byte enable). If the transfer size is 48 dwords and the first target byte address is N, the second transfer targets byte address N + 192.

If the number of transfers multiplied by the transfer size is larger than the target area (or if the transfers are non-stop), the transfers loop back to the beginning of the target area when they reach its end. For transfer sizes that are not a power of two, the exact behavior at wraparound is not specified. The Traffic Checker detects one error every time the transfers loop back to the beginning of the target area.

The target_size parameter should not exceed the size of one address mapping window, or at least should not exceed the size of the address mapping table.

The Bursting Slave module transforms read and write transactions on its Avalon^® -MM interface into PCIe memory read (MRd) and memory write (MWr) request packets. The Bursting Slave uses the Avalon^® -MM address provided on its 64-bit wide address bus directly as the PCIe address in the TLPs that it creates.

The Bursting Slave, with its 64-bit address bus, uses up the whole Avalon^® -MM address space and prevents other slaves from being connected to the same bus. In many cases, the user application only needs to access a few relatively small regions of the PCIe address space, and would prefer to dedicate a smaller address space to the Bursting Slave to be able to connect to other slaves.

The example design includes an address mapping module that maps sixteen 1 MB regions of the Avalon^® -MM memory space into sixteen 1 MB regions of the PCIe address space. The module occupies only 16MB of the Avalon^® -MM address space, and only needs a 24-bit wide address bus, leaving space for other Avalon^® -MM slaves.

The Bursting Master module transforms PCIe memory read and write request packets received from the PCIe system into Avalon^® -MM read and write transactions. The offset from the matching BAR is provided as the Avalon^® -MM address, and the number of the matching BAR is provided in a conduit synchronously with the address.

Although these signals are in a conduit separate from the Avalon^® -MM master interface, they are synchronous to it and can be treated as extensions of the address bus.

The BAR Interpreter simply concatenates the BAR number to the address bus to form a wider address bus that Platform Designer can now treat as a normal address bus and route to the various slaves connected to the BAR Interpreter.

This testbench simulates up to x16 variants. However, the provided BFM only supports x1 to x8 links. It supports x16 variants by downtraining to x8. To simulate all lanes of a x16 variant, you can create a simulation model to use in a testbench along with a third party BFM such as the Avery BFM. For more information refer to AN-811: Using the Avery BFM for PCI Express Gen3x16 Simulation on Intel Stratix 10 Devices.

When configured as an Endpoint variation, the testbench instantiates a design example and a Root Port BFM, which provides the following functions:

• A configuration routine that sets up all the basic configuration registers in the Endpoint. This configuration allows the Endpoint application to be the target and initiator of PCI Express transactions.
• A Verilog HDL procedure interface to initiate PCI Express transactions to the Endpoint.

This testbench simulates a single Endpoint DUT.

The testbench uses a test driver module, altpcietb_bfm_rp_gen3_x8.sv, to exercise the target memory and DMA channel in the Endpoint BFM. The test driver module displays information from the Root Port Configuration Space registers, so that you can correlate to the parameters you specify using the parameter editor. The Endpoint model is the user-specified Endpoint variation.

Your Application Layer design may need to handle at least the following scenarios that are not possible to create with the Intel testbench and the Root Port BFM:

• It is unable to generate or receive Vendor Defined Messages. Some systems generate Vendor Defined Messages and the Application Layer must be designed to process them. The Hard IP block passes these messages on to the Application Layer, which in most cases should ignore them. The Avalon^® -MM bridge itself ignores Vendor Defined Messages.
• It can only handle received read requests that are less than or equal to the currently set Maximum Payload Size. You can set this parameter in the parameter editor by going to IP Settings > PCI Express/PCI Capabilities > Device > Maximum payload size. Many systems are capable of handling larger read requests that are then returned in multiple completions.
• It always returns a single completion for every read request. Some systems split completions on every 64-byte address boundary.
• It always returns completions in the same order the read requests were issued. Some systems generate the completions out-of-order.
• It is unable to generate zero-length read requests that some systems generate as flush requests following some write transactions. The Application Layer must be capable of generating the completions to the zero length read requests.
• It uses a fixed credit allocation.
• It does not support multi-function designs.

If your PIPE simulation cycles between the Detect.Quiet (6'h00), Detect.Active (6'h01), and Polling.Active (6'h02) LTSSM states, the PIPE interface width may be incorrect. The width of the DUT top-level PIPE interface is 32 bits for Intel^® Stratix^® 10 devices.

Typically, PCI Express hardware bring-up involves the following steps:

1. System reset
2. Link training
3. BIOS enumeration

The following sections describe how to debug the hardware bring-up flow. Intel recommends a systematic approach to diagnosing bring-up issues as illustrated in the following figure.

The Physical Layer automatically performs link training and initialization without software intervention. This is a well-defined process to configure and initialize the device's Physical Layer and link so that PCIe packets can be transmitted. If you encounter link training issues, viewing the actual data in hardware should help you determine the root cause. You can use the following tools to provide hardware visibility:

• Signal Tap Embedded Logic Analyzer
• Third-party PCIe protocol analyzer

You can use Signal Tap Embedded Logic Analyzer to diagnose the LTSSM state transitions that are occurring on the PIPE interface. The ltssmstate bus encodes the status of LTSSM. The LTSSM state machine reflects the Physical Layer’s progress through the link training process. For a complete description of the states these signals encode, refer to Reset, Status, and Link Training Signals. When link training completes successfully and the link is up, the LTSSM should remain stable in the L0 state. When link issues occur, you can monitor ltssmstate to determine the cause.

A third-party protocol analyzer for PCI Express records the traffic on the physical link and decodes traffic, saving you the trouble of translating the symbols yourself. A third-party protocol analyzer can show the two-way traffic at different levels for different requirements.

For high-level diagnostics, the analyzer shows the LTSSM flows for devices on both side of the link side-by-side. This display can help you see the link training handshake behavior and identify where the traffic gets stuck. A traffic analyzer can display the contents of packets so that you can verify the contents. For complete details, refer to the third-party documentation.

The following figure provides an overview of the debug capability available when you enable all of the options on the Configuration, Debug and Extension Option tab of the Intel L-/H-Tile Avalon-ST for PCI Express IP component GUI.

As this figure illustrates, the PCIe* Link (pli*) commands provide access to the following registers:

• The PCI Express* Configuration Space registers
• LTSSM monitor registers
• ATX PLL dynamic partial reconfiguration I/O (DPRIO) registers from the dynamic reconfiguration interface
• fPLL DPRIO registers from the dynamic reconfiguration interface
• Native PHY DPRIO registers from the dynamic reconfiguration interface

The NPDME commands (currently called adme*_) provide access to the following registers

• ATX PLL DPRIO registers from the ATX PLL NPDME interface
• fPLL DPRIO registers from the fPLL NPDME interface
• Native PHY DPRIO registers from the Native PHY NPDME interface

When you enable, the PCIe* Link Inspector, the altera_pcie_s10_hip_ast_pipen1b module of the generated IP includes the PCIe* Link Inspector as shown in the figure below.

You drive the PCIe* Link Inspector from a System Console running on a separate test PC. The System Console connects to the PCIe* Link Inspector via a Native PHY Debug Master Endpoint (NPDME). An Intel^® FPGA Download Cable makes this connection.

You can also access low-level link status information from the PCIe Hard IP, XCVR or PLL blocks via the Link Inspector Avalon^® -MM Interface by enabling the Enable PCIe Link Inspector Avalon^® -MM Interface option in the IP GUI. See the section Enabling the Link Inspector for more details. When you enable this option, you do not need to use the System Console. The pli_avmm_* ports that are exposed connect directly to the LTSSM Monitor without going through an NPDME block.

When you generate a PCIe* design example (with a PCIe* IP instantiated) without enabling the Link Inspector, the following interfaces are not exposed at the top level of the PCIe* IP:

• FPLL reconfiguration interface (reconfig_pll0)
• ATXPLL reconfiguration interface (reconfig_pll1)
• Transceiver reconfiguration interface (xcvr_reconfig)
• Hard IP reconfiguration interface (hip_reconfig)

If you later want to enable the Link Inspector using the same design, you need to provide a free-running clock and a reset to drive these interfaces at the top level of the PCIe* IP. Intel recommends that you generate a new design example with the Link Inspector enabled. When you do so, the design example will include a free-running clock and a reset for all reconfiguration interfaces.

You enable the PCIe* Link Inspector on the Configuration Debug and Extension Options tab of the parameter editor. You must also turn on the following parameters to use the PCIe* Link Inspector:

• Enable transceiver dynamic reconfiguration
• Enable dynamic reconfiguration of PCIe read-only registers
• Enable Native PHY, LCPLL, and fPLL ADME for Transceiver Toolkit

To have access to the low-level link status information such as the information from the LTSSM, XCVR, and PLLs using the PCIe* Link Inspector from the top level of the PCIe IP, you can enable the Enable PCIe Link Inspector AVMM Interface option. This allows you to extract the information from the pli_avmm_* ports for link-level debugging without JTAG access. This optional debug capability requires you to build custom logic to read and write data from the PCIe* Link Inspector.

Figure 33. Enable the Link Inspector in the Intel L-/H-Tile Avalon-MM+ for PCI Express IP

By default, all of these parameters are disabled.

For the design example generation, a JTAG-to- Avalon^® Bridge instantiation is connected to the exported pli_avmm_* ports, so that you can read all the link information via JTAG. The JTAG-to- Avalon^® Bridge instantiation is to verify the pli_avmm_* ports through JTAG. Without the design example generation, the JTAG-to- Avalon^® Bridge instantiation is not present.

You can use the command ltssm_file2console to display the contents of a previously saved file on the TCL system console window. For example, if you previously used the command ltssm_save2file to save all the LTSSM states and other status information into a file, you can use ltssm_file2console to display the saved contents of that file on the TCL system console window.

This is a new command that is added in the 18.1 release of Intel^® Quartus^® Prime.

If you use ltssm_display or ltssm_save2file to read the LTSSM states and other status information, all that information is cleared after the execution of these commands. However, you can use the ltssm_save_oldstates command to overcome this hardware FIFO read clear limitation.

The first time you use ltssm_save_oldstates, it saves all states to the file provided at the command line. If you use this command again, it appends new readings to the same file and displays on the TCL console all the LTSSM states saved to the file.

This is a new command that is added in the 18.1 release of Intel^® Quartus^® Prime.

The basic Root Port BFM provides a Verilog HDL task‑based interface to test the PCIe* link. The Root Port BFM also handles requests received from the PCIe* link. The following figure provides an overview of the Root Port BFM.

The following descriptions provides an overview of the blocks shown in the Root Port BFM figure:

• BFM shared memory (altpcietb_g3bfm_shmem.v): The BFM memory performs the following tasks:
  • • Stores data received with all completions from the PCI Express link.
    • Stores data received with all write transactions received from the PCI Express link.
    • Sources data for all completions in response to read transactions received from the PCI Express* link.
    • Sources data for most write transactions issued to the link. The only exception is certain BFM PCI Express* write procedures that have a four-byte field of write data passed in the call.
    • Stores a data structure that contains the sizes of and the values programmed in the BARs of the Endpoint.

A set of procedures read, write, fill, and check the shared memory from the BFM driver. For details on these procedures, see BFM Shared Memory Access Procedures.

• BFM Read/Write Request Functions (altpcietb_g3bfm_rdwr.v): These functions provide the basic BFM calls for PCI Express* read and write requests. For details on these procedures, refer to BFM Read and Write Procedures.
• BFM Configuration Functions (altpcietb_g3bfm_rp.v ): These functions provide the BFM calls to request configuration of the PCI Express* link and the Endpoint Configuration Space registers. For details on these procedures and functions, refer to BFM Configuration Procedures.
• BFM Log Interface (altpcietb_g3bfm_log.v): The BFM log functions provides routines for writing commonly formatted messages to the simulator standard output and optionally to a log file. It also provides controls that stop simulation on errors. For details on these procedures, refer to BFM Log and Message Procedures.
• BFM Request Interface (altpcietb_g3bfm_req_intf.v): This interface provides the low-level interface between the altpcietb_g3bfm_rdwr.v and altpcietb_g3bfm_configure.v procedures or functions and the Root Port RTL Model. This interface stores a write-protected data structure containing the sizes and the values programmed in the BAR registers of the Endpoint. This interface also stores other critical data used for internal BFM management. You do not need to access these files directly to adapt the testbench to test your Endpoint application.
• Avalon‑ST Interfaces (altpcietb_bfm_vc_intf_ast_common.v): These interface modules handle the Root Port interface model. They take requests from the BFM request interface and generate the required PCI Express* transactions. They handle completions received from the PCI Express* link and notify the BFM request interface when requests are complete. Additionally, they handle any requests received from the PCI Express* link, and store or fetch data from the shared memory before generating the required completions.

The ebfm_bar procedures in altpcietb_bfm_rdwr.v implement read and write transactions to the Endpoint Application Layer. The procedures and functions listed below are available in the Verilog HDL include file altpcietb_bfm_rdwr.v.

• ebfm_barwr: writes data from BFM shared memory to an offset from a specific Endpoint BAR. This procedure returns as soon as the request has been passed to the Virtual Channel (VC) interface module for transmission.
• ebfm_barwr_imm: writes a maximum of four bytes of immediate data (passed in a procedure call) to an offset from a specific Endpoint BAR. This procedure returns as soon as the request has been passed to the VC interface module for transmission.
• ebfm_barrd_wait: reads data from an offset of a specific Endpoint BAR and stores it in BFM shared memory. This procedure blocks waiting for the completion data to be returned before returning control to the caller.
• ebfm_barrd_nowt: reads data from an offset of a specific Endpoint BAR and stores it in the BFM shared memory. This procedure returns as soon as the request has been passed to the VC interface module for transmission, allowing subsequent reads to be issued in the interim.

These routines take as parameters a BAR number to access the memory space and the BFM shared memory address of the bar_table data structure set up by the ebfm_cfg_rp_ep procedure. (Refer to Configuration of Root Port and Endpoint.) Using these parameters simplifies the BFM test driver routines that access an offset from a specific BAR and eliminates calculating the addresses assigned to the specified BAR.

The Root Port BFM does not support accesses to Endpoint I/O space BARs.

Before you issue transactions to the Endpoint, you must configure the Root Port and Endpoint Configuration Space registers.

The ebfm_cfg_rp_ep procedure executes the following steps to initialize the Configuration Space:

1. Sets the Root Port Configuration Space to enable the Root Port to send transactions on the PCI Express link.
2. Sets the Root Port and Endpoint PCI Express Capability Device Control registers as follows:
   1. Disables Error Reporting in both the Root Port and Endpoint. The BFM does not have error handling capability.
   2. Enables Relaxed Ordering in both Root Port and Endpoint.
   3. Enables Extended Tags for the Endpoint if the Endpoint has that capability.
   4. Disables Phantom Functions, Aux Power PM, and No Snoop in both the Root Port and Endpoint.
   5. Sets the Max Payload Size to the value that the Endpoint supports because the Root Port supports the maximum payload size.
   6. Sets the Root Port Max Read Request Size to 4 KB because the example Endpoint design supports breaking the read into as many completions as necessary.
   7. Sets the Endpoint Max Read Request Size equal to the Max Payload Size because the Root Port does not support breaking the read request into multiple completions.
3. Assigns values to all the Endpoint BAR registers. The BAR addresses are assigned by the algorithm outlined below.
   1. I/O BARs are assigned smallest to largest starting just above the ending address of BFM shared memory in I/O space and continuing as needed throughout a full 32-bit I/O space.
   2. The 32-bit non-prefetchable memory BARs are assigned smallest to largest, starting just above the ending address of BFM shared memory in memory space and continuing as needed throughout a full 32-bit memory space.
   3. The value of the addr_map_4GB_limit input to the ebfm_cfg_rp_ep procedure controls the assignment of the 32-bit prefetchable and 64-bit prefetchable memory BARS. The default value of addr_map_4GB_limit is 0.

      If addr_map_4GB_limit is set to 0, then the ebfm_cfg_rp_ep procedure assigns the 32‑bit prefetchable memory BARs largest to smallest, starting at the top of 32-bit memory space and continuing as needed down to the ending address of the last 32-bit non-prefetchable BAR. When addr_map_4GB_limit is set to 0, the ebfm_cfg_rp_ep procedure also assigns the 64-bit prefetchable memory BARs smallest to largest starting at the 4 GB address assigning memory ascending above the 4 GB limit throughout the full 64-bit memory space.

      However, if addr_map_4GB_limit is set to 1, the address map is limited to 4 GB. The ebfm_cfg_rp_ep procedure assigns 32-bit and 64-bit prefetchable memory BARs largest to smallest, starting at the 4 GB address and continuing as needed down to the ending address of the last 32-bit non-prefetchable BAR.

   4. The above algorithm cannot always assign values to all BARs when there are a few very large (1 GB or greater) 32-bit BARs. Although assigning addresses to all BARs may be possible, a more complex algorithm would be required to effectively assign these addresses. However, such a configuration is unlikely to be useful in real systems. If the procedure is unable to assign the BARs, it displays an error message and stops the simulation.
4. Based on the above BAR assignments, the ebfm_cfg_rp_ep procedure assigns the Root Port Configuration Space address windows to encompass the valid BAR address ranges.
5. The ebfm_cfg_rp_ep procedure enables master transactions, memory address decoding, and I/O address decoding in the Endpoint PCIe* control register.

The ebfm_cfg_rp_ep procedure also sets up a bar_table data structure in BFM shared memory that lists the sizes and assigned addresses of all Endpoint BARs. This area of BFM shared memory is write-protected. Consequently, any application logic write accesses to this area cause a fatal simulation error.

BFM procedure calls to generate full PCIe* addresses for read and write requests to particular offsets from a BAR use this data structure. This procedure allows the testbench code that accesses the Endpoint application logic to use offsets from a BAR and avoid tracking specific addresses assigned to the BAR. The following table shows how to use those offsets.

The configuration routine does not configure any advanced PCI Express capabilities such as the AER capability.

Besides the ebfm_cfg_rp_ep procedure in altpcietb_bfm_rp_gen3_x8.sv, routines to read and write Endpoint Configuration Space registers directly are available in the Verilog HDL include file. After the ebfm_cfg_rp_ep procedure runs the PCI Express I/O and Memory Spaces have the layout shown in the following three figures. The memory space layout depends on the value of the addr_map_4GB_limit input parameter. The following figure shows the resulting memory space map when the addr_map_4GB_limit is 1.

The following figure shows the resulting memory space map when the addr_map_4GB_limit is 0.

The following figure shows the I/O address space.

Enumeration assigns the Root Port interface device number 0 on internal bus number 0. Use the ebfm_cfg_rp_ep to assign the Endpoint to any device number on any bus number (greater than 0). The specified bus number is the secondary bus in the Root Port Configuration Space.

The BFM shared memory is 2 MBs. The BFM shared memory maps to the first 2 MBs of I/O space and also the first 2 MBs of memory space. When the Endpoint application generates an I/O or memory transaction in this range, the BFM reads or writes the shared memory.

The BFM includes procedures, functions, and tasks to drive Endpoint application testing. It also includes procedures to run the chaining DMA design example.

The BFM read and write procedures read and write data to BFM shared memory, Endpoint BARs, and specified configuration registers. The procedures and functions are available in the Verilog HDL. These procedures and functions support issuing memory and configuration transactions on the PCI Express link.

The ebfm_barwr procedure writes a block of data from BFM shared memory to an offset from the specified Endpoint BAR. The length can be longer than the configured MAXIMUM_PAYLOAD_SIZE. The procedure breaks the request up into multiple transactions as needed. This routine returns as soon as the last transaction has been accepted by the VC interface module.