The P-Tile Avalon^® memory mapped IP for PCIe combines the functionality of previous Avalon^® memory-mapped (Avalon-MM) and Avalon memory-mapped with direct memory access (DMA) interfaces. The IP core using the Avalon-MM interface removes many of the complexities associated with the PCIe protocol. It handles all of the Transaction Layer Packet (TLP) encoding and decoding, simplifying the design task. It also includes optional Read and Write Data Mover modules facilitating the creation of high-performance DMA designs. Both the Avalon-MM interface and the Read and Write Data Mover modules are implemented in soft logic. This IP Core natively supports Endpoint and Root Port configurations with Gen3/Gen4 data rates and x4/x8/x16 link widths. Gen1/Gen2 data rates and x1/x2 link widths are supported via link down-training.

This IP provides support for an Avalon^® memory mapped interface with DMA and is designed to optimize the performance of large-size data transfers. If you want to achieve maximum performance with small-size transfers, Intel recommends the use of the P-Tile Avalon^® streaming IP for PCIe.

IP versions are the same as the Intel Quartus Prime Design Suite software versions up to v19.1. From Intel Quartus Prime Design Suite software version 19.2 or later, IPs have a new IP versioning scheme.

Intel verifies that the current version of the Intel^® Quartus^® Prime Pro Edition software compiles the previous version of each IP core, if this IP core was included in the previous release. Intel reports any exceptions to this verification in the Intel IP Release Notes or clarifies them in the Intel^® Quartus^® Prime Pro Edition IP Update tool. Intel does not verify compilation with IP core versions older than the previous release.

The following terms define device support levels for Intel^® FPGA IP cores:

• Advance support—the IP core is available for simulation and compilation for this device family. Timing models include initial engineering estimates of delays based on early post-layout information. The timing models are subject to change as silicon testing improves the correlation between the actual silicon and the timing models. You can use this IP core for system architecture and resource utilization studies, simulation, pinout, system latency assessments, basic timing assessments (pipeline budgeting), and I/O transfer strategy (data-path width, burst depth, I/O standards tradeoffs).
• Preliminary support—the IP core is verified with preliminary timing models for this device family. The IP core meets all functional requirements, but might still be undergoing timing analysis for the device family. It can be used in production designs with caution.
• Final support—the IP core is verified with final timing models for this device family. The IP core meets all functional and timing requirements for the device family and can be used in production designs.

The following table shows the recommended FPGA fabric speed grades for all the configurations that the Avalon^® -MM IP core supports.

The Avalon^® -MM variants include an Avalon^® -MM DMA bridge implemented in soft logic. It operates as a front end to the hardened protocol stack. The resource utilization table below shows results for the Simple DMA dynamically generated design example.

The results are for the current version of the Intel^® Quartus^® Prime Pro Edition software.

The following table shows the support levels of the Avalon-MM IP core and design example in Intel^® Stratix^® 10 DX devices.

The following table shows the support levels of the Avalon-MM IP core and design example in Intel^® Agilex™ devices.

The four cores in the IP can be configured to support the following topologies:

In Configuration Mode 0, only the x16 core is active, and it operates in Gen3 x16 mode or Gen4 x16 mode.

In Configuration Mode 1, the x16 core and x8 core are active, and they operate as two Gen3 x8 cores or two Gen4 x8 cores.

In Configuration Mode 2, all four cores (x16, x8, x4_0, x4_1) are active, and they operate as four Gen3 x4 cores or four Gen4 x4 cores.

In the first two modes, the P-Tile Avalon^® -MM IP functions as an Endpoint (EP). In Root Port mode, it functions as a Root Port (RP).

The Avalon^® -MM Bridge consists of five main modules: Read Data Mover (RDDM), Write Data Mover (WRDM), Bursting Avalon^® -MM Master (BAM), Bursting Avalon^® -MM Slave (BAS) and Control Register Access (CRA). These modules are shown in Figure 2 and described below. Depending on the mode of operation, different modules in the IP core are enabled.

Here is a block diagram of the P-Tile Avalon^® -MM Bridge showing the main modules:

• Bursting Master (BAM): This module converts memory read and write TLPs initiated by the remote link partner and received over the PCIe link into Avalon^® -MM burst read and write transactions, and sends back CplD TLPs for read requests it receives. It can also function in a non-bursting mode.
• Bursting Slave (BAS): This module converts Avalon^® -MM read and write transactions initiated by the application logic into PCIe memory read and write TLPs to be transmitted over the PCIe link. This module also processes the CplD TLPs received for the read requests it sent. It can also function in a non-bursting mode.
• Read Data Mover (RDDM): This module uses PCIe memory read TLPs and Avalon^® -MM write transactions to move large amounts of data from the system memory in the PCIe space to the FPGA memory in the Avalon^® -MM space. It fetches descriptors from the system memory through one of its two Avalon^® -ST sink interfaces. These descriptors define the data transfers to be executed. The RDDM also reports the status of these data transfers via its Avalon^® -ST source interface.
• Write Data Mover (WRDM): This module uses PCIe memory write TLPs and Avalon^® -MM read transactions to move large amounts of data from your application logic in the Avalon^® -MM space to the system memory in the PCIe space. The WRDM also supports immediate writes, which are enabled by a bit in the descriptors that the WRDM receives via one of its Avalon^® -ST descriptor sink interfaces. For more details on immediate writes, refer to Write Data Mover Avalon -ST Descriptor Sinks. Similar to the RDDM, the WRDM also has its own Avalon^® -ST source interface to report the status of its data transfers.
• Control Register Access (CRA) Avalon-MM Slave (Root Port only): This module is used in Root Port mode only to issue accesses to the Endpoint's configuration space registers. It supports a single transaction at a time. It converts single-cycle, 32-bit Avalon^® -MM read and write transactions into PCIe configuration read and write TLPs (CfgRd0, CfgRd1, CfgWr0 and CfgWr1) to be sent over the PCIe link. This module also processes the completion TLPs (Cpl and CplD) it receives in return.

The Response Reordering module assembles and reorders completion TLPs received over the PCIe link for the Bursting Slave and the Read Data Mover. It routes the completions based on their tags.

No re-ordering is necessary for the completions sent to the CRA module as it only issues one request TLP at a time.

Endpoint applications typically need the Bursting Master to enable the host to provide information for the other modules.

The PHY clock domain (i.e. core_clk domain) is a dynamic frequency domain. The PHY clock frequency is dependent on the current link speed.

P-Tile has two reference clock inputs at the package level, refclk0 and refclk1. You must connect a 100 MHz reference clock source to these two inputs. Depending on the port mode, you can drive the two refclk inputs using either a single clock source or two independent clock sources.

In 1x16 and 4x4 modes, drive the refclk inputs with a single clock source (through a fanout buffer) as shown in the figure below.

There is only one PERST# (pin_perst_n) pin on P-Tile. Therefore, toggling pin_perst_n will affect the entire P-Tile. If the P-Tile x16 port is bifurcated into two x8 Endpoints, toggling pin_perst_n will affect both x8 Endpoints.

To reset each port individually, use the in-band mechanism like Hot Reset.

The following is an example where a single PERST# (pin_perst_n) is driven with independent refclk0 and refclk1. In this example, the add-in card (FPGA and Soc) is powered up first. P-Tile refclk0 is fed by the on-board free-running oscillator. P-Tile refclk1 driven by the Host becomes stable later. Hence, the PERST# is connected to the Host.

The P-Tile Avalon^® -MM IP for PCI Express contains Physical Medium Attachment (PMA) and PCI Express Physical Coding Sublayer (PCIe PCS) blocks for handling the Physical layer (PHY) packets. The PMA receives and transmits high-speed serial data on the serial lanes. The PCS acts as an interface between the PMA and the PCIe controller, and performs functions like data encoding and decoding, scrambling and descrambling, block synchronization etc. The PCIe PCS in the P-Tile Avalon^® -MM IP for PCI Express is based on the PHY Interface for PCI Express (PIPE) Base Specification 4.4.1.

In this IP, the PMA consists of up to four quads. Each quad contains a pair of transmit PLLs and four SerDes lanes capable of running up to 16 GT/s to perform the various TX and RX functions.

PLLA generates the required transmit clocks for Gen1/Gen2 speeds, while PLLB generates the required clocks for Gen3/Gen4 speeds. For the x8 and x16 lane widths, one of the quads acts as the master PLL source to drive the clock inputs for each of the lanes in the other quads.

The PMA performs functions such as serialization/deserialization, clock data recovery, and analog front-end functions such as Continuous Time Linear Equalizer (CTLE), Decision Feedback Equalizer (DFE) and transmit equalization.

The transmitter consists of a 3-tap equalizer with one tap of pre-cursor, one tap of main cursor and one tap of post-cursor.

The PHY layer uses a fixed 16-bit PCS-PMA interface width to output the PHY clock (core_clk). The frequency of this clock is dependent on the current link speed. Refer to Table 11 for the frequencies at various link speeds.

The Data Link Layer (DLL) is located between the Transaction Layer and the Physical Layer. It maintains packet integrity and communicates (by DLL packet transmission) at the PCI Express link level.

The DLL implements the following functions:

• Link management through the reception and transmission of DLL Packets (DLLP), which are used for the following functions:
  • Power management of DLLP reception and transmission
  • To transmit and receive ACK/NAK packets
  • Data integrity through the generation and checking of CRCs for TLPs and DLLPs
  • TLP retransmission in case of NAK DLLP reception or replay timeout, using the retry (replay) buffer
  • Management of the retry buffer
  • Link retraining requests in case of error through the Link Training and Status State Machine (LTSSM) of the Physical Layer

The DLL has the following sub-blocks:

• Data Link Control and Management State Machine—This state machine connects to both the Physical Layer’s LTSSM state machine and the Transaction Layer. It initializes the link and flow control credits and reports status to the Transaction Layer.
• Power Management—This function handles the handshake to enter low power mode. Such a transition is based on register values in the Configuration Space and received Power Management (PM) DLLPs. For more details on the power states supported by the P-Tile Avalon^® -MM IP for PCIe, refer to section Power Management Interface.
• Data Link Layer Packet Generator and Checker—This block is associated with the DLLP’s 16-bit CRC and maintains the integrity of transmitted packets.
• Transaction Layer Packet Generator—This block generates transmit packets, including a sequence number and a 32-bit Link CRC (LCRC). The packets are also sent to the retry buffer for internal storage. In retry mode, the TLP generator receives the packets from the retry buffer and generates the CRC for the transmit packet.
• Retry Buffer—The retry buffer stores TLPs and retransmits all unacknowledged packets in the case of NAK DLLP reception. In case of ACK DLLP reception, the retry buffer discards all acknowledged packets.
• ACK/NAK Packets—The ACK/NAK block handles ACK/NAK DLLPs and generates the sequence number of transmitted packets.
• Transaction Layer Packet Checker—This block checks the integrity of the received TLP and generates a request for transmission of an ACK/NAK DLLP.
• TX Arbitration—This block arbitrates transactions, prioritizing in the following order:
  • Initialize FC Data Link Layer packet
  • ACK/NAK DLLP (high priority)
  • Update FC DLLP (high priority)
  • PM DLLP
  • Retry buffer TLP
  • TLP
  • Update FC DLLP (low priority)
  • ACK/NAK FC DLLP (low priority)

The following figure shows the major blocks in the P-Tile Avalon^® -MM IP for PCI Express Transaction Layer:

The RAS (Reliability, Availability, and Serviceability) block includes a set of features to maintain the integrity of the link.

For example: Transaction Layer inserts an optional ECRC in the transmit logic and checks it in the receive logic to provide End-to-End data protection.

When the application logic sets the TLP Digest (TD) bit in the Header of the TLP, the P-Tile Avalon^® -MM IP for PCIe will append the ECRC automatically.

The TX block sends out the TLPs that it receives as-is. It also sends the information about non-posted TLPs to the Completion (CPL) Timeout Block for CPL timeout detection.

The following figure shows how the DMA example design that you can generate using the Intel^® Quartus^® Prime software interfaces with the P-Tile Avalon^® -MM IP to perform DMA operations. If you are not using the provided DMA example design, you need to implement your custom DMA Controller and BAR Interpreter in your application logic.

In this DMA example design, the BAM is used in non-bursting mode by the host to program the Control and Status registers of the DMA controller in the user Avalon^® -MM space. The DMA controller, after being programmed, sends descriptor-fetching instructions to the host via the RDDM. After the fetched descriptors are processed by the WRDM and RDDM, status and/or MSI-X messages are sent to the host via the WRDM in “Immediate” mode. In this mode, the data payload is embedded in bits [31:0] or [63:0] of the fetched descriptors that the WRDM receives (depending on whether a one- or two-dword immediate transfer is needed respectively). For more details on immediate transfers, refer to Write Data Mover Avalon -ST Descriptor Sinks.

The RDDM uses PCIe memory read TLPs and Avalon^® -MM write transactions (which can be bursting transactions) to move large amounts of data from the host memory in PCIe space to the local FPGA memory in Avalon^® -MM space. On the other hand, the WRDM uses PCIe memory write TLPs and Avalon^® -MM read transactions to move large amounts of data from the FPGA memory in Avalon^® -MM space to the host memory in PCIe space. The Data Movers' transfers are controlled by descriptors that are provided to the Data Movers through one of their Avalon^® -ST sink interfaces. The Data Movers report the transfers’ status through their Avalon^® -ST source interfaces.

The external Avalon^® -MM master can be a custom DMA controller that uses the Bursting Slave in the IP core to send memory reads and writes upstream. These memory reads and writes can be up to 512-bytes long. The reordering buffer in the IP core reorders the Completion TLPs received over the PCIe link and sends them to the Bursting Slave.

The Bursting Master provides the host with access to the registers and memory in the Avalon^® -MM address space of the FPGA. It converts PCIe memory reads and writes to Avalon^® -MM reads and writes.

Registers in the custom DMA controller can be programmed by software via the Bursting Master port.

The IP core must be able to generate and process configuration reads and writes to the Endpoint and to the Hard IP configuration registers. This is done via the Configuration Slave. Since the DMA controller resides on the Endpoint side, its control registers need to be programmed by the FPGA local processor. Using the Bursting Slave (in non-bursting mode), the local processor can program the Endpoint control registers for DMA operations. The Endpoint can also send updates of its DMA status to the local processor via the Bursting Master.

This chapter provides a reference for all the parameters that are configurable in the Intel^® Quartus^® Prime IP Parameter Editor for the P-Tile Avalon^® -MM IP for PCIe.

The following figure shows how to enable Root Port mode:

Figure 13. Intel P-Tile Avalon^® -MM Top-Level IP Parameter Editor for a Gen3x4 Hard IP in Root Port Mode

Depending on which Hard IP Mode you choose in the Top-Level Settings tab, you will see different tabs for setting the core parameters.

Figure 14. Intel P-Tile Avalon^® -MM Top-Level IP Parameter Editor for a x8 Hard IP Mode If you choose a x8 mode (either Gen4 or Gen3), the PCIe0 Settings and PCIe1 Settings tabs will appear.

For each core (PCIe0/PCIe1/PCIe2/PCIe3), the PCI Express / PCI Capabilities tab contains separate tabs for the device, MSI (Endpoint mode), ACS capabilities (Root Port mode), slot (Root Port mode), MSI-X, and legacy interrupt pin register parameters.

Figure 15. PCI Express / PCI Capabilities Parameters

The following table lists the default values of the Device ID registers. You can use the parameter editor to change the values of these registers.

Read Data Mover (RDDM) interface: This interface transfers DMA data from the PCIe system memory to the memory in Avalon^® -MM address space.

Write Data Mover (WRDM) interface: This interface transfers DMA data from the memory in Avalon^® -MM address space to the PCIe system memory.

Bursting Master (BAM) interface: This interface provides host access to the registers and memory in Avalon^® -MM address space. The Busting Master module converts PCIe Memory Reads and Writes to Avalon^® -MM Reads and Writes.

Bursting Slave (BAS) interface: This interface allows the user application in the FPGA to access the PCIe system memory. The Bursting Slave module converts Avalon^® -MM Reads and Writes to PCIe Memory Reads and Writes.

Control Register Access (CRA) interface: This optional, 32-bit Avalon-MM Slave interface provides access to the Control and Status registers. You must enable this interface when you enable address mapping for any of the Avalon-MM slaves or if interrupts are implemented. The address bus width of this interface is fixed at 15 bits. The prefix for this interface is cra*.

The modular design of the P-Tile Avalon^® -MM IP for PCIe lets you enable just the interfaces required for your application.

The figures below provide the top-level block diagrams of the P-Tile Avalon^® -MM IP with all interfaces while operating in Endpoint mode with Data Movers or in Endpoint mode. These interfaces are described in more details in following sections.

These interfaces are standard Avalon^® interfaces. For timing diagrams, refer to the Avalon Interface Specifications.

The Bursting Avalon^® -MM Master module has one user-visible Avalon^® -MM Master interface.

You enable this interface by turning On the Enable Bursting Avalon^® -MM Master interface option in the Avalon^® -MM Settings tab of the IP Parameter Editor.

The Bursting Avalon^® -MM Slave module has one user-visible Avalon^® -MM slave interface.

You enable this interface by turning On the Enable Bursting Avalon^® -MM Slave interface option in the IP Parameter Editor.

For more details on these interface signals, refer to the Avalon^® Interface Specifications.

The Bursting Avalon^® -MM Slave module supports bursting mode while operating in Root Port mode.

This interface provides the Read data from the Host memory to the user application. The rddm_address_o value is set within the descriptor destination address.

The Read Data Mover has two Avalon^® -ST sink interfaces to receive the descriptors that define the data transfers to be executed. One of the interfaces receives descriptors for normal data transfers, while the other receives descriptors for high-priority data transfers.

The descriptor format for the Read Data Mover is described in the section Descriptor Format for Data Movers.

The Read Data Mover internally supports two queues of descriptors. The priority queue has absolute priority over the normal queue. Use it carefully to avoid starving the normal queue.

If the Read Data Mover receives a descriptor on the priority interface while processing a descriptor from the normal queue, it switches to processing descriptors from the priority queue as soon as it has completed the current descriptor. The Read Data Mover resumes processing the descriptors from the normal queue once the priority queue is empty. Do not use the same descriptor ID simultaneously in the two queues as there would be no way to distinguish them on the Status Avalon^® -ST source interface.

The Read Data Mover handles one descriptor at a time. When a descriptor has been processed (the memory command has been issued to the PCIe link), the Read Data Mover will read the next descriptor from the priority or normal descriptor interface.

Software should only send new descriptors when the Read Data Mover has processed all previously sent descriptors. The P-Tile Avalon^® -MM IP indicates the completion of the Read Data Mover's data processing by performing an immediate write to the system memory using its Write Data Mover. For more details, refer to the Read DMA Example section in the P-tile Avalon Memory Mapped (Avalon-MM) IP for PCI Express Design Example User Guide (see the link in the Related Information below).

This interface does not have a ready input. The application logic must always be ready to receive status information for any descriptor that it has sent to the Read Data Mover.

The Read Data Mover copies over the application specific bits in the rddm_tx_data_o bus from the corresponding descriptor. A set priority bit indicates that the descriptor is from the priority descriptor sink.

A status word is output on this interface when the processing of a descriptor has completed, including the reception of all completions for all memory read requests.

This interface reads data from the Avalon-MM Read Master interface and writes it to the Host memory.

The wrdm_address_o value is set within the descriptor source address.

The Write Data Mover has two Avalon^® -ST sink interfaces to receive the descriptors that define the data transfers to be executed. One of the interfaces receives descriptors for normal data transfers, while the other receives descriptors for high-priority data transfers.

The descriptor format for the Write Data Mover is described in the section Descriptor Formats for Data Movers.

The Write Data Mover internally supports two queues of descriptors. The priority queue has absolute priority over the normal queue, so it should be used carefully to avoid starving the normal queue.

If the Write Data Mover receives a descriptor on the priority interface while processing a descriptor from the normal queue, it switches to processing descriptors from the priority queue after it has completed processing the current descriptor. The Write Data Mover resumes processing descriptors from the normal queue once the priority queue is empty. Do not use the same descriptor ID simultaneously in the two queues as there would be no way to distinguish them on the Status Avalon^® -ST source interface.

The Write Data Mover handles one descriptor at a time. When a descriptor has been processed, the Write Data Mover will read the next descriptor from the priority or normal descriptor interface.

Software should only send new descriptors when the Write Data Mover has processed all previously sent descriptors. The Write Data Mover indicates the completion of the its data processing by performing an immediate write to the system memory using the last descriptor in the descriptor table. For more details, refer to the Write DMA Example section in the P-tile Avalon Memory Mapped (Avalon-MM) IP for PCI Express Design Example User Guide (see the link in the Related Information below).

This interface does not have a ready input. The application logic must always be ready to receive status information for any descriptor that it has sent to the Write Data Mover.

The ready latency does not matter because there is no ready input.

The Write Data Mover copies over the application specific bits in the wrdm_tx_data_o bus from the corresponding descriptor. A set priority bit indicates that the descriptor was from the priority descriptor sink.

The Read and Write Data Movers uses descriptors to transfer data. The descriptor format is fixed and specified below:

Application-Specific Bits

Three application-specific bits (bits [151:149] ) from the Write Data Mover and Read Data Mover Status Avalon-ST Source interfaces control when interrupts are generated.

The External DMA Controller makes the decision whether to drop the status word and whether to generate an interrupt as soon as it receives the status word from the Data Mover. When the generation of an interrupt is requested, and the corresponding RI or WI register does enable interrupts, the DMA Controller generates the interrupt. It does so by queuing an immediate write to the Write Data Mover's descriptor queue (specified in the corresponding interrupt control register) using the MSI address and message data provided in that register.

Avalon^® -MM DMA operations are used to transfer large blocks of data. The P-Tile Avalon^® -MM IP for PCIe can support DMA operations with an external descriptor controller implemented in the user application.

The following figure shows the Avalon^® -MM DMA Bridge when a custom external descriptor controller drives the Read and Write Data Movers.

This configuration includes the PCIe Read DMA and Write DMA Data Movers. The custom DMA descriptor controller must connect to the following Data Mover interfaces:

• PCIe Read Descriptor Sinks: These are two 174-bit, Avalon^® -ST sink interfaces (for normal and priority descriptors). The custom DMA descriptor controller drives read descriptor table entries on this bus. For more details on this interface, refer to Read Data Mover Avalon -ST Descriptor Sinks.
• PCIe Write Descriptor Sinks: These are two 174-bit, Avalon^® -ST sink interfaces (for normal and priority descriptors). The custom DMA descriptor controller drives write descriptor table entries on this bus. For more details on this interface, refer to Write Data Mover Avalon -ST Descriptor Sinks.
• PCIe Read Data Mover Status Source: The Read Data Mover reports status to the custom DMA descriptor controller on this interface. For more details on this interface, refer to Read Data Mover Status Avalon -ST Source.
• PCIe Write Data Mover Status Source: The Write Data Mover reports status to the custom DMA descriptor controller on this interface. For more details on this interface, refer to Write Data Mover Status Avalon -ST Source.

In Gen3 x4 and Gen4 x4 Root Port modes, the IP core uses the 256-bit Avalon^® -MM bridge instead of the 512-bit Avalon^® -MM bridge for the performance purpose. In Root Port mode, DMA functionalities are not available. The table below shows the interfaces for the P-tile 256-bit Avalon^® -MM bridge.

The High Performance Avalon^® -MM Slave has a 256-bit-wide data bus. It supports up to 16-cycle bursts with dword granularity byte enable on the first and last cycles of a write burst and for single-cycle read bursts. It also supports optional address mapping when the address bus is less than 64-bit wide.

This interface is optional. You enable it by turning On the Enable Bursting Slave option in the GUI.

The bursting Avalon^® -MM master is always enabled in Root Port mode and is not associated with any BAR. Packets targeting addresses outside of the range of the base and limit registers are forwarded to the host via the HPRXM master. The bursting Avalon^® -MM master has a 256-bit-wide data bus and supports up to 16-cycle bursts with dword granularity byte enable on the first and last cycles of a write burst and on single-cycle read bursts. Byte granularity access is supported for single-cycle one-dword or smaller transactions.

The P-Tile Avalon^® -MM IP for PCIe natively supports 4, 8, or 16 PCIe lanes. Each lane includes a TX differential pair and an RX differential pair. Data is striped across all available lanes.

This interface includes the signals that are useful for debugging, such as the link status signal, LTSSM state outputs, etc. These signals are available when the optional Power Management interface is enabled.

The P-Tile Avalon^® -MM IP for PCI Express* supports Message Signaled Interrupts (MSI), MSI-X interrupts, and legacy interrupts. MSI and legacy interrupts are mutually exclusive.

Legacy interrupts, MSI, and MSI-X interrupts are all controlled and generated externally to the Avalon^® -MM IP to ensure total flexibility of allocating interrupt resources based on the user’s application needs.

To support domain-isolation, legacy interrupt messages, MSI, and MSI-X TLPs need to be sent with the appropriate source IDs.

The following figure shows an example integrating an external interrupt controller with the P-Tile Avalon^® -MM IP. The interrupt controller takes interrupt requests from the external DMA controller as well as those from the user application.

If legacy interrupts are enabled at IP configuration time, the user’s interrupt controller generates legacy interrupts by asserting the intx_req_i input signal which causes the PCIe Hard IP to send the corresponding interrupt message. Use of legacy interrupts to signal the completion of DMA transfers is not recommended as their ordering with respect to the DMA traffic is not guaranteed.

If MSI or MSI-X are enabled at IP configuration time, the external interrupt controller can generate MSI/MSI-X transactions by issuing memory writes to the Bursting Slave or using the immediate write feature of the Write Data Mover, especially if signaling the completion of a DMA transfer by the Write Data Mover. The interrupt controller gets the address and data information to generate the MSI/MSI-X messages from the MSI or MSI-X capability registers in the Transaction Layer in the P-Tile IP.

MSI interrupts are signaled on the PCI Express link using a single dword Memory Write TLP. The user application issues an MSI request (MWr) through the Avalon^® -ST interface and updates the configuration space register using the MSI interface.

For more details on the MSI Capability Structure, refer to Figure 55.

The Mask Bits register and Pending Bits register are 32 bits in length each, with each potential interrupt message having its own mask bit and pending bit. If bit[0] of the Mask Bits register is set, interrupt message 0 is masked. When an interrupt message is masked, the MSI for that vector cannot be sent. If software clears the mask bit and the corresponding pending bit is set, the function must send the MSI request at that time.

You should obtain the necessary MSI information (such as the message address and data) from the configuration output interface (tl_cfg_*) to create the MWr TLP in the format shown below to be sent via the Avalon^® -ST interface.

The following figure shows the timings of msi_pnd_* signals in three scenarios. The first scenario shows the case when the MSI pending bits register is not used. The second scenario shows the case when only physical function 0 is enabled and the MSI pending bits register is used. The last scenario shows the case when four physical functions are enabled and the MSI pending bits register is used.

There are 32 possible MSI messages. The number of messages requested by a particular component does not necessarily correspond to the number of messages allocated. For example, in the following figure, the Endpoint requests eight MSIs but is only allocated two. In this case, you must design the Application Layer to use only two allocated messages.

The following table describes three example implementations. The first example allocates all 32 MSI messages. The second and third examples only allocate 4 interrupts.

MSI interrupts generated for Hot Plug, Power Management Events, and System Errors always use Traffic Class 0. MSI interrupts generated by the Application Layer can use any Traffic Class. For example, a DMA that generates an MSI at the end of a transmission can use the same traffic control as was used to transfer data.

The following figure illustrates a possible implementation of the Interrupt Handler Module with a per vector enable bit in the Application Layer. Alternatively, the Application Layer could implement a global interrupt enable instead of this per vector MSI.

The P-Tile Avalon^® -MM IP provides a Configuration Intercept Interface. User soft logic can monitor this interface to get MSI-X Enable and MSI-X function mask related information. User application logic needs to implement the MSI-X tables for all PFs and VFs at the memory space pointed to by the BARs as a part of your Application Layer.

For more details on the MSI-X related information that you can obtain from the Configuration Intercept Interface, refer to the MSI-X Registers section in the Registers chapter.

MSI-X is an optional feature that allows the user application to support large amount of vectors with independent message data and address for each vector.

When MSI-X is supported, you need to specify the size and the location (BARs and offsets) of the MSI-X table and PBA. MSI-X can support up to 2048 vectors per function versus 32 vectors per function for MSI.

A function is allowed to send MSI-X messages when MSI-X is enabled and the function is not masked. The application uses the Configuration Output Interface (address 0x0C bit[5:4]) or Configuration Intercept Interface to access this information.

When the application needs to generate an MSI-X, it will use the contents of the MSI-X Table (Address and Data) and generate a Memory Write through the Avalon^® -ST interface.

You can enable MSI-X interrupts by turning on the Enable MSI-X option under the PCI Express/PCI Capabilities tab in the parameter editor. If you turn on the Enable MSI-X option, you should implement the MSI-X table structures at the memory space pointed to by the BARs as a part of your Application Layer.

The MSI-X Capability Structure contains information about the MSI-X Table and PBA Structure. For example, it contains pointers to the bases of the MSI-X Table and PBA Structure, expressed as offsets from the addresses in the function's BARs. The Message Control register within the MSI-X Capability Structure also contains the MSI-X Enable bit, the Function Mask bit, and the size of the MSI-X Table. For a picture of the MSI-X Capability Structure, refer to Figure 57.

MSI-X interrupts are standard Memory Writes, therefore Memory Write ordering rules apply.

Example:

If the application needs to generate an MSI-X interrupt (vector 1), it will read the MSI-X Table information, generate a MWR TLP through the Avalon^® -ST interface and assert the corresponding PBA bits (bit[1]) in a similar fashion as for MSI generation.

The generated TLP will be sent to address 0x00000001_BBBB0000 and the data will be 0x00000002. When the MSI-X has been sent, the application can clear the associated PBA bits.

Hot Plug support means that the device can be added to or removed from a system during runtime. The Hot Plug Interface in the P-Tile Avalon^® -MM IP for PCIe allows an Intel FPGA with this IP to safely provide this capability.

This section describes the signals reported by the on-board hot plug components in the Downstream Port. This interface is available only if the Slot Status Register of the PCI Express Capability Structure is enabled.

Software programs the device into a D-state by writing to the Power Management Control and Status register in the PCI Power Management Capability Structure. The power management output signals indicate the current power state. The IP core supports the two mandatory power states: D0 (full power) and D3 (preparation for a loss of power). It does not support the optional D1 and D2 low-power states.

The correspondence between the device power states (D states) and link power states (L states) is as follows:

P-Tile does not support ASPM.

The Transaction Layer configuration output (tl_cfg) bus provides a subset of the information stored in the Configuration Space. Use this information in conjunction with the app_err* signals to understand TLP transmission problems.

The table below provides the tl_cfg_add_o[4:0] to tl_cfg_ctl_o[15:0] mapping.

The Hard IP reconfiguration interface is an Avalon^® -MM slave interface with a 21‑bit address and an 8‑bit data bus. It is also sometimes referred to as the User Avalon^® -MM Interface. You can use this interface to dynamically modify the value of configuration registers. Note that after a warm reset or cold reset, changes made to the configuration registers of the Hard IP via the Hard IP reconfiguration interface are lost as these registers revert back to their default values.

In Root Port mode, the application logic uses the Hard IP reconfiguration interface to access its PCIe configuration space to perform link control functions (such as Hot Reset, link disable, or link retrain).

Reading and Writing to the Hard IP Reconfiguration Interface

Reading from the Hard IP reconfiguration interface of the P-Tile Avalon^® -MM IP for PCI Express retrieves the current value at a specific address. Writing to the reconfiguration interface changes the data value at a specific address. Intel recommends that you perform read-modify-writes when writing to a register, because two or more features may share the same reconfiguration address.

Modifying the PCIe configuration registers directly affects the behavior of the PCIe device.

Figure 27. Timing Diagram to Perform Read and Write Operations Using the Hard IP Reconfiguration Interface

The User Avalon^® -MM interface provides access to the configuration registers and the IP core registers. This interface includes an 8-bit data bus and a 21-bit address bus (which contains the byte addresses).

The following diagram and table show the address offsets for physical function 0 (PF0), User Avalon^® -MM Port Configuration Register and Debug (DBI) Register.

Targeting PF Configuration Space Registers

User application needs to specify the offsets of the targeted PF registers.

For example, if the application wants to read the MSI Capability Register of PF0, it will issue a Read with address 0x0050 to target the MSI Capability Structure of PF0.

Targeting VSEC Registers

User application needs to program the VSEC field (0x104068 bit[0]) first. Then all accesses from the user Avalon^® -MM interface starting at offset 0xD00 will be translated to VSEC configuration space registers.

DEBUG_DBI_ADDR register is located at user Avalon^® -MM offsets 0x104204 to 0x104207 (corresponding to byte 0 to byte 3). For example, the d_done bit is bit 7 at byte address 0x104207.

DEBUG_DBI_DATA register is located at user Avalon^® -MM offsets 0x104200 to 0x104203 (corresponding to byte 0 to byte 3).

The PHY reconfiguration interface is an optional Avalon^® -MM slave interface with a 26‑bit address and an 8‑bit data bus. Use this bus to read the value of PHY registers. Refer to Table 62 for details on addresses and bit mappings for the PHY registers that you can access using this interface.

These signals are present when you turn on Enable PHY reconfiguration on the Top-Level Settings tab using the parameter editor.

Please note that the PHY reconfiguration interface is shared among all the PMA quads.

Reading from the PHY Reconfiguration Interface

Reading from the PHY reconfiguration interface of the P-Tile Avalon^® -MM IP for PCI Express retrieves the current value at a specific address.

The PCIe* controller IP contains a set of port bifurcation muxes to remap the four controller PIPE lane interfaces to the shared 16 PCIe* PHY lanes. The table below shows the relationship between PHY lanes and the port mapping.

As you bring up your PCI Express system, you may face issues related to FPGA configuration, link training, BIOS enumeration, data transfer, and so on. This chapter suggests some strategies to resolve the common issues that occur during bring-up.

You can additionally use the P-Tile Debug Toolkit to identify the issues.

The following sections describe the flow to debug link issues during the hardware bring-up. Intel recommends a systematic approach to diagnosing issues as illustrated in the following figure.

The Physical Layer automatically performs link training and initialization without software intervention. This is a well-defined process to configure and initialize the device's Physical Layer and link so that PCIe packets can be transmitted.

Flow Chart for Debugging Link Training Issues

Use the flow chart below to identify the potential cause of the issue seen during link training when using the P-Tile Avalon^® -MM IP for PCI Express.

Note: (*) Redo the equalization using the Link Equalization Request 8.0 GT/s bit of the Link Status 2 register for 8.0 GT/s or Link Equalization Request 16.0 GT/s bit of the 16.0 GT/s Status Register.

Use the following debug tools for debugging link training issues observed on the PCI Express link when using the P-tile Avalon^® -MM IP for PCI Express.

You can use utilities like lspci, setpci to obtain general information of the device like link speed, link width etc.

Example: To read the negotiated link speed for the P-Tile device in a system, you can use the following commands:

sudo lspci –s $bdf -vvv

-s refers to “slot” and is used with the bus/device/function number (bdf) information. Use this command if you know the bdf of the device in the system topology.

sudo lspci –d <1172>:$did -vvv

-d refers to device and is used with the device ID (vid:did). Use this command to search using the device ID.

Figure 36.  lspci Output

The LnkCap under Capabilities indicates the advertised link speed and width capabilities of the device. The LnkSta under Capabilities indicates the negotiated link speed and width of the device.

Using the SignalTapII Logic Analyzer, you can monitor the following top-level signals from the P-Tile Avalon^® -MM IP for PCI Express to confirm the failure symptom for any port type (Root port, Endpoint or TLP Bypass) and configuration (Gen4/Gen3).

Use the Hard IP reconfiguration interface and PHY reconfiguration interface on the P-Tile Avalon^® -MM IP for PCI Express to access additional registers (for example, receiver detection, lane reversal etc.).

Using the Hard IP Reconfiguration Interface

Refer to the section Hard IP Reconfiguration Interface for details on this interface and the associated address map.

The following table lists the address offsets and bit settings for the PHY status registers. Use the Hard IP Reconfiguration Interface to access these read-only registers.

Using the PHY Reconfiguration Interface

Refer to the section PHY Reconfiguration Interface for details on how to use this interface.

Follow the steps below to access registers in Table 62 using the PHY reconfiguration interface.

1. Enable the PHY reconfiguration interface using the IP Parameter Editor.
2. Set the Quad and address offset from which you want to read the status by performing a read operation from the address xcvr_reconfig_addr_i[25:0] using the PHY reconfiguration interface signals.
   • xcvr_reconfig_read_i = 1’b1
   • xcvr_reconfig_addr_i[25:0] = {5-bit Quad mapping, 21-bit address offset}. Refer to Table 62 for the address offset and bit mapping.
   • xcvr_reconfig_readdata_o[7:0] = Refer to Table 62 for the address offset and bit mapping.

There are many possible reasons causing the PCIe link to stop transmitting data. The PCI Express base specification defines three types of errors, outlined in the table below:

Use the debug tools mentioned in the next two sections for debugging link training issues observed on the PCI Express link when using the P-Tile Avalon^® -MM IP for PCI Express.

Each PCI Express compliant device must implement a basic level of error management and can optionally implement advanced error management. The PCI Express Advanced Error Reporting Capability is an optional Extended Capability that may be implemented by PCI Express device functions supporting advanced error control and reporting.

The P-Tile Avalon^® -MM IP for PCI Express implements both basic and advanced error reporting. Error handling for a Root Port is more complex than that of an Endpoint. In this P-Tile Avalon^® -MM IP for PCI Express, the AER capability is enabled by default.

Use the AER capability of the PCIe Hard IP to identify the type of error and the protocol stack layer in which the error may have occurred. Refer to the PCI Express Capability Structures section of the Configuration Space Registers appendix for the AER Extended Capability Structure and the associated registers.

Use the following debug tools for second-level debug of any issue observed on the PCI Express link when using P-Tile:

Using the Hard IP Reconfiguration Interface

• Refer to the section Hard IP Reconfiguration Interface for details on this interface and the address map.

Using the PHY Reconfiguration Interface

• Refer to the section PHY Reconfiguration Interface for details on this interface and the address map.

The P-Tile Debug Toolkit is a System Console-based tool for P-Tile that provides real-time control, monitoring and debugging of the PCIe links at the Physical, Data Link and Transaction layers.

The P-Tile Debug Toolkit allows you to:

• View protocol and link status of the PCIe links per port.
• View PLL and per-channel status of the PCIe links per port.
• Control the channel analog settings.
• View the receiver eye and measure the eye height and width.
• Indicate the presence of a re-timer connected between the link partners.

The following figure provides an overview of the P-Tile Debug Toolkit in the P-Tile Avalon^® -MM IP for PCI Express.

When you enable the P-Tile Debug Toolkit, the intel_pcie_ptile_avmm module of the generated IP includes the Debug Toolkit modules and related logic as shown in the figure above.

Drive the Debug Toolkit from a System Console. The System Console connects to the Debug Toolkit via an Native PHY Debug Master Endpoint (NPDME). Make this connection via an Intel FPGA Download Cable.

Provide a clock source (50 MHz - 125 MHz, 100 MHz recommended clock frequency) to drive the xcvr_reconfig_clk clock. Use the output of the Reset Release Intel FPGA IP to drive the ninit_done, which provides the reset signal to the NPDME module.

To enable the P-Tile Debug Toolkit in your design, enable the option Enable Debug Toolkit in the PCIe Configuration, Debug and Extension options tab of the Intel FPGA P-Tile Avalon^® -MM IP for PCI Express.

When using bifurcated ports, you can enable the Debug Toolkit for each bifurcated port by enabling the option Enable Debug Toolkit on each of the bifurcated ports.

Use the design example you compiled by following the Quick Start Guide to familiarize yourself with the P-Tile Debug Toolkit. Follow the steps in the Generating the Design Example and Compiling the Design Example to generate the SRAM Object File, (.sof) for this design example.

To use the P-Tile Debug Toolkit, download the .sof to the Intel Development Kit. Then, open the System Console and load the design to the System Console as well. Loading the .sof to the System Console allows the System Console to communicate with the design using NPDME. NPDME is a JTAG-based Avalon-MM master. It drives Avalon-MM slave interfaces in the PCIe design. When using NPDME, the Intel Quartus Prime software inserts the debug interconnect fabric to connect with JTAG.

The following sections describe the different tabs and features available in the Debug Toolkit.

A. Main View

The main view tab lists a summary of the transmitter and receiver settings per channel for the given instance of the PCIe IP.

B. Toolkit Parameters

The Toolkit parameters window has 2 sub-tabs.

B.1. P-Tile Information

This lists a summary of the P-Tile PCIe IP parameter settings in the PCIe IP Parameter Editor when the IP was generated, as read by the P-Tile Debug Toolkit when initialized.

When using bifurcated ports, you will see all the P-Tile information for each port for which the Debug Toolkit has been enabled.

All the information is read-only.

Use the Get P-tile Info button to read the settings.

Figure 39. Example of P-Tile Parameter Settings

B.2. PCIe Configuration Space

This lists a summary of the P-Tile PCIe configuration settings of the PCIe configuration space registers, as read by the P-Tile Debug Toolkit when initialized.

All the information is read-only.

Use the Read cfg space button to read the settings.

Figure 40. Example of P-Tile PCIe Configuration Settings

C. Channel Parameters

The channel parameters window allows you to monitor and control the transmitter and receiver settings for a given channel. It has the following 2 sub-windows.

C.1. TX Path

This tab allows you to monitor and control the transmitter settings for the channel selected. Use the TX Refresh button to read the settings, TX Apply Ch to apply the settings to the selected channel, and TX apply all to apply the settings to all channels.

Figure 41. Example of Transmitter Settings

C.1. RX Path

This tab allows you to monitor and control the receiver settings for the channel selected. Use the RX Refresh button to read the settings, RX Apply Ch to apply the settings to the selected channel, and RX apply all to apply the settings to all channels.

Figure 42. Example of Receiver Settings

Eye Viewer

The P-Tile Debug Toolkit supports running eye tests for Intel devices with P-Tile. The Eye Viewer tool allows you to set up and run eye tests, monitoring bit errors.

1. In the System Console Tools menu option, click on Eye View Tool.
   Figure 43. Opening the Eye Viewer

   
2. This will open a new tab Eye View Tool next to the Main View tab. Choose the instance and channel for which you want to run the eye view tests.
   Figure 44. Opening the Instance and Channel

   
3. Choose the eye vertical step setting from the drop-down menu. The eye view tool allows you to choose between vertical step sizes of 1, 2, 4, 8.
   Note: The time taken for the eye view tool to draw the eye varies with different vertical step sizes (8 results in a faster eye plot when compared to 1).
   Figure 45. Choosing the Step Size

   
4. The messages window displays information messages to indicate the eye view tool's progress.
   Figure 46. Eye View Tool Messages

   
5. Once the eye plot is complete, the eye height, eye width and eye diagram are displayed.
   Figure 47. Sample Eye Plot

   

To enable the Link Inspector, enable the option Enable Debug Toolkit in the PCIe Configuration, Debug and Extension Options tab. The PCIe Link Inspector is enabled by default if Enable Debug Toolkit is enabled.

Figure 48. Enabling the Link Inspector

The Link Inspector is found under the PCIe Link Inspector tab after opening the Debug Toolkit:

Figure 49. View of the Link Inspector

When the Dump LTSSM Sequence to Text File button is initially clicked, a text file (ltssm_sequence_dump_p*.txt) with the LTSSM information is created in the location from where the System Console window is opened. Depending on the PCIe topology, there can be up to four text files. Subsequent LTSSM sequence dumps will append to the respective files.

Figure 50. Example LTSSM Sequence Dump (Beginning)

Figure 51. Example LTSSM Sequence Dump (End)

Each LTSSM monitor has a FIFO storing the time values and captured LTSSM states. The FIFO is written when there is a state transition. When you want to dump the LTSSM sequence, a single read of the FIFO status of the respective core is performed. Depending on the empty status and how many entries are in the FIFO, successive reads are executed.

In addition to accessing the Endpoint's configuration space registers by sending Configuration Read/Write TLPs via the Avalon^® -ST interface, the application logic can also gain read access to these registers via the Configuration Output Interface (tl_cfg*). Furthermore, the Hard IP Reconfiguration Interface (a User Avalon^® -MM interface) also provides read/write access to these registers.

For signal timings on the User Avalon^® -MM interface, refer to the Avalon^® Interface Specifications document.

The table PCIe Configuration Space Registers describes the registers for each PF. To calculate the address for a particular register in a particular PF, add the offset for that PF from the table Configuration Space Offsets to the byte address for that register as given in the table PCIe Configuration Space Registers.

This document uses the following abbreviations when describing register accesses.

The Corresponding Section in PCIe Specification column in the tables in the Configuration Space Registers section lists the appropriate sections of the PCI Express Base Specification that describe these registers.

This read-only register returns the JTAG Silicon ID. Intel programming software uses this JTAG ID to ensure that is is using the correct SRM Object File (*.sof).

These registers are only good for Port 0 ( PCIe* Gen4 x16). They are blocked for the other Ports.

This register provides a user configurable device or board ID so that the user software can determine which .sof file to load into the device.

This register is only available for Port 0 ( PCIe* Gen4 x16). It is blocked for the other Ports.

This register provides up to eight I/O pins each for Application Layer Control and Status requirements. This feature supports Partial Reconfiguration of the FPGA fabric. Partial Reconfiguration only requires one input pin and one output pin. The other seven I/Os make this interface extensible.

This register reports the status of the internally checked errors that are uncorrectable. When these specific errors are enabled by the Uncorrectable Internal Error Mask register, they are forwarded as Uncorrectable Internal Errors.

This register controls which errors are forwarded as internal uncorrectable errors.

The Correctable Internal Error Status register reports the status of the internally checked errors that are correctable. When these specific errors are enabled by the Correctable Internal Error Mask register, they are forwarded as correctable internal errors. This register is for debug only. Only use this register to observe behavior, not to drive custom logic

This register controls which errors are forwarded as internal correctable errors.