AN 838: Interoperability between Intel Arria 10 NBASE-T Ethernet Solution with Aquantia* Ethernet PHY Reference Design

RSS

AN-838 | 2018.01.12

AN 838: Interoperability between Intel Arria 10 NBASE-T Ethernet Solution and Aquantia Ethernet PHY Reference Design

The reference design demonstrates packets transfer between the Intel^® Arria^® 10 NBASE-T Ethernet solution and the Aquantia Ethernet PHY.

The Intel^® Arria^® 10 NBASE-T Ethernet solution implements an Intel^® Arria^® 10 Low Latency Ethernet 10G MAC with 10G Universal Serial Media Independent Interface (USXGMII) configuration connected to the 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP core, while the Ethernet PHY is using the Aquantia AQR105 Ethernet PHY device. This reference design demonstrates packets transmission using 1, 2.5, 5, and 10 Gigabit per second (Gbps) speed rates.

Figure 1. Reference Design Block Diagram

Reference Design Features

The reference design demonstrates the following features:

Auto-negotiation procedure between Intel^® Arria^® 10 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP core and Aquantia AQR105 Ethernet PHY.
Ethernet packet transfer between Intel^® Arria^® 10 and Aquantia AQR105 Ethernet PHY.
Sequential random burst transfer with configurable number of packets, payload data type, and payload size for each burst.
Statistics for Intel^® Arria^® 10 Low Latency 10G MAC, traffic generator, and traffic monitor developed by Intel.
Throughput monitoring using internal tool.

Reference Design Components

The following table describe the components available in the reference design:

Table 1. Reference Design Components
Component	Description
10G USXGMII Design Example Components
Intel FPGA Low Latency Ethernet 10G MAC IP core	The Intel FPGA Low Latency Ethernet 10G MAC IP core with the following configuration: Speed: 1G/2.5G/5G/10G (USXGMII) Datapath options: TX & RX Enable ECC on memory blocks: Not selected Enable supplementary address: Selected Enable statistics collection: Selected Statistics counters: Memory-based Use legacy XGMII Interface: Not selected Use legacy Avalon Memory-Mapped Interface: Not selected Use legacy Avalon Streaming Interface: Selected
PHY	The 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP with USXGMII variant.
Channel address decoder	Decodes the addresses of the components in each Ethernet channel.
Multi-channel address decoder	Decodes the addresses of the components used by all channels, such as the Master ToD module.
Top address decoder	Decodes the addresses of the top-level components, such as the Traffic Controller.
Transceiver Reset Controller	The Intel FPGA Transceiver PHY Reset Controller IP core. Resets the transceiver.
ATX PLL	Generates a TX serial clock for the Intel^® Arria^® 10 transceiver.
fPLL	Generates clocks for all design components
Traffic controller	The traffic controller consists of: Traffic generator: generate burst packets to the MAC for transmission. Traffic monitor: receive burst packets from MAC.
JTAG to Avalon Master Bridge	This IP core provides a connection between the System Console and Platform Designer (Standard) through a physical interface. The System Console initiates Avalon^® Memory Mapped transactions by sending encoded streams of bytes through the bridge's physical interface.
Aquantia Ethernet PHY components
Aquantia AQR105 Ethernet PHY	Ethernet PHY device on the Aquantia 28nm AQrate* ARQ105 evaluation board.

Getting Started Intel Arria 10 Multi Speed Ethernet Solution and Aquantia Ethernet PHY Reference Design

Hardware Requirements

The reference design requires the following hardware tools:

Two Intel^® Arria^® 10 GX Transceiver Signal Integrity (SI) development kits
Two Aquantia 28nm AQrate* ARQ105 evaluation boards
Two 10 GbE SFP+ direct attach copper cables
One CAT 6A Ethernet cable

Software Requirements

The reference design requires the following software tools:

Intel^® Quartus^® Prime Standard Edition version 17.1
Future Technology Device International (FTDI) driver
Aquantia USXGMII firmware ¹
Aquantia Ethernet PHY GUI ¹

¹ Download all Aquantia related firmware and software from the Aquantia Customer Portal in the related links.

Generating the 10G USXGMII Ethernet Example Design

Procedure

Figure 2. Example Design Tab

Select Tools > IP Catalog to open the IP Catalog and select Low Latency Ethernet 10G MAC.
The IP parameter editor appears.
Specify a top-level name and the folder for your custom IP variation, and the target device. Click OK.
To generate a design example, select a design example preset from the Presets library and click Apply. When you select a design, the system automatically populates the IP parameters for the design.
The Parameter Editor automatically sets the parameters required to generate the design example. Do not change the preset parameters in the IP tab.
Specify the parameters in the Example Design tab.
Click the Generate Example Design button.

The software generates all design files in sub-directories. You require these files to run simulation, compilation, and hardware testing.

Design Example Parameters

Table 2. Parameters in the Example Design Tab
Parameter	Description
Select Design	Available example designs for the IP parameter settings. When you select an example design from the Preset library, this field shows the selected design.
Example Design Files for Simulation or Synthesis	The files to generate for the different development phase. Simulation—generates the necessary files for simulating the example design. Synthesis—generates the synthesis files. Use these files to compile the design in the Intel^® Quartus^® Prime software for hardware testing and perform static timing analysis.
Generate File Format	The format of the RTL files for simulation—Verilog or VHDL.
Select Board	Supported hardware for design implementation. When you select an Intel FPGA development board, the `Target Device` is the one that matches the device on the Development Kit. If this menu is grayed out, there is no supported board for the options that you select. Intel^® Arria^® 10 GX Transceiver Signal Integrity Development Kit: This option allows you to test the design example on selected Intel FPGA IP development kit. This selection automatically selects the `Target Device` to match the device on the Intel FPGA IP development kit. If your board revision has a different device grade, you can change the target device. Custom Development Kit: This option allows you to test the design example on a third party development kit with Intel FPGA IP device, a custom designed board with Intel FPGA IP device, or a standard Intel FPGA IP development kit not available for selection. You can also select a custom device for the custom development kit. No Development Kit: This option excludes the hardware aspects for the design example.
Change Target Device	Select this parameter to display and select all devices for the Intel FPGA IP development kit.
Specify Number of Channels	The number of Ethernet channels.
Enable ADME support	Turn on this option to enable Transceiver ADME feature.
Partial Reconfiguration Ready	When this option is enabled, the generated hierarchy of the design example is compliance with the partial reconfiguration flow, where there is clear separation between hard IP and soft IP, without any functionality changes. Hard IPs such as Native PHY, JTAG, transmitter PLL, and FPLL are instantiated at the top-level wrapper of design example.

Reference Design Walk Through

Setting Up Aquantia AQR105 Evaluation Board

To set up the Aquantia 28nm AQrate* AQR105 evaluation board, refer to Aquantia Customer Portal in the related links.

Setting Up Intel Arria 10 GX Transceiver SI Development Kit

Follow the steps to setup the Intel^® Arria^® 10 GX transceiver SI development kit.

Generate the Intel^® Arria^® 10 10G USXGMII Ethernet design example using Intel^® Quartus^® Prime Standard Edition version 17.1. Refer to the Intel FPGA Low Latency Ethernet 10G MAC Design Example User Guide for Intel^® Arria^® 10 Devices in the related links.
Connect Intel^® FPGA Download Cable from both development kits to your host.
Launch the Clock Control (ClockController.exe) tool from the Intel^® Arria^® 10 GX Transceiver Signal Integrity Installation Package. The tool is available in arria10GX_10ax115sf45_si_v15.1\examples\board_test_system\ directory.
If you have two Intel^® FPGA Download Cables connected to a single host, the following prompt pops up when you launch the Clock Control tool. Select a cable number to set the clocks for each development kit.
Figure 3. Intel^® FPGA Download Cable Selection
Close the Intel^® FPGA Download Cable selection prompt.
Set Y5 to 644.53125 MHz and Y6 to 125 MHz to the Intel^® Arria^® 10 GX Transceiver SI development kit.

Figure 4. Clock Controller Setting
Repeat step 3 to set the clocks for the subsequent Intel^® Arria^® 10 GX Transceiver SI development kit.
In Intel^® Quartus^® Prime software, open altera_eth_top.qpf file.
Click Processing > Start Compilation to compile the design example.
Configure the FPGA using the generated configuration file, altera_eth_top.sof.
This reference design uses the same host to control both Intel^® Arria^® 10 GX Transceiver SI development kits. Duplicate the system console folder from LL10G_10G_USXGMII/hwtesting directory and rename the folders to identify system console for each development kit. This reference design uses system_console_pod13_A and system_console_pod13_B folder names as an example.
Open basic.tcl file from LL10G_10G_USXGMII/hwtesting/system_console_pod13_B/basic and set the System Console path from the system_console_pod13_B to 1 using the following command:
set port_id [lindex [get_service_paths master] 1];
In the Intel^® Quartus^® Prime software, launch the System Console tool from Tools > System Debugging Tools > System Console.
In the System Console, change the working directory to LL10G_10G_USXGMII/hwtesting/system_console_pod13_A.
Initialize the design command list using the following command:
source main.tcl
Initialize the Ethernet channel and enable auto-negotiation test using the following command:
```
SET_CHANNEL_BASE_ADDR 1
SETPHY_USXGMII_AN
```
Repeat step 12 to launch a new system console.
In the System Console, change the working directory to LL10G_10G_USXGMII/hwtesting/system_console_pod13_B.
Repeat step 14 and 15 to initialize the Ethernet channel, enable auto-negotiation, and the design command list.
The Aquantia evaluation board shows auto-negotiation status as completed and system is connected once auto-negotiation between Intel 1G/2.5G/5G/10G Multi-rate Ethernet PHY and Aquantia AQR105 Ethernet PHY has completed successfully.

Figure 5. Aquantia Ethernet PHY GUI

Running Basic Packet Transfer

Follow these steps to run basic packet transfer using the reference design once the links between Intel 1G/2.5G/5G/10G Multi-rate Ethernet PHY and Aquantia AQR105 Ethernet PHY are successfully connected:

Depending on which Intel^® Arria^® 10 GX Transceiver SI development kit you would like the loop back mechanism to be, enable the Avalon^® ST interface loop back in the System Console using the following command:
```
SET_TRAFFIC_CONTROLLER_STD_CHANNEL_BASE_ADDR 1
SET_AVALON_ST_LOOPBACK_ENA
```
In the System Console from the Intel^® Arria^® 10 GX Transceiver SI development kit without the Avalon^® ST interface loop back, use the following command to start packets transfer. This reference design connects channel 1 of the transceiver to the SFP+ interface.
TEST_EXT_LB <channel> <speed> <burst_size>.

Example: TEST_EXT_LB 1 10G 80000000.

The following diagram shows an example of successful packets transfer during basic packet transfer test.

Figure 6. Example of Basic Transfer Test Result

Changing Speed between 1 Gbps to 10Gbps

Follow these steps to change the transfer speed for the reference design. The following table shows the reference design supported speeds.

Table 3. Speed Mode Supported by the Reference Design
Speed Mode set in Aquantia AQR105 Ethernet PHY	Speed Mode set in Intel^® Arria^® 10 USXGMII Ethernet PHY	Speed set after Auto-negotiation Completed
1 Gb/2.5 Gb/5 Gb/10 Gb	1 Gbps/2.5 Gbps/5 Gbps/10 Gbps	10 Gbps
	1 Gbps/2.5 Gbps/5 Gbps	5 Gbps
	1 Gbps/2.5 Gbps	2.5 Gbps
	1 Gbps	1 Gbps
1 Gb/2.5 Gb/5 Gb	1 Gbps/2.5 Gbps/5 Gbps/10 Gbps	5 Gbps
	1 Gbps/2.5 Gbps/5 Gbps	5 Gbps
	1 Gbps/2.5 Gbps	2.5 Gbps
	1 Gbps	1 Gbps
1 Gb/2.5 Gb	1 Gbps/2.5 Gbps/5 Gbps/10 Gbps	2.5 Gbps
	1 Gbps/2.5 Gbps/5 Gbps
	1 Gbps/2.5 Gbps
	1 Gbps	1 Gbps
1 Gb	1 Gbps/2.5 Gbps/5 Gbps/10 Gbps	1 Gbps
	1 Gbps/2.5 Gbps/5 Gbps
	1 Gbps/2.5 Gbps
	1 Gbps

Important: Ensure all transfers have completed before starting these procedure.

To change speed on the Aquantia AQR105 Ethernet PHY, turn off the current selected speed and enable the desired new speed in the Aquantia GUI.
During auto-negotiation process, both links advertise the supported speed for each links. Connection is established when both links are operating in the same speed.
Click Restart Autonegotiation in the Aquantia GUI. Configure the same speed on both the Aquantia evaluation board to have the same speed transfer for the reference design.
Restart auto-negotiation on both Intel 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP cores using the following command in System Console:
```
SET_CHANNEL_BASE_ADDR 1
SETPHY_USXGMII_AN_RESTART
```
Use the following command to display the Intel 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP cores register settings:
CHKPHY_STATUS

Figure 7. Successful Auto-Negotiation Status for Intel 1G/2.5G/5G/10G Multi-rate Ethernet PHYRefer to USXGMII Ethernet PHY Configuration and Status Registers for register description.
Repeat the steps in Running Basic Packet Transfer to ensure the packet transmission is successfully running in the configured speed.

USXGMII Ethernet PHY Configuration and Status Registers

Table 4. 10G USXGMII Ethernet PHY Configuration and Status Registers Description
Register Name	Address	Description	Access	HW Reset Value
`usxgmii_control`	0x400	Bit [0]: `USXGMII_ENA:` 0 : 10GBASE-R mode 1 : USXGMII mode	RW	0x0
		Bit [1]: `USXGMII_AN_ENA` is used when `USXGMII_ENA` is set to 1: 0: Disables USXGMII Auto-Negotiation and manually configures the operating speed with the `USXGMII_SPEED` register. 1: Enables USXGMII Auto-Negotiation, and automatically configures operating speed with link partner ability advertised during USXGMII Auto-Negotiation.	RW	0x1
		Bit [4:2]: `USXGMII_SPEED` is the operating speed of the PHY in USXGMII mode and `USE_USXGMII_AN` is set to 0. 3’b000: Reserved 3’b001: Reserved 3’b010: 1G 3’b011: 10G 3’b100: 2.5G 3’b101: 5G 3’b110: Reserved 3’b111: Reserved	RW	0x0
		Bit [8:5]: Reserved	—	—
		Bit [9]: RESTART_AUTO_NEGOTIATION Write 1 to restart Auto-Negotiation sequence The bit is cleared by hardware when Auto-Negotiation is restarted	RWC (hardware self-clear)	0x0
		Bit [15:10]: Reserved	—	—
		Bit [30:16]: Reserved	—	—
`usxgmii_status`	0x401	Bit [1:0]: Reserved	—	—
		Bit [2]: `LINK_STATUS` indicates link status for USXGMII all speeds 1: Link is established 0: Link synchronization is lost, a 0 is latched	RO	0x0
		Bit [3]: Reserved	—	—
		Bit [4]: Reserved	—	—
		Bit [5]: `AUTO_NEGOTIATION_COMPLETE` A value of 1 indicates the Auto-Negotiation process is completed.	RO	0x0
		Bit [15:6]: Reserved	—	—
		Bit [31:16]: Reserved	—	—
Reserved	0x402:0x404	—	—	—
`usxgmii_partner_ability`	0x405	Bit [0]: Reserved	—	—
		Bit [6:1]: Reserved	—	—
		Bit [7]: `EEE_CLOCK_STOP_CAPABILITY` Indicates whether or not energy efficient ethernet (EEE) clock stop is supported. 0: Not supported 1: Supported	RO	0x0
		Bit [8]: `EEE_CAPABILITY` Indicates whether or not EEE is supported. 0: Not supported 1: Supported	RO	0x0
		Bit [11:9]: `SPEED` 3'b000: 10M 3'b001: 100M 3'b010: 1G 3'b011: 10G 3'b100: 2.5G 3'b101: 5G 3'b110: Reserved 3'b111: Reserved	RO	0x0
		Bit [12]: `DUPLEX` Indicates the duplex mode. 0: Half duplex 1: Full duplex	RO	0x0
		Bit [13]: Reserved	—	—
		Bit [14]: `ACKNOWLEDGE` A value of 1 indicates that the device has received three consecutive matching ability values from its link partner.	RO	0x0
		Bit [15]: `LINK` Indicates the link status. 0: Link down 1: Link up	RO	0x0
		Bit [31:16]: Reserved	RO	—
Reserved	0x406:0x411	—	—	—
`usxgmii_link_timer`	0x412	Auto-Negotiation link timer. Sets the link timer value in bit [19:14] from 0 to 2 ms in approximately 0.05-ms steps. You must program the link timer to ensure that it matches the link timer value of the external NBASE-T PHY IP Core. The reset value sets the link timer to approximately 1.6 ms. Bits [13:0] are reserved and always set to 0.	[19:14]: RW [13:0]: RO	[19:14]: 0x1F [13:0]: 0x0
Reserved	0x413:0x41F	—	—	—
`phy_serial_loopback`	0x461	Bit [0] 0: Disables the PHY serial loopback 1: Enables the PHY serial loopback	RW	0x0
		Bit [15:1]: Reserved	—	—
		Bit [31:16]: Reserved	—	—

Debugging the Reference Design

To debug problems encountered during system bring up, you may change the reference design connection to one of the following setups:

Loopback mode on Intel^® Arria^® 10 GX Transceiver SI development kit using SFP+ loopback adapter module. This setup removes Aquantia 28nm AQrate* ARQ105 evaluation board in the design.
Loopback modes on Aquantia AQR105 Ethernet PHY. This setup removes Intel^® Arria^® 10 NBASE-T Ethernet solution in the design.

Setting Up Loopback Mode for Intel Arria 10 GX Transceiver SI Development Kit

Follow these steps to setup loopback mode on Intel^® Arria^® 10 GX Transceiver SI Development Kit.

Figure 8. Loopback Mode Setup for Intel^® Arria^® 10 GX Transceiver SI Development Kit

Remove the Aquantia's 28nm AQrate* ARQ105 evaluation board from the Intel^® Arria^® 10 GX Transceiver SI Development Kit and connect the SFP+ loopback adapter module to the SFP+ connector on both the development kits.
Verify the Clock Control on both the development kit are set to the following values:
- Y5 is set to 644.53125 MHz
- Y6 is set to 125 MHz
Set correct port ID for the System Console, if you are using the same host for both Intel^® Arria^® 10 GX Transceiver SI Development Kit. Refer to Setting Up Intel Arria 10 GX Transceiver SI Development Kit to setup the System Console's port ID.
Example:
- System Console from LL10G_10G_USXGMII/hwtesting/system_console_pod13_A is set to 0.
- System Console from LL10G_10G_USXGMII/hwtesting/system_console_pod13_B is set to 1.
Run the basic packet transmission test using the following command and observe the transmission result for error packets.
TEST_EXT_LB <channel> <speed> <burst_size>

Setting Aquantia AQR105 Ethernet PHY to Loopback Mode

The Aquantia AQR105 Ethernet PHY provides network and system loopback modes for system debug and diagnostic. You may refer to the Aquantia AQR105 Single-Port AQrate* Ethernet PHY Transceiver datasheet in Aquantia Customer Portal to setup loopback mode in AQR105 devices.

Document Revision History for AN 838: Interoperability between Intel Arria 10 NBASE-T Ethernet Solution and Aquantia Ethernet PHY Reference Design

Date	Version	Changes
January 2018	2018.01.12	Initial release.