sfp port support

[jepio]

I'm using a clearfog board (rev 1.2, cex7 rev 1.5) with u-boot and the LSDK-20.04 release. I have plugged an SFP+ transceiver into the upper left slot of the SFP+ cage and created a dpni interface:

ls-addni dpmac.9

This gives me an eth1 interface, but I can't bring the link up and /dev/dpaa2_mc_console reports "internal link is down" on dpmac.9. Is there something I am missing? Is this expected to work or are there any preliminary patches that are needed to make this work. If it requires patches, i'd be happy to test them.

[rabeeh]

Which SFP+ module are you using?
Can you test with direct-attach cable (DAC)? Maybe there are some I2C commands you need to issue for the module to start initiate a link-up?

Notice that as of today; all the lx2160a_build that uses NXP releases do not have CONFIG_SFP enabled; which means that all the configurations that are needed for SFP+ module to start operation must be done manually (or use DAC cable that there is no configuration).
If you figure out how to init your SFP+ module I would be happy to add it to the instruction list.

As for the future; I can check with Russell King and mainline support when CONFIG_SFP support is ready
in detecting the inserted module and modifying the DPAA2 drivers accordingly.

[jepio]

I was testing using an Intel X520 adapter, I think it only supports Intel SFP+ modules. I'll check the exact module tomorrow. To be honest it's my first time dealing with 10G ethernet, so I still need to learn how it works.

Notice that as of today; all the lx2160a_build that uses NXP releases do not have CONFIG_SFP enabled; which means that all the configurations that are needed for SFP+ module to start operation must be done manually (or use DAC cable that there is no configuration).

I found Russels work, took his device tree (http://git.armlinux.org.uk/cgit/linux-arm.git/tree/arch/arm64/boot/dts/freescale/fsl-lx2160a-clearfog-cx.dts?h=cex7), and re-built the NXP provided kernel with CONFIG_SFP. That was my naive attempt to get it working.

I'll try to find a DAC cable to test. If I find out how to init the SFP+ module, i'll submit instructions.

[NadavLevi]

@jepio Hey there, have you succeed with the SFP problem? Did you used Optical cable or DAC?

[jepio]

I finally found a DAC cable lying around - a short Dell one (https://1000servers.com/dell-sfp-to-sfp-10gbe-twinax-dac-cable-1m-3-3f-ft-p-n-v250m-0v250m-new.html) - and performed a quick loopback test through the upper 2 ports of the 10G cage. dpmac.7 and dpmac.9. Things work like a charm.

I'm still searching for the initialization required for SFP+ transceivers to work - I'll try Russels kernel when I get the chance.

[NadavLevi]

I finally found a DAC cable lying around - a short Dell one (https://1000servers.com/dell-sfp-to-sfp-10gbe-twinax-dac-cable-1m-3-3f-ft-p-n-v250m-0v250m-new.html) - and performed a quick loopback test through the upper 2 ports of the 10G cage. dpmac.7 and dpmac.9. Things work like a charm.

I'm still searching for the initialization required for SFP+ transceivers to work - I'll try Russels kernel when I get the chance.

@jepio
Which SFP+ have you tried to use? Was it single mode or multimode? If you have the serial number of the SFP+ i would like to have it.

[eqvinox]

Hi all!

since this is the top result when googling lx2160a sfp, I'll add my own investigation & results here.

0. DAC cables — most seem to work. Did not check further.

1. FTLX8571D3BCV. Does not work as-is due to incorrect rate select.

   This module is a dual 1G/10G SFP+ transceiver and uses the RS0 pin to switch between 1G and 10G. (reference INF-8074 page 21:

   pin 7 - Rate Select - Note 4:
   Low or Open – reduced bandwidth,
   High– full bandwidth

   The Honeycomb LX2k has these pins open, so the selected rate is low=1G. This can be changed by writing the bit over I₂C on the SFF-8472 interface (A2h reg 6Eh bit 3) but this is not handled anywhere.

   This issue affects all Finisar FTLX………V SFPs (this includes a lot of branded modules, e.g. some Intel, Cisco, Juniper, etc.) Probably other multi-rate SFPs too.

   NB: the schematic shows RS0 pin connections to resistors, but they are DNP (R492/R489/R485/R113). Also, only 3 of the 4 slots show the connection in the schematic (note AB_RS0, BB_RS0 and BT_RS0 are connected but AT_RS0 is not!)

2. FTLX8571D3BCL and FTLX1471D3BNL. These work but lose link after a while (ca. 30min).

   I'm honestly rather confused how this could happen. Link comes up, but after a while it goes down in only one direction. The switch → LX2k link is UP, but the LX2k → switch is DOWN. Seems to happen after 15min — 60min. Before this happens, it works correctly.

   Guessing that something in the TX direction on the LX2k is broken. I tried increasing the TX amplitude by writing to SERDES LN{4,5,6,7}TECR0 (0b000000 in bits 0:5 for 1.0× amplitude, 0x10808307 ⇒ 0x10808300), but this did not help (maybe U-boot or the Linux driver changes it back? But this was only a guess anyway.)

3. Same SFPs as 2. but with a TI DS125DF111 retimer inserted - works.

   I have a custom board with 2 SFP slots and a DS125DF111 chip between the slots and inserted it before the same SFPs, i.e. connected as:

   [Honeycomb LX2k ⋄ SFP-slot] - DAC cable - [SFP-slot ⋄ DS125DF111 ⋄ SFP-slot ⋄ SFP] - fiber - [SFP ⋄ switch]

   This has worked for several days so it does not seem to have the "lose link after a while" problem. (This is what made me try to increase the TX amplitude through LNxTECR0, I thought maybe the DS125DF111 is fixing the signal levels.)

   The DS125DF111 chip has an eye opening monitor built in, but I haven't implemented this for my board yet. Might be able to get it later.

So — not much luck yet, but hopefully some of these issues can be fixed. I'll keep posting here when I have more time to try around random things.

Cheers,

equi

P.S.: I have carrier board rev. 1.2, CEx7 module rev. 2.1. I have the variant without QSFP cage but will probably solder it on in a while, and then try testing with 40G/4×10G QSFP modules. Note the 1.2 revision does not have the retimer chips for 100G.