Opened 3 years ago

Closed 2 years ago

#16932 closed bug (invalid)

Network breakage with emulex 10Gbps NIC

Reported by: kallisti5 Owned by: axeld
Priority: normal Milestone: Unscheduled
Component: Network & Internet/Stack Version: R1/beta2
Keywords: Cc:
Blocked By: Blocking:
Platform: All

Description (last modified by kallisti5)

Networking works for a short period of time before decreasing to being broken when connecting to the internet via a emulex PCIe 10Gbps NIC.

Configuration:

  • Haiku x86_64. hrev55041
  • Emulex Corporation OneConnect 10Gb NIC
  • 10Gbps DAC cable to a SFP+ switch.

On startup, network is working as expected. Functionality slowly declines until all network connectivity hangs.

Attachments (1)

issue.png​ (336.6 KB ) - added by kallisti5 3 years ago.
example of network connectivity loss

Download all attachments as: .zip

Change History (10)

comment:1 by kallisti5, 3 years ago

Resolution: → junk
Status: new → closed

(thought it was cloud-init messing with stuff on my desktop)... it wasn't

Last edited 3 years ago by kallisti5 (previous) (diff)

comment:2 by kallisti5, 3 years ago

Resolution: junk
Status: closed → reopened

Reopening. Still happening after uninstalling cloud-init.

comment:3 by kallisti5, 3 years ago

Description: modified (diff)

comment:4 by kallisti5, 3 years ago

  • Bootup
  • Can ping google.com
  • Attempt pkgman update
  • OS-wide network connectivity grinds to a stop
  • attempt ifconfig (interface) auto-config
  • stuck "link configuring" in ifconfig
  • one CPU core maxed at 100% on network traffic.

comment:5 by kallisti5, 3 years ago

interrupts: 

int 65, enabled: 1, handled 771, unhandled 0
  emulex_oce:oce_fast_isr     .... handled 771
int 66, enabled: 1, handled 6, unhandled 0
  emulex_oce:oce_fast_isr     .... handled 6

by kallisti5, 3 years ago

Attachment: issue.png​ added

example of network connectivity loss

comment:6 by kallisti5, 3 years ago

Relevant emulex_oce logs: 

KERN: pci_reserve_device(8, 0, 0, emulex_oce)
KERN: pci_reserve_device(8, 0, 1, emulex_oce)
KERN: package_daemon [34210306:   991] [emulex_oce] (oce) bus_alloc_resource(3, [16], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [24], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: package_daemon [34290281:   991] [emulex_oce] (oce) bus_alloc_resource(3, [32], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: package_daemon [35055118:   991] [emulex_oce] (oce) bus_alloc_resource(1, [1], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: package_daemon [35237959:   991] if_initname(0xffffffffe2bf2000, oce, 49)
KERN: [emulex_oce] emulex_oce: /dev/net/emulex_oce/0
KERN: emulex_oce: init_driver(0xffffffff81cbbe70)
KERN: package_daemon [35272562:   991] [emulex_oce] (oce) bus_alloc_resource(3, [16], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [24], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: package_daemon [35355371:   991] [emulex_oce] (oce) bus_alloc_resource(3, [32], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: package_daemon [35462625:   991] [emulex_oce] (oce) bus_alloc_resource(1, [1], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: package_daemon [35643804:   991] if_initname(0xffffffffe2bf1800, oce, 51)
KERN: [emulex_oce] emulex_oce: /dev/net/emulex_oce/1
KERN: package_daemon [35671322:   991] emulex_oce: init_driver(0xffffffff81cbbe70)
KERN: loaded driver /boot/system/add-ons/kernel/drivers/dev/net/emulex_oce
KERN: [net/emulex_oce/0] compat_open(0x2)
KERN: [emulex_oce] (oce) Interface Down
KERN: [emulex_oce] (oce) Interface Up
KERN: /dev/net/emulex_oce/0: media change, media 0x900033 quality 1000 speed 1410065408
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_DISCOVER to 255.255.255.255:67
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_OFFER from 192.168.1.1
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_REQUEST for 192.168.1.140 to 255.255.255.255:67
KERN: [net/emulex_oce/1] compat_open(0x2)
KERN: [emulex_oce] (oce) Interface Down
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_ACK from 192.168.1.1
KERN: [emulex_oce] (oce) Interface Up
KERN: /dev/net/emulex_oce/1: media change, media 0x20 quality 1000 speed 0
DAEMON 'DHCP': /dev/net/emulex_oce/0: DHCP status = No error

interfaces: 

/dev/net/emulex_oce/0
        Hardware type: Ethernet, Address: 00:90:fa:04:42:00
        inet addr: 192.168.1.140, Bcast: 192.168.1.255, Mask: 255.255.255.0
        MTU: 1500, Metric: 0, up broadcast link auto-configured
        Receive: 2159 packets, 0 errors, 1868612 bytes, 0 mcasts, 0 dropped
        Transmit: 1073 packets, 37 errors, 84545 bytes, 0 mcasts, 0 dropped
        Collisions: 0

/dev/net/emulex_oce/1
        Hardware type: Ethernet, Address: 00:90:fa:04:42:04
        inet addr: --, Bcast: --, Mask: --
        MTU: 1500, Metric: 0, up broadcast
        Receive: 0 packets, 0 errors, 0 bytes, 0 mcasts, 0 dropped
        Transmit: 0 packets, 0 errors, 0 bytes, 0 mcasts, 0 dropped
        Collisions: 0

The "speed 0" might be a sign that Haiku's having issues with the negotiated 10Gbps. The ping "no buffer space available" is interesting as well.

comment:7 by kallisti5, 3 years ago

  • ping sendto ENOBUFS, perror("ping: sendto");
  • headers/posix/sys/socket.h
  • src/system/libnetwork/socket.cpp
  • _kern_sendto meh
  • src/system/kernel/fs/socket.cpp
  • common_sendto GET_SOCKET_FD_OR_RETURN, sStackInterface

Enabling a bunch of tracing in src/add-ons/kernel/network/stack and will report back.

comment:8 by kallisti5, 3 years ago

debugging went... poorly. Tracker locked up, ifconfig hung, ping hung.

Not really much in debug info so far... 

KERN: net: create net_socket 0xffffffffde6f8600 (1.2.17):
KERN: net:   [0] 0xffffffffdf663a68  network/protocols/udp/v1
KERN: net:   [1] 0xffffffffde5b51e0  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffde6f8600
KERN: net: create net_socket 0xffffffffde6f8600 (1.2.17):
KERN: net:   [0] 0xffffffffdf6636a8  network/protocols/udp/v1
KERN: net:   [1] 0xffffffff8086c6c0  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffde6f8600
KERN: net: create net_socket 0xffffffffde6f8600 (1.2.17):
KERN: net:   [0] 0xffffffffdf6636a8  network/protocols/udp/v1
KERN: net:   [1] 0xffffffff8086c6c0  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffde6f8600
KERN: net: create net_socket 0xffffffffde6f8600 (1.2.17):
KERN: net:   [0] 0xffffffffdf6636a8  network/protocols/udp/v1
KERN: net:   [1] 0xffffffff8086c6c0  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffde6f8600
KERN: net: create net_socket 0xffffffffde6f8600 (1.2.17):
KERN: net:   [0] 0xffffffffdf6634c8  network/protocols/udp/v1
KERN: net:   [1] 0xffffffff8097f6c8  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffde6f8600
KERN: net: create net_socket 0xffffffffdf5f6800 (1.2.17):
KERN: net:   [0] 0xffffffffdf590918  network/protocols/udp/v1
KERN: net:   [1] 0xffffffffde966c00  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffdf5f6800
KERN: net: create net_socket 0xffffffffdf5f6800 (1.2.17):
KERN: net:   [0] 0xffffffffdf590918  network/protocols/udp/v1
KERN: net:   [1] 0xffffffffde966c00  network/protocols/ipv4/v1
KERN: net: delete net_socket 0xffffffffdf5f6800

$ cat syslog | grep emulex
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_DISCOVER to 255.255.255.255:67
DAEMON 'DHCP': /dev/net/emulex_oce/0: Timeout shift: 500 msecs (try 1)
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_DISCOVER to 255.255.255.255:67
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_OFFER from 192.168.1.1
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_REQUEST for 192.168.1.140 to 255.255.255.255:67
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_OFFER from 192.168.1.1
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_ACK from 192.168.1.1
DAEMON 'DHCP': /dev/net/emulex_oce/0: DHCP status = No error
KERN: pci_reserve_device(8, 0, 0, emulex_oce)
KERN: pci_reserve_device(8, 0, 1, emulex_oce)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [16], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [24], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [32], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(1, [1], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] emulex_oce: /dev/net/emulex_oce/0
KERN: emulex_oce: init_driver(0xffffffff81d64e70)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [16], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [24], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [32], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(1, [1], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] emulex_oce: /dev/net/emulex_oce/1
KERN: emulex_oce: init_driver(0xffffffff81d64e70)
KERN: loaded driver /boot/system/add-ons/kernel/drivers/dev/net/emulex_oce
KERN: get_device_interface: ask "network/devices/loopback/v1" for /dev/net/emulex_oce/0
KERN: get_device_interface: ask "network/devices/ethernet/v1" for /dev/net/emulex_oce/0
KERN: net: Interface 0xffffffffde8772a8: new "/dev/net/emulex_oce/0", device interface 0xffffffffde713600
KERN: [net/emulex_oce/0] compat_open(0x2)
KERN: [emulex_oce] (oce) Interface Down
KERN: [emulex_oce] (oce) Interface Up
KERN: /dev/net/emulex_oce/0: media change, media 0x900033 quality 1000 speed 1410065408
KERN: get_device_interface: ask "network/devices/loopback/v1" for /dev/net/emulex_oce/1
KERN: get_device_interface: ask "network/devices/ethernet/v1" for /dev/net/emulex_oce/1
KERN: net: Interface 0xffffffffde8777e8: new "/dev/net/emulex_oce/1", device interface 0xffffffffde713200
KERN: [net/emulex_oce/1] compat_open(0x2)
KERN: [emulex_oce] (oce) Interface Down
KERN: [emulex_oce] (oce) Interface Up
KERN: /dev/net/emulex_oce/1: media change, media 0x20 quality 1000 speed 0
KERN: net:   /dev/net/emulex_oce/0 matches
KERN: net:   /dev/net/emulex_oce/0 matches
KERN: net:   /dev/net/emulex_oce/0 matches
KERN: pci_reserve_device(8, 0, 0, emulex_oce)
KERN: pci_reserve_device(8, 0, 1, emulex_oce)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [16], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [24], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [32], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(1, [1], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] emulex_oce: /dev/net/emulex_oce/0
KERN: emulex_oce: init_driver(0xffffffff81dece70)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [16], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [24], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(3, [32], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] (oce) bus_alloc_resource(1, [1], 0x0, 0xffffffffffffffff, 0x1,0x6)
KERN: [emulex_oce] emulex_oce: /dev/net/emulex_oce/1
KERN: emulex_oce: init_driver(0xffffffff81dece70)
KERN: loaded driver /boot/system/add-ons/kernel/drivers/dev/net/emulex_oce
KERN: get_device_interface: ask "network/devices/loopback/v1" for /dev/net/emulex_oce/0
KERN: get_device_interface: ask "network/devices/ethernet/v1" for /dev/net/emulex_oce/0
KERN: net: Interface 0xffffffffde9757e8: new "/dev/net/emulex_oce/0", device interface 0xffffffffdea71b00
KERN: [net/emulex_oce/0] compat_open(0x2)
KERN: [emulex_oce] (oce) Interface Down
KERN: [emulex_oce] (oce) Interface Up
KERN: /dev/net/emulex_oce/0: media change, media 0x900033 quality 1000 speed 1410065408
KERN: get_device_interface: ask "network/devices/loopback/v1" for /dev/net/emulex_oce/1
KERN: get_device_interface: ask "network/devices/ethernet/v1" for /dev/net/emulex_oce/1
KERN: net: Interface 0xffffffffde975708: new "/dev/net/emulex_oce/1", device interface 0xffffffffdea71000
KERN: [net/emulex_oce/1] compat_open(0x2)
KERN: [emulex_oce] (oce) Interface Down
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_DISCOVER to 255.255.255.255:67
KERN: net:   /dev/net/emulex_oce/0 matches
KERN: [emulex_oce] (oce) Interface Up
KERN: /dev/net/emulex_oce/1: media change, media 0x20 quality 1000 speed 0
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_OFFER from 192.168.1.1
DAEMON 'DHCP': /dev/net/emulex_oce/0: Send DHCP_REQUEST for 192.168.1.140 to 255.255.255.255:67
KERN: net:   /dev/net/emulex_oce/0 matches
DAEMON 'DHCP': /dev/net/emulex_oce/0: Received DHCP_ACK from 192.168.1.1
DAEMON 'DHCP': /dev/net/emulex_oce/0: DHCP status = No error

comment:9 by kallisti5, 2 years ago

Resolution: → invalid
Status: reopened → closed

emulex_oce support was pulled. It was poorly maintained on FreeBSD

Note: See TracTickets for help on using tickets.