Updated Broadcom Tigon3 ethernet driver

[pieterpan]

It would be nice if Haiku supported the latest broadcom gigabit ethernet chips. Currently Nathan's ​driver is in SVN, but it does not support new chips like the one in my laptop (BCM 5787M) Unfortunately the code his driver is based on was not updated much, and linux switched to a new codebase. So I see 2 options:

Port the ​linux driver or use the FreeBSD network driver compatibility layer to use the ​FreeBSD driver. Most likely the last option would be easiest.

[axeld]

The old FreeBSD 6.2 driver does not support your device, the new 7.0 driver does, though. I'll see if the new driver will work with our compatibility layer as well.

[pieterpan]

Thanks, just let me know when I can help test it, if it works with the compat layer.

[pieterpan]

Well I tried to make this driver compile with the compat framework (based on the rtl8139), but I'm getting errors left and right, trying to include files from the freebsd cvs. This will take forever for me to fix without the appropriate knowledge. I hope you can get it to work...

[PieterPanman]

I managed to get it to compile (with some cheats), but it does not work quite yet. See the attached diff for what I did to make it compile. I still have to implement m_cljget, but I get stuck on other things first. Besides, if it is called, it will panic, then I'll implement it :) It's something to do with jumbo frames.

When I start Haiku with the driver added to the image, it gets to the red rocket, where things go haywire. (without this driver it boots fine) See the attached pictures of the debugging output: it is very exciting to at least see it is identifying my device and initializing it.

My suspicion is that I'm messing up the interrupts, because everything else is rather bothered by my driver's actions. I have very little driver writing experience, so this is mostly trial and error for me.

I also tried these two below, but that doesn't help at all. Probably this device generates interrupts, and this does not apply. NO_HAIKU_CHECK_DISABLE_INTERRUPTS(); NO_HAIKU_REENABLE_INTERRUPTS();

I also tried with the latest driver from ​http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/bge/ but they now added m_collapse, which still needs to be added to the layer. So I figured I'd focus on the current driver, can always update later.

Any advise would be greatly appreciated.

[PieterPanman]

I don't see the way to fix these issues. I attached another picture of the ints command in kdl, maybe it is useful. I can always try out fixes/patches and produce debugging output on request.

[axeld]

It's pretty obvious that there shouldn't be 2.9 million interrupts during boot. So it looks like the interrupt is acknowledged on the PCI bus before the device has cleared it, so that it fires again as soon as it can.

You can try to see if removing the USB drivers helps in any way, although I would doubt it.

The solution should be to clear interrupts of the device in the interrupt handler routine, like the 3com driver does, and store the interrupts that happened somewhere for evaluation in the service thread.

[PieterPanman]

Removing the USB drivers didn't really help, though after I did so, it actually paniced once on its own, something with if and MII. I didn't take a picture, sorry. Other 6 reboots resulted in the interrupt handler taking 100% cpu and me needing to F12 it to KDL. It still shares irq 10 with ahci, as haiku boots from ahci drive. When I set AHCI configuration to disabled in the bios, the ide driver actually also takes irq 10, so it doesn't matter.

I'm reading the changes for the 3com driver, along with the glue code, and the interrupt wrapper stuff in bus.c. If I understand correctly (how it should be for bge), the service thread ("bge intr handler") loops infinitely and waits on a sem. As soon as an interrupt comes in at intr_wrapper, it disables interrupts using the glue code. This code either returns 0 if it is not for bge, OR it clears the interrupt, disables them and sets the interrupt status in the extra variable in sc. Now the intr_wrapper releases the sem and the service thread calls the bge_intr. This handles all available interrupts (the first interrupt status is taken from the sc variable) after which the service thread re-enables interrupts.

I hope I can make this work :)

[PieterPanman]

Hmm well partial success I suppose. It no longer gets stuck on the bge_intr, but it does panic a few times on mii_phy_setmedia in net_server.

PANIC: invalid ife->ifm_data (0xa) in mii_phy_setmedia
Welcome to Kernel Debugging Land...
Thread 58 "net_server" running on CPU 0

I will attach the backtrace.

After a few continue's, haiku boots and shows the desktop. With ifconfig I can actually see the correct mac address, so somethings are working. Until I plug in the cable, then mouse and keyboard input stop working, but the screen keeps refreshing. With processcontroller I can see many different threads all take up 50% cpu (dualcore system) one after another. (scsi scheduler, link state checker, and some others). Cannot get into KDL. I will also attach the diff of my work so far.

[PieterPanman]

backtrace of netserver problems with mii_phy_setmedia

[PieterPanman]

diff that get bge partially working (until you plug in the cable)

[PieterPanman]

At least I get the mac address in ifconfig

[Adek336]

With the diffs it still gets a lot of interrupt hits. Commenting out the code that enables interrupts in bge_init_locked prevents the interrupt storm, but then no packets can be sent or received.

How about trying DEVICE_POLLING? It needs atomic_readandclear_32(), ether_poll_register() and ether_poll_deregister() to be implemented, though.

[jackburton]

This looks exactly like bug #2803, although the NIC is different, it shares with this that uses the bsd network compatibility layer.

[PieterPanman]

Adek336, if you can, give it a try. I'm stuck studying for my exams right now. I'm not sure if this is related to #2803. I think we just need someone with proper know how to apply some driver magic. Right now that's not me :)

[Adek336]

update if_bge.c to 1.198.2.9

[Adek336]

Update the driver to a version a bit newer (FreeBSD has a much newer version but I didn't dare to try it). The newer driver supports BCM5906. There seems to be a bit different intending style between the old and new sources so the diff file grew big.

[Adek336]

works

[Adek336]

fix

[Adek336]

fix

[Adek336]

Normal operations now work. However, the driver doesn't recognize link up/link down media changes. Also, for some reason mii_phy_add_media() used to intentionally add an invalid ifmedia struct. If that was supposed to induce an error to get the user's attention, I don't think we should quietly ignore the cause. The main change was in haiku_disable_interrupts(): the new way of checking if the interrupt came from the network adapter involves reading the BGE_PCI_PCISTATE register, just like the Linux driver does. That solved the interrupt storm.

[PieterPanman]

Nice, when I get some time I'll try it out. Probably after Thursday. Thanks for your efforts so far! I'll definitely test it and let you know

[anevilyak]

Replying to Adek336:

Normal operations now work. However, the driver doesn't recognize link up/link down media changes. Also, for some reason mii_phy_add_media() used to intentionally add an invalid ifmedia struct.

Does that driver include all the possible mii PHYs that that chipset can be paired with? Maybe one's missing?

[Adek336]

        if (sc->mii_capabilities & BMSR_ANEG) {
                ADD(IFM_MAKEWORD(IFM_ETHER, IFM_AUTO, 0, sc->mii_inst),
                    MII_NMEDIA);        /* intentionally invalid index */
                PRINT("auto");
        }

BMSR_ANEG is commented as "autonegotation capable". I think that if a PHY was missing it would fallback to ukphy and work well.

[Adek336]

Hm. It stopped working.

[Adek336]

I booted FreeBSD 7.0 and loaded a modified driver which prints out the value of statusword variable in bge_intr(). I got some statusword=0x0 messages and a whole lot of statusword=0x400000 messages. Next I got a statusword=0x401000 message and the storm ended, additional debugging messages from other parts of the driver told me the driver was once again fully operational.

So, the storm happens not only in Haiku, but also in FreeBSD! However FreeBSD is able to tame it.

After rebooting, the Haiku driver worked once again!

This also means the patches I sent earlier today are irrelevant to the problem.

[Adek336]

update if_bgereg.h to 1.73.2.4

[Adek336]

Almost done, now the driver is indeed able to tame an interrupt storm. I'll post the patch when I figure out how to make svn diff without including the long changes coming from upgrading if_bge.c ?

[PieterPanman]

Nice, I'm looking forward to it. I think just posting the full diff to what is currently in svn is also good. If you want, just download the old FreeBSD if_bge.c as if_bge_orig.c and do "diff if_bge_orig.c if_bge.c"

[Adek336]

works, needs minor changes

[axeld]

Can you provide the verbatim version of if_bge.c you used as well?

We always have the verbatim important vendor versions in the vendor branch: vendor/freebsd/RELENG_7_BP/dev/bge in this case. This makes it possible to easily update to newer versions without losing the changes.

[Adek336]

I can't attach them because of Trac problems but they are ​http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/src/sys/dev/bge/if_bge.c?rev=1.198.2.9;content-type=text%2Fplain and ​http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/src/sys/dev/bge/if_bgereg.h?rev=1.73.2.4;content-type=text%2Fplain .

These are 4 months old, they are the first revisions to support my card.

As a sidenote, I found your comment by sheer luck when I wanted to check something in this ticket - trac didn't send me an update email.

[PieterPanman]

Hmm unfortunately it doesn't work yet with my network chip. It does boot properly, and the interrupts seem to be handled. But it does not give an ip address, and setting one manually does not help. I cannot ping, nor go online. To get it to boot, I had to disable my USB drivers, as they also use that irq, and I sometimes get more than 99% vectors unhandled during boot. With USB removed, the broadcom chip only shares the IRQ with AHCI, and it at least boots.

Relevant syslog output when it boots properly.

KERN: src/add-ons/kernel/drivers/network/broadcom_bcm570x/dev/bge/if_bge.c: bge_link_upd: 4428
KERN: src/add-ons/kernel/drivers/network/broadcom_bcm570x/dev/bge/if_bge.c: bge_link_upd: 4433
KERN: bge_link_upd, sc->bge_link=0x0

ifconfig output, after a while. It does not receive responses to dhcp requests, but all of a sudden it did receive 2 packets.

loop	Hardware Type: Local Loopback, Address: none
	inet addr: 127.0.0.1, Mask: 255.0.0.0
	MTU: 16384, Metric: 0, up loopback link
	Receive: 0 packets, 0 errors, 0 bytes, 0 mcasts, 0 dropped
	Transmit: 0 packets, 0 errors, 0 bytes, 0 mcasts, 0 dropped
	Collisions: 0

/dev/net/broadcom_bcm570x/0
	Hardware Type: Ethernet, Address: 00:16:d4:dc:ca:f2
	inet addr: 192.168.1.101, Bcast: 192.168.255.255, Mask: 255.255.1.0
	MTU: 1500, Metric: 0, up broadcast auto-configured
	Receive: 2 packets, 0 errors, 497 bytes, 0 mcasts, 0 dropped
	Transmit: 44 packets, 0 errors, 4134 bytes, 0 mcasts, 0 dropped
	Collisions: 0

[PieterPanman]

ok update: Something is not going well with the sharing of interrupt with AHCI, because every time bge tries to send a packet, also KERN: [34mahci:[0m AHCIPort::ScsiSynchronizeCache port 0 occurs. When USB is enabled, I get even more fails. See the 3 attached syslogs. I have added some debug output in the bge_txeof and bge_rxeof, only when stuff needs to happen. Immediately you see error responses of USB and AHCI.

[PieterPanman]

syslog with all 3 drivers on same irq: AHCI, USB, Broadcom

[PieterPanman]

syslog with 2 drivers on same irq: AHCI, Broadcom

[PieterPanman]

syslog with only AHCI

[PieterPanman]

with additional debug output

[PieterPanman]

diff adjusted for new location and name of driver. Also includes debugging output to generate above syslog

[PieterPanman]

See attached syslog and new diff that generated this syslog.

At some point, bge_intr_status is: 
sc->bge_intr_status                       0x401000
and at another time it is
sc->bge_intr_status                       0x400000

#define	BGE_MACSTAT_LINK_CHANGED	0x00001000
#define	BGE_MACSTAT_MI_COMPLETE		0x00400000

When it is 0x401000, it indeed goes to the link changed function

Not sure what to do with it now. I really don't get why I get all the KERN: [34mahci:[0m AHCIPort::ScsiSynchronizeCache port 0 lines after a transmit.

To be continued later

[Adek336]

glue.c, __haiku_disable_interrupts

        {       
                return 0;
        } 
+        ktrace_printf("bge: (status & BGE_MACSTAT_LINK_CHANGED) = 0x%x, "
+                   "sc->transmitting=0x%x, pci_read_config(sc->bge_dev, "
+                   "BGE_PCI_PCISTATE,4)&BGE_PCISTATE_INTR_STATE)=0x%x, "
+                   "sc->transmitting=0x%x\n",
+                        (status & BGE_MACSTAT_LINK_CHANGED),
+                        sc->transmitting,
+                        pci_read_config(sc->bge_dev, BGE_PCI_PCISTATE,4)
+                                & BGE_PCISTATE_INTR_STATE,
+                        sc->transmitting
+                );
+        if ((status & BGE_MACSTAT_LINK_CHANGED) == 0)
                sc->transmitting = 0;

        // Clear the interrupt, the bge intr handler will take care of it
        BGE_SETBIT(sc, BGE_PCI_MISC_CTL, BGE_PCIMISCCTL_MASK_PCI_INTR);
        bge_writembx(sc, BGE_MBX_IRQ0_LO, 1);

how about this? or dprintf instead of ktrace_printf if you don't have tracing enabled.

[PieterPanman]

With Adek's additional debugging output

[PieterPanman]

Axel, maybe you can already commit the verbatim version in the vendor branch, and the patch that at least makes it work for Adek. This will make the diffs a bit more readable for debugging my chip, and maybe it will work for others.

Thanks for your hints Adek336, we're getting there. I was wondering, you say it works for you... Do you have to share the interrupt with other devices? I'm wondering if something strange is going on there. Maybe the AHCI does something to the interrupts for bge? The attached syslog is me trying to ping a few times, unplugging and plugging it back in. Actually it sort of received one packet, as you see, but the ping didn't work.

[korli]

Hello guys,

I checked in an updated version of if_bge, with part of patches from yours. Missing stuff in freebsd compat layer should be OK hopefully. As I don't understand why and where make changes for HAIKU only, I kept things as is.

IMO disabling interrupts in bge_intr() another time shouldn't hurt much. The important stuff is to know exactly if the interrupt is yours, and only then handle it.

Please check against hrev28604 and provide patches against this version. Thank for your help, very appreciated.

[korli]

Oh and I noticed there is only the generic ukphy.c included. It could be useful to check with brgphy.c for instance.

[Adek336]

BCM5906M works well with this patch

[Adek336]

I share IRQ 10 between Tigon3 and at least AHCI - I'll check more thoroughly when I sort out problems with the laptop's power supply.

[PieterPanman]

Thanks for your new patch, It sort of works, but I still do not get an IP address, and something goes wrong with media detection. It also does not give the correct link state in the network tray application. Most likely it has to do with the incomplete

if (sc->mii_capabilities & BMSR_ANEG) {

part in fbsd_mii_physubr.c.

I only get two lnDbg, so this (in bge_link_update) returns false:

			if (!sc->bge_link &&
			    mii->mii_media_status & IFM_ACTIVE &&
			    IFM_SUBTYPE(mii->mii_media_active) != IFM_NONE) {

I also tried to make this work my way, I got it just about as far as your solution. However, to determine if this is our interrupt, I did the following:

+	uint32 pcistate = pci_read_config(sc->bge_dev, BGE_PCI_PCISTATE,4);
+	uint32 pci_interrupted = !(pcistate & BGE_PCISTATE_INTR_STATE);
+	if (!pci_interrupted)
 		return 0;

From the tigon driver I found that BGE_PCISTATE_INTR_STATE is true means the interrupt is NOT from the broadcom chip. So checking for this should be enough?

I also had to move the bge_link_update call to the haiku_disable_interrupts(device_t dev), because it would never get called due to the interrupt storm.

Actually, I now believe the KERN: [34mahci:[0m AHCIPort::ScsiSynchronizeCache port 0 is simply the harddrive being flushed the new syslog contents, and probably not an irq problem.

Lastly, I do send and receive packets, but I do not get an ip address, and I cannot ping anyone. I will attach my syslog and my own patch

[PieterPanman]

syslog with adek's latest patch

[PieterPanman]

my own take at the problem with korli's new version

[korli]

I added brgphy to broadcom570x in hrev28637.

[PieterPanman]

Sweet, it works for me too now. I have cleaned up my patch a lot, using some of Adek's ideas. However, it is a lot simpler, as it doesn't keep track of any variables. Adek, can you please test my patch with your chip? If it works with yours maybe Korli can commit it. I managed to check out almost the entire svn tree with it! (until I ran out of space ;-) )

What doesn't work yet is:

• link state detection in the Network Deskbar applet. I'm not sure if this works at all, can someone confirm
• DHCP detection if you plug the cable in after you booted. It doesn't notice the cable is plugged in
• bringing down and up the interface results in the read errors to go up rapidly.

Interestingly, in the syslog you see it detects the link up and link down events.

[PieterPanman]

now working syslog

[PieterPanman]

cleaned up and working version

[PieterPanman]

Korli, Thanks for the updates and brgphy. Just curious, why do you check for IFF_DRV_RUNNING in the broadcom440x? Shouldn't you still handle the interrupt in that case?

From broadcom440x glue.c in haiku_disable_interrupts:

istat = CSR_READ_4(sc, BFE_ISTAT);
if (istat == 0 || (sc->bfe_ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) {
    HAIKU_INTR_REGISTER_LEAVE();
    return 0;
}

[korli]

Replying to PieterPanman:

Korli, Thanks for the updates and brgphy. Just curious, why do you check for IFF_DRV_RUNNING in the broadcom440x? Shouldn't you still handle the interrupt in that case?

This is because it's done this way in bfe_intr():

/* not expecting this interrupt, disregard it */
	if (istat == 0 || (ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) {
		BFE_UNLOCK(sc);
		return;
	}

[karmak]

Hi,

here on a Thinkpad R52 (BCM5751M), Haiku installed on ATA hard disk, built from Linux.

• With stock broadcom570x driver (from SVN), system hangs (hard) after rocket icon has appeared.
• with broadcom_1.diff (from PeterPanman) applyed, the network works fine all the time I was using it (surfing with ffox 2.0.0.18, wget, netpositive, fetching 10000 mails in IMAP configuration…), more than one hour (after that i got a crash in intel_accelerant, will file another bug later).
  • sometimes network seems stalled (with either ffox, wget or net+) ; clicking pause/unpause (or stopping wget and restarting it with -c) resolves the situation and download continues normally. The stall happened at max one time for each download i tryed ; i couldn't dertermine reliably if the problems comes from inside haiku, the driver or from outside.
  • the deskbar applet always reports "link down", even though everything works normally (network cable pluged in, DHCP OK, downloading in progress…).

I didn't try to unplug/plug cable to see if it changes anything. Will do next time I boot Haiku. I didn't try with against-26….diff since broadcom_1.diff alone makes my card work.

Tell me if you have some other tests I should do, or syslog or anything you want to help this.

[PieterPanman]

Karmak, thanks for testing. Good to know it works for you too. Adek, can you also try it with my patch? Then it can be committed to the tree and enabled by default. I haven't yet gotten any stalled downloads. I just downloaded the latest ubuntu iso at 1600 kB/sec without a hickup. Regarding the icon in NetworkStatus, it reports the same as ifconfig. It only shows a connection if both 'up' and 'link' are true. Only 'up' is true here. Maybe this is also the reason why it doesn't get an ip address through dhcp when you plug it in after booting. Bug in the freebsd compat layer?

/dev/net/broadcom570x/0
        Hardware Type: Ethernet, Address: 00:16:d4:dc:ca:f2
        inet addr: 192.168.1.37, Bcast: 192.168.1.255, Mask: 255.255.255.0
        MTU: 1500, Metric: 0, up broadcast auto-configured
        Receive: 559660 packets, 0 errors, 846238348 bytes, 0 mcasts, 0 dropped
        Transmit: 306325 packets, 0 errors, 20346875 bytes, 0 mcasts, 0 dropped
        Collisions: 0

[PieterPanman]

Cleaned up some more and added #ifndef HAIKU stuff

[PieterPanman]

It is amazing how simple this patch ends up being, compared to my first attempts... Nice and clean, and it still works!

[Adek336]

I'll let you know as soon as my laptop gets fixed!

[PieterPanman]

That sucks, hope they fix it soon. Thanks for the note.

[karmak]

With broadcom_2.diff I see no difference, meaning that my network card works perfectly, at the expand of connecting cable BEFORE booting (else it will never work).

What is strange is that kernel seems to detect the link UP/DOWN events (i noticed that in syslog), but they are not reflected in the the NetStatus deskbar applet.

[PieterPanman]

Korli, you seem to have fixed the link state detection problem with hrev28705. It now correctly detects the link changes. Please commit my patch, I think we're done :) If not we can always change it again. Thanks for all your help, testing and thinking out loud :)

[korli]

Applied in hrev28712.

[Adek336]

Have my patches helped? I've found out BGE_PCI_PCISTATE PCI register determines interrupt source.

[PieterPanman]

Yes, I wouldn't have been able to do it without your patches to start out with. I'm very happy I can now access the internet from my laptop, it makes Haiku much more fun :)

[Adek336]

I can now confirm that the trunk driver works with my network card :-)

[PieterPanman]

Very nice indeed! Good to hear you have your laptop back, and thanks for your help with it.