Beta Systemcore Lockup Analysis

Discovery

An FRC team (5199) experienced unexpected reboots caused by the beta Systemcore hardware watchdog. The team reproduced this on multiple units. Two other teams reported similar failures and had similarities that helped us narrow-down the cause.

Reproduction

After the SoCal Showdown event, the team stayed for several hours to allow us to collect peripheral and interrupt statistics from the team’s robot. The robot was simulated using these statistics and the failure was reproduced on other Systemcore units. The failure was traced back to a combination of Ethernet RX traffic and SPI Bus traffic in either direction, as neither one alone can excite the failure. CPU utilization and CPU frequency affect the frequency of the failure, Ethernet TX traffic does not excite the failure, and Wi-Fi traffic does not excite the failure. Ethernet RX traffic and SPI bus traffic were artificially increased to reproduce the failure 30-200 seconds after boot. At 600 Mbps of Ethernet RX traffic, CAN Bus utilization of 2% - 10% on three buses or a CAN Bus utilization of ~100% on one bus triggers the failure. At < 10 Mbps of Ethernet RX traffic, three buses at 70% utilization trigger the failure.

The failure appears to be a lockup of the BCM2712 PCIe bridge excited by the RP1 GPIO driver. For any GPIO write, the driver performs six writes to a PCIe configuration register that stalls the CPU’s write path to guarantee monotonicity when power-saving modes are enabled. This particular stall is unnecessary as Systemcore deliberately disables PCIe power management features. The driver erroneously applies the writes in all power modes. We have a hypothesis regarding the underlying failure mechanism that is being confirmed by component vendors. The failure was reproduced with a stock kernel which absolves Systemcore kernel patches. PREEMPT_RT accelerates the failure, but it is not a requirement for the failure. The PCIe bridge does not seem to raise an error and is not publicly documented. The failure causes all CPU cores to lock, limiting the utility of ramoops logs.

Resolution

An OS update (Alpha/Beta 17) mitigates the failure by eliminating 56 unnecessary PCIe configuration writes per CAN Bus frame (350k - 850k writes/second/bus in high-utilization scenarios). It also polls for smaller SPI Bus transactions, reducing SPI controller interrupt counts in one configuration from 52k interrupts/second to 460 interrupts/second and reducing PCIe transactions by 27%. With these changes, Systemcore can operate with heavily loaded CAN Buses, ~100% CPU utilization, and ~1Gbps Ethernet RX activity. The team that most easily reproduced the issue confirmed that Beta 17 appears to resolve the lockup. Limelight is in communication with component vendors regarding the underlying failure mechanism.

Action for Users

Update Systemcore OS to Alpha/Beta 17 or later.