LSI Cards: A Primer

I've been working on building a home server as of late, and for it, I ended-up needing an HBA (Host Bus Adapter). Of all the SATA-capable HBAs out there, the LSI ones are by far the most-reliable and well-known. The trouble is, there are a lot of LSI HBAs out there, and this made it tricky to decide which to choose. After months of research and trying out three separate cards, I came to learn quite a lot about these things. This document compiles the core of those learnings into one singular reference sheet.

Background

My requirements

I have a 3-way mirror of HDDs for my vdev and a 3-way mirror of SSDs for my svdev. All are SATA devices. I will be running the card in a PCIe 4.0 x4 slot because [1] I only need a max of 2475 MB/s over PCIe for my drives (well-below the theoretical max of 3.9 GB/s for PCIe 3.0 x4) and [2] because that slot positions the card where it will get its rated LFM (200 LFM @ 55°C for all the LSI cards I've seen) from my system's case fans.

I have 6 drives needing attachment. A single mini-SAS connector is serviced by 4 PHYs, which means they can break out into 4 SATA cables. Accordingly, I need two mini-SAS-to-SATA cables, for a total of 8 usable SATA connections.

Corporate history

While I call these cards "LSI", not all of the ones I've been looking at are/were actually by LSI: This is because Avago bought LSI on 6 May 2014. Avago then on 28 May 2015 acquired Broadcom and rebranded as such. Avago/Broadcom continued to put out LSI-like HBAs after its acquisition of LSI and continued using the same version numbering and tooling and such, so people have continued to refer to them as "LSI" cards, even though nothing after the 2014 acquisition actually has "LSI" printed on it.

It is worth noting that post-LSI cards tend to be made per different, arguably questionable philosophies.

Model numbers

LSI HBA model numbers can be broken-down as follows:

• The second digit is the generation of card (NOT the PCIe version).
• The number after the hyphen is the number of PHYs (and therefore the number of SATA connections).
• The letter after the hyphen tells you if the ports are internal (i) or external (e).

Relevant cards

Right off the bat, the 9200-8i is eliminated: it is PCIe 2.0, meaning it'd be capped at only 2000 MB/s at my x4 PCIe width, which is below my 2475 MB/s requirement.
Also immediately eliminated is the 9500-8i, as it is far-too-expensive.
That leaves us with the 9300-8i, the 9305-16i*, and the 9400-8i. (*The 9305 series has no -8i available.). Let's dig in.

(Side-note: the 9300-16i is a monstrosity: two 9300-8is wired together. It accordingly has about twice the power consumption and twice the heat output. Do not ever get a 9300-16i; always opt instead for the 9305-16i if you can find one.)

Important links

It took me a very long time to hunt through thousands of inconsistently-named and sorted entries on Broadcom's site to find the latest possible versions of all the firmware needed to update and service these cards. I have compiled the results of that labor here:

• LSI 9300-8i (SAS3008) (Firmware and Patch) (Flashing Utility)
• Broadcom 9305-16i (SAS3216) (Firmware) (Flashing Utility)
• Broadcom 9400-8i (SAS3408) (Firmware) (Flashing Utility)
• Broadcom's software search

Tables

Models

ASICs

Supplementary info

• We can see from the above tables that the 9400 series does not have a true passthrough mode, which means it is a bad idea to use it with ZFS. Compared with a direct SATA connection to my motherboard, this card incurred about a 9% read penalty for my SATA SSDs (hdparm -t). As if this weren't already bad-enough, Broadcom locks you out of configuring a bunch of relevant settings (like staggered spin-up) because these cards are supposed to be dumb HBAs (They aren't.). So not only do you have a large performance penalty, but you don't even get to avail yourself of the firmware's various features.
• The 9305-16i has status LEDs right next to its PCIe bracket, which is handy. One of them is its heartbeat, and the other is an overheating indicator. The 9300-8i only has a heartbeat LED, which is near the middle of the board; it's still possible to see it from outside the case, just a little trickier. The 9400-8i has LEDs for PSoC heartbeat, system heartbeat, and temperature; but they are difficult to make out through its PCIe bracket's grill because they are deep, the PSoC heartbeat is under the heatsink, and the other two block each other.
• The 9300-8i and the 9400-8i have simple black-painted aluminium heatsinks, with fins that accept airflow from both the front and the side. And because these cards only have 2 mini-SAS ports, only half of the fronts of their heatsinks are blocked. The 9300-8i's heatsink is quite small, though. The 9305-16i has an elaborate unpainted aluminium heatsink with a copper heatpipe and contact plate. Part of the heatsink is cut-away to fit an extra 2 mini-SAS ports, which are present on the -24i version but absent on the -16i version. This heatsink's fins do not accept side airflow, and the 4 mini-SAS ports completely block front airflow to the heatsink. Accordingly, the layout of the 9305-16i's heatsink is airflow-defective; but despite this, it nevertheless runs substantially cooler than the 9300-8i, as its large size and copper elements more than make up for its poor fin situation. In my tests (6 drives attached and idling), the 9300-8i idles at 66°C, the 9305-16i at 53°C, and the 9400-8i at 43°C. All remained at acceptable temperatures with the rated LFM at half the rated air temperature.
• The 9300-8i and 9305-16i have pins for UART and SBR recovery, both reserved for internal Broadcom use. The 9400-8i has pins for UART and mode selection, both reserved for internal Broadcom use.

Conclusions

• The 9400 series should never be used for ZFS unless NVMe drives are being used (in which case you have no choice) — it is simply too degradative for performance.
• If you only need to use two SAS ports, then the only reason to choose the 9305-16i over the 9300-8i is if you are struggling to keep the 9300-8i cool or if it is important to you to have an LED to indicate overheating. In all other cases, you should buy the cheaper 9300-8i.
  • The second SAS core of the 9305-16i is nice in theory but should not matter in practice for this topology and load. Nevertheless, if you do get a 9305-16i, it is probably best to place your two SAS cables onto different cores.
  • While the 9300-16i likely has a slightly higher worst-case power-draw than the 9300-8i due to possessing the same CPU and twice as many PHYs, its PHYs being 4 years newer could mean that under partial loads the 9305-16i may require less power than the 9300-8i; accordingly, if you can find these two cards for the same price, the 9305-16i may save a tiny bit of money over time relative to the 9300-8i. This is, however, speculative, and the difference is surely pretty small; accordingly, this should not be a primary consideration in which card to buy.