LSI MegaRAID SAS

1. Card information

MegaRAID SAS is the current high-end RAID controllers series by LSI.
It is fully hardware RAIDs controllers supporting RAID5, at least, with SAS or SATA interfaces.
If you're a looking for information about MegaRAID SCSI connectors, please look at LSIMegaRAID instead.

All theses card can be used with stock Linux kernel which includes a working driver.
It's quite new and thus, may be missing in some not-up-to-date distributions.

There is currently no known opensource tool for theses cards.
Some old MegaRAID SAS can be used with megactl, but none of current cards works.
However LSI provide megacli, a proprietary management command line utility which is rather hard to use.

2. Linux kernel drivers

Driver	Supported cards
megaraid_sas	LSI MegaRAID SAS

megaraid_sas is part of mainstream Linux kernel and should be available in all current distributions.
However, please that most of old distributions won't have this driver.

If your card use megaraid_mm or megaraid_mbox driver, please look at LSIMegaRAID instead.

Some lspci -nn output examples:

• 02:0e.0 RAID bus controller [0104]: Dell PowerEdge Expandable RAID controller 5 ![1028:0015]
• 01:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04) ![1000:0060]
• 04:0e.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS ![1000:0411]

3. Management and reporting tools

megactl includes a SAS compliant binary named megasasctl. It seems to work on old card but fails with the new one.
If megasasctl doesn't work for you, you will have to use the proprietary cli utility from LSI: megaclisas.

For managing the card there are no alternatives to megaclisas.

3.1. megactl

Despites megasasctl doesn't seem to work with recent cards, you should really give it a try.

3.1.1. Quickstart and output example

Print current controller status:

server:~# megasasctl
a0       PERC 5/i Integrated      encl:1 ldrv:1  batt:good
a0d0      136GiB RAID 1   1x2  optimal
a0e8s0     136GiB  a0d0  online
a0e8s1     136GiB  a0d0  online

[root@server ~]# megasasctl
a0       PERC 5/i Integrated      encl:1 ldrv:2  batt:good
a0d0       67GiB RAID 1   1x2  optimal
a0d1      836GiB RAID 5   1x4  optimal
a0e8s0      68GiB  a0d0  online
a0e8s1      68GiB  a0d0  online
a0e8s2     279GiB  a0d1  online
a0e8s3     279GiB  a0d1  online
a0e8s4     279GiB  a0d1  online
a0e8s5     279GiB  a0d1  online

There are several switches which are interresting:

• -H: Only print lines which are not ok.
  If nothing is printer, everything is fine
• -B: Ignore batttery problems when running -H.
  In fact megasasctl can't define if your controller has a battery or not.
  If you don't have one, use this parameter.

3.1.2. Periodic checks

You can write your own script around megasasctl to check your adapter status health periodically. However, I already did this for you. See megaraid-status below.

3.2. megaraid-status

3.2.1. About megaraid-status

megaraidsas-status is a wrapper script around megactl with periodics checks.
It is available in the packages repository too.

The packages comes with a python wrapper around megasasctl and an initscript that periodic run this wrapper to check status.
It keeps a file with latest status and thus is able to detect RAID status changes and/or brokeness.
It will log a ligne to syslog when something failed and will send you a mail.
Until arrays are healthy again a reminder will be sent each 2 hours.

3.2.2. Wrapper output example

server:~# megaraidsas-status
-- Arrays informations --
-- ID | Type | Size | Status
a0d0 | RAID 1 | 136GiB | optimal

-- Disks informations
-- ID | Model | Status | Warnings
a0e8s0 | SEAGATE ST3146854SS 136GiB | online
a0e8s1 | SEAGATE ST3146854SS 136GiB | online

[root@server ~]# megaraidsas-status
-- Arrays informations --
-- ID | Type | Size | Status
a0d0 | RAID 1 | 67GiB | optimal
a0d1 | RAID 5 | 836GiB | optimal

-- Disks informations
-- ID | Model | Status | Warnings
a0e8s0 | FUJITSU MBA3073RC 68GiB | online
a0e8s1 | FUJITSU MBA3073RC 68GiB | online
a0e8s2 | SEAGATE ST3300656SS 279GiB | online
a0e8s3 | SEAGATE ST3300656SS 279GiB | online
a0e8s4 | SEAGATE ST3300656SS 279GiB | online
a0e8s5 | SEAGATE ST3300656SS 279GiB | online

3.3. megacli

3.3.1. About megacli

megacli is a proprietary tool by LSI which can perform both reporting and management for MegaRAID SAS cards.
However it's really hard to use because it's use tones of command line parameters and there's no documentation.

3.3.2. Quickstart and output example

Get all adapters status and config:

server:~# megacli -AdpAllInfo -aAll
Adapter #0

==============================================================================
                    Versions
                ================
Product Name    : PERC 5/i Integrated
Serial No       : 12345
FW Package Build: 5.2.1-0067

                    Mfg. Data
                ================
Mfg. Date       : 00/00/00
Rework Date     : 00/00/00
Revision No     : @��A
Battery FRU     : N/A

                Image Versions In Flash:
                ================
Boot Block Version : R.2.3.12
BIOS Version       : MT28-8
MPT Version        : MPTFW-00.10.61.00-IT
FW Version         : 1.03.40-0316
WebBIOS Version    : 1.03-04
Ctrl-R Version     : 1.04-019A
[...]

Logical drive 0 on adapter 0 status and type:

server:~# megacli -LDInfo -L0 -a0
Adapter 0 -- Virtual Drive Information:
Virtual Disk: 0 (Target Id: 0)
Name:raid1
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:237824MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:2
Span Depth:1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default

Exit Code: 0x00

Display, disable or enable automatic rebuild on adapter 0:

server:~# megacli -AdpAutoRbld -Dsply -a0

Adapter 0: AutoRebuild is Enabled.

Exit Code: 0x00
server:~# megacli -AdpAutoRbld -Dsbl -a0

Adapter 0: AutoRebuild is Disabled.

Exit Code: 0x00
server:~# megacli -AdpAutoRbld -Enbl -a0

Adapter 0: AutoRebuild is Enabled.

Exit Code: 0x00

Get and modify rebuild rate:

server:~# megacli -AdpGetProp RebuildRate -a0

Adapter 0: Rebuild Rate = 30%

Exit Code: 0x00
server:~# megacli -AdpSetProp RebuildRate 60 -a0

Adapter 0: Set rebuild rate to 60% success.

Exit Code: 0x00

Show physical disks from first controller:

server:~# megacli -PDList -a0
[...]
Enclosure Device ID: 32
Slot Number: 1
Device Id: 1
Sequence Number: 9
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 140014MB [0x11177328 Sectors]
Non Coerced Size: 139502MB [0x11077328 Sectors]
Coerced Size: 139392MB [0x11040000 Sectors]
Firmware state: Rebuild
SAS Address(0): 0x5000c5000c8579d1
SAS Address(1): 0x0
Connected Port Number: 1(path0)
Inquiry Data: SEAGATE ST3146855SS     S5283LN6CNGM
Foreign State: None

We can see that disk 32,1 (enclosure id = 32, slot = 1) is currently rebuilding (firmware state).

Let's check this operation progress:

server:~# megacli -PDRbld -ShowProg -PhysDrv [32:1] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 1 Completed 51% in 10 Minutes.

Create a RAID6 array with megacli

Let's assume we have a server with two MegaRAID SAS cards. The first one is already setup but we have just plugged a disk bays on the second card.

List physical disks on second card (only print enclosure and slots numbers):

server:~# megacli -PDlist -a1 | grep -e '^Enclosure Device ID:' -e '^Slot Number:'
Enclosure Device ID: 0
Slot Number: 0
Enclosure Device ID: 0
Slot Number: 1
Enclosure Device ID: 0
Slot Number: 2
Enclosure Device ID: 0
Slot Number: 3
Enclosure Device ID: 0
Slot Number: 4
Enclosure Device ID: 0
Slot Number: 5
Enclosure Device ID: 0
Slot Number: 6

Now we have all enclosure and slot number. Let's create the new array:

server:~# megacli -CfgLdAdd -r6 [0:0,0:1,0:2,0:3,0:4,0:5,0:6] -a1

Adapter 1: Created VD 0

Adapter 1: Configured the Adapter!!

Exit Code: 0x00

3.3.3. Periodic checks

You can write your own script around megacli to check your adapter status health periodically. However, I already did this for you. See megaclisas-status below.

3.4. megaclisas-status

3.4.1. About megaclisas-status

megaclisas-status is a wrapper script around megacli that report summarized RAID status with periodic checks feature.
It is available in the packages repository too.

The packages comes with a python wrapper around megacli and an initscript that periodic run this wrapper to check status.
It keeps a file with latest status and thus is able to detect RAID status changes and/or brokeness.
It will log a ligne to syslog when something failed and will send you a mail.
Until arrays are healthy again a reminder will be sent each 2 hours.

3.4.2. Wrapper output example

server:~# megaclisas-status
-- Controller informations --
-- ID | Model
c0 | PERC 5/i Integrated

-- Arrays informations --
-- ID | Type | Size | Status | InProgress
c0u0 | RAID1 | 238G | Optimal | None

-- Disks informations
-- ID | Model | Status
c0u0p0 | ATA ST3250620NS 3BKH 5QE4PW70 | Online
c0u0p1 | ATA WDC WD2500YS-18S6C07 WD-WCANY3450500 | Online

Another example:

server:~# megaclisas-status
-- Controller informations --
-- ID | Model
c0 | PERC 6/i Integrated

-- Arrays informations --
-- ID | Type | Size | Status | InProgress
c0u0 | RAID5 | 279G | Optimal | Background Initialization: Completed 42%, Taken 46 min.

-- Disks informations
-- ID | Model | Status
c0u0p0 | SEAGATE ST3146855SS S5283LN6CLF9 | Online
c0u0p1 | SEAGATE ST3146855SS S5283LN6CNGM | Online
c0u0p2 | SEAGATE ST3146855SS S5283LN6CGW4 | Online

The same example but with second disk rebuilding:

server:~# megaclisas-status
-- Controller informations --
-- ID | Model
c0 | PERC 6/i Integrated

-- Arrays informations --
-- ID | Type | Size | Status | InProgress
c0u0 | RAID5 | 279G | Degraded | None

-- Disks informations
-- ID | Model | Status
c0u0p0 | SEAGATE ST3146855SS S5283LN6CLF9 | Online
c0u0p1 | SEAGATE ST3146855SS S5283LN6CNGM | Rebuild
c0u0p2 | SEAGATE ST3146855SS S5283LN6CGW4 | Online

There is at least one disk/array in a NOT OPTIMAL state.

A server with two controllers: The first one with a RAID1 array working fine. A second one with a RAID6 arrays of 7 drives with one offline (the array has just been created so it's under initilisation too).

server:~# megaclisas-status
-- Controller informations --
-- ID | Model
c0 | PERC 6/i Integrated
c1 | PERC 6/E Adapter

-- Arrays informations --
-- ID | Type | Size | Status | InProgress
c0u0 | RAID1 | 69G | Optimal | None
c1u0 | RAID6 | 3574G | Partially Degraded | Background Initialization: Completed 0%, Taken 2 min.

-- Disks informations
-- ID | Model | Status
c0u0p0 | SEAGATE ST373455SS S5283LQ44AGP | Online
c0u0p1 | SEAGATE ST373455SS S5283LQ44ELN | Online
c1u0p0 | SEAGATE ST3750630SS MS049QK1DQWD | Online
c1u0p1 | SEAGATE ST3750630SS MS049QK1DQT9 | Online
c1u0p2 | SEAGATE ST3750630SS MS049QK11NJY | Online
c1u0p3 | SEAGATE ST3750630SS MS049QK1DQM5 | Online
c1u0p4 | Unknown | Offline
c1u0p5 | SEAGATE ST3750630SS MS049QK1DQNV | Online
c1u0p6 | SEAGATE ST3750630SS MS049QK1DQX0 | Online

There is at least one disk/array in a NOT OPTIMAL state.

3.5. About /dev/megaraid_sas_ioctl_node

All theses tools requires this device node to be created.
It has to be done by hand.

Proprietary tools creates the device node at startup.
I made some wrappers around binaries from megactl package to create the node if it doesn't exist yet.

4. BIOS upgrade from a Linux system

Dell cards can be flashed using firmware-tools.
See ​http://linux.dell.com/wiki/index.php/Repository/firmware for more informations.
However this will only work on RedHat, CentOS, SuSE and Fedora. Even Ubuntu is listed in the wikipage, LSI card upgrade is not supported.
We use a Fedora 8 nfsroot booted by PXE to update our Dell's firmware running Debian.

We don't have any informations yet about upgrading non-Dell cards.