SAS and SATA HBA options for NAS and Servers

As a follow-up to my previous post, Flashing IT Firmware to the LSI SAS9211-8i HBA, I have been looking at other options for HBAs. Sometimes it might be appropriate to have an HBA that supports both internal and external drives or you might need support for more than eight drives. These are some additoinal solutions that should be considered.

All of the controllers mentioned here have been selected based on the need for drives to be used in some sort of software RAID solution such as Linux with md RAID or FreeBSD with ZFS. While there are other more advanced controllers with advanced hardware RAID support, those features would go completely unused in the intended use. To that end, using the IT firmware is preferred and all of these controllers support IT firmware. As I covered in the previously mentioned post, the process of flashing to IT firmware is a little more complicated but is worth the trouble.

Unless otherwise noted, the controllers can support up to 256 devices if used with an SAS expander. This is a good option but increases costs and only really makes sense if you plan on having a very large number of drives in your system.

LSI SAS9211-8i

The original HBA I was writing about was the LSI SAS9211-8i which supports eight internal drives through the use of two SFF-8087 internal connectors. These require breakout cables to directly support Serial ATA drives. The 3ware Multi-lane Internal SFF-8087 to four SATA cable seems to be the best value for the breakout cable. In the case of the SAS9211-8i, you would need two of these cables. I have been extremely happy with this card in my storage server (write-up coming soon) and its performance has been exceptional. Note that this controller is a PCI Express 2.0 x8 card so make sure you have an appropriate PCI Express slot.

Qty	Component		Cost		Link
1	LSI Logic SAS9211-8i 6Gb/s SAS/SATA PCIe x8 Card		$234.99		Amazon
2	3ware Multi-lane Internal SFF-8087 to 4 SATA Cable		$16.88		Amazon
	Total		$268.75
	Controller cost per drive		$33.59

LSI SAS9211-4i

As I covered in another post, High-End Mini-ITX NAS Server with ECC, another choice would be the LSI SAS9211-4i which is identical to the SAS9211-8i except that it only has a single SFF-8087 port and therefore supports only four Serial ATA hard drives. The same 3ware cable is needed but only one this time. This controller is a PCI Express 2.0 x4 card which is more common on many motherboards.

Qty	Component		Cost		Link
1	LSI Logic SAS9211-4i 6Gb/s SAS/SATA PCIe x4 Card		$167.99		Amazon
1	3ware Multi-lane Internal SFF-8087 to 4 SATA Cable		$16.88		Amazon
	Total		$184.87
	Controller cost per drive		$46.22

LSI SAS9201-16i

In the case that four or eight drives is not enough, another option is the LSI SAS9201-16i which is a similar controller to the other two but supports sixteen drives via four SFF-8087 ports which require four of the 3ware cables. I am considering this controller for my storage server since I already have eight drives installed and am thinking about adding at least three more drives. Like the SAS9211-8i, this card also uses a PCI Express 2.0 x8 slot but can support up to 512 devices using an SAS expander.

Qty	Component		Cost		Link
1	LSI Logic SAS9201-16i 6Gb/s SAS/SATA PCIe x8 Card		$410.05		Amazon
4	3ware Multi-lane Internal SFF-8087 to 4 SATA Cable		$16.88		Amazon
	Total		$477.57
	Controller cost per drive		$29.85

LSI SAS9212-4i4e

Another interesting option pointed out by John in the comments on a previous post is the LSI SAS9212-4i4e controller. This is a more unusual controller since it features four internal Serial ATA ports (not an SFF-8087 port) and one external SFF-8088 port to connect to an external enclosure or SAS expander. It can support up to 256 devices with an SAS expander like the 4i and 8i models. It also features a PCI Express 2.0 x8 interface.

The LSI9212-4i4e is available for $251.99 from Amazon but, in this case, there is an even better bonus. There is an identical but IBM-branded version of this card also available and it is $198.27 from Amazon (thanks to John for pointing this one out too). Saving a little over $50 is definitely worth it since the card is identical in every way. This is not an unusual situation and happens frequently especially with IBM and Intel selling their own versions of LSI cards.

Other Options

There are, of course, other options out there. I particularly like LSI controllers due to their optional IT firmware rather than being stuck with IR firmware. Generally speaking a NAS design is all contained in one computer which makes an internal controller preferable. There are other excellent LSI controllers that are designed to only use external enclosures using SFF-8088 ports. These include the LSI SAS9201-16e, a PCIe x8 card with four SFF-8088 ports for sixteen external drives without SAS expanders and 512 devices with SAS expanders. Another option is the LSI SAS9205-8e, a PCIe x8 card with two external SFF-8088 ports for eight external drives unless used with an SAS expander where it can support an amazing 1024 devices.

There are a few other controllers but these HBAs cover the best options for nearly all situations where hardware RAID is not a priority. Some of those mentioned also support RAID 0, 1, 1E, or 10 but with the IR firmware. If you are planning to use hardware RAID, it is probably better to look at other options that include on-card battery backup and additional cache memory.

Conclusion

Of the cards mentioned, the most intriguing to me is the IBM-branded version of the LSI SAS9212-4i4e. I am considering adding this card to my storage server since it would provide the additional four internal ports I need while also providing an external SFF-8088 port for an enclosure. I have not made a firm decision yet.

Since I already have the SAS9211-8i, I might either replace it with a SAS9201-16i, add an SAS9211-8i, or add the IBM-branded SAS9212-4i4e. The other temporary alternative is to use some of the five currently-unused Serial ATA 3Gb/s ports on the motherboard. These are slower (higher-end PCIe controllers seem to be faster than motherboard ports) but might do the job for now.

Flashing IT Firmware to the LSI SAS9211-8i HBA

In the process of upgrading my storage server, I wanted to upgrade the firmware on my recently acquired LSI SAS9211-8i SAS/SATA 6Gb/s PCI Express card to the latest IT version rather than the IR version it ships with. Apparently this process is a little bit easier on older 3Gb/s cards but not quite as simple on the 6Gb/s models. In addition, things get complicated with newer motherboards as well. Here are the steps to switch from IR to IT firmware.

For those of you that have not read up on the two firmware types, the IR Firmware is the Integrated RAID firmware. This is the typical hardware RAID firmware that the internal cards ship with. The IT firmware, on the other hand, is the Initiator Target firmware which is designed to allow an SAS HBA to connect to up to 256 devices uses SAS Expanders. The IT firmware is typically shipped on external cards. The IT firmware does not have any RAID capabilities and simply acts as an HBA. The reason I wanted the IT firmware was so that I would have no IR firmware to worry about since I was going to be using Linux md software RAID anyway.

Before you do anything else, I would recommend that you remove any other HBAs from the system, especially LSI HBAs. It is possible to specify which controller you want to flash but it is safer to just have a single controller in the system.

The first step in this process is to download the appropriate files from LSI Logic's web site. These are located in the support downloads section for the SAS9211-8i. It would be nice if the needed files were clearly explained but, sadly, that is not the case. Although Windows will never touch this system, you first need to download the Windows and MSDOS package because that is where the actual firmware files are available. I am downloading the latest Phase 12 firmware in order to have future support for larger than 2TB hard drives. LSI knowledge base article 16399 lists what versions are required for larger than 2TB support. The name of this package is:

9211_8i_Package_P12_IR_IT_Firmware_BIOS_for_MSDOS_Windows.zip

Now that we have the actual firmware, we need a way to install it. I initially downloaded the Installer P12 for Linux package. The included sas2flash utility was easily able to upgrade the firmware on the SAS9211-8i to the latest IR version but threw an error when I tried to flash with IT firmware.

Some searching ensued which led me to several LSI knowledge base articles. The first, number 16266, is entitled Flashing IT firmware over IR firmware (and vice versa) for SAS HBAs. In summary, the article suggests that I need to create a DOS bootable USB flash drive or similar, copy the appropriate firmware and sas2flsh DOS utility to the drive and update that way. The important information is the command sequence needed for a 6Gb/s HBA.

sas2flsh -o -e 6

This command puts the sas2flsh command into advanced mode (-o) and then erases the flash (-e 6) firmware and BIOS. Apparently rebooting at this point would cause the HBA to cease functioning as the article warns:

DO NOT REBOOT. If you do reboot, or if you attempt to flash the firmware and/or BIOS image and it does not flash correctly, you will have to RMA the controller.

After erasing the flash, you need both the IT firmware and the BIOS file. These are both in the previously mentioned archive for Windows and MSDOS. The relevant files are 2118it.bin and mptsas2.rom. The next command actually does the flashing.

sas2flsh -o -f 2118it.bin -b mptsas2.rom

After flashing is complete you can verify that version 12 firmware is loaded with the command:

sas2flsh -listall

This process seemed simple enough. I reasoned that perhaps the article uses a DOS bootable USB drive since that is the way most people will do the flashing but the Linux sas2flash program should work fine as well. This was not the case. Every time I attempted to erase the flash I would get back an error that the erase had failed.

Perhaps using DOS was important and so I proceeded with those instructions. I created the DOS bootable USB drive according to the recommendations in article 16266. Using bootflashdos as suggested apparently requires a 2GB or less USB drive probably due to FAT limitations. I simply used a 2GB SD card in a USB SD card reader. Once bootflashdos was done I copied sas2flsh, 2118it.bin, and mptsas2.rom to the SD card.

I booted from the SD card and attempted to run sas2flsh but was met with an error: "ERROR: Failed to initialize PAL. Exiting program." Fortunately, article 16266 mentions the error at the end and directs you to article 15807. The gist of article 15807 is that Intel 3000 or 5000-based boards have optionROM limitations which cause the above error. The easiest solution is to use a different system. Alternately, you can use the UEFI installer.

At this point, I have already tried using Linux sas2flash and DOS sas2flsh and and now I have to get another version for UEFI. I went back to the LSI site and downloaded the Installer P12 for UEFI archive. This provided the sas2flash.efi program. I copied this program to the same DOS bootable USB drive and proceeded to reboot my system.

For some reason on my board, the SuperMicro X9SCL+-F, the video output seems to come on a little late in the boot process so I can never seem to see the key commands for getting into the BIOS or bringing up the Boot Menu. This is worse when an HBA is installed that also has its own BIOS messages. Through a little bit of trial and error I figured out that F11 brings up the Boot Menu. I selected the option for UEFI Shell and was dropped into an unfamiliar command line interface.

Through more trial and error and prodigious use of the help command, I figured out that the map command lists the device tree. The names are a little complex but it looked to me like the device aliased to fs1: was the USB flash drive. I guessed correctly. In order to access fs1:, I first had to mount it.

mount fs1:

...seemed to do the trick. I then used the command...

fs1:

...rather than the more logical use of the cd fs1: command which, of course, failed. Using the _ls_ command gave me a listing of the files in the directory of the USB drive. The command I needed was sas2flash.efi. Running through the same series of commands from above...

sas2flash.efi -o -e 6
sas2flash.efi -o -f 2118it.bin -b mptsas2.rom

...resulted in a successful flash of the HBA. Finally this process was complete. I rebooted the system and accessed the BIOS interface with Ctrl-C. This confirmed that the IT firmware was loaded.

This process was much more complex than I had anticipated. Although I recognize that every scenario cannot be documented clearly and in order, it would have been nice if the LSI site had made this process a little simpler. In particular, it was very irritating that the downloads page does not list files by date under their respective sections. It would seem logical to have all the latest versions of the installer at the top of the list and then older versions in order going down. Instead the files are arranged haphazardly in no particular order but loosely grouped by intended system.

The documentation provided with the various installer files was fairly complete but should have had better examples of a simple flash operation. There are a myriad of flags to the various versions of the flashing program but only a few are needed in normal operation. I had to resort to searching the knowledge base to get a clear idea of what flags were needed.

Once it was clear what flags and that the UEFI version of the installer was needed, it was a painless process. Getting to that point was a much longer process that it should have been.

Update: I have also written another post detailing other similar controllers that are also capable of being used with IT firmware.

Xserve G5 and WD2003FYYS 2TB Drives

Over the past couple of weeks, I have been working with a school on getting a 2GHz Xserve G5 up and running to replace a PowerMac G5 operating as a server. The Xserve had no memory so we acquired 4GB of memory. In early 2011, I had upgraded the PowerMac G5 to dual Western Digital WD2003FYYS 2TB drives in a RAID 1 configuration.<! data-preserve-html-node="true"--more-->

All of this had been working perfectly but then the Xserve G5 was donated to the school and it was time to move the drives. After the memory arrived I installed it and attempted to just move the drives over. I had also initially setup the RAID 1 using SoftRAID (an excellent product by the way).

When I attempted to boot the Xserve with the 2TB drives, it just refused to boot and flashed the infamous folder error meaning it cannot find a boot drive. Although unlikely, I thought perhaps SoftRAID was preventing the drives from booting in the Xserve G5 and so I used SoftRAID to convert the on-disk format to Apple RAID.

I was also unable to boot from the Mac OS X 10.5 boot DVD. After some additional research, it occurred to me that the Xserve only had a CD drive and not a DVD drive. With that discovery, I decided to create a bootable FireWire drive with Mac OS X 10.5.8 as a means of booting the system.

Using the FireWire drive worked perfectly and allowed me to boot the system and get a better idea of what could be keeping the 2TB drives from being recognized. Oddly, one of the 2TB drives would finally mount in the Xserve after a considerable delay but the second drive would only mount after a long wait and only once in every ten reboots or so. If the second drive ever mounted, its paritions would never be properly recognized as part of a RAID set.

I started digging for more information about what could be causing this very strange behavior. Contrary to what some other users experienced, the drive trays did not show an orange light but instead showed a normal green light but only one drive would show any activity as indicated by the blue light.

After some searching and a great deal of trial and error, I came across a post on the Apple discussion forums detailing how another user had the very same WD2003FYYS drives and was able to get them to work perfectly by putting a jumper on pins 5 and 6 which causes the drive to only operate at 1.5Gb/s instead of the typical SATA II speed of 3.0Gb/s.

Once I set these jumpers, the drives came right up. Both RAID sets showed as degraded even though each RAID set had two partitions as expected. Disk Utility showed no further information so I moved to the Terminal. I ran:

diskutil listRAID

The output showed me that each RAID set had three partitions and one was marked as missing. I used the following command to remove the extra parition:

diskutil removeFromRAID (replace with missing partition's UUID) disk3

Obviously "replace with partition's UUID" refers to the UUID of the extra partition. The RAID set itself will also have a disk reference such as disk3, disk4, etc. In this case, both disk3 and disk4 are RAID sets made up of two partitions, one from each 2TB drive. This allows a separate boot RAID set and data RAID set.

Once I had removed the extra partitions from each set, the RAID sets immediately changed to online from the previous degraded state. This whole process was more complex than I expected but finally works as expected.

If you are facing a similar upgrade, I have read in a variety of posts that Seagate drives seem to work without issues while Western Digital drives tend to require the 1.5Gb/s jumper workaround. Hopefully I will not run into this exact problem again, but if I do, it should be an easy fix.

Build Your Own 8 Hard Drive Storage Array

If you work in a multimedia field or have very large collections of video files of some sort, large storage arrays become necessary. There are several ways to go about building a multimedia storage array. In this article, we explore one of the possible solutions. We are all waiting for the wonderful new Thunderbolt arrays but they will be expensive and likely will not be available until fall. In the mean time, this is a great eight drive array for a great price.

Connectivity

First off, we need to address connectivity. What type of interface should you be using and what can you use? The answer to this question depends on what system you are using to connect to your storage array. If you are using a Mac Pro or other system with PCIe slots, you have the best options. The fastest readily available way of connecting to an array is eSATA which allows for external SATA II 3Gb/s connections. In addition, some interface cards offer the option of eSATA Port Multiplier support which essentially allows up to five SATA drives in a Port Multiplier-aware enclosure to connect through one cable. This works great but does impose a bottleneck for the drives at somewhere around 300MB/s available for all the drives running through that port. For this article we are only focusing on an eSATA enclosure that support Port Multiplier technology.

For bulk storage a bottleneck of 300MB/s is not all that critical but for day-to-day video editing or other high performance needs it is not ideal. The advent of new SATA III interface cards which operate at 6Gb/s will start to solve this bottleneck and allow upwards of 600MB/s throughput but requires enclosures and drives to support 6Gb/s as well.

As a reference point, USB 2.0 operates at 480Mb/s which works out to be 40MB/s or so at its maximum. FireWire 800 can sustain about double that speed at 80MB/s or so. When talking about those connectivity methods 300MB/s eSATA sounds pretty good. Once again, for most uses it is more than adequate.

Enclosure and Drives

In a market where external drive enclosures tend to be expensive, one company has separated itself from the pack with great value and high quality products. Those are the products of Sans Digital. They have a substantial line of hard drive enclosures. We will focus on a few of their products that fit our needs the best.

Since we are building a storage array, we need space for a number of drives. In our case we want to be able to use eight SATA drives. This brings us to the Sans Digital TR8M eSATA Port Multiplier enclosure for eight SATA drives.

As you can see from the picture there are essentially eight drive sleds that slide into the enclosure. At the back of the enclosure are two eSATA ports that correspond to four drives each using SATA Port Multiplier technology. We are currently using this enclosure with eight Hitachi 1TB drives which are currently $60.48. At $0.06 per gigabyte they are a good deal for 7200RPM SATA drives but larger 2TB drives reach a better value.

The current best value for bulk storage is the Hitachi 2TB CoolSpin 5400RPM drive for $79.99 ($0.04/GB). This drive is slower than the Hitachi 2TB 7200RPM drive but those go for $116.34 right now ($0.058/GB). Another option would be the Hitachi 3TB CoolSpin 5400RPM drive drive at $149.99 ($0.05/GB) or the faster Hitachi 3TB 7200RPM drive at $174.99 ($0.058/GB).

Interface Card

In addition to the enclosure and eight hard drives, you will want a fast interface card that supports SATA Port Multiplier technology. By far the best interface card I have used is the Sonnet E4p 4-port eSATA Port Multiplier-aware PCIe x4 card .

This card works great in a Mac Pro and a PC as well. The Sonnet cards in particular seem to get excellent performance with their Marvell chipset. Because of its SATA Port Multiplier support this card actually supports 20 drives if you have the proper enclosures.

RAID

The final part of the setup once you have your enclosure, all eight drives, and your interface card is to create your RAID array. In Mac OS X, RAID 0 (striping) and RAID 1 (mirroring) are supported in Disk Utility along with combinations of those such as RAID 10 (stripe of mirrors) and RAID 0+1 (mirror of stripes). Windows also offers software RAID capabilities but we will not address those capabilities in this article. Essentially creating a RAID array allows you to combine the storage space of the drives in different ways to have larger volumes in which to storage your data.

It is important to remember that you do not have to buy all eight drives at once. You can add drives in groups. For example, if you know you need storage now but only need 2TB you could buy two 2TB drives and create a RAID 1 (mirroring) or buy two 1TB drives and create a RAID 0 (striping). In addition, you do not have to use a RAID array at all. It is perfectly possible to leave all eight drives as separate drives and copy data to them individually. You can also mix and match different RAID arrays out of a selection of the drives. We are going to focus on creating a single storage space.

In brief, a two drive RAID 1 creates an identical mirror of your data on two physical drives. RAID 1 is designed to protect you against the failure of a drive. On the other hand, a two drive RAID 0 stripes chunks of your data over both drives. RAID 0 spreads your data out across both drives so accessing it becomes faster but at the cost of protection. If one drive fails everything is gone since most files are spread between both disks.

The combination or nested RAID levels as they are called provide additional protection and performance. RAID 10 or 1+0 is created by taking sets of RAID 1 mirrored drives and striping those together. Let’s say you have four drives. The first two drives would be in a RAID 1 and the second two drives would also be in a RAID 1. Then you would create a RAID 0 with two “drives” that happen to be the two individual RAID 1 sets. This provides the protection of RAID 1 since one drive out of each RAID 1 set can fail but provides the increased performance of RAID 0 by striping the two RAID 1 sets together.

RAID 0+1 is similar to RAID 10 but uses the opposite combination of RAID levels. In this case with four drives you would create two separate RAID 0 arrays and then create a RAID 1 array out of the RAID 0 arrays. The disadvantage in RAID 0+1 is that a single drive cannot fail in both RAID 0 arrays or all data is lost. RAID 10 has the advantage here.

Conclusion

We are currently using a TR8M enclosure with eight 1TB drives in a RAID 10 array which provides 4TB of usable space. We are using this array for backups right now which is why it is setup with a RAID 10 rather than a RAID 0 which would provide 8TB of space. We have been extremely happy with this array and the next step is upgrading it to 2TB or 3TB drives for even more storage. Whether you choose to use RAID 1, RAID 0, RAID 10, or some combination of individual drives and RAID levels, this setup is a very solid one that works great and provides excellent performance and value.

Seagate Chasing iPad Users with GoFlex Satellite

Today, Seagate announced the new GoFlex Satellite Mobile Wireless Storage device. This device is specifically designed for iPad users who want more space to store media. It is quite an innovative product and quick intriguing. What is it and why it is so innovative?

The GoFlex Satellite Mobile Wireless Storage System is a 500GB hard drive that also creates an 802.11n Wi-Fi network and has a rechargeable battery that is said to last up to 5 hours. A user loads media by means of USB 3.0/2.0 from your Mac or PC similar to a regular hard drive. After the media is synced to the GoFlex Satellite, you can access it from your iPad or iPhone with Seagate's GoFlex Media app.

Although the video Seagate released only shows the iPad and other marketing materials also emphasize the iPad, the system requirements state that it is compatible with "iPad, smartphone, tablet or computer with a web browser and Wi-Fi connectivity (802.11 b/g/n)" for viewing the media stored on the device.

The specs say that you can stream media over Wi-Fi to three iPads at once. This might be a good solution for families needing more storage.

Forking over money for a 64GB iPad can seem daunting particularly when more than one is purchased for a family. I have owned a 64GB iPad and it really does not hold as much HD video content as you would think. The GoFlex Satellite might just solve this problem. Being able to carry 500GB of video content sure would be nice for long trips or being able to have a nice selection of options to watch.

The GoFlex Satellite 500GB will be available in July for $199.99 and is currently available for pre-order from Amazon.com. The full press release is available from Seagate's web site. Seagate also has a product page up as well.