I came across a puzzling dilemma today as I was provisioning some Sun Fire V120 servers for some tests. Although somewhat old now, a 650MHz Sun Fire V120 with 3GB of memory does just fine for a lot of tasks like DNS, some basic mail processing, and static web content. One of the dilemmas I have faced in the past was running two disks in a RAID mirror. OpenBSD has had softraid(4) for quite a while but OpenBSD 5.1 adds support for the root filesystem on softraid(4) for amd64, i386, and sparc64 systems. There was only one problem.
I found some excellent documentation for amd64 and i386 in the form of Josh Grosse's Provisioning Root on Softraid article at the OpenBSD Journal. This except from the article shows how you would do the installboot "magic" needed for root on softraid(4) to work.
# /usr/mdec/installboot -v /boot /usr/mdec/biosboot sd0
boot: /boot proto: /usr/mdec/biosboot device: /dev/rsd0c
sd0: softraid volume with 2 disk(s)
sd0: installing boot loader on softraid volume
/boot is 3 blocks x 16384 bytes
wd0d: installing boot blocks on /dev/rwd0c, part offset 209744
master boot record (MBR) at sector 0
partition 3: type 0xA6 offset 64 size 4193216
/boot will be written at sector 64
wd1d: installing boot blocks on /dev/rwd1c, part offset 209744
master boot record (MBR) at sector 0
partition 3: type 0xA6 offset 64 size 4193216
/boot will be written at sector 64The only problem is that sparc64 is a single-stage boot environment whereas amd64 and i386 are two-stage boot environments. As Josh Grosse stated, "I never tested root-on-softraid on single-stage booting architectures. I don't have any handy. You'll have to test that yourself!"
Unfortunately, there was no follow-up to this mailing list post. Since OpenBSD is known for its excellent man pages, I thought looking there might be a good idea. The installboot(8) man page for sparc64 gave the following example.
# cp /usr/mdec/ofwboot /ofwboot
# /usr/mdec/installboot /usr/mdec/bootblk sd0aThe problem was, rather than using the hardware device name like sd0, sparc64 uses a partition name, in this case sd0a. This goes back to sparc64 being a single-stage boot environment rather than a two-stage boot environment like amd64 and i386. After trying various incarnations of this command with sd0a and sd2a, I finally figured out the right command. You really will want to use the -v flag for installboot to see what's going on. Otherwise, you will be in the dark.
# /usr/mdec/installboot -v /usr/mdec/bootblk sd2
bootblk: /usr/mdec/bootblk
device: /dev/rsd2c
sd2: softraid volume with 2 disk(s)
sd2: installing boot loader on softraid volume
sd0d: installing boot blocks on /dev/rsd0c
sd1d: installing boot blocks on /dev/rsd1cOf course the device names such as sd2 will vary based on your system. My Sun Fire V120 has two 36GB SCSI disks which show up as sd0 and sd1 so when I created my softraid(4) volume it shows up as sd2. If you have two IDE disks like a Sun Fire V120 I have, then sd0 will likely be your softraid(4) volume and wd0 and wd1 will be your two disks.
The Install Process
What I did to actually create the softraid(4) came straight out of Provisioning Root on Softraid. I booted up using the bsd.rd RAMDISK kernel and select s for (S)hell at the installation prompt. In the shell, I followed a little different sequence since this is a sparc64 system, not amd64 or i386.
# disklabel -E sd0
Label editor (enter '?' for help at any prompt)
> p
OpenBSD area: 0-71132959; size: 71132959; free: 71132959
# size offset fstype [fsize bsize cpg]
c: 71132959 0 unused
> a a
offset: [0]
size: [71132959] 100m
Rounding size to cylinder (2889 sectors): 205119
FS type: [4.2BSD]
> a b
offset: [205119]
size: [70927840] 3g
Rounding size to cylinder (2889 sectors): 6292242
FS type: [swap]
> a d
offset: [6497361]
size: [64635598]
Rounding size to cylinder (2889 sectors): 64635597
FS type: [4.2BSD] raid
> p
OpenBSD area: 0-71132959; size: 71132959; free: 1
# size offset fstype [fsize bsize cpg]
a: 205119 0 4.2BSD 2048 16384 1
b: 6292242 205119 swap
c: 71132959 0 unused
d: 64635597 6497361 RAID
> q
Write new label?: [y]
# disklabel sd0 > protofile
# disklabel -R sd1 protofile
# bioctl -c 1 -l sd0d,sd1d softraid0
sd2 at scsibus4 targ 1 lun 0: SCSI2 0/direct fixed
sd2: 31560MB, 512 bytes/sector, 64635069 sectorsYour numbers will likely be somewhat different but this is the main idea. The second p prints out what your disklabel should look like with an a partition to load your kernels from, b for swap, and d for softraid(4).
After this, type exit to get back to the installation menu and select I for (I)nstall. Continue through the installation and select sd2 or whatever your softraid(4) disk is called when partitioning. Leave sd0 and sd1 alone in the installer. When the install finishes, format sd0a and sd1a
# newfs /dev/rsd0a
# newfs /dev/rsd1aBy default, the installer ramdisk has /mnt and /mnt2 mount points. You will need a third mount point.
# mount /dev/sd0a /mnt2
# mkdir /mnt3
# mount /dev/sd1a /mnt3As long as sd2a, your softraid(4) root, is still mounted on /mnt, you need to do a couple of steps.
# cd /mnt
# cp bsd* /mnt2/
# cp bsd* /mnt3/
# cp ofwboot /mnt2/
# cp ofwboot /mnt3/For OpenBSD/sparc64 to boot, it is essential that ofwboot be in your sd0a and sd1a filesystems. I found this out the hard way as well. With all of this done, use installboot to finish your install.
# /mnt/usr/mdec/installboot -v /mnt/usr/mdec/bootblk sd2
bootblk: /usr/mdec/bootblk
device: /dev/rsd2c
sd2: softraid volume with 2 disk(s)
sd2: installing boot loader on softraid volume
sd0d: installing boot blocks on /dev/rsd0c
sd1d: installing boot blocks on /dev/rsd1cWith that, you're done with the basic install. One other note, you will want to add your swap partitions to your fstab since they will not be automatically picked up. With that done, my fstab looks like this.
# cat /etc/fstab
ed82495b1ef94968.a / ffs rw,softdep 1 1
fd22a39d1637d863.b none swap sw
68396a680d88772f.b none swap swAs you can see, I am using DUIDs rather than disk references like sd2a. The first entry is my softraid(4) root partition, sd2a on this system. The second two entries are the sd0b and sd1b swap partitions. If you're wondering how to find the DUIDs, this will get you the information for sd0.
# disklabel sd0 | grep duid
duid: fd22a39d1637d863With the install fully done and booting to root on softraid(4), it's time to start installing the other software I need.