Slice 1 swap 8.00GB # swap space
Slice 2 backup 146.35GB # whole disk
Slice 3 var 10.00GB # /var
Slice 4 unassigned 8.00GB # dump device
Slice 5 unassigned 0
Slice 6 unassigned 90.00GB # soft partitions mirror
Slice 7 unassigned 34.78MB # SVM state database
- Configure networking on the server as appropriate.
- Configure the network-based Service Processor for future administration. This is Highly Recommended.
Disk Configuration
We use Solaris Volume Manager (previously Solstice) to provide redunancy and management for our system. For this description, the first disk (primary mirror) is c1t0d0 and the second disk (secondary mirror) is c1t1d0.
- Duplicate the slice layout of the boot disk on the drive that will be its mirror:
# prtvtoc /dev/rdsk/c1t0d0s2 fmthard -s - /dev/rdsk/c1t1d0s2
- Configure the new dedicated dump device. This is done so that we don't have to depend on a working swap mirror to retrieve a kernel core dump:
# dumpadm -d /dev/dsk/c1t0d0s4
- Initialize the state database replicas for Solaris Volume Manager. This gives us four state replicas, two per disk:
# metadb -a -f -c 2 c1t0d0s7 # metadb -a -c 2 c1t1d0s7
- Create the /, swap, and /var volumes as follows:
# metainit -f d10 1 1 c1t0d0s0
# metainit -f d11 1 1 c1t0d0s1
# metainit -f d13 1 1 c1t0d0s3
# metainit d20 1 1 c1t1d0s0
# metainit d21 1 1 c1t1d0s1
# metainit d23 1 1 c1t1d0s3
# metainit d0 -m d10
# metainit d1 -m d11
# metainit d3 -m d13
- Set up the system to boot from the mirror: # metaroot d0
- Change the entries for swap and /var in /etc/vfstab to point to their new locations (/dev/md/dsk/d1, /dev/md/dsk/d3). Don't forget to change the /dev/rdsk entries to /dev/md/rdsk as well.
- Write down the name of the second disk in the mirror indicated in bold below:
# ls -l /dev/rdsk/c1t1d0s0 lrwxrwxrwx 1 root root 47 Feb 22 11:38 /dev/rdsk/c1t1d0s0 ->devices/pci@1c,600000/scsi@2/sd@1,0:a,raw
# init 0
- Create an OpenBoot PROM alias for the bootable disks. Some controllers require you to replace sd with disk; you can verify this with the show-disks command before running the nvalias commands. For the V210 and the V240, it appears we should make this substitution. Also, remove ,raw from the end of the device name. You can also look at the output of devalias for examples; in particular, look at disk0 and disk1.
ok nvalias bootdisk /pci@1c,600000/scsi@2/disk@0,0:a
ok nvalias mirrdisk /pci@1c,600000/scsi@2/disk@1,0:a
ok setenv boot-device bootdisk mirrdisk
ok boot
Once booted, attach the submirrors to complete the mirroring:
# metattach d0 d20
# metattach d1 d21
# metattach d3 d23
* Mirror the remaining free space on the disks for future soft partitions. Later, you can allocate soft partitions from d6 for whatever you like:
# metainit d16 1 1 c1t0d0s6
# metainit d26 1 1 c1t1d0s6
# metainit d6 -m d16
# metattach d6 d26
- For example, to allocate a 10g soft partition that would be mounted at /data:
# metainit d100 -p d6 10g
# newfs -m 0 /dev/md/dsk/d100
# mkdir /data
- Then, edit /etc/vfstab to include the new mountpoint and set it to mount during boot.
Patching
Now we will install the Solaris 9 Recommended and Security patches, last updated Feb 21, 2007. Download 9_Recommended_Security-20070221.zip. Once it's on the system, run the following commands to install:
# unzip -d 9_Recommended_Security-20070221 9_Recommended_Security-20070221.zip ... unzip output ...
# cd 9_Recommended_Security-20070221/9_Recommended
# ./install_cluster
Answer `y' to start installing patches.
Many of the patches will fail to install, but this is (usually) because the patch has already been applied or is not needed on the system. Check the patch log at /var/sadm/install_data/Solaris_9_Recommended_Patch_Cluster_log for details of the process.
FC3510 Setup
Here we install the software required to access the FC3510 storage array, and then we configure it.
SAN 4.4.12 Installation
Next we install the SAN software, which includes drivers for the Fibre Channel host adapter.
Download SAN 4.4.12: SAN_4.4.12_install_it.tar.Z. Once it's on the system, run the following commands to install:
# zcat SAN_4.4.12_install_it.tar.Z tar xvf - ... tar output ...
# cd cd SAN_4.4.12_install_it
# ./install_it
Answer `y' to start installing the software.
If the installation succeeds, reboot the system:
# shutdown -y -i6 -g0
It would be wise to watch the system console while it reboots in case errors appear during boot related to the new patches.
StorEdge 3000 Family Software Installation
The Sun StorEdge 3000 Family Software includes the command line utility (sccli) used to manage the external array.
Download the Sun StorEdge 3000 Family Software: 2.2_sw_solaris-sparc.zip and2.3_smis_provider.zip. Once on the system, run the following commands to install:
# unzip -d 2.2_sw_solaris-sparc 2.2_sw_solaris-sparc.zip ... unzip output ...
# pkgadd -d 2.2_sw_solaris-sparc/solaris/sparc Answer `all' to install all packages, and then `y' to any questions. # unzip -d 2.3_smis_provider 2.3_smis_provider.zip ... unzip output ... # pkgadd -d 2.3_smis_provider Answer `all' to install all packages, and then `y' to any questions.
FC3510 Firmware Upgrade
Now we will upgrade the firmware on the FC3510.
Download patch 113723-15: 113723-15.zip. Decompress the patch with unzip and read the section entitled "Patch Installation Instructions" inside of README.113723-15. This README documents the upgrade steps better than I could do here.
Array Configuration
We need to delete any existing LUN mappings, and then delete the logical drives themselves. Run sccli to enter the configuration tool. It should connect to the 3510 automatically.
First, display the current LUN mappings (this is an example and may not match what you see): sccli> show lun-maps
Ch Tgt LUN ld/lv ID-Partition Assigned Filter Map
-----------------------------------------------------------------0 40 0 ld0 1A6C4238-00 Primary
For each LUN mapping, run unmap Ch.Tgt.LUN. Do this starting with the highest numbered LUNs and work your way down to 0. For example, using the above output:
sccli> unmap 0.40.0
sccli>
Exit sccli and run the following:
# devfsadm -Cv ... output regarding device changes, if any ...
Now restart sccli, and display the logical drives:
sccli> show logical-drives
LD LD-ID Size Assigned Type Disks Spare Failed Status ------------------------------------------------------------------------
ld0 1A6C4238 58.59GB Primary RAID0 2 0 0 Good Write-Policy Default StripeSize 128KB
And delete each logical drive by running delete logical-drive LD for each logical drive. For example:
sccli> delete logical-drive ld0
This operation will result in the loss of all data on the logical drive.
Are you sure? y
sccli: ld0: deleted logical drive
Now we have a clean array, ready for a new configuration. We have 6 73GB drives in each array, we will configure 5 of the disks into a RAID5 configuration with a spare sixth disk. This should give us about 292GB of usable storage. Note the following examples were done using only 5 disks, so the size numbers will be less than a 6 disk system. Make sure that your configuration uses all but 1 disk for the RAID5 array and the last disk for a spare.
First, we configure SCSI channel 0. This setting should be the same as the default config, but we'll make sure:
sccli> configure channel 0 host pid 40 --reset
sccli: shutting down controller...
sccli: controller is shut down
sccli: resetting controller... sccli: controller has been reset
Now we will set the cache parameters for the array. Since this is a single-controller array, we need to make sure we're using write-through caching, as write-back caching is dangerous without a redundant controller. Also, we set the array to optimize for random access:
sccli> set cache-parameters random write-through
Changes will not take effect until controller is reset
Do you want to reset the controller now? y
sccli: resetting controller...
sccli: controller has been reset
Now we're ready to create our logical disk. Type the following commands into sccli to display the disks in the system, configure a RAID5 logical disk, and configure a global spare drive. Remember, make sure that your configuration uses all but 1 disk for the RAID5 array and the last disk for a spare:
sccli> show disks
Ch Id Size Speed LD Status IDs
Rev ----------------------------------------------------------------------------
2(3) 0 68.37GB 200MB NONE FRMT FUJITSU MAT3073F SUN72G 0602 S/N000513B02RF7 WWNN
500000E010FC3EF0
2(3) 1 68.37GB 200MB NONE FRMT FUJITSU MAT3073F SUN72G 0602 S/N 000512B02DYP WWNN 500000E010F8CF60
2(3) 2 68.37GB 200MB NONE FRMT FUJITSU MAT3073F SUN72G 0602 S/N 000512B02E3S WWNN 500000E010F8D410
2(3) 3 68.12GB 200MB NONE NEW FUJITSU MAT3073F SUN72G 0602 S/N 000513B02RN8 WWNN 500000E010FC8500
2(3) 4 68.37GB 200MB NONE FRMT FUJITSU MAT3073F SUN72G 0602 S/N 000514B02VRY WWNN 500000E010FE8100
sccli> create logical-drive raid5 2.0,2.1,2.2,2.3 primary global-spare 2.4
sccli> map ld0 0.40.0 sccli> show disks
Ch Id Size Speed LD Status IDs
Rev ----------------------------------------------------------------------------
2(3) 0 68.37GB 200MB ld0 ONLINE FUJITSU MAT3073F SUN72G 0602 S/N 000513B02RF7 WWNN 500000E010FC3EF0
2(3) 1 68.37GB 200MB ld0 ONLINE FUJITSU MAT3073F SUN72G 0602 S/N 000512B02DYP WWNN 500000E010F8CF60
2(3) 2 68.37GB 200MB ld0 ONLINE FUJITSU MAT3073F SUN72G 0602 S/N 000512B02E3S WWNN 500000E010F8D410
2(3) 3 68.37GB 200MB ld0 ONLINE FUJITSU MAT3073F SUN72G 0602 S/N 000513B02RN8 WWNN 500000E010FC8500
2(3) 4 68.37GB 200MB GLOBAL STAND-BY FUJITSU MAT3073F SUN72G 0602 S/N 000514B02VRY WWNN 500000E010FE8100
sccli> show logical-drives
LD LD-ID Size Assigned Type Disks Spare Failed Status ------------------------------------------------------------------------
ld0 7D1F7008 204.35GB Primary RAID5 4 1 0 Good I Write-Policy Default StripeSize 32KB
sccli> show map
We are now done with the array configuration. Exit =sccli= and run:
# devfsadm -Cv
... output regarding device changes, if any ...
Host Configuration
We will now configure the drive array for access from the host system.
Slice Setup
Now, when you run format, you should see the new device and should be able to configure it:
# format
Searching for disks...done
c2t40d0: configured with capacity of 204.34GB
AVAILABLE DISK SELECTIONS:
0. c1t0d0 /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000011c63f6cd7,0 1. c1t1d0 /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100000c50010967,0 2. c2t40d0 /pci@9,600000/SUNW,qlc@1,1/fp@0,0/ssd@w216000c0ff88655a,0 Specify disk (enter its number): 2
selecting c2t40d0
[disk formatted]
Disk not labeled. Label it now? y
... format menu, choose `p' and `p' again ...
And there's the disk, right on target 40 where we put it. You should now see the slices in the new LUN. Reconfigure the slices such that all space is given to the first slice (i.e., it matches the s2 slice). It should look something like this:
partition> p
Current partition table (unnamed):
Total disk cylinders available: 52723 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 - 52722 20 4.34GB (52723/0/0)
428532544
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 52722 204.34GB (52723/0/0)
428532544
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
label the disk and exit the format utility.
Solaris Volume Manager Setup
We need to add another set of SVM state replicas to the new disk and create an SVM submirror so we can allocate soft partitions:
# metadb -a -c 2 c2t40d0s0
# metainit d7 1 1 c2t40d0s0
At this point we should have two SVM devices suitable for soft partition allocation: d6 and d7. d6 has the extra space from the disks internal to the server, and d7 has the entire RAID5 array from the FC3510. For example, to allocate a 50GB soft partition from the RAID5 array (we'll call it d101), you would run the following command:
# metainit d101 -p d7 50g
You could then run newfs on /dev/md/dsk/d101 and otherwise treat it as a standard block device.
