Running Linux applications in Solaris Linux branded zones

Saturday, February 19, 2011 at 8:40 PM
While playing around with the latest version of Nevada this week, I decided to see how well Linux branded zones work. In case your not following the Sun development efforts, Linux branded zones allow you to run Linux ELF executables unmodified on Solaris hosts. This is pretty interesting, and I definitely wanted to take this technology for a test drive. After reading through the documentation in the brandz community, I BFU’ed my Nevada machine to the latest nightly build, and installed the packages listed on the brandz download page. Since brandz currently only supports CentOS 3.0 – 3.7 and the Linux 2.4 kernel series, I first had to download the three CentoS 3.7 iso images (branded zones currently don’t support CentOS 3.8 without some hacking):

$ cd /home/matty/CentOS

$ wget http://www.gtlib.gatech.edu/pub/centos/3.7/isos/i386/CentOS-3.7-i386-bin1of3.iso

$ wget http://www.gtlib.gatech.edu/pub/centos/3.7/isos/i386/CentOS-3.7-i386-bin2of3.iso

$ wget http://www.gtlib.gatech.edu/pub/centos/3.7/isos/i386/CentOS-3.7-i386-bin3of3.iso

After I retrieved the ISO images, I needed to create a branded zone. Creating Linux branded zones is a piece of cake, and is accomplished by running the zonecfg utility with the “-z” option and a name to assign to your zone, and then specifying one or more parameters inside the zone configuration shell. Here is the configuration I used with my test zone:

$ zonecfg -z centostest

centostest: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:centostest> create -t SUNWlx
zonecfg:centostest> add net
zonecfg:centostest:net> set physical=ni0
zonecfg:centostest:net> set address=192.168.1.25
zonecfg:centostest:net> end
zonecfg:centostest> set zonepath=/zones/centostest
zonecfg:centostest> set autoboot=true
zonecfg:centostest> verify
zonecfg:centostest> commit
zonecfg:centostest> exit
This zone configuration is pretty basic. It contains one network interface (when you boot the zone, a virtual interface is configured on that interface with the address passed to the address attribute), a location to store the zone data, and it is configured to automatically boot when the system is bootstrapped. Next I needed to install the CentOS binaries in the zone. To install the CentOS 3.7 binaries in the new zone I created, I ran the zoneadm utility with the ‘install’ option, and passed the directory with the CentOS ISO images as an argument:

$ zoneadm -z centostest install -v -d /home/matty/CentOS

Verbose output mode enabled.
Installing zone "centostest" at root "/zones/centostest"
Attempting ISO-based install from directory:
"/home/matty/CentOS"
Checking possible ISO
"/home/matty/CentOS/CentOS-3.7-i386-bin1of3.iso"...
added as lofi device "/dev/lofi/1"
Attempting mount of device "/dev/lofi/1"
on directory "/tmp/lxisos/iso.1"... succeeded.
Checking possible ISO
"/home/matty/CentOS/CentOS-3.7-i386-bin2of3.iso"...
added as lofi device "/dev/lofi/2"
Attempting mount of device "/dev/lofi/2"
on directory "/tmp/lxisos/iso.2"... succeeded.
Checking possible ISO
"/home/matty/CentOS/CentOS-3.7-i386-bin3of3.iso"...
added as lofi device "/dev/lofi/3"
Attempting mount of device "/dev/lofi/3"
on directory "/tmp/lxisos/iso.3"... succeeded.
Checking for distro "/usr/lib/brand/lx/distros/centos35.distro"...
Checking iso file mounted at "/tmp/lxisos/iso.1"...
read discinfo file "/tmp/lxisos/iso.1/.discinfo"
ISO "/tmp/lxisos/iso.1": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 1
Checking iso file mounted at "/tmp/lxisos/iso.2"...
read discinfo file "/tmp/lxisos/iso.2/.discinfo"
ISO "/tmp/lxisos/iso.2": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 2
Checking iso file mounted at "/tmp/lxisos/iso.3"...
read discinfo file "/tmp/lxisos/iso.3/.discinfo"
ISO "/tmp/lxisos/iso.3": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 3
Checking for distro "/usr/lib/brand/lx/distros/centos36.distro"...
Checking iso file mounted at "/tmp/lxisos/iso.1"...
read discinfo file "/tmp/lxisos/iso.1/.discinfo"
ISO "/tmp/lxisos/iso.1": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 1
Checking iso file mounted at "/tmp/lxisos/iso.2"...
read discinfo file "/tmp/lxisos/iso.2/.discinfo"
ISO "/tmp/lxisos/iso.2": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 2
Checking iso file mounted at "/tmp/lxisos/iso.3"...
read discinfo file "/tmp/lxisos/iso.3/.discinfo"
ISO "/tmp/lxisos/iso.3": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 3
Checking for distro "/usr/lib/brand/lx/distros/centos37.distro"...
Checking iso file mounted at "/tmp/lxisos/iso.1"...
read discinfo file "/tmp/lxisos/iso.1/.discinfo"
ISO "/tmp/lxisos/iso.1": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 1
Added ISO "/tmp/lxisos/iso.1" as disc 1
Checking iso file mounted at "/tmp/lxisos/iso.2"...
read discinfo file "/tmp/lxisos/iso.2/.discinfo"
ISO "/tmp/lxisos/iso.2": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 2
Added ISO "/tmp/lxisos/iso.2" as disc 2
Checking iso file mounted at "/tmp/lxisos/iso.3"...
read discinfo file "/tmp/lxisos/iso.3/.discinfo"
ISO "/tmp/lxisos/iso.3": Serial "1144177644.47"
Release "CentOS [Disc Set 1144177644.47]" Disc 3
Added ISO "/tmp/lxisos/iso.3" as disc 3
Installing distribution 'CentOS [Disc Set 1144177644.47]'...
Installing cluster 'desktop'
Installing zone miniroot.
Installing miniroot from ISO image 1 (of 3)
RPM source directory: "/tmp/lxisos/iso.1/RedHat/RPMS"
Attempting to expand 30 RPM names...
Installing RPM "SysVinit-2.85-4.4.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "basesystem-8.0-2.centos.0.noarch.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "bash-2.05b-41.5.centos.0.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "beecrypt-3.0.1-0.20030630.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "bzip2-libs-1.0.2-11.EL3.4.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "coreutils-4.5.3-28.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "elfutils-0.94-1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "elfutils-libelf-0.94-1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "filesystem-2.2.1-3.centos.1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "glibc-2.3.2-95.39.i586.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "glibc-common-2.3.2-95.39.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "gpm-1.19.3-27.2.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "initscripts-7.31.30.EL-1.centos.1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "iptables-1.2.8-12.3.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "iptables-ipv6-1.2.8-12.3.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "kernel-utils-2.4-8.37.14.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "laus-libs-0.1-70RHEL3.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "libacl-2.2.3-1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "libattr-2.2.0-1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "libgcc-3.2.3-54.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "libtermcap-2.0.8-35.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "ncurses-5.3-9.4.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "pam-0.75-67.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "popt-1.8.2-24_nonptl.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "rpm-4.2.3-24_nonptl.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "rpm-libs-4.2.3-24_nonptl.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "setup-2.5.27-1.noarch.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "termcap-11.0.1-17.1.noarch.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "zlib-1.1.4-8.1.i386.rpm" to miniroot at
"/zones/centostest"...
Installing RPM "centos-release-3-7.1.i386.rpm" to miniroot at
"/zones/centostest"...
Setting up the initial lx brand environment.
System configuration modifications complete!
Duplicating miniroot; this may take a few minutes...

Booting zone miniroot...
Miniroot zone setup complete.

Installing zone 'centostest' from ISO image 1.
RPM source directory: "/zones/centostest/root/iso/RedHat/RPMS"
Attempting to expand 667 RPM names...
Installing 433 RPM packages; this may take several minutes...

Preparing... ##################################################
libgcc ##################################################
setup ##################################################
filesystem ##################################################
hwdata ##################################################
redhat-menus ##################################################
mailcap ##################################################
XFree86-libs-data ##################################################
basesystem ##################################################
gnome-mime-data ##################################################

[.....]
After the brandz installer finished installing the CentOS 3.7 RPMs, I used the zoneadm ‘boot’ option to start the zone:

$ zoneadm -z centostest boot

To view the console output while the zone was booting, I immediately fired up the zlogin utility to console into the new Linux branded zone, and ran a few commands to see what the environment looked like after the zone was booted:

$ zlogin -C centostest

[Connected to zone 'centostest' console] [ OK ]
Activating swap partitions: [ OK ]
Checking filesystems [ OK ]
Mounting local filesystems: [ OK ]
Enabling swap space: [ OK ]
modprobe: Can't open dependencies file /lib/modules/2.4.21/modules.dep (No such file or directory)
INIT: Entering runlevel: 3
Entering non-interactive startup
Starting sysstat: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting automount: No Mountpoints Defined[ OK ]
Starting cups: [ OK ]
Starting sshd:[ OK ]
Starting crond: [ OK ]
Starting atd: [ OK ]
Rotating KDC list [ OK ]

CentOS release 3.7 (Final)
Kernel 2.4.21 on an i686

centostest login: root
$ uname -a

Linux centos 2.4.21 BrandZ fake linux i686 i686 i386 GNU/Linux
$ cat /proc/cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 6
model name : Intel Celeron(r)
stepping : 5
cpu MHz : 1662.136
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
flags : fpu pse tsc msr mce cx8 sep mtrr pge cmov mmx fxsr sse sse2 ss
Yum works swell in a branded zone, and most of the tools you typically use work out of the box. Linux branded zones are wicked cool, and I can see tons of uses for them. Some folks are dead set on running Linux instead of Solaris, which means they can’t take advantage of things like ZFS, FMA and DTrace. If you need to better understand your application and the way it interacts with the system, or if you want to take advantage of the stability the Solaris kernel brings to production system, you can fire up a branded zone and run your application transparently on a Solaris system.

Build a Guest Domain

at 8:30 PM
A guest domain is made up of the following components:


CPU

MAU (Cryptographic Thread)
Memory
Networking
Storage
The control domain will partition CPU threads as VCPU's for the guest domain. Each CPU core has an MAU for cryptographic processing. Only one logical domain using the CPU threads in a core can have control over this thread. So it's important to decide if your guest domain will require one. Memory is partitioned in 8K segments. Networking is handled by connecting a virtual network interface to a virtual switch in one of the service domains. In our example, I configured each physical interface as a separate virtual switch in the control/service (a.k.a. primary) domain. Storage can come from a wide variety of sources:


Local Disk

SAN LUN
Virtual Disk Image File
Disk Slice
ZFS Volume
The T2000 for example has four drive bay that could be used, but obviously that doesn't leave us with a lot of flexibility or space. SAN storage can be used with greater flexibility since it's remote and can easily be migrated or replicated. It's possible to create a sparse file and use it as a virtual disk. This has the advantage of being stored on local, SAN, or even NAS. The fact that files can be used opens the door for very flexible options. Using a disk slice is also possible, but it can not be used for jumpstart installation. One could create ZFS volumes and use them as storage for logical domains as well. However, it can not be used for jumpstart installation. However, it makes for easy allocation of storage for applications. You can even take SAN LUN's and create a ZFS pool and export it into a logical domain. For our example, I'll use two virtual disk image files created on a ZFS file system and use SVM mirroring:) The following will be configured:


4 x VCPU's
1 x MAU
4GB's RAM

2 x 10GB Virtual Disk Image Files
2 x Network Ports


# ldm add-domain ldom1
# ldm add-vcpu 4 ldom1
# ldm add-mau 1 ldom1
# ldm add-memory 4G ldom1
# mkfile 10g /ldoms/vdisk1_10gb.img
# mkfile 10g /ldoms/vdisk2_10gb.img
# ldm add-vdiskserverdevice /ldoms/vdisk1_10gb.img vdisk1@primary-vds0
# ldm add-vdiskserverdevice /ldoms/vdisk2_10gb.img vdisk2@primary-vds0
# ldm add-vdisk vdisk1 vdisk1@primary-vds0 ldom1
# ldm add-vdisk vdisk2 vdisk2@primary-vds0 ldom1
# ldm add-vnet vnet0 primary-vsw0 ldom1
# ldm add-vnet vnet1 primary-vsw2 ldom1
# ldm set-variable auto-boot\?=false ldom1
# ldm set-variable local-mac-address\?=true ldom1
# ldm set-variable boot-device=/virtual-devices@100/channel-devices@200/disk@0 ldom1
# ldm bind-domain ldom1
# ldm start ldom1


So with the above commands we allocated the vcpu's, mau, and the memory. Then created the virtual disk images files, added them as virtual disk devices to the primary domain's VDS service, and finally added them as virtual disks to the guest domain. Then attached virtual networks, set auto-boot to false in the OBP(yes that's right each logical domain gets its own OBP), set local-mac-address to true, and set the default boot device. Finally we've bound the configuration and started the guest domain. So what do we get?


# ldm list-bindings ldom1
Name: ldom1
State: active
Flags: transition
OS:
Util: 0.2%
Uptime: 1d 6h 43m
Vcpu: 4
vid pid util strand
0 4 0.7% 100%
1 5 0.1% 100%
2 6 0.1% 100%
3 7 0.0% 100%
Mau: 1
mau cpuset (4, 5, 6, 7)
Memory: 4G
real-addr phys-addr size
0x4800000 0x104800000 4G
Vars: auto-boot?=false
boot-device=/virtual-devices@100/channel-devices@200/disk@0
local-mac-address?=true
Vldcc: vldcc0 [Domain Services]
service: primary-vldc0 @ primary
[LDC: 0x0]
Vnet: vnet0 [LDC: 0x2]
mac-addr=0:14:4f:fb:c4:ef
service: primary-vsw0 @ primary
[LDC: 0x1]
Vnet: vnet1 [LDC: 0xd]
mac-addr=0:14:4f:fb:24:b6
service: primary-vsw2 @ primary
[LDC: 0xc]
Vdisk: vdisk1 vdisk1@primary-vds0
service: primary-vds0 @ primary
[LDC: 0x17]
Vdisk: vdisk2 vdisk2@primary-vds0
service: primary-vds0 @ primary
[LDC: 0x18]
Vcons: [via LDC:25]
ldom1@primary-vcc0 [port:5000]


As you can see, everything that's been previously configured is listed. Some important things to note are the MAC addresses for the network interfaces (which are assigned automatically) and the Vcons port for the console. So now we can jumpstart our domain:


# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldom1" in group "ldom1" ....
Press ~? for control options ..

Sun Fire T200, No Keyboard
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.26.0.build_07, 4096 MB memory available, Serial #66831599.
Ethernet address 0:14:4f:fb:c4:ef, Host ID: 83fbc4ef.



{0} ok show-nets
a) /virtual-devices@100/channel-devices@200/network@1
b) /virtual-devices@100/channel-devices@200/network@0
q) NO SELECTION
Enter Selection, q to quit: a
/virtual-devices@100/channel-devices@200/network@1 has been selected.
Type ^Y ( Control-Y ) to insert it in the command line.
e.g. ok nvalias mydev ^Y
for creating devalias mydev for /virtual-devices@100/channel-devices@200/network@1
{0} ok boot /virtual-devices@100/channel-devices@200/network@1 - install
Boot device: /virtual-devices@100/channel-devices@200/network@1 File and args:
- install
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
SunOS Release 5.10 Version Generic_118833-33 64-bit
Copyright 1983-2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
WARNING: machine_descrip_update: new MD has the same generation (1) as the old MD
whoami: no domain name
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet1...
Configured interface vnet1
Attempting to configure interface vnet0...
Skipped interface vnet0
Setting up Java. Please wait...
Extracting windowing system. Please wait...
Beginning system identification...
Searching for configuration file(s)...
...
So after the guest domain is finished jumpstarting, we can take a look around.


# psrinfo -vp
The physical processor has 4 virtual processors (0-3)
UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
# psrinfo -v
Status of virtual processor 0 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:15.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
# prtdiag -v
System Configuration: Sun Microsystems sun4v Sun Fire T200
Memory size: 4096 Megabytes

========================= CPUs ===============================================

CPU CPU
Location CPU Freq Implementation Mask
------------ ----- -------- ------------------- -----
MB/CMP0/P0 0 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P1 1 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P2 2 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P3 3 1000 MHz SUNW,UltraSPARC-T1


========================= IO Configuration =========================

IO
Location Type Slot Path Name Model
----------- ----- ---- --------------------------------------------- ------------------------- ---------

========================= HW Revisions =======================================

System PROM revisions:
----------------------
OBP 4.26.0.build_07 2007/02/14 19:20

IO ASIC revisions:
------------------
Location Path Device Revision
-------------------- ---------------------------------------- ------------------------------ ---------
# df -h
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d0 7.8G 2.2G 5.5G 30% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 5.1G 1.1M 5.1G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
7.8G 2.2G 5.5G 30% /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
7.8G 2.2G 5.5G 30% /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
swap 1.6G 0K 1.6G 0% /tmp
swap 5.1G 32K 5.1G 1% /var/run
# metastat
d1: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d21
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4194600 blocks (2.0 GB)

d11: Submirror of d1
State: Okay
Size: 4194600 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d0s1 0 No Okay No


d21: Submirror of d1
State: Okay
Size: 4194600 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d1s1 0 No Okay No


d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16644000 blocks (7.9 GB)

d10: Submirror of d0
State: Okay
Size: 16644000 blocks (7.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d0s0 0 No Okay No


d20: Submirror of d0
State: Okay
Size: 16644000 blocks (7.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d1s0 0 No Okay No


Device Relocation Information:
Device Reloc Device ID
c0d1 No -
c0d0 No -
# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vnet0: flags=9040843 mtu 1500 index 2
inet 192.168.1.2 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp1
ether 0:14:4f:fb:c4:ef
vnet0:1: flags=1000843 mtu 1500 index 2
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
vnet1: flags=9040843 mtu 1500 index 3
inet 192.168.1.3 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp1
ether 0:14:4f:fb:24:b6
# uptime
10:20pm up 5 min(s), 1 user, load average: 0.02, 0.11, 0.06

So now we have a guest domain running Solaris 10 Update 3, with SVM mirrored boot drives that are really sparse files, IPMP on virtual NICs, four CPU's, 4GB's RAM, etc

Oracle Solaris 11 Express Released!

at 8:23 PM
Solaris 11 Express has finally been released! This has been a long time in coming and I'm very excited to finally see this day. Just so that folks are clear, this is a full express release with support for developers, system administrators, evaluators, enthusiasts, etc. on x86 and SPARC! It is totally free to use as long as it is not used on production. As you can see on the main link above, Oracle is selling a full suite of support for Solaris 11 Express, if you are looking for support or to use it in production. Oracle is dead serious about Solaris, so make no mistake about it!

Needless to say, I'll be busy downloading and upgrading my systems to this release. I'll make some additional blog posts once I have things in place to take it for a full spin on both x86 and SPARC. I'll leverage my Ultra 20, some VirtualBox instances, and some LDoms to make things interesting!

LDOM Installation

at 8:19 PM
Before you begin, the following is required:


sun4v based server (SunFire T1000/T2000, Sun Netra T2000, or Sun Netra CP3060 Blade).
Solaris 10 Update 3 (HW 11/06) or Solaris Express (Build 57 or higher) installed.
Logical Domains 1.0 Early Access
The first step is into install the firmware included with the LDOM software bundle. The firmware will contain the ALOM CMT, Post, OBP, and hypervisor updates. You must load the corresponding firmware for your platform. There are two methods for doing this. You can download the firmware to the ALOM CMT using FTP or you can upload it from your currently installed Solaris instance. The later is much simpler:)


# cd Firmware/tools # ./sysfwdownload ../Sun_System_Firmware-6_4_0_build_07-Sun_Fire_T2000.bin

.......... (10%).......... (20%).......... (30%).......... (40%).......... (51%)
.......... (61%).......... (71%).......... (81%).......... (92%)........ (100%)

Download completed successfully.


This will upload the firmware to your ALOM CMT. Make sure that you upload the corresponding firmware for your platform. Now you need to shutdown your Solaris instance:


# shutdown -y -g0 -i5


Now you can upgrade the firmware from the ALOM CMT console:


sc> showkeyswitch
Keyswitch is in the NORMAL position.
sc>
SC Alert: Host system has shut down.
flashupdate -s 127.0.0.1

SC Alert: System poweron is disabled.
................................................................................
................................................................................
......

Update complete. Reset device to use new software.

SC Alert: SC firmware was reloaded
sc> resetsc
Are you sure you want to reset the SC [y/n]? y

The firmware is now updated and the SC has been reset. Once it is done resetting, verify the version of the firmware:


sc> showhost
Sun-Fire-T2000 System Firmware 6.4.0_build_07 2007/02/14 22:07

Host flash versions:
Hypervisor 1.4.0_build_07 2007/02/14 21:52
OBP 4.26.0.build_07 2007/02/14 19:20
POST 4.26.0.build_07 2007/02/14 19:51

The version should match the version info in the firmware bin file name. Now you can power on your server and proceed to the installation of the LDOM software. Depending on the OS you are running, you may have to apply the patches that are included in the Patches directory first.

For example, if you are running Solaris 10 Update 3, you will need to install 118833-36 and reboot. Then you'll have to install patches 125043-01 and T124921-02, then reboot. This is not required if you are running build 57 or higher of Nevada (OpenSolaris, Solaris Express, etc.).

Now it's time to install the LDOM software for what will become the control domain. The software package includes JASS to secure the control domain. Remember, the control domain is similar to the SC on a Sun Fire 15K. You don't want it to be used for anything other than administering the platform. You can install the SUNWjass and SUNWldm package with the install-ldm script under the Install directory. Or you can install them manually. If you already have secured the control domain, you may not need JASS, it's up to you:)


# Install/install-ldm
Welcome to the LDoms installer.

You are about to install the domain manager package that will enable
you to create, destroy and control other domains on your system. Given the capabilities of the domain manager, you can now change the security configuration of this Solaris instance using the Solaris Security Toolkit.
Select a security profile from this list:
a) Hardened Solaris configuration for LDoms (recommended)
b) Standard Solaris configuration
c) Your custom-defined Solaris security configuration profile
Enter a, b, or c [a]: a
The changes made by selecting this option can be undone through the
Solaris Security Toolkit’s undo feature. This can be done with the
’/opt/SUNWjass/bin/jass-execute -u’ command.
Installing LDoms and Solaris Security Toolkit packages.

Installation of was successful.
...
Verifying that all packages are fully installed. OK.
Enabling services: svc:/ldoms/ldmd:default
Running Solaris Security Toolkit 4.2.0 driver ldm_control-secure.driver.
...
Solaris Security Toolkit hardening executed successfully; log file
/var/opt/SUNWjass/run//jass-install-log.txt. It will not
take effect until the next reboot. Before rebooting, make sure SSH or
the serial line is setup for use after the reboot.
Then reboot your control domain.

Introduction to LDOM's

at 8:14 PM
Logical domains are discrete instances of the Solaris OE running independently within a virtualized environment. Each logical domain has its own virtual cpu, memory, OBP, console, networking, storage, and I/O components. These components are configured with a combination of different technologies.


sun4v Platform Hypervisor
Logical Domain Management Software
Solaris OE
The hypervisor provides the mechanism for masking and virtualizing the resources on the platform. The hypervisor is a light-weight software layer built into the ALOM CMT firmware. The hypervisor also helps to abstract the low-level hardware details from logical domains.

The logical domain management software is the nexus for control and configuration of the hypervisor. This software provides a CLI to controlling and configuring the resources that define each logical domain. Only one logical domain can run the management software. This logical domain is known as the "primary" or control domain. More about the different LDOM types in a moment.

The Solaris OE provides support for the sun4v platform, dynamic reconfiguration, and virtual devices. At this time, you need Solaris 10 Update 3 (11/06) or Nevada build 57. It's not possible to use Solaris 9 or below for LDOM's, as the platform support is not there.

There are four types of LDOM's that can be created:


Control Domain
Service Domain
I/O Domain
Guest Domain
The control domain is the first installed LDOM or instance of Solaris on the platform. This LDOM contains the Logical Domain Management (SUNWldm) software for managing the platform. It is from this LDOM that all of the hardware platform specifics are visible and configurable. The control and configuration of the platform is communicated through LDC's (Logical Domain Channels). It is through this mechanism that the configuration, virtual devices, and virtual services communications are relayed.

A service domain is an LDOM that has control over either one or more PCI-E controllers. It consists of an instance of the Solaris OE. No additional software is required, the control domain will configure the virtualized devices and service within a service domain. The service domain will then service the I/O for these virtualized components for guest domains to utilize. The service domain has direct control over the hardware under its PCI-E controller. There are only two in the Sun Fire T2000. So only two service domains are configurable, one of which must also be the control domain. It is important to remember that the control domain is one of the service domains. If a second service domain is created, this is called a Split PCI-E configuration. More about that later.

An I/O domain is exactly like a service domain, except for the fact that none of its devices or services are virtualized for guest domains. This is useful if you have an application that requires direct access to a PCI-E device for performance or some other reason.

A guest domain is a consumer of virtualized devices and services. Meaning that it does not virtualize any devices or services for other domains. It is independent of other guest domains. However, it is dependent upon the service domains that provide its virtual devices and services. A guest domain consist of its own instance of Solaris OE. This is where your applications will typically live as consuming resources in the control or services domains affects the platform as a whole.

While a fully configured Sun Fire T2000 has a total of 32 CPU threads, it's probably not a good idea to create 32 LDOM's. As this would under power the control and services domains.

The next post will be about the installation of the firmware, patches, and Logical Domain Management software.

Configuring the Control Domain (LDOM)

at 8:13 PM
Now it's time to configure the resources for your control domain! The first step is to make sure that the required SMF services are running:


# svcs -a | grep ldom
online Mar_20 svc:/ldoms/ldmd:default
online Mar_20 svc:/ldoms/vntsd:default


The ldmd service is responsible for controlling the platform and the vntsd service is responsible for providing the virtual terminal services for your logical domains. If they are not running, enable them. You should then be able to run the /opt/SUNWldm/bin/ldm command:


# /opt/SUNWldm/bin/ldm list
Name State Flags Cons VCPU Memory Util Uptime
primary active -t-cv SP 32 32G 0.8% 3d 16h 27m

As you can see, all 32 vcpu's and all of the memory are assigned to the primary (a.k.a. control) domain. We must free up these resources and create the basic infrastructure to support guest domains.


# /opt/SUNWldm/bin/ldm add-vdiskserver primary-vds0 primary
# /opt/SUNWldm/bin/ldm add-vconscon port-range=5000-5100 primary-vcc0 \
primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g0 primary-vsw0 primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g1 primary-vsw1 primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g2 primary-vsw2 primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g3 primary-vsw3 primary
# /opt/SUNWldm/bin/ldm set-mau 1 primary
# /opt/SUNWldm/bin/ldm set-vcpu 4 primary
# /opt/SUNWldm/bin/ldm set-memory 4G primary


The above creates the virtual disk server for servicing storage, the virtual terminal console ports, virtual switch for each physical network port, one crypto unit, 4 vcpu's, and 4GB's of memory for the primary domain. This sets up enough resources for the primary domain, which acts as a control and service domain for the platform. Now we need to store this configuration into the ALOM CMT and reboot.


# /opt/SUNWldm/bin/ldm list-config
factory-default [current]
# /opt/SUNWldm/bin/ldm add-config initial
# /opt/SUNWldm/bin/ldm list-config
factory-default [current]
initial [next]
# shutdown -y -g0 -i6


This stores the configuration and activates it. When the control domain comes back up, you'll notice that the available cpu and memory has changed:


# ldm list primary
Name State Flags Cons VCPU Memory Util Uptime
primary active -t-cv SP 4 4G 0.9% 3d 16h 39m
# psrinfo -vp
The physical processor has 4 virtual processors (0-3)
UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
shou18leng01:~ $ psrinfo -v
Status of virtual processor 0 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:23.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:27.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:27.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:27.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
# prtdiag -v | grep -i mem
Memory size: 4096 Megabytes
Now we are ready to create our first guest domain! Watch out for the next post.

root passwd change permission denied

Friday, February 11, 2011 at 7:31 AM
Error Message:

# passwd root
New Password:
Re-enter new Password:
Permission denied
#

#grep passwd /etc/nsswitch.conf
passwd: files nis
#

Use the below syntax to change the password

# passwd -r files
passwd: Changing password for root
New Password:
Re-enter new Password:
passwd: password successfully changed for root
#

Solaris | Powered by Blogger | Entries (RSS) | Comments (RSS) | Designed by MB Web Design | XML Coded By Cahayabiru.com