Vic/LDOMS

From Summerseas

Jump to: navigation, search

Logical Domains, LDOMS
LDom's were introduced with the 11/06 release of Solaris 10 and provide the OS with additional virtualization capabilities. LDom's exploit the multithread architecture of Sun's new SPARC T1 and T2 processors. It is especially interesting that logical domains on these systems can be created from virtual and physical hardware. For example, on a T2000 system with 1 processor I can create one domain with with access to one of the PCI buses and access to virtual disk/net/console devices exported from another domain.

Contents

[edit]

Required Hardware

  • Sun Sparc T1 and T2 based servers (T1000 and T2000 servers and the new T5000 servers)
[edit]

Required Software

  • T1 Architecture
    • Primary Domain Sol10_u3, Guest Domains Sol10_u3
  • T2 Architecture
    • Primary Domain Sol10_u4, Guest Domains Sol10_u3
[edit]

LDOM Docs and S/W Download

[edit]

Required Firmware

  • T1 6.5.x
  • T2 7.0.x
  • The 6.5.x s/w is required for LDOM but a note in the Blueprint doc mentioned that at least 1 pre-6.3 firmware must be installed before upgrading to a newer version or the system could become inoperable. The patch files do not mention this warning but out of caution I installed the older version first then the newer version.
  • When I first checked the versions with sc> showhost there was no firmware version loaded. This must be normal for early T2000's.
  • Follow instructions in Install.info to update the firmware. Note, unless you have the net mgmt interface up you will not be able to use the ftp server method. The alternative method works without issue.
  • NOTES
    • 6.6.x is recommended for UltraSparc T1
    • 7.1.x is recommended for UltraSparc T2
    • 7.1.x is required for UltraSparc T2 Plus
[edit]

Required Patches

[edit]

Required Patches

[edit]
LDom S/W 1.01
  • Sol10U4 Requires no patches for LDoms
  • Sol10U3
    • 124921-02 at a minimum
    • 125043-01 at a minimum
    • 118833-36 at a minimum
  • Firmware for the T2000
    • 127576-01 Sun Fire and SPARC Enterprise T2000 Server Firmware
[edit]
LDom S/W 1.02
  • At a minimum, Solaris 10 U3 plus the following patches, are needed for all domains on UltraSPARC T1 servers and guest domains on UltraSPARC T2 and T2+ servers. UltraSparc T2 servers will require Solaris 10 U4 in the control domain. These required patches are available on http://sunsolve.sun.com.
    • 124921-02
    • 125043-01
  • The following patch, which obsoletes the previous patch list, is recommended by Sun:
    • 127111-08
  • NOTE-1 - Patch 127111-08 is a kernel patch and has dependencies which would result in the newest rev of the Leadville drivers.
  • Bug Fixes
  • Bugs Fixed for LDoms 1.0.2
  • Sun recommends the latest patch be installed. The following LDoms bugs were fixed

for the LDoms 1.0.2 software release:

  • Fixed in Patch 127111-08
    • 6578761 System hangs in ds_cap_fini() and ds_cap_init()
    • 6593231 Domain Services logging facility must manage memory better
    • 6616313 cnex incorrectly generates interrupt cookies
    • 6630945 vntsd runs out of file descriptor with very large domain counts
    • 6501039 rebooting multiple guests continuously causes a reboot thread to hang 52 Logical Domains (LDoms) 1.0.2 Release Notes • February 2008
    • 6527622 Attempt to store boot command variable during a reboot can time out
    • 6589682 IO-DOMAIN-RESET (Ontario-AA): kern_postprom panic on tavor-pcix configuration (reboot)
    • 6605716 halting the system should not override auto-boot? on the next poweron
    • 6519849 vnet hot lock in vnet_m_tx affecting performance
    • 6530331 vsw when plumbed and in prog mode should write its mac address into HW
    • 6531557 format(1m) does not work with virtual disks
    • 6536262 vds occasionally sends out-of-order responses
    • 6544946 Adding non existent disk device to single cpu domain causes hang
    • 6566086 vdc needs an I/O timeout
    • 6573657 vds type-conversion bug prevents raw disk accesses from working
    • 6575216 Guests may lose access to disk services (VDS) if IO domain is rebooted
    • 6578918 disk image should have a device id
  • New in 1.02
    • Sun SPARC® Enterprise T5140 and T5240 Servers
    • Up to 64 domains on Sun UltraSPARC® T2 processors
    • Up to 128 domains on Sun UltraSPARC T2 Plus processors
[edit]

Sun BUGs with Work Arounds

  • 6528156, 6522993 and 2153029 - VDS services exclusively opens devices for export. This fails for devices held open by multipathing and volume management software such as VxDMP or SDS. This is fixed in Sol10U5 and a work-around is available for Sol10U3 and U4.
  • Work-around - Disable the open exclusive option by setting vd_open_flags to 0x3:
    • On the running system: echo 'vd_open_flags/W 0x3' | mdb -kw
    • In /etc/system: set vds:vd_open_flags = 0x3
  • 6575216
    • Work-around - Add the following to /etc/system on the service domain...
      • set vds:vds_dev_delay = 60000000
      • set vds:vds_dev_retries = 10
[edit]

Issues Noted

  • 1. Providing guest domains with whole labeled disks didn't work. The guest domain saw the disk but I could not change the label from within the domain. So I provided the guest domain a large file from the disk server domain and this worked well. Also, if I partition the disk to suit the install that works as well.
  • 2. The "net" and "vdisk" OBP alias entries were not correct in the newly created domains.
  • 3. I setup the LDOM clients in jumpstart using the MAC reported from the "banner" command within the LDOM OBP but when I tried to jumpstart the LDOM I noted that it was trying to get an IP for a different MAC. I then updated /etc/ethers and re-added the client and jumpstart succeeded after that. Note - Use "ldm list-domain -l" from the control domain to get the correct MAC for the guest.
  • 4. In Solaris 10 U4 it is best to plumb the VSW device and unplumb the physical. Otherwise there may be no IP connectivity between the control domain and the guests. This is supposed to be fixed already in opensolaris.
  • 5. If you install from a flash archive created on a non-ldom T2000 you may run into a problem where cfgadm doesn't work; devfsadm -C fixes this problem.
  • 6. ZFS is unable to create EFI labels on vdisks backed by files in the I/O domain.
[edit]

Install the LDom Management S/W

  • This basically consists of running an install script which does some checks, adds a couple of packages and enables the LDOM service.
  • Follow these instructions to install and enable the S/W.
    • NOTE - It is optional to install the security component. The security component just adds RBAC support.
[edit]

Initial Configuration of the Control Domain

  • NOTE-1 - (From the Admin Guide) If you are not using ZFS to deliver disk services, 1 GB of memory should be adequate. If you are using ZFS to deliver disk services, assign a complete core of 4 virtual CPUs and at least 4 GB of memory. You may need to assign additional complete cores for heavier I/O loads.
  • NOTE-2 - LDom management requires that the following services are started...
    • online 11:23:14 svc:/ldoms/ldmd:default
    • online 16:46:11 svc:/ldoms/vntsd:default (Needed for the domain consoles)
  • Create default services in the Control domain. These services will provide services to the guest domains.
  • The following steps create a virtual disk server and a virtual console server.
    • ldm add-vds primary-vds0 primary
    • ldm add-vcc port-range=5000-5100 primary-vcc0 primary
  • The next 2 steps create virtual switch devices which may be used to create virtual NICs for the guests. With Solaris 10 you must begin using the vsw device instead of the physical decive. For example, after creating vsw0 you should move /etc/hostname.e1000g2 to /etc/hostname.vsw0 before rebooting.
    • ldm add-vsw mac-addr=<MAC Addr> net-dev=e1000g2 vsw0 primary
    • ldm add-vsw mac-addr=<MAC Addr> net-dev=e1000g3 vsw1 primary
  • Now verify that things look correct...
    • ldm list-services primary
  • The next 2 steps par down resources on the control domain so that they will be available for guests.
    • ldm set-memory 1G primary
    • ldm set-vcpu 4 primary
  • Now save the config..
    • ldm add-config initial
  • Verify that the new config will be applied during the next boot.
    • ldm list-config
  • If you're satisfied with the configuration then shutdown and power cycle the host. When the host reboots login and verify the configuration and if things look good then you're ready to create a guest domain.
[edit]

Configuration of a Guest Domain

[edit]
Configuring a guest domain for iSCSI
  • ldm add-domain iSCSI_LDOM
  • ldm add-vcpu 4 iSCSI_LDOM
  • ldm add-memory 2500M iSCSI_LDOM
  • ldm add-vnet vnet2 vsw0 iSCSI_LDOM
  • ldm add-vnet vnet3 vsw1 iSCSI_LDOM
  • ldm add-vdsdev /ldomstorage/iSCSI_LDOM vol2@primary-vds0
  • ldm add-vdisk vdisk2 vol2@primary-vds0 iSCSI_LDOM
  • ldm set-var auto-boot\?=true iSCSI_LDOM
  • IMPORTANT - The following step is required or the guest will hang on "Configuring Devices" during jumpstart.
  • ldm set-var boot-device=vdisk2 iSCSI_LDOM
  • ldm bind-domain iSCSI_LDOM
  • ldm start-domain iSCSI_LDOM
  • ldm list -l iSCSI_LDOM
  • ldm list-config
[edit]
Configuring a guest domain with a HBA
  • In order to give a HBA to a guest domain one of the PCI buses initially owned by the control domain must be removed from the control domain and added to the guest domain. CAUTION - Take care to avoid removing the bus used by the control domain for booting. My T2000 has a PCI-X LSI Logic adapter in the pci@7c0 bus. The control domain must keep that bus. Newer T2000 models will include embedded LSI controllers for the internal disk so it is important to double check the bus configuration before removing a bus from the control domain. Also verify that the control domain is not using any nic interfaces on the bus to be removed.
  • On my T2000 I found the following PCI bus arrangement:
    • pci@780
      • e1000g0
      • e1000g1
      • PCI-E/0
    • pci@7c0
      • e1000g2
      • e1000g3
      • PCI-E/1
      • PCI-E/2
      • PCI-X/0
      • PCI-X/1
  • To determine which bus you're booting from simply examine the device path of the root slice...
  • [root@sunt2000-shu01--->]ls -l /dev/dsk/c1t0d0s0
  • lrwxrwxrwx 1 root root 65 Oct 11 15:19 /dev/dsk/c1t0d0s0 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0:a
  • The system sunt2000-shu01 is booting from /dev/dsk/c1t0d0s0 which is on pci@7c0
  • Now hopefully we're ready to remove PCI bus, pci@780.
    • ldm rm-io pci@780 primary (Note - I named my control domain "primary"
    • ldm add-io pci@780 VxDOM (Adding a pci bus to a LDOM named "VxDOM"
[edit]

"ldm" Command Usage

  • The ldm command is used to create and manage domains. There can be only one logical domain manager and it must be installed on the control domain.
  • Command usage is here.
[edit]

Boot disk notes

  • LDom's may be booted from a vdisk exported from the virtual disk server. The virtual disk server can export either a physical device like c3t0d0s2 or a file like /bootfiles/ldom1. If exporting a physical disk as an ldom boot-disk it must be the whole disk (s2).
  • To export a file simply use mkfile to create a boot-disk file of appropriate size.
  • An interesting aspect of using a file as a vdisk is that it can be copied and manipulated from the exporting domain which is usually the control domain.
  • The boot image can be mounted using lofi in the control domain which is convenient for editing the configuration files. The caveat is that the OS installed on the vdisk must have the root slice starting on cylinder zero. A potential work-around for this caveat would be to use this tool to split the disk image into slices and then lofi mount and edit the root slice. Once edited the image can be concatenated back together.
  • The following jumpstart profiles are designed to do just that using a 9gb file based vdisk. 

# Profile for an initial install
install_type    initial_install
system_type     standalone
geo             N_America
cluster         SUNWCXall
partitioning    explicit
filesys         rootdisk.s0     0:27454    /
filesys         rootdisk.s1     free     swap

# This profile is for installing with a flash archive
#
install_type    flash_install
archive_location nfs 192.168.96.100:/solaris/flars/sol10u4_flar
partitioning    explicit
filesys         c0d1s0     0:27454    /
filesys         c0d1s1     free     swap
[edit]

ZFS in a Guest Domain

  • ZFS is supported in a guest domain with no restrictions for physical disks that the guest has direct access to; like iSCSI provided disks or fibre channel disks if the guest owns the HBA. ZFS on virtual devices does have limitations. When creating a zpool in a guest domain from VDISK disks you must provide a slice instead of the whole disk because Zpool will fail trying to put an EFI label on the vdisk.
Retrieved from "http://www.vicengle.org/wiki/Vic/LDOMS"
Personal tools