Tips and Techniques

Using SSH with RSYNC

• Rsync is an extremely useful file transfer tool. It is comonly used to keep various collections of files in sync for http and ftp servers. Rsync can handle symbolic links and other special files and it can do incremental transfers which saves time and bandwidth when keeping collections in sync. Rsync can use rsh as the file copy transport but for added security an alternate transport protocol, such as ssh, can be specified.

• Using ssh offers host and data security and should be used anywhere security is desired which is pretty much everywhere in IT these days. To use ssh with rsync simply add the "--rsh=" option to the rsync command line as in the following example:

• rsync --rsh="ssh -l webadm" -aq /var/www/html remotehost:/var/www/html

• The command above will prompt for a ssh login/password however this can be handled by setting up public key authentication for the webadm user.

Port Forwarding with OpenSSH

• With OpenSSH it is possible to log into a remote server and forward a port on the remote server back through the encrypted tunnel to a port on the local server.
  • Create the initial tunnel from the local server, "ssh -R 1956:localhost:22 some_remote_server".
  • Go to the remote server and use the tunnel, "ssh -p 1956 localhost"
• In the above example it would be possible to open a tunnel through a firewall by logging into a system outside the firewall. Usually outbound ssh connections are allowed by firewall rules. This is risky and probably not a good idea.

Copying VTOC from Disk to Disk

• Sometimes it is convenient to partition one disk the same as another.
  • "prtvtoc /dev/rdsk/cxtxdxs2 | fmthard -s - /dev/rdsk/cxtxdxs2"

Mounting an ISO Image as a Filesystem

• It is convenient to be able to use an ISO image directly rather than having to burn it to CD first. The lofiadm command makes this possible.
  • lofiadm -a /path/to/iso
  • mount -F hsfs -o ro /device/from/step1 /desired/mountpoint

Logging debug authentication info

• In some situations it is helpful log debug level authentication messages. This can be accomplished by adding the following to /etc/syslog.conf:
  • auth.debug /var/log/authlog
  • Note - The white space above should be tab(s).
  • Make sure /var/log/authlog exists.
  • Restart syslogd following the syslog.conf edit.

Logging sudo info

• It is advisable to keep an eye on how sudo is used on a system to ensure that users have the limited root access that you expect. To log sudo commands add the following to /etc/syslog.conf:
  • local2.debug /var/log/sudo.log
  • Note - The white space above should be tab(s).
  • Make sure /var/log/sudo.log exists.
  • Restart syslogd following the syslog.conf edit.

Associating a Process ID with an open TCP/UDP Port

• Tracking down network connections is a frequent requirement of the sysadmin. While troubleshooting problems it is often required to determine if a network port is open and often you may need to know which process opened the port. Sorting out connections to a central backup server is a good example. Suppose you see that an oracle backup is hung on the backup server. This situation could seriously impact the backup server because often it results in tape drive and media resources becoming unavailable to other clients. With a little work you can trace the network connection back to an oracle process on the remote server attempting the backup and then ask the DBA's to look into the problem or terminate the backup process.

• To begin, find the connections from the oracle server to the backup server, "netstat -a | grep oracleserver". The results will show connections the connection ports for both sides of the connection. Usually a central backup server like Legato Networker or Veritas Netbackup will use a large range of connection ports. This makes it a little difficult to say with certainty the the connection(s) you see in netstat is the backup server connection. You can use lsof from www.sunfreeware.com to find the PID associated with local address port from your netstat command.

• Here is an example of tracing network connections from a Windows Exchange server to a Solaris Netbackup server:

  root@netbck1#  netstat -a |grep msg
  netbck1-alt.760      msg4.bpcd            33207      0 2622328      0 ESTABLISHED
  netbck1-alt.981      msg4.581             64240      0 64240      4 ESTABLISHED
  netbck1-alt.1386     msg4.631             64240      0   164      0 ESTABLISHED
  netbck1-alt.660      msg4.bpcd            33167      0 2622328      0 ESTABLISHED
  netbck1-alt.953      msg4.1014            64240      0 64240      2 ESTABLISHED
  netbck1-alt.1030     msg4.830             64240      0   164      0 ESTABLISHED
  
  The command above shows 6 connections from msg4 to netbck1. Now we need
  to see which processes those connections are talking to.
  
  root@netbck1# lsof -i tcp:760 -i tcp:981 -i tcp:1386 -i tcp:660 -i tcp:953 -i tcp:1030
  COMMAND PID USER   FD   TYPE        DEVICE     SIZE/OFF NODE NAME
  bptm     59 root    7u  IPv4 0x30011b06920 0x1fca32f000  TCP netbck1-alt:1386->msg4:631 (ESTABLISHED)
  bpbrm    61 root    7u  IPv4 0x3000e40b950    0t5209462  TCP netbck1-alt:760->msg4:bpcd (ESTABLISHED)
  bpbrm    61 root   10u  IPv4 0x300065f8e58    0t1042428  TCP netbck1-alt:981->msg4:581 (ESTABLISHED)
  bptm     76 root    8u  IPv4 0x3001d2d17e8 0x1fca22f800  TCP netbck1-alt:1030->msg4:830 (ESTABLISHED)
  bpbrm    81 root    7u  IPv4 0x300125f0f98    0t5209422  TCP netbck1-alt:660->msg4:bpcd (ESTABLISHED)
  bpbrm    81 root   10u  IPv4 0x300065fcd10    0t1042408  TCP netbck1-alt:953->msg4:1014 (ESTABLISHED)
  

• This shows that all 6 connections are to various Netbackup processes. If msg4 was a Solaris or Linux server it would be trivial also to find the processes associated with the connections on that side as well.

• Solaris offers some tools in /usr/proc/bin that will allow you to accomplish the same thing as lsof just not as easily. See the man page for pfiles and pmap etc.

Associating a Disk device with an HBA

• Sometimes a large system has many fibre channel HBA's in addition to HVD/LVD controllers which connect maybe hundreds of disks to the server. It is sometimes difficult to keep strait which disks are connected via which controller. Here is an easy to follow procedure to help sort this out.

• First choose a disk like c7t34d19s2 for example. Then display the full device name for that disk.

  bash-2.03# ls -l /dev/dsk/c7t34d19s2
  ....  /dev/dsk/c7t34d19s2 -> ../../devices/sbus@7,0/fcr@0,400000/sd@22,13:c
  

• The identifying part of the device link is /sbus@7,0/fcr@0,400000. The next step is to find the kernel device driver responsible for this disk device. Solaris keeps this information in /etc/path_to_inst so all we need to do is grep out the device like so:

  bash-2.03# grep /sbus@7,0/fcr@0,400000 /etc/path_to_inst |head -1
  "/sbus@7,0/fcr@0,400000" 1 "jnic146x"
  

• As you can see the jnic146x kernel driver handles this path. Another bit of useful information is that this is instance 1 of the jnic146x driver. That tells me that this device would be referenced as jnic146x1 in /kernel/drv/sd.conf and /kernel/drv/jnic146x.conf where I could find persistent bindings and target and lun configurations.

Hot Swapping with Dynamic Reconfiguration

Suppose you suddenly need 1Gb/sec network bandwidth but you have no GigE interface installed. You can add the interface on the fly with Sun's Enterprise or Sun Fire class servers. Here are the basic steps using cfgadm and ifconfig.

• Insert an I/O board with a ge interface installed into an available board slot. Look at /var/adm/messages to verify the the system sees the new board. Also look at the output of cfgadm -al and verify that the new I/O board is listed. It should show that it is disconnected and unconfigured.

• Next run "cfgadm -c connect device" (Device will be something like sysctrl0:slot5). Now if you run cfgadm -al again you will notice that the device is now connected. This step pauses the system for several seconds so don't panic and hope that your kernel also doesn't panic.

• Configure the new I/O board with "cfgadm -c configure device". Cfgadm -al will now show that the device is configured and /var/adm/messages will show that the appropriate devices were created including a device for the ge interface.

• Now we just need to configure the interface, "ifconfig ge0 plumb" and "ifconfig ge0 inet 10.252.0.135 netmask 255.255.255.0 up". Assuming that you connected a cable to the interface you should be ready to start using the interface.