D-kriptik Blog

Since there was some interest in the previous post, here are a few more notes and tips. Most often I forget to mention those standard tools we all take for granted, so I threw in some of those in as well. Oh, and I won’t be keeping up this extremely rapid posting rate (i.e., 2 in about week). (Note: If I played with any of what is written here, it was on Ubuntu 8.10 or FreeBSD 7.1.)

1. By default in Firefox 3, when opening Tools->Add-ons, Firefox sends queries to *.mozilla.com in order to populate the “Get Add-ons” pane. This pane and its “paneful” behavior can be disabled by setting the “extensions.getAddons.showPane” preference to “false” through, say, “about:config”.
2. Using the “-n” option is your friend for listing network address information with utilities like “lsof” (e.g., “lsof -n”), “netstat” (e.g., “netstat -an”), and “iptables” (e.g., “iptables -n –list”), as it prevents the resolving of numeric network addresses, which can be annoying, revealing, and time consuming.
3. Dislike when ps truncates rather than wraps? The “w” (wide output – 132 columns) or “ww” (unlimited width) option is your friend. If you need finer-grained control than that for some reason, you could try setting a “COLUMNS” environment variable in your shell to override the width determined by ps automatically. Or, in a nod to a later bullet point in this post, you may have additional ps options, like the “–cols” or “–columns” options that can be used to specify these settings in the GNU/Linux ps world.
4. In much of the *n[iu]x world, if you want to know how many lines, words, or bytes are in a file (or stdin), “wc” is often where its at.
5. Want to log to syslog using a command from a shell in much of the *n[iu]x world? Try “logger”.
6. In the much of the *n[iu]x world, ldd can be used to list the shared object (i.e., dynamically linked library) dependencies of a binary. (A question of what stock commands there are available to do this came up on a non-technical mailing list I am on recently.)
7. On GNU/Linux, lsof can be used to list open files, and it is quite a powerful utility. (A question of what stock commands there are available to do this came up on a non-technical mailing list I am on recently.) For example, “lsof +L 1″ will list all open files that have a zero link count (e.g., open files that are unlinked from the file system – this is useful to play with if you have ever wondered why you can upgrade packages/ports in place that have currently running applications).

   Now, on GNU/Linux, you can dig into “/proc”, which contains a wealth of kernel information. Browsing the contents of “/proc/[pid]” can be used to access the process information for a particular process, and, coming back to our example, this includes the contents of files open by that process – e.g., “/proc/[pid]/fd/[fd]” for files open by the process and “/proc/[pid]/exe” for the executable of the process itself, both of which can be used to access the contents of files open by the process with a zero link count.

   On FreeBSD, fstat can be used to list open files; however, it does not have quite the same flexibility as lsof. For example, listing open files with a zero link count is not simply a matter of command line options to fstat, although you could hack a script together utilizing fstat with other standard tools available on FreeBSD, hopefully unlike the following.

   #!/bin/sh
   
   # This is worse than the worst kludge. Do not use it.
   
   dofind()
   {
           # This find is ugly and potentially huge. Not to mention, pkill could
           # run amuck. Perhaps you could use info gathered from, say,
           # "ps wwe -p $pid -o command=" to help refine these searches,
           # but, like I said, all of this herein is worse than the worst kludge.
   
           find -x $mount -inum $inum -exec pkill -SIGINT -f -n -g 0 -s 0 \"find -x $mount -inum $inum -exec pkill\" \; 2>/dev/null
           return $?
   }
   
   doout()
   {
           echo "$cmd $pid $user $fd $szdv $mode $rw $inum $mount$1" `[ $1 ] || ps ww -p $pid -o command= 2>/dev/null`
   }
   
   fstat_unlink()
   {
           fstat $* 2>/dev/null | while read user cmd pid fd mount inum mode szdv rw junk; do
                   case $fd in
                           # We have the header
                           "FD")
                                   doout " CommandWithArgs"
                                   ;;
                           # We have a special
                           [!0-9]*)
                                   dofind
                                   [ $? -eq 1 ] && doout
                                   ;;
                           # We have a standard except non-inodes
                           *[0-9])
                                   dofind
                                   [ $? -eq 1 ] && doout
                                   ;;
                   esac
           done
   }
   
   fstat_unlink $*
   

   Now, FreeBSD has variants of /proc (procfs and linprocfs), but these are not quite as free with the information as on Linux. This means our example of extracting information from open but unlinked files may be a bit more involved; however, you can still fall back on standard tools on FreeBSD to do it, such as fsdb and dd. For example, once you have a list of the inodes and devices for unlinked files, then you can access FFS inode data with “fsdb -r” using its “inode” and “blocks” CLI commands to get the disk blocks of the relevant inodes, and then use “dd if= bs= count= skip=” to dump the disk blocks associated with the inode, perhaps into a file.

   Or, you could just write some code. However, if you dig into the FreeBSD ports, you will find that lsof is available there. As are tools/toolkits like foremost and sleuthkit. No need to reinvent the wheel.

8. Little tweaks to common system applications can be tricky business in the *n[iu]x world. For example, take the stock find on Ubuntu GNU/Linux (8.10) and FreeBSD (7.1). In this particular GNU/Linux world, our find comes with a “-quit” action that can be used to exit find after it has found particular results; however, in this particular BSD Unix world, we find that our find does not support “-quit”, yet we can combine pkill with the “-exec” primary/action of find to have a “find” process kill itself after it has found particular results. A form of this use is illustrated by the kludge script above. Also, being specifically tied to this particular BSD Unix world, that same kludge script uses “-x” over the equivalent “-xdev”, but, in this particular GNU/Linux world, find provides solely “-xdev”. Yet another example, the find in this particular BSD Unix world has the “-X” option to allow for safe use with xargs, but the find in this particular GNU/Linux world sticks with the good old “-print0″ and “xargs -0″ technique.

   With all this lovely variation, it is often good practice to at least know if not use the more standard, common, portable variants of commands and their options, as you never know where you may find yourself or your scripts ending up.

9. Having worked with iptables quite a bit of late has served to remind me of the elegance of pf.
10. bangbang, baby. shell-fu.

I have been lax on my note taking, but here are a few notes and tips that have come up over the last few months. (I think I did better with formatting this time, after some mockery for this post.)

1. Firefox 3 is configured to download a BBC news feed (Bookmarks->Bookmarks Toolbar->Latest Headlines) by default. You can just remove the offending bookmark to do away with this behavior.
2. The way import and export of OpenPGP keys with multiple sub-keys works in GnuPG (1.4.9) is a bit counter-intuitive to me. If you generate a new sub-key pair for an OpenPGP key in a GnuPG key ring and then export the public and secret components of that OpenPGP key, and then try importing that OpenPGP key into another GnuPG key ring that already has an older version of that OpenPGP key (i.e., the version prior to the addition of a new sub-key pair), the already existing and unchanged components of the OpenPGP key will remain unchanged (as expected) but only the public component of the new sub-key will be imported (as unexpected). In order to get the complete OpenPGP key imported, you will first have to delete the public and private components of the key from the key ring into which you want to merge this updated key (back up the key ring first), and then import the key. Also, you may want to (re)set the trust of the key after doing so.
3. On FreeBSD, “pkg_deinstall” is quite helpful in pruning your installed ports tree, especially for the clean up of dependencies for ports that are being, or have been, removed – in particular the “-r” and “-R” options, which have the same meaning as with “pkg_info” (in reverse) or “portupgrade”. Also, I find running “portupgrade -an” useful to see what will be upgraded prior to actually doing the upgrades.
4. Using a box running Ubuntu (8.10) server (without X and its children) with iptables for IP forwarding with NAT is quite easy. An extremely helpful concept to keep in mind is that, within the “filter” table (the default table), the “INPUT” and “OUTPUT” chains are for packets terminating or originating at the box respectively, while the “FORWARD” chain is for packets flowing through the box. So, say you have a box already configured with “filter” table “INPUT” and “OUTPUT” rules that make you comfortable; adding “filter” table “FORWARD” rules does not generally require modification to these existing “filter” table rules, although you may want/need to tweak things a bit for your new setup. However, you do need to remember to consider which “INPUT” and “OUTPUT” rules you also want applied to “FORWARD” rules and set things up accordingly. Also, the “nat” table is independent of the “filter” table and its rules are used to mangle packets rather than filter them.

   With that said, this HOWTO contains a helpful example set of rules for IP forwarding with NAT using iptables.

5. On Ubuntu (8.10) server (without X and its children), dhclient has the ability to run scripts before (“/etc/dhcp3/dhclient-enter-hooks.d/”) or after (“/etc/dhcp3/dhclient-exit-hooks.d/”) it configures interfaces from the information gathered through DHCP, a nice supplement to the good old “/etc/network/pre-if-up.d/” and “/etc/network/if-up.d/”, and “/etc/network/pre-if-down.d/” and “/etc/network/if-down.d/”. So, for example, you can use this to reload your iptables rules, if, say, you have rules tied to information that was configured via DHCP. Or, to invoke ddclient, if you have a public IP address assigned via DHCP and use a dynamic DNS service.

   E.g. a very generic ddclient script,

   sudo vi /etc/dhcp3/dhclient-exit-hooks.d/ddclient
   

   And add something like the following to it,

   (Note: This is an illustration and has not been tried by me exactly as it is here. Actually, ddclient seems like a rather poor example for this, but oh well.)

   # Note: only really makes a slight bit of sense if an interface
   # is configured with a DHCP assigned public IP address, and that
   # is the interface used by ddclient.
   
   DDCLIENT_CONF=/etc/ddclient.conf
   DDCLIENT_CACHE=/var/cache/ddclient/ddclient.cache
   DDCLIENT_USER=ddclient
   
   rerun_ddclient() {
           [ -e /usr/sbin/ddclient ] || return
   
           [ -e $DDCLIENT_CONF ] || return
   
   # Note: ddclient performs this sort of check itself as well.
           if [ -e $DDCLIENT_CACHE ] && [ -n "$old_ip_address" -a "$old_ip_address" = "$new_ip_address" ]; then
                   return
           fi
   
   # (I know, not /bin/sh -e, but habit.)
   
   # Runs ddclient as user other than root, but you must have
   # configured such a user and set ddclient up to work with them.
   #       /usr/bin/sudo -u $DDCLIENT_USER /usr/sbin/ddclient >/dev/null 2>&1 || true
   
   # ddclient is run as root here.
           /usr/sbin/ddclient >/dev/null 2>&1 || true
   }
   
   ddclient_whattodo() {
           case $reason in
                   BOUND|RENEW|REBIND|REBOOT)
                           rerun_ddclient
                           ;;
                   EXPIRE|FAIL|RELEASE|STOP)
                           ;;
                   *)
                           ;;
           esac
   }
   
   ddclient_whattodo
   

   The manpage for “dhclient-script” details the various “$reason”’s for why dhclient-script is being invoked as well as the other variables available, such as “$new_ip_address” and “$old_ip_address”, that can be used to determine the actions you want your scripts to take. Also, “/sbin/dhclient-script” uses “run-parts –list” to generate the list of scripts to be invoked, and this list is alphabetical. So, if your scripts need to be invoked in a certain order, keep that in mind.

   Note: While outside the scope being discussed here, if you are using the Gnome (2.24) NetworkManager to manage your network interfaces, then dhclient is invoked with the “-sf” option to override the use of the standard “/sbin/dhclient-script” with “/usr/lib/NetworkManager/nm-dhcp-client.action”, which does not run the scripts in “/etc/dhcp3/dhclient-enter-hooks.d/” or “/etc/dhcp3/dhclient-exit-hooks.d/”. You may still make due with using the good old “/etc/network/pre-if-up.d/” and “/etc/network/if-up.d/”, and “/etc/network/pre-if-down.d/” and “/etc/network/if-down.d/”, which have generally been fine for my use cases. Or, perhaps you could probably write a script to monitor D-Bus for a “PropertiesChanged” signal from “org.freedesktop.NetworkManager.DHCP4Config” and then check if a new IP has been assigned. Or, maybe there is a more standard plugin mechanism. Who knows, I haven’t played with it.

6. On Ubuntu (8.10) server (without X and its children), dhclient has this habit of overwriting “resolv.conf” by default. This is useful in many cases, but, if not, you can override it creating a custom “make_resolv_conf()” function in “/etc/dhcp3/dhclient-enter-hooks.d”.

   E.g.,

   sudo vi /etc/dhcp3/dhclient-enter-hooks.d/donotmodifyresolvconf
   

   And add something like the following to it,

   make_resolv_conf() {
           return
   }
   

   Note: While outside the scope being discussed here, if you are using the Gnome (2.24) NetworkManager to manage your network interfaces, then, as noted previously, dhclient is invoked with the “-sf” option to override the use of the standard “/sbin/dhclient-script”. So, to configure this sort of DHCP settings for a particular interface in Gnome using the NetworkManager, you could instead go to Main Menu->System->Preferences->Network Configuration, select the interface you want to configure and click “Edit”, go to the “IPv4 Settings” tab, and select “Automatic (DHCP) addresses only” along with filling in the static DNS information. Also, see this for underlying details of the settings available.

7. The Linux kernel “printk” setting determines what kernel messages will be displayed on the console. On a stock Ubuntu (8.10) server install (without X and its children, and using the stock syslog), one can use “sysctl kernel.printk” or “cat /proc/sys/kernel/printk” to display this setting, which will look something like “4 4 1 7″. What these numbers mean is as follows,

   include/linux/kernel.h

   #define	KERN_EMERG	"<0>"	/* system is unusable			*/
   #define	KERN_ALERT	"<1>"	/* action must be taken immediately	*/
   #define	KERN_CRIT	"<2>"	/* critical conditions			*/
   #define	KERN_ERR	"<3>"	/* error conditions			*/
   #define	KERN_WARNING	"<4>"	/* warning conditions			*/
   #define	KERN_NOTICE	"<5>"	/* normal but significant condition	*/
   #define	KERN_INFO	"<6>"	/* informational			*/
   #define	KERN_DEBUG	"<7>"	/* debug-level messages			*/
   

   kernel/printk.c

   /* printk's without a loglevel use this.. */
   #define DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */
   
   /* We show everything that is MORE important than this.. */
   #define MINIMUM_CONSOLE_LOGLEVEL 1 /* Minimum loglevel we let people use */
   #define DEFAULT_CONSOLE_LOGLEVEL 7 /* anything MORE serious than KERN_DEBUG */
   

   int console_printk[4] = {
   	DEFAULT_CONSOLE_LOGLEVEL,	/* console_loglevel */
   	DEFAULT_MESSAGE_LOGLEVEL,	/* default_message_loglevel */
   	MINIMUM_CONSOLE_LOGLEVEL,	/* minimum_console_loglevel */
   	DEFAULT_CONSOLE_LOGLEVEL,	/* default_console_loglevel */
   };
   

   So, at the time of this writing, we see that “kernel.printk=”7 4 1 7″” is the 2.6 Linux kernel default; on Ubuntu (8.10), “kernel.printk” is set to “4 4 1 7″ by default (see “/etc/sysctl.d/10-console-messages.conf”). Let’s say I want “kernel.printk” to be set to “6 4 1 7″.

   There are lots of ways to accomplish this. Besides “klogd -c” or the old “echo /proc/sys/kernel/printk”, sysctl can be used to set “/proc/sys” parameters at runtime (e.g., “sudo sysctl kernel.printk=”6 4 1 7″”) on Ubuntu (8.10), but such changes are lost at reboot and let’s say I want this setting applied at each boot. Ordinarily, if you want certain “/proc/sys” parameters to be automatically set at boot and you are not doing so anywhere else (e.g., in some other rc.d script), you either add files in “/etc/sysctl.d” containing the parameter settings, or you add/modify parameters in “/etc/sysctl.conf”.

   Now, on Ubuntu (8.10), the settings in “/etc/sysctl.d/*.conf” are applied after those in “/etc/sysctl.conf”, which means “/etc/sysctl.d/*.conf” settings will override “/etc/sysctl.conf” settings. Since Ubuntu (8.10) uses “/etc/sysctl.d/10-console-messages.conf” to set “kernel.printk=”4 4 1 7″”, I need to either edit that conf or add my own that is set after that one. To do things “properly,” I should go with the latter, by, say, copying “10-console-messages.conf” to “60-console-messages.conf” (60 is the standard prefix for user customizations) and then editing “60-messages-console.conf” to contain “kernel.printk=”6 4 1 7″”.

   I guess, rather than overriding “printk” defaults, I could instead configure the stock syslog by, say, modifying its default configuration file (“/etc/syslog.conf”) to display log entries to the console, and this could be used to cover both system logs and kernel messages. Leaving alone the Ubuntu “kernel.printk” default of “4 4 1 7″, by adding a line like “kern.info;kern.!err /dev/console”, kernel messages of priority “info” (6) or more critical should be displayed to the console (the “kern.!err” is there because we are already displaying “err” and higher priority messages by virtue of the Ubuntu default “kernel.printk” configuration). Going broader than just the kernel, a line like “*.info /dev/console” would cover all system log entries of priority “info” or more critical. Generally though, if I want kernel messages dumped to the console, I work within the parameters of the kernel itself, namely “kernel.printk”.

   Coming back to iptables, the “log-level” option of iptables “LOG” rules can be used to tweak what will hit the console. Now, unless iptables is told to use a specific loglevel, the system default seems to be used. So, with the “4 4 1 7″ setting used by default on Ubuntu (8.10), iptables “LOG” rules without the “–log-level” option will not result in console messages, and only those “LOG” rules with a “–log-level err” or more serious will hit the console. With the “7 4 1 7″ kernel default, iptables “LOG rules” without the “–log-level” option will result in console messages, and only those “LOG” rules with a “–log-level debug” will not hit the console.

   Tying this into syslog a bit, if, say, there are quite a bit of kernel log entries generated by iptables and whittling back the “LOG” rules is not satisfactory, syslog rules could be used to split the kernel logs into pieces to help with manual review and remove redundant logging. So, using the “–log-level debug” in “LOG” rules and adding an entry like “kern.=debug -/var/log/kern.debug” in “/etc/syslog.conf” would place all iptables log entries (and other kernel messages with a priority of “debug”) into “/var/log/kern.debug”, while the default “/var/log/kern.log” would still contain all kernel messages (and the default “/var/log/debug” would still contain most debug log entries and the default “var/log/syslog” would still contain most log entries). Or, you could split the kernel logs up into, say, “kern.debug”, “kern.info;kern.!err”, and “kern.err”, and then use “–log-level” rules to determine to which kernel log the iptables “LOG” rules are logged. Combining such a split of the kernel logs with removing kernel messages from the default logs could help reduce redundancy as well, although care should be taken because sometimes applications depend on the default logs being, well, set to the defaults. (On a side note, if you are doing more than just simple basics with syslog, it may be time to consider replacements like rsyslog and syslog-ng.)

   (Whew, I hope I typed all that correctly.)

8. Having had to rebuild djbdns a bit recently and remembering this post, I updated “/etc/event.d/svscan” to handle a system shutdown a bit friendlier.

   E.g.,

   sudo vi /etc/event.d/svscan
   

   And add something like the following to it,

   (Note: The following is an example and has not been tried by me exactly as it is here. FYI, “svscanboot” might be located somewhere other than “/command”.)

   # svscan - daemontools -- http://www.froyn.net/blosxom/blosxom.cgi/2007/1/12
   #
   
   # This service maintains an svscan process from the point the system is
   # started until it is shut down again.
   
   start on runlevel 2
   start on runlevel 3
   start on runlevel 4
   start on runlevel 5
   
   stop on runlevel 0
   stop on runlevel 1
   stop on runlevel 6
   
   exec /command/svscanboot
   respawn