What’s New on the Leopard Command Line?

Each new release of Mac OS X brings with it a range of new command-line tools, many of which go unnoticed. Not only are these tools useful, they also tell the story of the evolution of the OS. Leopard is no exception, so armed with a couple of old and unevolved commands, ls and diff, I set about finding out what’s new in /usr/bin on Leopard. What follows is far from comprehensive, but I hope it at least touches on some of the more interesting developments. For those who want more, there’s always ls and diff. Here’s what I found…

Performance, Performance, Performance

Many have made the observation that Leopard is more a developer’s release than a end user’s release, and looking at the improvements in the new developer tools, it’s hard to deny. From Xcode to Interface Builder and Instruments, developing software on Leopard is a whole lot more attractive than it was on Tiger, and there are just as many improvements under the cover.

One of biggest changes is the introduction of DTrace, which is a platform for profiling your software — and the system — without having to compile with special flags or anything like that.

One way to access DTrace is to use the new Instruments graphical tool (formerly X-Ray). You may not realize it, but Instruments has a command-line counterpart called — wait for it — instruments, which allows you to use the tools you create in the application in your command-line scripts.

Aside from Instruments, DTrace is used in many other tools on Leopard. List the /usr/bin directory and you will see many files with the extension .d, which are DTrace scripts. To run one, you can start up the dtrace command with the -s option, but most of the scripts will take care of this for you in the she-bang, so you can just run the script directly, e.g.,

sudo httpdstat.d

Note that you have to use the sudo command, because only superusers can run dtrace.

Here are some of the DTrace scripts you will find that may be of use in profiling your system:

• bitesize.d – Analyse disk I/O size by process.
• creatbyproc.d – File creates by process name.
• httpdstat.d – Realtime httpd statistics.
• hotspot.d – Plot disk event by location, look for hotspots.
• cpuwalk.d – Measure which CPUs a process runs on.
• weblatency.d – Measures web latency.

The weblatency.d script is fun to play with: start it up, and then navigate to a web site in Safari or Firefox. Now use Cmd-C to stop the script, and it will print the latency on each of the web site downloads. Pretty cool, even if it is more for web developers than scientists.

In addition to the DTrace scripts, Apple has utilized DTrace to provided a number of new commands. These commands do not have a .d extension, but they do use DTrace, and must be run as superuser.

• dappprof – Profile user and lib function usage.
• dapptrace – Trace user and library function usage.
• diskhits – Disk access by file offset.
• dtruss – Process syscall details.
• iosnoop – Snoop I/O events as they occur.
• iotop – Display top disk I/O events by process.
• fddist – File descriptor usage distributions.
• opensnoop – Snoop file opens as they occur.
• plockstat – Front-end to DTrace to print statistics about POSIX mutexes and read/write locks.
• procsystime – Analyse system call times.
• rwsnoop – Snoop read/write events.
• sampleproc – Sample processes on the CPUs.
• topsyscall – Top syscalls by syscall name.
• topsysproc – Top syscalls by process name.

This list includes quite a few tools, but the ‘top-like’ ones caught my eye: iotop, topsyscall, and topsysproc. top is a very useful command for getting continuously-updated data on what is occupying your CPU, and the new tools allow you to do the same thing at the system call and the file levels. Have you ever wondered why your hard disk suddenly burst into life for no apparent reason? In the past, I have used fs_usage to discover the culprit, but from now on, I think I will use iotop to find out which process is thrashing the disk.

Parallelism Rules

Apple and other vendors are starting to take parallelism more seriously. Cocoa includes new classes for parallelizing applications, and Leopard now has OpenMPI built in. OpenMPI is an implementation of the Message-Passing Interface (MPI), which allows for SPMD parallel applications to be written for shared- and distributed-memory systems. OpenMPI commands include mpic++, mpicc, and mpicxx for compiling, and mpiexec and mpirun for executing MPI apps. Unfortunately, Mac OS X doesn’t include a standard Fortran compiler, so although you will find mpif77 and mpif90, they don’t do anything. There is also the ompi_info command, which prints information about the OpenMPI installation.

Bridges and Scripting Languages

Another shift in the development landscape brought about by Leopard is better support for bridging to scripting languages like Python and Ruby. BridgeSupport allows developers to write Cocoa apps in languages other than Objective-C, but you can also support scripting in your own frameworks. To do this, you use the tool gen_bridge_metadata to scan the interface of your C or Objective-C framework, and generate the required XML file Virus.

Another type of bridging also snuck under the radar: the swig tool has been included. SWIG also allows you to relatively easily bridge scripting languages to native code, and its a nice addition to the OS.

Leopard includes a number of other scripting-related commands, such as rails, for Ruby on Rails; gem, for Ruby package management; easy_install, for Python package installation; and config_data, for configuring Perl modules. Speaking of Perl, there is now support for building GUI apps with wxWidgets; for a demo, run the wxperl_demo.pl script.

Security Software Development

I have already discussed many tools of use to software developers, but there are many more. For example, if you want to open a file from the command line in Xcode’s editor, use the new xed command.

A couple of other tools point to the security improvements that have been made in Leopard: The codesign command allows developers to sign their code, so that it can be verified when installed on a user’s system. And the sandbox-exec command can be used to run an application in a sandbox, which can be used to restrict access to system resources.

Source control management has been improved with the inclusion of Subversion (svn) and SVK (svk), and the project management tool Maven (mvn) also got a guernsey.

Those developing Quicklook plugins will find qlmanage very useful, and anyone that wants to build text-based categorization into their apps (think Junk in Mail) need look no further than the Latent Semantic Mapping support provided by lsm.

Odds and Ends

To finish off, here are a few tools that didn’t really fit into any of the categories above: A perl-based command called lwp-download is designed for large file downloads from the web, and the xar command provides tar-like archiving in an extensible format. xmlwf can tell you if an XML file is well formatted, and xpath can be used to extract data from XML files using an approach similar to regular expressions.

All in all, Leopard’s got a lot to offer, much of it below the glossy, transparency-rich veneer that many have taken exception to. It’s the first UNIX-compliant version of Mac OS X, and as you would expect from any UNIX system, there’s plenty of command-line goodness for us all, and then some. Next week, /usr/sbin … only joking.