This page contains a loosely-categorized list of software I've created and that I can talk about more openly. It's not an exhaustive list.
- Various aspects of “cloud” storage
- fileXray—Software for analyzing, dissecting, monitoring, scavenging, and performing forensic investigation on HFS+ volumes
- MacFUSE—Mechanism for creating arbitrary file systems in user space on Mac OS X
- AncientFS—Ancient Unix file systems on Mac OS X, Linux, and FreeBSD
- UnixFS—A General-purpose layer for writing Unix-style MacFUSE-based file systems
- UnixFS-based user-space versions of the UFS, System V, and Minix file system families
- GrabFS—A file system that contains live screenshots of all on-screen and off-screen windows
- procfs—A MacFUSE-based process file system for Mac OS X
- sshfs—A Mac OS X port of the FUSE-based SSH file system from Linux
- DiskFS—An in-kernel file system for mounting disk devices as regular files
- hfsdebug — A file system debugger for HFS+ volumes
- fslogger — A file system activity monitor based on the FSEvents mechanism in Mac OS X
- A database-backed searchable file system
- Simulation of "interesting" hypothetical disk drives for even more interesting file system applications
Security and Trusted Computing
- A Trusted Platform Module (TPM) device driver for Mac OS X
- A Mac OS X Port of the TrouSerS Trusted Software Stack
- Software emulation of the TPM
- Mechanism for "unspoofable" secure communication in insecure environments
- Mechanism for autonomic security of managed network systems (firmware and software)
Miscellaneous Low-Level Software
- XBinary—Extended binary format support for Mac OS X
- gcore—Handcrafting process core dumps on Mac OS X
/dev/kmemdevice driver for Mac OS X
- Software for general-purpose use of motion sensors in mainstream computers
- Software for general-purpose use of light sensors in mainstream computers
- Software for receiving/manipulating mouse, keyboard, and infrared remote events
- Software for manipulating keyboard LEDs
- Software for experimenting with the binary protection mechanism in Mac OS X
- Linux and Mac OS X ports of the preamble patching mechanism used by google-perftools
- Mechanisms for remotely and seamlessly "shepherding" processes
- A user-space mechanism for editing environment variables in other ("remote") processes
- An in-kernel mechanism for loading code into other ("remote") processes
- An implementation of anonymous POSIX semaphores for Mac OS X
Virtualization, Sandboxing, and Quality of Service
A Virtualized Solaris Operating System with Quality of Service, 1999—2000
I created a "virtualized" version of Sun's Solaris operating system. The idea is to divide the operating system into multiple virtual environments by creating a software layer in the kernel. Each virtual environment is capable of running arbitrarily complicated existing applications unmodified. A complicated application, say the Oracle database suite, would typically exercise most components/subsystems of the operating system. Resources are made available to a virtual instance with Quality of Service. Moreover, applications in one virtual system instance are in complete isolation from applications in other instances on the same "real" machine.
Each virtual instance can be managed—administered, configured, rebooted, shutdown etc.—completely independently of the others and is visible as the "normal" operating system to applications within it. Note that this is different from an emulator: there is only one instance of the system kernel, but the APIs have been virtualized within the kernel. This results in a much higher virtual instance performance than would be possible with an emulator.
Specific virtualization components include, but are not limited to, the following:
- Virtual system calls
- Virtual uid 0 (each instance has its own "root" user)
- Fair share network scheduler
- Per-virtual OS resource limits on memory, CPU, and link
- Virtual sockets and TLI (including the sockets port space)
- Virtual IP address space
- Virtual NFS
- Virtual disk driver and enhanced VFS (each instance sees its own physical disk that may be resized dynamically, and partitioned as usual)
- Virtual System V IPC layer (each instance gets its own IPC namespace)
/dev/kmem(each instance can access
/dev/kmemappropriately without compromising other instances, or the system in general)
/procfile system (each instance gets its own
/procwith only its processes showing up)
- Virtual syslog facility
- Virtual device file system
- Overall system management layer
Note that this was product quality software and all work was done without ever having seen the source code for Solaris, which was closed and proprietary to Sun at that time and their source license had enough "wrong" strings attached from my company's point of view.
I've been asked sometimes how this virtualization project (referred to as "V" from now on) relates to, or is different from the "Zones" feature in Solaris 10, or the FreeBSD "jail" subsystem. Here are some thoughts on this, assuming the reader is familiar with Solaris Zones and FreeBSD "jail":
- "V"'s goals are different (loftier, in many ways) from the others: it strives to give you the benefits of an OS emulator (or a real OS running on a hardware platform emulator) with far less overhead. Like others, it provides you isolated environments in which you can securely run applications, but unlike others, its isolated environments are very much like the full-blown underlying OS. For example, a virtual environment in "V" comes up just like a normal system (its own
initand startup scripts). Having said that, the "Zones" feature does appear to be very similar.
- "V" lets you install and run most applications within a virtual instance, except those that access hardware directly. Since there is a virtual disk driver, applications that want to access disk(s) directly are allowed. The capacity of this virtual disk can be altered dynamically, even if there is a file system on it.
- Like others (to varying degrees), "V" lets you assign per-virtual-instance resource limits on CPU, memory, disk space, network bandwidth etc.
- "V" is implemented as a set of loadable kernel modules without any need for the kernel source. It can be dynamically introduced into a running system.
- The project was started in 1999 (on Solaris 7, later carried over to Solaris 8) and was largely done in 2000. At that time, there was no talk of Solaris "Zones" and "jail" was not as mature as it is today. It makes me happy to see Sun heading in a similar direction today with "Zones". (And they do have the kernel source!)
- At the time of this writing, a Solaris Zone cannot be an NFS server, whereas a "V" instance could.
- At the time of this writing, a Solaris Zone does not allow the
mknodsystem call, while "V" did.
- There are most likely features in Zones that "V" did not.
Resource Management for Quality of Service in a Custom Operating System, 1998—1999
I worked on introducing Resource Management for Quality of Service in a custom operating system derived from FreeBSD. This included work on schedulers for CPU, network, and disk; a pseudo file system based resource management API; and a resource management layer to provide seamless quality of service to legacy, unmodified applications.
Quality of Service Support for Legacy Applications, 1998—1999
While working on a FreeBSD-derived operating system with Quality of Service (QoS), I extended the NFS, HTTP, and FTP protocols to make them QoS-enabled. The extensions were such that legacy user applications could benefit from QoS without having to be modified in any way.
Socket Credential Query Mechanism, 2006
I created a kernel mechanism for Mac OS X to allow a user-space caller to query the credentials (such as the user and process identifiers) of the peer of a TCP socket involved in a local connection. The interface is easy-to-use, lightweight, unspoofable, and extensible.
Signaled Receiver Processing, 1998—1999
Protocol processing of received packets in BSD Unix is interrupt-driven and may cause scheduling anomalies that are unacceptable in systems that provide Quality of Service (QoS) guarantees. In collaboration with other researchers, I implemented an alternate mechanism—Signaled Receiver Processing (SRP)—that generates a signal to the receiving process when a packet arrives. The default action of this signal is to perform protocol processing asynchronously. However, a receiving process may catch, block or ignore the signal and defer protocol processing until a subsequent receive call. In any case, protocol processing occurs in the context of the receiving process and is correctly charged. Therefore, SRP allows the system to enforce and honor QoS guarantees.
Embedded Software/Firmware Programming, 1998—
I've created software for various embedded/firmware environments over the years. My Towers Of Hanoi hobby project includes software for the Sega Dreamcast Game Console, the Nintendo Gameboy Advance Hand-held System, and the RioCar MP3 Player. My other related experiments include creating software for the ProGear WebPad, running TCP/IP over Infra-red, and populating a Debian Linux distribution on an ARM-based embedded system for on-board software development, running 5th edition UNIX on the Nintendo Game Boy Advance, and projects involving MIPS, PowerPC, and x86 development boards.
For examples of programming in the firmware environment, see:
For examples of low-level bringup, see:
- Test-Driving Linux on an Intel-Based Macintosh
- Knoppix on the Intel-Based Macintosh
- BAMBIOS: Legacy BIOS-Based Booting on the Intel-Based Macintosh
High Level Software
- Software Strategies for "Linux on the Desktop"
- One-Click Migration of User "Personalities" (All Configuration and Personal Data) between Different Operating Systems
- Extensions to Perl and Python Runtimes to Support Signed and/or Encrypted Scripts and Modules
- Design of Software Licensing Schemes
A Linux/Unix Software Stack for Implementing and Deploying Arbitrary Web Services, 2002
This is a highly extensible system for implementing, deploying, and managing web services and their APIs. I envisioned, designed, and implemented this system. Later I led the project's further development and management. Its key components and features include the following:
- Object-Oriented Plug-in Architecture: The system is modular in that new services can be plugged-in. The plug-ins themselves are dynamically loaded in a fine-grained manner.
- N-Level Recursive: The system can hierarchically instantiate itself, thereby enabling "reseller" support, with each reseller level being arbitrarily customizable and configurable.
- Enterprise Database Support: A database abstraction layer makes the system capable of using most of the major databases. The database to use is selected through a single setting. Configurations for IBM DB2, Oracle, and Postgres were bundled.
- Platform Independence: The system is implemented entirely in Perl, and is capable of functioning fully with no use of platform dependent features of the language.
- Well-defined APIs: The system's set of APIs is exported via multiple mechanisms: Web Services (SOAP), XML-RPC over HTTP/HTTPS and raw XML over SSH (primarily for legacy support). The system allows easy definition of custom APIs through powerful XML libraries built on top of a DOM parser.
- Error Subsystem: The exception based error subsystem collects all errors encountered along with extensive context information, and is capable of reporting the information in a detailed debugger-style stack-trace, presented as one or more of the following: pre-formatted text, XML (for parsing by a GUI, say) and pre-formatted HTML (for ready inclusion by a GUI, say).
- The Registry: This module consists of a configuration and settings manager, and an extensible validation library. This provides a flexible way to store, retrieve, and validate arbitrarily complicated data. The validation library can perform various kinds of checks on arbitrarily typed data (like "email address list", "list of FQDNs", etc.). The module employs shared memory for better performance and scalability on systems that support System V IPC.
- The Auth Module: This module provides various methods of authorizing and authenticating requests.
- Internationalization Support: All messages in the system are internationalized.
- Black Box Clients: The system provides black box clients on various platforms—Linux, Solaris, and Windows 2000—for the APIs it exports. These clients are self-contained, and stream XML over their standard input/output.
Flexible and Automated System Installation, 1997
I extended the "kick-start" installation procedure of Red Hat Linux to simplify it and to support additional installation methods. I designed a hands-off installation scheme for x86 systems wherein special-purpose boot code can be placed in the PROM of the network card and upon booting, the user is given the option of either installing an operating system on the computer, running a minimal system in RAM, or booting entirely off the network with an NFS root filesystem.
An Audit and System Call Interception Mechanism for the Linux Kernel, 1997
I designed and implemented a dynamically loadable system call audit mechanism for the Linux kernel. Using this mechanism, it is possible to arrange for arbitrary context sensitive operations to happen upon execution of a system call. For example, a set of system calls can be denied to certain users. Arbitrary system time can be returned to specific applications—for testing time related bugs, say, like the erstwhile Y2K. It is also possible to associate the execution of certain user level programs when certain actions occur. For example, users can associate passwords with files, etc.
Other Esoteric Pursuits
Many Systems on a PowerBook, 2003
A large number of operating systems running in virtual machines.
I created myriad implementations of the Towers Of Hanoi program. Please go to the Hanoimania page for details.
Obfuscated Escapes, 1994—
Some examples of code obfuscated on purpose.