go-diskfs

go-diskfs is a go library for performing manipulation of disks, disk images and filesystems natively in go.

You can do nearly everything that go-diskfs provides using shell tools like gdisk/fdisk/mkfs.vfat/mtools/sgdisk/sfdisk/dd. However, these have the following limitations:

• they need to be installed on your system
• you need to fork/exec to the command (and possibly a shell) to run them
• some are difficult to run without mounting disks, which may not be possible or may be risky in your environment, and almost certainly will require root privileges
• you do not want to launch a VM to run the excellent libguestfs and it may not be installed

go-diskfs performs all modifications natively in go, without mounting any disks.

Usage

Note: detailed go documentation is available at godoc.org.

Concepts

go-diskfs has a few basic concepts:

• Disk
• Partition
• Filesystem

Disk

A disk represents either a file or block device that you access and manipulate. With access to the disk, you can:

• read, modify or create a partition table
• open an existing or create a new filesystem

Partition

A partition is a slice of a disk, beginning at one point and ending at a later one. You can have multiple partitions on a disk, and a partition table that describes how partitions are laid out on the disk.

Filesystem

A filesystem is a construct that gives you access to create, read and write directories and files.

You do not need a partitioned disk to work with a filesystem; filesystems can be an entire disk, just as they can be an entire block device. However, they also can be in a partition in a disk

Working With a Disk

Before you can do anything with a disk - partitions or filesystems - you need to access it.

• If you have an existing disk or image file, you Open() it
• If you are creating a new one, usually just disk image files, you Create() it

The disk will be opened read-write, with exclusive access. If it cannot do either, it will fail.

Once you have a Disk, you can work with partitions or filesystems in it.

Partitions on a Disk

The following are the partition actions you can take on a disk:

• GetPartitionTable() - if one exists. Will report the table layout and type.
• Partition() - partition the disk, overwriting any previous table if it exists

As of this writing, supported partition formats are Master Boot Record (mbr) and GUID Partition Table (gpt).

Filesystems on a Disk

Once you have a valid disk, and optionally partition, you can access filesystems on that disk image or partition.

• CreateFilesystem() - create a filesystem in an individual partition or the entire disk
• GetFilesystem() - access an existing filesystem in a partition or the entire disk

As of this writing, supported filesystems include FAT32 and ISO9660 (a.k.a. .iso).

With a filesystem in hand, you can create, access and modify directories and files.

• Mkdir() - make a directory in a filesystem
• Readdir() - read all of the entries in a directory
• OpenFile() - open a file for read, optionally write, create and append

Note that OpenFile() is intended to match os.OpenFile and returns a godiskfs.File that closely matches os.File

With a File in hand, you then can:

• Write(p []byte) to the file
• Read(b []byte) from the file
• Seek(offset int64, whence int) to set the next read or write to an offset in the file

Read-Only Filesystems

Some filesystem types are intended to be created once, after which they are read-only, for example ISO9660/.iso and squashfs.

godiskfs recognizes read-only filesystems and limits working with them to the following:

• You can GetFilesystem() a read-only filesystem and do all read activities, but cannot write to them. Any attempt to Mkdir() or OpenFile() in write/append/create modes or Write() to the file will result in an error.
• You can CreateFilesystem() a read-only filesystem and write anything to it that you want. It will do all of its work in a "scratch" area, or temporary "workspace" directory on your local filesystem. When you are ready to complete it, you call Finalize(), after which it becomes read-only. If you forget to Finalize() it, you get... nothing. The Finalize() function exists only on read-only filesystems.

Example

There are examples in the examples/ directory. Here is one to get you started.

The following example will create a fully bootable EFI disk image. It assumes you have a bootable EFI file (any modern Linux kernel compiled with CONFIG_EFI_STUB=y will work) available.

import diskfs "github.com/diskfs/go-diskfs"

espSize int := 100*1024*1024 // 100 MB
diskSize int := espSize + 4*1024*1024 // 104 MB


// create a disk image
diskImg := "/tmp/disk.img"
disk := diskfs.Create(diskImg, diskSize, diskfs.Raw, diskfs.SectorSizeDefault)
// create a partition table
blkSize int := 512
partitionSectors int := espSize / blkSize
partitionStart int := 2048
partitionEnd int := partitionSectors - partitionStart + 1
table := PartitionTable{
	type: partition.GPT,
	partitions:[
		Partition{Start: partitionStart, End: partitionEnd, Type: partition.EFISystemPartition, Name: "EFI System"}
	]
}
// apply the partition table
err = disk.Partition(table)


/*
 * create an ESP partition with some contents
 */
kernel, err := ioutil.ReadFile("/some/kernel/file")

fs, err := disk.CreateFilesystem(0, diskfs.TypeFat32)

// make our directories
err = fs.Mkdir("/EFI/BOOT")
rw, err := fs.OpenFile("/EFI/BOOT/BOOTX64.EFI", os.O_CREATE|os.O_RDRWR)

err = rw.Write(kernel)

Tests

There are two ways to run tests: unit and integration (somewhat loosely defined).

• Unit: these tests run entirely within the go process, primarily test unexported and some exported functions, and may use pre-defined test fixtures in a directory's testdata/ subdirectory. By default, these are run by running go test ./... or just make unit_test.
• Integration: these test the exported functions and their ability to create or manipulate correct files. They are validated by running a docker container with the right utilities to validate the output. These are run by running TEST_IMAGE=diskfs/godiskfs go test ./... or just make test. The value of TEST_IMAGE will be the image to use to run tests.

For integration tests to work, the correct docker image must be available. You can create it by running make image. Check the Makefile to see the docker build command used to create it. Running make test automatically creates the image for you.

Integration Test Image

The integration test image contains the various tools necessary to test images: mtools, fdisk, gdisk, etc. It works on precisely one file at a time. In order to avoid docker volume mounting limitations with various OSes, instead of mounting the image -v, it expects to receive the image as a stdin stream, and saves it internally to the container as /file.img.

For example, to test the existence of directory /abc on file $PWD/foo.img:

cat $PWD/foo.img | docker run -i --rm $INT_IMAGE mdir -i /file.img /abc

Plans

Future plans are to add the following:

• embed boot code in mbr e.g. altmbr.bin (no need for gpt since an ESP with /EFI/BOOT/BOOT<arch>.EFI will boot)
• ext4 filesystem
• Joliet extensions to iso9660
• Rock Ridge sparse file support - supports the flag, but not yet reading or writing
• squashfs sparse file support - currently treats sparse files as regular files
• qcow disk format