2.10. Harddisk Images based on redologs

This section describes how the three new disk images "undoable", "growing", and "volatile" are implemented in Bochs 2.1. It also applies to the write support the "vvfat" disk image mode in Bochs 2.4.6.

2.10.1. Description

The idea behind volatile and undoable disk images is to have a read-only base file, associated with one redolog file. In case of vvfat, a directory is associated with the redolog file.

Reading a sector is done from the redolog file if it contains the sector, or from the base file / vvfat directory otherwise.

Sectors written go to the redolog, so base image files are opened in read only mode in this configuration.

The redolog is designed in a way so it starts as a small file and grows with every new sectors written to it. Previously written sectors are done in place. Redolog files can not shrink.

The redolog is a growing file that can be created on the fly.

Now, it turns out that if you only use a redolog without any base image file, you get a "growing" disk image.

So "undoable", "volatile", "growing" and "vvfat" harddisk images classes are implemented on top of a redolog class.

2.10.2. How redologs works ?

At the start of a redolog file, there is a header, so Bochs can check whether a file is consistent. This header is also checked when the automatic type and size detection is selected.

The generic part of the header contains values like type of image, and spec version number.

The header also has a specific part. For redologs, the number of entries of the catalog, the extent, bitmap and disk size are stored.

In a redolog, the disk image is divided in a number of equal size "extents". Each extent is a collection of successive 512-bytes sectors of the disk image, preceeded by a n*512bytes bitmap.

the n*512bytes bitmap defines the presence (data has been written to it) of a specific sector in the extent, one bit for each sector. Therefore with a 512bytes bitmap, each extent can hold up to 4k blocks

Typically the catalog can have 256k entries. With a 256k entries catalog and 512bytes bitmaps, the redolog can hold up to 512GiB

Note: All data is stored on images as little-endian values

2.10.2.1. Header

At the start of a redolog file, there is a header. This header is designed to be reusable by other disk image types.

The header length is 512 bytes. It contains :

Table 2-8. Generic header description

Start position in bytesLength in bytesData typeDescriptionPossible values
0 32 string magical value Bochs Virtual HD Image
32 16 string type of file Redolog
48 16 string subtype of file Undoable, Volatile, Growing
64 4 Bit32u version of used specification 0x00010000, 0x00020000
68 4 Bit32u header size 512

The current version of the header is 0x00020000 (2.0) - see below for details.

Table 2-9. Redolog specific header description

Start position in bytesLength in bytesData typeDescription
72 4 Bit32u number of entries in the catalog
76 4 Bit32u bitmap size in bytes
80 4 Bit32u extent size in bytes
84 4 Bit32u timestamp in FAT format ("undoable" mode only - otherwise reserved)
88 8 Bit64u disk size in bytes

The reserved field between "extent" and "disk" has been added in redolog version 2.0 to fix an alignment bug on some platforms. It is now used for consistency check of the "undoable" mode. When creating the redolog file, the timestamp of the read-only file is stored there (in FAT format). After that, the "undoable" mode init code compares the timestamp of the r/o file with the one stored in the redolog.

2.10.2.2. Catalog

Immediately following the header, there is a catalog containing the position number (in extents) where each extent is located in the file.

Each position is a Bit32u entity.

2.10.2.3. Bitmap

Each extent starts with a bitmap block of n*512 bytes size. Each byte of the bitmap stores the write status of 8 coresponding disk sectors in the extent (1 = data written).

2.10.2.4. Extent

This is a collection of successive 512-bytes sectors of the disk image. The bitmap preceeding this data block contains the write status of each sector.

2.10.3. Parameters

The following tables shows what parameters are used when creating redologs or creating "growing" images :

Table 2-10. How number of entries in the catalog and number of blocks by extents are computed

Catalog entriesCatalog size(KiB)Bitmap size (B)Extent size (KiB)Disk Max Size
5122142MiB
5122284MiB
1k4288MiB
1k441616MiB
2k841632MiB
2k883264MiB
4k16832128MiB
4k161664256MiB
8k321664512MiB
8k32321281GiB
16k64321282GiB
16k64642564GiB
32k128642568GiB
32k12812851216GiB
64k25612851232GiB
64k256256102464GiB
128k5122561024128GiB
128k5125122048256GiB
256k10245122048512GiB
256k1024102440961TiB
512k2048102440962TiB
512k2048204881924TiB
1024k4096204881928TiB
1024k409640961638416TiB
2048k819240961638432TiB

2.10.4. Redolog class description

The class redolog_t(); implements the necessary methods to create, open, close, read and write data to a redolog. It also contains methods for the subtype and consistency check and for the save/restore support. Managment of header catalog and sector bitmaps is done internally by the class.

2.10.4.1. Constants

#define STANDARD_HEADER_MAGIC     "Bochs Virtual HD Image"
#define STANDARD_HEADER_VERSION   (0x00020000)
#define STANDARD_HEADER_SIZE      (512)
These constants are used in the generic part of the header.

#define REDOLOG_TYPE "Redolog"
#define REDOLOG_SUBTYPE_UNDOABLE "Undoable"
#define REDOLOG_SUBTYPE_VOLATILE "Volatile"
#define REDOLOG_SUBTYPE_GROWING  "Growing"
These constants are used in the specific part of the header.

#define REDOLOG_PAGE_NOT_ALLOCATED (0xffffffff)
This constant is used in the catalog for an unwritten extent.

2.10.4.2. Methods

redolog_t(); instanciates a new redolog.

int make_header(const char* type, Bit64u size); creates a header structure in memory, and sets its type and parameters based on the disk image size. Returns 0.

int create(const char* filename, const char* type, Bit64u size); creates a new empty redolog file, with header and catalog, named filename of type type for a size bytes image. Returns 0 for OK or -1 if a problem occured.

int create(int filedes, const char* type, Bit64u size); creates a new empty redolog file, with header and catalog, in a previously opened file described by filedes, of type type for a size bytes image. Returns 0 for OK or -1 if a problem occured.

int open(const char* filename, const char* type, Bit64u size); opens a redolog file named filename, and checks for consistency of header values against a type and size. Returns 0 for OK or -1 if a problem occured.

int open(const char* filename, const char* type, Bit64u size, int flags); opens a redolog file with flags applied. This allows to open a redolog in read-only mode. All other parameters and the return value are similar to the default open() method above.

void close(); closes a redolog file.

off_t lseek(off_t offset, int whence); seeks at logical data offset offset in a redolog. offset must be a multiple of 512. Only SEEK_SET and SEEK_CUR are supported for whence. Returns -1 if a problem occured, or the current logical offset in the redolog.

ssize_t read(void* buf, size_t count); reads count bytes of data of the redolog, from current logical offset, and copies it into buf. count must be 512. Returns the number of bytes read, that can be 0 if the data has not previously be written to the redolog.

ssize_t write(const void* buf, size_t count); writes count bytes of data from buf to the redolog, at current logical offset. count must be 512. Returns the number of bytes written.

Bit64u get_size(); returns the size stored in the "disk" field in the header. This is used for size autodetection feature ("growing" mode) and the consistency check ("undoable" mode).

Bit32u get_timestamp(); returns the value of the "timestamp" field in the header (only used by the "undoable" mode).

bx_bool set_timestamp(Bit32u timestamp); writes the timestamp to the header. This is done by the "undoable" mode init code if get_timestamp() returns 0 or the redolog is newly created.

static int check_format(int fd, const char *subtype); checks the format of the file with descriptor fd. Returns HDIMAGE_FORMAT_OK if the subtype matches the requested one. This is used for for the image mode autodetection feature.

bx_bool save_state(const char *backup_fname); copies the redolog file to a new file backup_fname. This is used by the hdimage save/restore feature.

2.10.5. Disk image classes description

"volatile" and "undoable" disk images are easily implemented by instanciating a device_image_t object (base image) and a redolog_t object (redolog).

"growing" disk images only instanciates a redolog_t object.

Class names are undoable_image_t, volatile_image_t and growing_image_t.

When using these disk images, the underlying data structure and layout is completely hidden to the caller. Then, all offset and size values are "logical" values, as if the disk was a flat file.

2.10.5.1. Constants

#define UNDOABLE_REDOLOG_EXTENSION ".redolog"
#define UNDOABLE_REDOLOG_EXTENSION_LENGTH (strlen(UNDOABLE_REDOLOG_EXTENSION))
#define VOLATILE_REDOLOG_EXTENSION ".XXXXXX"
#define VOLATILE_REDOLOG_EXTENSION_LENGTH (strlen(VOLATILE_REDOLOG_EXTENSION))
These constants are used when building redolog file names

2.10.5.2. undoable_image_t methods

undoable_image_t(Bit64u size, const char* redolog_name); instanciates a new undoable_image_t object. This disk image logical length is size bytes and the redolog filename is redolog_name.

int open(const char* pathname); opens the disk image pathname in read-only mode, as an undoable disk image. The image mode of this base image is auto-detected. All supported disk image modes can be used here. The associated redolog will be named pathname with a UNDOABLE_REDOLOG_EXTENSION suffix, unless set in the constructor. Returns 0 for OK or -1 if a problem occured.

void close(); closes the base image and its redolog.

off_t lseek(off_t offset, int whence); seeks at logical data position offset in the undoable disk image. Only SEEK_SET and SEEK_CUR are supported for whence. Returns -1 if a problem occured, or the current logical offset in the undoable disk image.

ssize_t read(void* buf, size_t count); reads count bytes of data from the undoable disk image, from current logical offset, and copies it into buf. count must be 512. Returns the number of bytes read. Data will be read from the redolog if it has been previously written or from the base image otherwise.

ssize_t write(const void* buf, size_t count); writes count bytes of data from buf to the undoable disk image, at current logical offset. count must be 512. Returns the number of bytes written. Data will always be written to the redolog.

bx_bool save_state(const char *backup_fname); calls the related redolog_t method to save the image state.

void restore_state(const char *backup_fname); called by the hdimage restore code. Copies the backup file to the original location and overwrites the existing redolog file.

2.10.5.3. volatile_image_t methods

volatile_image_t(Bit64u size, const char* redolog_name); instanciates a new volatile_image_t object. This disk image logical length is size bytes and the redolog filename is redolog_name plus a random suffix.

int open(const char* pathname); opens the disk image pathname in read-only mode, as a volatile disk image. The image mode is auto-detected. The associated redolog will be named pathname with a random suffix, unless set in the constructor. Returns 0 for OK or -1 if a problem occured.

void close(); closes the base image and its redolog. The redolog is deleted/lost after close is called.

off_t lseek(off_t offset, int whence); seeks at logical data position offset in the volatile disk image. Only SEEK_SET and SEEK_CUR are supported for whence. Returns -1 if a problem occured, or the current logical offset in the volatile disk image.

ssize_t read(void* buf, size_t count); reads count bytes of data from the volatile disk image, from current logical offset, and copies it into buf. count must be 512. Returns the number of bytes read. Data will be read from the redolog if it has been previously written or from the base image otherwise.

ssize_t write(const void* buf, size_t count); writes count bytes of data from buf to the volatile disk image, at current logical offset. count must be 512. Returns the number of bytes written. Data will always be written to the redolog.

bx_bool save_state(const char *backup_fname); calls the related redolog_t method to save the image state.

void restore_state(const char *backup_fname); called by the hdimage restore code. Copies the backup file to the original location and overwrites the existing redolog file.

2.10.5.4. growing_image_t methods

growing_image_t(Bit64u size); instanciates a new growing_image_t object. This disk image logical length is size bytes.

int open(const char* pathname); opens the growing disk image pathname, Returns 0 for OK or -1 if a problem occured.

void close(); closes the growing disk image.

off_t lseek(off_t offset, int whence); seeks at logical data position offset in the growable disk image. Only SEEK_SET and SEEK_CUR are supported for whence. Returns -1 if a problem occured, or the current logical offset in the grwoing image.

ssize_t read(void* buf, size_t count); reads count bytes of data from the growing disk image, from current logical offset, and copies it into buf. count must be 512. Returns the number of bytes read. The buffer will be filled with null bytes if data has not been previously written to the growing image.

ssize_t write(const void* buf, size_t count); writes count bytes of data from buf to the growing disk image, at current logical offset. count must be 512. Returns the number of bytes written.

static int check_format(int fd, Bit64u imgsize); checks the format of the file with descriptor fd. Returns HDIMAGE_FORMAT_OK if the file format matches the "growing" one. This is used for the image mode autodetection feature.

bx_bool save_state(const char *backup_fname); calls the related redolog_t method to save the image state.

void restore_state(const char *backup_fname); called by the hdimage restore code. Copies the backup file to the original location and overwrites the existing redolog file.