Win3mu - Part 9 - Path Mapping

This is Part 9 in a series of articles about building Win3mu — a 16-bit Windows 3 emulator. If you’re not sure what this is or why I’m doing it you might like to read from the start.

This post describes Win3mu’s path mapper which provides a mapping between the guest and host file systems. It also gives a quick overview of the Machine class which ties all of this together before revisiting the ALU with some more testing to chase down some weird CPU behaviour.

Path Mapper

Windows 3 has no concept of multiple users nor file permissions — it’s all one big free for all. One of the things I wanted to do with Win3mu is take control of this a little and present the VM with a sand-boxed file system.

The idea here is to be able to put Win16 programs in a standard location like “C:\Program Files” but have saved files go to a per-user folder. Since these programs often save data (eg: high score files) to the same folder as the program file there needs to be support for separate read-only and read-write mapping.

The path mapper is configured with a set of mount points each with the following settings:

• guest — the path as seen by the guest program (inside the emulator)
• host — the path on the host machine
• hostWrite — an optional host path where written files will be saved.

The path mapper works so that a file is read from the “host” path until it’s modified after which it’s copied to the “hostWrite” folder where all read and write operations then take place.

No Long File Names

Win3mu only supports 8.3 file names:

• only valid 8.3 filenames on the host will be visible to the VM and
• only valid 8.3 filenames will be accepted from the guest.

This only applies to paths visible inside the VM. ie: “Program Files” as part of a mount point is fine since the VM never sees that.

Variables

Having to define path mappings specific to each program would be tedious so Win3mu includes a simple variable replacement mechanism to expand strings containing embedded variables with a syntax like this:

This is a variable: $(variableName).  So is $(this).

I stole this class from Cantabile and updated it to support variables useful to Win3mu:

• $(AppName) — the name of the running 16-bit program file with path and extension removed.
• $(AppFolder) — the folder where the 16-bit program was loaded from
• $(Win3muFolder) — the folder where Win3mu is installed
• $(AppData), $(MyDocuments) etc… and other special system folders
• $(ax), $(bx), etc… for all the CPU registers
• $(asm) — disassembly of the current CPU instruction
• $(annotations) — annotations for the current CPU instruction (ie: the value of referenced registers and memory contents).

Path Mapper Defaults

With support for variables, Win3mu’s default path mapper configuration can now be written like this and it will work fairly seamlessly for most programs:

{
    "mountPoints":
    {
        "C:\\WINDOWS":
        {
            "host": "$(Win3muFolder)\\FILES\\WINDOWS",
            "hostWrite":  "$(AppData)\\Win3mu\\$(AppName)\\FILES\\WINDOWS",
        },
        "C:\\$(AppName)":
        {
            "host": "$(AppFolder)",
            "hostWrite": "$(AppData)\\Win3mu\\$(AppName)\\FILES\\$(AppName)",
        },
    },
}

For example, say the following Win 16 program is run:

C:\Program Files\Win16\Games\ski.exe

The following folder mappings will be created:

C:\WINDOWS 
 - reads from C:\Program Files\Win3mu\Files\Windows
 - writes to C:\Users\brad\AppData\Roaming\Win3mu\Files\Windows
 
C:\SKI
 - reads from C:\Program Files\Win16\Games
 - writes to C:\Users\brad\AppData\Roaming\Win3mu\ski

…and the running program will think of itself as C:\SKI\SKI.EXE.

Config Files

As you can tell from the above example Win3mu’s config files are JSON files.

Win3mu also supports config file merging. The idea here is that rather than editing the default config file, it’s used as a starting point with settings overridden by merging per-user/per-app configuration files over the top.

The final configuration is produced by merging the following files:

• $(Win3muFolder)\config.json — the default win3mu config file.
• $(AppData)\Win3mu\config.json — per-user config
• $(AppFolder)\config.json — per app folder config
• $(AppFolder)\$(AppName).json — per app config

By default the merge simply replaces existing keys in the base config with values from the overriding file. There’s a special syntax however to indicate that’s key should be merged instead of replaced.

eg: to add a new mount point while leaving the default ones in place the mountPoints key can be prefixed with “merge:”

{
    "merge:mountPoints":
    {
        "C:\\MYDATA":
        {
            "host": "$(MyDocuments)\\MYDATA"
        }
    }
}

There’s also “delete:xxx” to delete a key from the parent config.

Beside mount points for the path mapper the rest of the config file is mostly related to debug settings and logging.

CPU Related Variables

The CPU related variables described above allow for creating detailed execution logs. The logExecutionFormat variable specifies the format of the logging:

"logExecutionFormat": "$(cs):$(ip): $(asm,-30) $(annotations)",

Which gives an output like this:

0142:3267: or ax,ax                         ax=0x0002
0142:3269: jl 0x32A0                        
0142:326B: mov ax,0x0060                    ax=0x0002
0142:326E: imul si                          si=0x0003
0142:3270: mov bx,ax                        bx=0x0168, ax=0x0120
0142:3272: mov ax,di                        ax=0x0120, di=0x0006
0142:3274: mov cx,ax                        cx=0x0006, ax=0x0006

These execution logs have proven invaluable in getting Win3mu to run some programs.

The Machine Class

I’ve mentioned the Machine class in previous posts but thought I should elaborate a little on it.

The Machine class represents the entire virtual machine. It derives from the Sharp86 CPU class and provides the implementation of RaiseInterrupt and IBus (which it delegates to the global heap). It also holds references to other supporting modules and services:

class Machine : CPU, IBus
{
    // Main entry point
    public int RunProgram(string programName, string commandTail, int nCmdShow);

    // Services
    public ModuleManager ModuleManager { get; }
    public VariableResolver VariableResolver { get; }
    public PathMapper PathMapper { get; }
    public DosApi Dos { get; }
    public GlobalHeap GlobalHeap { get; }
    public LocalHeap SystemDataHeap { get; }
    public StringHeap StringHeap { get; }
    public Dictionary<string, string> Environment { get; }
    public Messaging Messaging { get; }

    // Emulated modules
    public Kernel Kernel { get; }
    public User User { get; }
    public GDI GDI { get; }
    
    // Thunking
    public void CallVM(uint lpfnProc);
    public uint CreateSystemThunk(Action handler, ushort popStack, bool preserveAX);
    
    // Interrupt handling
    public override void RaiseInterrupt(byte interruptNumber) { };
    
    // IBus implementation delegates to globalHeap
    
    // Also properties for most things in the config file
    
}

A couple of notes:

• RunProgram is the main entry point to emulator. There’s a small stub program that creates a Machine instance and calls this method.
• The DosApi class provides an implementation of DOS interrupt services and is called from Machine’s RaiseInterrrupt for relevant interrupts.
• The SystemDataHeap is a system owned local heap used to temporarily store structures that the 16-bit program needs to be able to access.
• The StringHeap is a place to permanently store strings that the 16-bit program might access.
• Environment is the DOS environment which is read from the config file
• Messaging is a class that handles all the complexities of Windows messaging — converting messages, calling WNDPROCs etc…

The main point here is that the Machine class is the central place that ties everything together and most other services hold a reference back to it.

Brute Force Testing the ALU

OK, that’s enough about these supporting services for now — let’s get back to the business of writing an emulator…

I mentioned in the post about the CPU that I was a little paranoid about CPU bugs and that to test the ALU there were:

tests for every operation and important flags after each

That it only tested “important flags” was nagging in the back of my mind and when I started to see some weird behaviour (eg: WordZap only painting one tile instead of 8) I decided I needed a lot more confidence with the ALU — and especially with the flags.

The flags register is a special register in the CPU that indicates additional information about the result of the last operation. eg: if the result of a operation is zero the Z flag would be set, if a signed add overflows the Carry flag would be set etc…

Testing the flags register is particularly tricky because:

• Some flags can be difficult to understand and have slightly different semantics on different instructions
• Some are specified in the Intel docs as “undefined”
• There are many boundary conditions
• Some flags aren’t affected by some operations

In the end I decided that precise unit tests were the wrong way to test this and something a bit more brute force was in order.

I decided to write a C++/ASM program that executed each instruction (on a real processor) with specific combinations of input values and flags, captured the result and wrote it all out to a text file.

eg: Here’s the function to capture the the behaviour of the 16-bit adc instruction:

ushort adc16(ushort inFlags, ushort a, ushort b, ushort* outFlags)
{
	__asm
	{
		push word ptr[inFlags]
		popf

		mov ax, word ptr[a]
		adc ax, word ptr[b]

		pushf
		mov esi, dword ptr[outFlags]
		pop word ptr[esi]
	}
}

Each instruction was executed with a set of input values designed to cover all the boundary conditions. For example, the 16-bit binary operations were tested with every possible combination of 2 values from this table:

0x0000, 0x0001, 0x0002, 0x0003, 0x0004, 0x0005, 0x0006, 0x0007,
0x0008, 0x0009, 0x000A, 0x000B, 0x000C, 0x000D, 0x000E, 0x000F,
0x3FF0, 0x3FF1, 0x3FF2, 0x3FF3, 0x3FF4, 0x3FF5, 0x3FF6, 0x3FF7,
0x3FF8, 0x3FF9, 0x3FFA, 0x3FFB, 0x3FFC, 0x3FFD, 0x3FFE, 0x3FFF,
0x4000, 0x4001, 0x4002, 0x4003, 0x4004, 0x4005, 0x4006, 0x4007,
0x4008, 0x4009, 0x400A, 0x400B, 0x400C, 0x400D, 0x400E, 0x400F,
0x7FF0, 0x7FF1, 0x7FF2, 0x7FF3, 0x7FF4, 0x7FF5, 0x7FF6, 0x7FF7,
0x7FF8, 0x7FF9, 0x7FFA, 0x7FFB, 0x7FFC, 0x7FFD, 0x7FFE, 0x7FFF,
0x8000, 0x8001, 0x8002, 0x8003, 0x8004, 0x8005, 0x8006, 0x8007,
0x8008, 0x8009, 0x800A, 0x800B, 0x800C, 0x800D, 0x800E, 0x800F,
0xBFF0, 0xBFF1, 0xBFF2, 0xBFF3, 0xBFF4, 0xBFF5, 0xBFF6, 0xBFF7,
0xBFF8, 0xBFF9, 0xBFFA, 0xBFFB, 0xBFFC, 0xBFFD, 0xBFFE, 0xBFFF,
0xC000, 0xC001, 0xC002, 0xC003, 0xC004, 0xC005, 0xC006, 0xC007,
0xC008, 0xC009, 0xC00A, 0xC00B, 0xC00C, 0xC00D, 0xC00E, 0xC00F,
0xFFF0, 0xFFF1, 0xFFF2, 0xFFF3, 0xFFF4, 0xFFF5, 0xFFF6, 0xFFF7,
0xFFF8, 0xFFF9, 0xFFFA, 0xFFFB, 0xFFFC, 0xFFFD, 0xFFFE, 0xFFFF,

Each instruction was executed twice — once with all the flags set and once with all the flags cleared. The idea here is to capture information about which flags change and to test cases where the flags affect the output value. Also, it captured which instructions generated exceptions (eg: division by zero).

The output of this was a gigantic text file with over 600,000 test cases and expected results.

;instruction, inFlags, oper1, oper2, result, outFlags
.
.
.
imul8 0000 ff fe 0002 0202
imul8 08d5 ff fe 0002 0202
imul8 0000 ff ff 0001 0202
imul8 08d5 ff ff 0001 0202
div16 0000 00000000 0000 ????     ; Indicates exception
div16 08d5 00000000 0000 ????
div16 0000 00000001 0000 ????
div16 08d5 00000001 0000 ????
div16 0000 00000000 0001 00000000 0202
div16 08d5 00000000 0001 00000000 0ad7
div16 0000 00000001 0001 00000001 0202
div16 08d5 00000001 0001 00000001 0ad7

Next, I wrote a C# program that read this file and put the ALU through its paces. Initially about 200,000 cases failed — nearly all related to undefined flags and exceptions that the ALU wasn’t generating.

In the end I fixed every operation to match exactly what a real processor does — even the undefined behaviour of “multi-bit rotate through carry” instructions and matched all the exception cases:

I then re-ran WordZap… and the same problem still happened!

Turns out I’d implemented an increment instruction without using the ALU — so the flags weren’t getting updated at all for that instruction. Still, this was definitely a worthwhile exercise and I now have a lot more confidence with it.

Now that I’ve covered off these supporting services the next post will return to building the Windows emulation — specifically how reflection is used to minimize the amount of code required to implement API methods.