I have a project student who will be doing some work on the hypervisor this year. The focus of his project is to make a hypervisor environment which is easy to use, and can be used to teach some computer architecture and operating systems concepts, without being as complex as a modern architecture.
In preparation for this, I have been thinking about what basic hardware and firmware features are needed to support this, and allow him to work on the hypervisor without having to resynthesise the whole FPGA every time he makes a small change. This has also served as a welcome distraction from the DDR memory controller problems I have been facing.
The first thing is that we need a way for traps to the hypervisor to be distinguishable from a hardware reset. Until now, the two have entered the same vector, and thus a trap to the hypervisor was basically a way of asking the machine to reset.
The hypervisor state registers map to $D640-$D67F. In hypervisor mode, these can be used to modify the CPU and memory mapper state. From a running program they are used to trap into the hypervisor. Previously only $D67F would trap into the hypervisor in this way. The bottom 6 address bits are used to calculate the entry vector in the hypervisor, resulting in entry points at $8000-$80FC, i.e. using four-byte multiples. I could have used 3 bytes to save a little space, but I didn't want to have to implement a multiply-by-3 in the CPU address decode logic. Hardware reset enters at $8100, so there are 64 traps plus reset.
These traps provide a mechanism that is a cross between the INT instruction on x86 (which triggers an "interrupt" using a vector at an address computed from the argument), and the SYSCALL type instruction found on many CPU types, that traps to the privileged mode, and passes some arguments through. The key difference is that on the 45GS10, the trap is straight through to hypervisor mode, and all CPU state is automatically saved and restored for you. Thus even though we are only at 48MHz, it is theoretically possible to perform somewhere around 8 million hypervisor traps per second -- provided you don't want them to actually do anything.
The final addition is a write-once flag that can be used by the hypervisor to see if it has been upgraded since power-on. Even CPU reset doesn't clear the flag. This is used to allow the kickstart ROM to know when it should try to replace itself by loading a specially named file (KICKUP.65) from the SD card. This will allow new kickstart ROMs to be easily loaded by putting them on the SD card, and then power-cycling the FPGA board.