In another blog post I'm writing at the moment, I'm exploring how I will do the UI for the new MEGAphone work. Part of that involves me figuring out how I can possibly support the full Unicode gammut -- including emojis, that are not more or less a required part of SMS communications.
In that post, I considered putting them into files on the FAT32 file system, or stashing them in a set of specially formatted D81 or D65 disk images. But all of those approaches require more than 1 SD card sector read to fetch a random glyph, and in some cases many such requests.
So I came up with the idea of putting linear slabs of shared resources into the system partition, and having a Hypervisor call that lets a user program ask for an arbitrary 512 byte SD card sector of that data. For fonts in the MEGAphone software, this would allow the program to work out and store the sector offset within the shared resources for the font(s) it needs to use, and then just adding an offset to that to retrieve the SD card sector with the required font glyph to load into chip RAM.
In this blog post I'm going to design and implement this facility in the hypervisor, and make a test program that will let me check that it's all working. From there, I'll work on some kind of utility that will let me modify the contents of the system partition to install the fonts. That might end up being a Linux-based utility that works directly on the SD card for speed --- since a full Unicode font will be about 300MB in the pre-rendered format that I'll be using (more about that in the other blog post, when it's done).
I'm going to track the addition of this functionality into HYPPO via Issue #897.
So let's start by making the system partition code in the Hypervisor read and store the starting sector and size of this slab.
Now I have to remember how I made it possible to update the hypervisor using m65 -k. From memory this can be done safely from the MEGA65's READY prompt, as that means that it's in user-space, and thus it's safe to overwrite the hypervisor.
... except that that doesn't seem to do it.
Okay, so because the Hypervisor RAM is only writeable when the CPU is in Hypervisor mode, we can't do it from the READY prompt any more. But we also need it to happen in some situation where the Hypervisor isn't really running any code, since otherwise the CPU will go off into la-la land, and bork things.
I am using an old version of m65, though. Maybe in newer versions -k looks after this, e.g., by setting the CPU to single-step mode, then causing the CPU to start a Hypervisor trap, and then stopping the CPU once in Hypervisor mode, writing the new version of the Hypervisor code, then resetting and resuming the CPU, so that it boots the new HYPPO fresh. If not, that would be the way to do it.
I'm tracking this via Issue #222, and pull request #223.
Now with m65 recompiled with that fix, I can quickly and easily test HYPPO builds. That's already made it _way_ more comfortable and convenient to find and fix a bug in my initial implementation of the of the shared resource thing.
So now I can implement the HYYPO trap that tries to read the sector. Because we are going to use A,X,Y and Z as the Q pseudo-register to make it easy for a user program to access, we will have to use up a whole trap address, but that's okay.
So far, the trap code looks like this:
readsharedresourcetrap:
stq $d681
cpq syspart_resources_area_size
bcs bad_syspart_resource_sector_request
ldq syspart_resources_area_start
adcq $d681
;; Ask SD card to read the sector.
lda #$02
sta $d680
;; Note that we don't wait for the request to finish -- that's
;; up to the end-user to do, so that we don't waste space here
;; in the hypervisor, where space is at an absolute premium.
clc
rts
bad_syspart_resource_sector_request:
sec
rts
In principle, that should be all we need.
The next step is to make a unit test for this, checking that it does cause a sector to get read, and that requests for out-of-range sectors fails.
I'll use the unit test framework that we have in mega65-tools.
Those tests use CC65, which I can't remember it's call convention for 32-bit args. So I'll just stash the 32-bit resource sector number into some convenient memory location that it doesn't use. Maybe $7F0-$7F3, i.e., the end of the screen RAM.
The test will then make the HYPPO call, and store the register values from HYPPO back where it read them from. Like this:
.export _test_resource_read
.p4510
.segment "CODE"
.proc _test_resource_read: near
lda $7f0
ldx $7f1
ldy $7f2
ldz $7f3
sta $d644 ; Trigger Hypervisor trap
nop ; CPU delay slot required after any hypervisor trap
sta $7f4
stx $7f5
sty $7f6
stz $7f7
php
pla
sta $7f8
rts
.endproc
Pulling it all together, we can now easily run tests like this:
paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg
/home/paul/Projects/mega65/mega65-tools/cc65/bin/cl65 --config src/tests/tests_cl65.cfg -I src/mega65-libc/cc65/include -O -o src/tests/test_897.prg --mapfile src/tests/test_897.map --listing src/tests/test_897.list src/tests/test_897_asm.s src/tests/test_897.c src/mega65-libc/cc65/libmega65.a
2025-06-07T07:10:52.445Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-07T07:10:52.454Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T07:10:52.454Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T07:10:52.660Z NOTE loading file 'src/tests/test_897.prg'
2025-06-07T07:10:52.841Z NOTE running
2025-06-07T07:10:52.841Z NOTE Entering unit test mode. Waiting for test results.
2025-06-07T07:10:52.841Z NOTE System model: UNKNOWN MODEL $06
2025-06-07T07:10:52.841Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-07T07:10:52.841Z NOTE System ROM version: M65 V920395
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z FAIL (Issue#0897, Test #000 - HYPPO RESOURCE READ TEST FAILED)
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:10:52.845Z NOTE terminating after completion of unit test
For those who haven't seen the unit test framework, it's quite nifty: The unit test running on the real hardware communicates test results back via the serial monitor interface to the m65 program, allowing it to directly report the test results.
Because I haven't implemented everything yet, the test claims to fail.
So now let's fix the implementation of our trap -- it should return from the Hypervisor by jumping to one of the return routines:
readsharedresourcetrap:
stq $d681
cpq syspart_resources_area_size
bcs bad_syspart_resource_sector_request
ldq syspart_resources_area_start
adcq $d681
;; Ask SD card to read the sector.
lda #$02
sta $d680
;; Note that we don't wait for the request to finish -- that's
;; up to the end-user to do, so that we don't waste space here
;; in the hypervisor, where space is at an absolute premium.
jmp return_from_trap_with_success
bad_syspart_resource_sector_request:
;; Return "illegal value" if trying to read beyond end of region
lda #dos_errorcode_illegal_value
jmp return_from_trap_with_failure
So now if we ask for an illegal sector number, we should get this illegal value error, which we know has the value 0x11 from the source.
So let's now update our test to check whether the trap is defined or not, and whether reading sector 0 results in success or an error:
// Try reading sector 0 of resource area
resource_sector = 0;
*(unsigned long *)0x7f0 = resource_sector;
if (test_resource_read() == 0) {
unit_test_ok("");
}
else {
unit_test_fail("hyppo call helper failed");
}
return_value = *(unsigned long *)0x7f4;
carry = PEEK(0x7f8)&1;
if (!carry) {
switch(PEEK(0x7f4)) {
case 0xff: unit_test_fail("hyppo returned 'trap not implemented'"); break;
case 0x11: // dos_errorcode_illegal_value
unit_test_fail("reading resource sector 0 failed -- no resource area in system partition?");
break;
default: {
snprintf(message,80,"hyppo resource read failed with unknown error $%02x",PEEK(0x7f4));
unit_test_fail(message);
}
}
unit_test_report(ISSUE_NUM, 0, TEST_DONEALL);
return;
}
This routine now checks whether the Hypervisor trap is implemented or not, and then whether reading sector 0 works. We need both of those to succeed, if we are going to able able to test that it works to actually read system resources.
With the fixed m65 -k option to easily load a HICKUP.M65 file containing an updated Hypervisor, I can now easily load the version that supports this new hypervisor call:
paul@bob:~/Projects/mega65/mega65-core$ make bin/HICKUP.M65 && m65 -k bin/HICKUP.M65
And then execute the test, and see the result:
paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg
...
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z FAIL (Issue#0897, Test #001 - READING RESOURCE SECTOR 0 FAILED -- NO RESOURCE AREA IN SYSTEM PARTITION?)
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:31:16.102Z NOTE terminating after completion of unit test
And look at that --- we can see it now passes the check for the hypervisor trap being implemented (Test #000), but fails reading the first sector of the shared resource area (Test #001). This is exactly what we expect, since the SD card in this MEGA65 doesn't (yet) contain a shared resource area.
In fact, I haven't even made the utility to create them. So let's tackle that next.
Currently the MEGA65 FDISK utility is the thing that's responsible for creating the system partition. So I'll probably just tweak that to by default create a reasonable size system resource partition, perhaps 1GB or so, but maybe also have the option to select a different size.
I'm tracking that issue here: https://github.com/MEGA65/mega65-fdisk/issues/28
The FDISK program is really quite old and crusty now, but it does it's job. There is an outstanding issue to totally refactor it, so while I will try to not add to the mess, I'm not going to go overboard with making this new feature overly sophisticated.
Okay, done a first cut at that. It will try to allocate half the system partition for shared resources. It also increases the target SYSPART size from 2GiB to 4GiB, thus allowing 2GiB for shared resources by default.
Now to backup my SD card, and try reformatting it!
Okay, so after a bit of procrastination, it's time to just pull the SD card, copy it all off, and put it back in, and run MEGA65 FDISK on it.
But then when I tried to boot with it, it was all borked up.
Basically the FAT file system seems to be messed up. But if I power on with the new HICKUP, it does see that we have $400000 sectors reserved for resources, which equates to 2GiB, so that's promising!
Okay, so why on earth is the FAT file system all messed up, then?
First thing to try: Downgrade to FDISK prior to the changes I've made, and see if it formats up fine with that.
Nope. Same thing. So probably not something that I've done.
Okay, so after a second go, including copying all my files back on, it's fine.
Who knows what went wrong.
The main thing is it is booting again.
Although, oddly it's running the intro disk, even though I don't have a default disk image to mount at boot set.
Hmmm... And now the new HICKUP is showing 0 sectors reserved for shared resources. Did I accidentally format it with the old version of FDISK? Yup --- looks like that was the case.
After I copied my files back into place, it was all fine.
And, the test to read sector 0 of the shared resource area works now:
paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg
...
2025-06-07T13:30:30.590Z NOTE 2025-06-07T13:30:30.590Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.590Z NOTE 2025-06-07T13:30:30.590Z PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.592Z NOTE 2025-06-07T13:30:30.592Z PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ TEST PASSED)
2025-06-07T13:30:30.592Z NOTE 2025-06-07T13:30:30.592Z DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.592Z NOTE terminating after completion of unit test
So now I can work on making the test more complete, by doing a binary search of the address space to determine the exact size of the shared resource partition.
Like this:
resource_sector = 0xffffffffUL;
for(i=31;i>=0;i--) {
*(unsigned long *)0x7f0 = resource_sector;
if (test_resource_read() != 0) unit_test_fail("hyppo trap failed");
carry = PEEK(0x7f8)&1;
if (!carry) {
resource_sector -= (1UL <<i);
}
}
if (resource_sector==0xffffffffUL || (!resource_sector)) {
unit_test_fail("failed to determine size of shared resource area");
} else {
snprintf(message,80,"determined shared resource area is $%08lx sectors",
resource_sector + 1);
unit_test_fail(message);
}
printf("Shared resource area = $%08lx sectors.\n",resource_sector+1);
And it works, correctly measuring the size!
paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg
/home/paul/Projects/mega65/mega65-tools/cc65/bin/cl65 --config src/tests/tests_cl65.cfg -I src/mega65-libc/cc65/include -O -o src/tests/test_897.prg --mapfile src/tests/test_897.map --listing src/tests/test_897.list src/tests/test_897_asm.s src/tests/test_897.c src/mega65-libc/cc65/libmega65.a
2025-06-07T13:41:32.282Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-07T13:41:32.296Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T13:41:32.296Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T13:41:32.502Z NOTE loading file 'src/tests/test_897.prg'
2025-06-07T13:41:32.691Z NOTE running
2025-06-07T13:41:32.691Z NOTE Entering unit test mode. Waiting for test results.
2025-06-07T13:41:32.691Z NOTE System model: UNKNOWN MODEL $06
2025-06-07T13:41:32.691Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-07T13:41:32.691Z NOTE System ROM version: M65 V920395
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ SECTOR 0 TEST PASSED)
2025-06-07T13:41:32.695Z NOTE 2025-06-07T13:41:32.695Z PASS (Issue#0897, Test #002 - DETERMINED SHARED RESOURCE AREA IS $00400000 SECTORS)
2025-06-07T13:41:32.696Z NOTE 2025-06-07T13:41:32.696Z DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.696Z NOTE terminating after completion of unit test
Okay, so now the last thing to test is that it actually causes the SD card to read the sector.
First, do the SD card registers get the correct sector number written into them? I'll test this by just making sure that they don't end up with all zeroes. Anyway, nope, they don't. Found and fixed -- the bug was in the hypervisor code (the interested reader can look at the code snippet for the hypervisor trap above and see if they can spot it --- leave a comment if you find it :)
So then the next step is to check that the SD sector buffer actually changes contents. That worked fine, without further changes (although I had to make sure the SD card was not busy, before making the HYPPO call, because requesting a sector read while the SD card is busy causes it to lock up).
Next up, then, is to come up with a simple file system for the shared resource area. It needs to be super simple to parse. I'm thinking it could be as simple as the first sector containing version information and the number of resources in the area, followed by one sector per file, that contains the name of the file/resource, and the start sector within the resource area, and the length of the resource in both sectors and bytes. After the final resource directory sector, we have a completely empty sector, so that a program scanning it can just look for that empty sector and know to stop, without even having to track the resource count, in case that's easier to implement in a given situation.
So let's make a utility that can build the shared resource area from a set of files and their names. Since it's just boring boiler-plate kind of code, I used ChatGPT to do it faster. Not sure if it actually was any faster to write. But we can now do things like this:
paul@bob:~/Projects/mega65/mega65-tools$ rm bin/shresls ; make bin/shres bin/shresls && bin/shres foop mega65-screen-000000.png,31 README.md,23 && bin/shresls foop
make: 'bin/shres' is up to date.
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
Resource Table (2 entries):
Idx Start Sectors Bytes Flags Name
---- ------- -------- ---------- ---------- -------------------------
0 4 9 4242 0x80000000 mega65-screen-000000.png
1 13 13 6436 0x00800000 README.md
MEGA65 Shared Resource File: foop
Declared resources: 2
Resource Table (2 entries):
Idx Start Sectors Bytes Flags Name
---- ------- -------- ---------- ---------- -------------------------
0 4 9 4242 0x80000000 mega65-screen-000000.png
1 13 13 6436 0x00800000 README.md
SHA1 Bytes Check Name
--------------------------------------------------------------------------------
40ce6cfcaaf14d11a8c3ffa71fd4726087613eee 4242 MATCH mega65-screen-000000.png
3b31809d69f814d2173484855072333db65190cb 6436 MATCH README.md
paul@bob:~/Projects/mega65/mega65-tools$
Next step is to make a tool to extract or insert a shared resource area onto/off of an actual SD card. My plan here is to make the existing tool I've made above to work out if it's writing to an SD card, in which case, it will find the SYSPART in there, and write the resources directly into there.
Okay, so I think I have writing the resources to SD card working, as well as updating shresls so that it can read and check them on a real disk image, too:
paul@bob:~/Projects/mega65/mega65-tools$ rm bin/shresls ; make bin/shres bin/shresls && sudo bin/shres /dev/sdb mega65-screen-000000.png,31 README.md,23
make: 'bin/shres' is up to date.
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
INFO: Attempting to open shared resource file or disk image '/dev/sdb'
INFO: Found MEGA65 SYSPART at sector 0x014d7ffe
INFO: Found MEGA65 SYSPART shared resource area of 2048 MiB.
DEBUG: area_start=0x00029b0ffc00, area_length=0x80000000
Resource Table (2 entries):
Idx Start Sectors Bytes Flags Name
---- ------- -------- ---------- ---------- -------------------------
0 4 9 4242 0x80000000 mega65-screen-000000.png
1 13 13 6436 0x00800000 README.md
paul@bob:~/Projects/mega65/mega65-tools$ make bin/shresls && sudo bin/shresls /dev/sdb
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
INFO: Attempting to open shared resource file or disk image '/dev/sdb'
INFO: Found MEGA65 SYSPART at sector 0x014d7ffe
INFO: Found MEGA65 SYSPART shared resource area of 2048 MiB at sector 2048 of SYSPART.
MEGA65 Shared Resource File: /dev/sdb
Declared resources: 2
Resource Table (2 entries):
Idx Start Sectors Bytes Flags Name
---- ------- -------- ---------- ---------- -------------------------
0 4 9 4242 0x80000000 mega65-screen-000000.png
1 13 13 6436 0x00800000 README.md
SHA1 Bytes Check Name
--------------------------------------------------------------------------------
40ce6cfcaaf14d11a8c3ffa71fd4726087613eee 4242 MATCH mega65-screen-000000.png
3b31809d69f814d2173484855072333db65190cb 6436 MATCH README.md
So now I just need to make some tests for scanning the shared resource list, and trying to read them. Actually, all we probably need is to check for the magic string in the shared resource area. This is because for that to be found, it means that the SD card access is working, as well as the sector address calculation.
And it works!
mega65-tools$ make src/tests/test_897.prg && m65 -4 -r -u src/tests/test_897.prg
make: 'src/tests/test_897.prg' is up to date.
2025-06-13T07:15:45.034Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-13T07:15:45.048Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-13T07:15:45.048Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-13T07:15:45.254Z NOTE loading file 'src/tests/test_897.prg'
2025-06-13T07:15:45.447Z NOTE running
2025-06-13T07:15:45.447Z NOTE Entering unit test mode. Waiting for test results.
2025-06-13T07:15:45.447Z NOTE System model: UNKNOWN MODEL $06
2025-06-13T07:15:45.447Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-13T07:15:45.447Z NOTE System ROM version: M65 V920395
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ SECTOR 0 TEST PASSED)
2025-06-13T07:15:45.452Z NOTE 2025-06-13T07:15:45.452Z PASS (Issue#0897, Test #002 - DETERMINED SHARED RESOURCE AREA IS $00400000 SECTORS)
2025-06-13T07:15:45.455Z NOTE 2025-06-13T07:15:45.455Z PASS (Issue#0897, Test #003 - SD CARD BUSY CLEARED ON RESET)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z PASS (Issue#0897, Test #004 - SD CARD REGISTERS WRITTEN TO)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z PASS (Issue#0897, Test #005 - SD CARD BUSY CLEARED)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z PASS (Issue#0897, Test #006 - SD CARD BUFFER CONTENTS CHANGE)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z PASS (Issue#0897, Test #007 - READ MAGIC STRING FROM SHARED RESOURCE AREA)
2025-06-13T07:15:45.457Z NOTE 2025-06-13T07:15:45.457Z DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.457Z NOTE terminating after completion of unit test
Okay, so now I have all the ingredients here for this to work -- HYPPO support for the shared resource area, patched MEGA65 FDISK to create the section in the system partition, linux-based tools to populate and interrogate the shared rescource area quickly and easily, and unit tests that provide proof-of-concept for accessing this area, and checking that the loaded HYPPO has support for it.
So I'm going to leave it at that for now, and I'll document writing a library to open and read from shared resources in a separate blog post.
I'll also pursue getting the HYPPO changes rolled into mega65-core development branch, and updated FDISK into mega65-fdisk development branch.
No comments:
Post a Comment