MEGAphone software tools and laying the foundation

In a recent post, I worked out how I am going to store SMS conversations, contacts, phone state and a pile of other stuff in D81 disk images, for convenient manipulation.

So now I need some tools that let me create those disk images and prepare their contents. Now, some of these functions will be needed on the MEGAphone, and others on the Linux build system. So I should try to make as many of them as common as possible to save re-work. This will also allow me to make unit tests for stuff that needs to run on the MEGA65.

So let's start with code that can take a 800KB blank disk image (or literally empty 800KB file on Linux) and populate it with a valid 1581 disk label and BAM sectors.

This is fairly straight-forward:

Disk header + first BAM sector on track 40 sector 0 (note I'm talking about physical 512 byte sectors here, because that's what the MEGA65 FDC sees).

Second BAM sector in first half of track 40 sector 1 + first 8 directory entries.

The wikipedia page for the 1581 has a reasonable run-down on what goes in the header, as well as the BAM sectors. There is an even better description here.

Now, for almost all these weird disk images for the telephony software,  we also want to allocate all data blocks in a single sequential file, as far as the 1581 file system is concerned, so that if someone VALIDATEs such a disk image Nothing Bad Will Happen. Unlike, say, GEOS formatted disks.

To make the code runnable on Linux and on the MEGA65, I'll need to make a Hardware Abstraction Layer (HAL), just as we did with the MEGA65 FDISK program -- so this isn't the first time we've done this kind of thing.  The correct HAL include will be selected at compile time, and the necessary files linked in:

#include <stdio.h>

extern unsigned char sector_buffer[512];
#define SECTOR_BUFFER_ADDRESS ((unsigned long long) &sector_buffer[0])

void hal_init(void);
void lfill(unsigned long long addr, unsigned char val, unsigned int len);
void lcopy(unsigned long long src, unsigned long long dest, unsigned int len);
void write_sector(unsigned char drive_id, unsigned char track, unsigned char sector);
char mount_d81(char *filename, unsigned char drive_id);
char create_d81(char *filename);

One of the challenges for the actual code to handle the disk images etc, is that it needs to be as compact as possible for compilation with CC65.  Also, we have to remember that an int on CC65 is 32 bits, not 16 bits. Also, function calls are not particularly cheap.

Using our HAL, we can produce fairly compact code to assemble and write the sectors: 

void format_image_fully_allocated(char drive_id,char *header)  
{
  char i,j;

  /* Header + BAM Sector 1
     ---------------------------------------- */
  
  // Erase sector buffer
  lfill(SECTOR_BUFFER_ADDRESS,0x00,512);
  // Clear header name
  lfill(&header_template[4],0xa0,16);
  // Set header name
  for(i=0;(i<16) && (header[i]);i++) {
    header_template[4+i] = ascii_to_petscii(header[i]);
  }
  lcopy(header_template,SECTOR_BUFFER_ADDRESS, sizeof(header_template));

  // BAM is easy: Just a valid header, and leave all bitmap bytes $00, to
  // indicate that there is no free space on the disk.
  lcopy(bam_template,SECTOR_BUFFER_ADDRESS + 0x100,sizeof(bam_template));
    
  write_sector(drive_id,40,0);
 

Note that for CC65 compatibility, we also have to declare variables at the start of blocks, because it generally uses an older C dialect.

So now I should make a tool that tries to create the set of D81s and test it under Linux, and then on the MEGA65.

This will start with a CONTACT0.D81 and STATE.D81 files.  On Linux we can also create the directory structure. But I don't think we have the MKDIR Hypervisor call on the MEGA65 implemented yet. The MKFILE call does seem to be implemented, though -- but only supports creating normal files with sizes that are multiples of 512KB (well, allocation happens in multiples of 512KB -- it will write whatever specific file size is requested into the length field).  So in theory we could use that as the basis for MKDIR, just changing the attribute byte and adding the . and .. special directory entries.

But let's instead just pike out, and require that the directory structure be created elsewhere for now, and only create it if we are on Linux.  We can insert the SD card into a reader on the Linux machine as an easy way to bootstrap things. Otherwise I'll be lost for half a week or more just getting MKDIR to work, especially since I don't know if MKFILE is working reliably.

In fact, there is probably an argument for fully populating the whole structure with empty D81s.  1,580 contacts, each with an 800KB D81 for contacts and some supporting directory structure is still only a little over 1GB. And having everything pre-allocated will only speed things up.  The main issue with doing that is when we sort the contacts, the mapping to the contact directories would change.  

We can deal with that by pre-populating a contact UID in each otherwise empty contact, so that when the contact lists get sorted the underlying physical contact directory doesn't change. We then just have to make sure we erase the message log for a contact, and any other stuff we have in the contact directory, whenever we delete a contact, so that the message thread doesn't get accidentally associated with some new contact later on.

But first, let's check if I'm generating valid D81 files.

I found a couple of entertaining bugs: My check for not writing over the directory track when creating the sector chain was checking for track 0x40, not track 40 (decimal). Whoops!

I also realised I hadn't set the sector count for the pseudo-file I create that links to the chained sectors on the whole disk, so it was showing as 0 blocks.  It's only cosmetic, but still worth fixing.

The next thing is that I know I've screwed up the sector linking somehow. 

It should show monotonically increasing sector numbers in the links every 256 (0x100) bytes). But it looks like this (in correct links in bold).

00000000  01 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  01 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200  01 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000300  01 02 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000400  01 05 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000500  01 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000600  01 07 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000700  01 04 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000800  01 09 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000900  01 05 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000a00  01 0b 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000b00  01 06 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................| 

So the links in the start of each 512 byte physical sector look fine, but those of the 2nd logical sector in each physical sector are wrong: I had forgotten to put the x2 in that I had for the first.

That got it better, but the final sector of the disk pointed to track 81, instead of having an end-of-file marker.  Fixed that, and now c1541 can extract the file without error

c1541 -attach ../PHONE/STATE.D81 -extract

$ ls -latr tmp/all\ sectors 
-rw-rw-r-- 1 paul paul 802639 Jul 28 10:14 'tmp/all sectors'

This looks encouraging.  There should be 79 tracks (80 minus 1 for the directory track) x 20 physical sectors x 2 logical sectors per physical sector x 254 bytes per sector =  802640 bytes.  The file is only 1 byte short. So maybe the end of file marker should be 0x00 not 0xFE. Yup --- that got the extra byte.

So the question is whether the file contents correctly represents the contents of the disk, and the sectors are in order. To make this easy to check, I added some code that writes the track and sector into the first data bytes of each sector, so that when I extract the "file", I can check the content of that file for the monotonically increasing track and sector numbers.

       // First half always links to 2nd half of physical sector
      lpoke(SECTOR_BUFFER_ADDRESS+0x000,i);
      lpoke(SECTOR_BUFFER_ADDRESS+0x001,j*2+1);

#define CONTENT_TEST
#ifdef CONTENT_TEST
      lpoke(SECTOR_BUFFER_ADDRESS+0x000+2,i);
      lpoke(SECTOR_BUFFER_ADDRESS+0x001+2,j*2+1);
#endif      
      
      // Second half points to next physical sector, or to first
      // sector of next track (or track 41 if we are on track 39,
      // so that we skip the directory).
      if (j<(20-1)) {
    lpoke(SECTOR_BUFFER_ADDRESS+0x100,i);
    lpoke(SECTOR_BUFFER_ADDRESS+0x101,j*2+1+1);
#ifdef CONTENT_TEST
    lpoke(SECTOR_BUFFER_ADDRESS+0x100+2,i);
    lpoke(SECTOR_BUFFER_ADDRESS+0x101+2,j*2+1+1);
#endif
      } else {
    lpoke(SECTOR_BUFFER_ADDRESS+0x100, (i==39)?41:(i+1));
    lpoke(SECTOR_BUFFER_ADDRESS+0x101,0);
#ifdef CONTENT_TEST
    lpoke(SECTOR_BUFFER_ADDRESS+0x100+2, (i==39)?41:(i+1));
    lpoke(SECTOR_BUFFER_ADDRESS+0x101+2,0);
#endif
      }

      if (i==80&&j==(20-1)) {
    // Terminate chain at end of disk

    lpoke(SECTOR_BUFFER_ADDRESS+0x100, 0x00);
    lpoke(SECTOR_BUFFER_ADDRESS+0x101, 0xff);
#ifdef CONTENT_TEST
    lpoke(SECTOR_BUFFER_ADDRESS+0x100+2, 0x00);
    lpoke(SECTOR_BUFFER_ADDRESS+0x101+2, 0xff);
#endif
      }

This is all great, and I can see the data markers in the file:

$ hexdump tmp/all\ sectors

0000000 0101 0000 0000 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
*
00000f0 0000 0000 0000 0000 0000 0000 0000 0201
0000100 0000 0000 0000 0000 0000 0000 0000 0000
*
00001f0 0000 0000 0000 0000 0000 0000 0301 0000
0000200 0000 0000 0000 0000 0000 0000 0000 0000
*
00002f0 0000 0000 0000 0000 0000 0401 0000 0000
0000300 0000 0000 0000 0000 0000 0000 0000 0000
*
So that's all good. The markers are every 254 bytes, because the first two bytes of each sector are used for the link to the next track and sector.  This is why the position of the markers looks like it moves backwards.

That's all great.

What has me confused is that while the markers are sector then track. I was getting quite worried, until I remembered that hexdump by default shows 16 bit values, i.e, with byte order swapped. Why? No idea. But the point is that it works: I'm creating D81 files that have all sectors allocated into one big chain, and it's all working. 

So now I can work on some tools that take a fictional contact list and message stream of messages between contacts, and smoosh them into the correct D81 files --- and then make a tool that goes the other way, i.e., extracts a similar canonical list of contacts and messages from the disk images. With those tools, I'll then be in a position to be able to easily test the actual routines for receiving and logging messages etc.

So let's start by coming up with a simple canonical format for contacts with first and last name, fictional phone numbers, and then generating message traffic.  This sounds like a good job for some vibe coding with an LLM. And indeed it's done a pretty fine job quickly for me, letting me generate files like this:

CONTACT:+9912208959:Donald:Hall:
CONTACT:+9992142067:Jeffrey:Jones:
CONTACT:+9991073060:Nicole:Mccarty:
CONTACT:+9911406450:Alyssa:Esparza:
CONTACT:+9961792880:Sophia:Sanchez:
CONTACT:+9971265517:Derek:Perry:
CONTACT:+9995659445:Jason:Williams:
CONTACT:+9912477184:Erica:Mack:
CONTACT:+9997053229:Rita:Lee:
CONTACT:+9975118477:Christopher:Tate:
MESSAGERX:+9971265517:1325413834:sed tempor et sit labore ipsum:
MESSAGERX:+9961792880:1325401304:😁 MEGA65 lorem elit sed:
MESSAGERX:+9997053229:1325377469:eiusmod amet MEGAphone consectetur tempor et sit sit:
MESSAGERX:+9912477184:1325340580:✨🍑🐶🐶🍏 sit eiusmod incididunt dolore sed:
MESSAGERX:+9991073060:1325349012:🐨📱🎉🐹😳🐸 tempor amet ut aliqua amet ut sit elit consectetur sed:
MESSAGERX:+9911406450:1325396204:adipiscing elit lorem elit dolore labore labore sed magna elit 🥚😉😚🤬📱🍿:
MESSAGETX:+9961792880:1325418258:🛑🙄 sed amet adipiscing amet labore ipsum incididunt:
MESSAGETX:+9912208959:1325369837:amet dolor labore 🤯🍳📚😴🍊:
MESSAGETX:+9911406450:1325447029:magna adipiscing et ipsum dolor dolore incididunt sed ut:
MESSAGERX:+9911406450:1325514412:et amet sed do amet adipiscing ut ut et amet:

The only mistake it made was to use +99 as an unallocated prefix, instead of +999. But that was an easy fix. 

 So now I need to make the Linux tool that populates the CONTACT0.D81 with the contacts from such a file. Once I have that working, then I can work on the logging of messages.

We want to convert the CONTACT lines above into the binary format I defined in an earlier blog post. But before we can do that, we need to be able to encode and decode fields in records. We'll also want to allocate and free records, too. So I'll tackle all that first.

Let's start with building our 512 - 2x2 = 508 byte records (split by the track and sector marker for the 2nd 256-byte logical sector). We can easily go from packed to sectorised format:

void sectorise_record(unsigned char *record,
              unsigned char *sector_buffer)
{
  lcopy((unsigned long)&record[0],
        (unsigned long)&sector_buffer[2],254);
  lcopy((unsigned long)&record[254],
        (unsigned long)&sector_buffer[256+2],254);
}

void desectorise_record(unsigned char *sector_buffer,
            unsigned char *record)
{
  lcopy((unsigned long)&sector_buffer[2],
        (unsigned long)&record[0],254);
  lcopy((unsigned long)&sector_buffer[256+2],
        (unsigned long)&record[254],254);
}

This lets us then just treat the records as 508 contiguous blocks. We can then have very nice simple code to add, delete and find fields in these records:

char append_field(unsigned char *record, unsigned int *bytes_used, unsigned int length,
          unsigned char type, unsigned char *value, unsigned int value_length)
{
  // Insufficient space
  if (((*bytes_used)+1+1+value_length)>=length) return 1;
  // Field too long
  if (value_length > 511) return 2;
  // Type must be even
  if (type&1) return 3;

  record[(*bytes_used)++]=type|(value_length>>8);
  record[(*bytes_used)++]=value_length&0xff;
  lcopy((unsigned long)value,(unsigned long)&record[*bytes_used],value_length);
  (*bytes_used)+=value_length;

  return 0;
}

char delete_field(char *record, unsigned int *bytes_used, unsigned char type)
{
  unsigned int ofs=2;      // Skip record number indicator
  unsigned char deleted=0;

  while(ofs<=(*bytes_used)&&record[ofs]) {
    unsigned int shuffle = 1 + 1 + ((record[ofs]&1)?256:0) + record[ofs+1];

    if ((record[ofs]&0xfe) == type) {
      // Found matching field
      // Now to shuffle it down.
      // On MEGA65, lcopy() has "special" behaviour with overlapping copies.
      // Basically the first few bytes will be read before the first byte is written.
      // The nature of the copy that we are doing, shuffling down, is safe in this context.
      lcopy((unsigned long)&record[ofs+shuffle],(unsigned long)&record[ofs],(*bytes_used)-ofs-shuffle);
      (*bytes_used) -= shuffle;

      deleted++;
      
      // There _shouldn't_ be multiple fields with the same type in a record, but who knows?
      // So we'll just continue and look for any others.
    } else ofs+=shuffle;
  }
  return deleted;
}

char *find_field(char *record, unsigned int bytes_used, unsigned char type, unsigned int *len)
{
  unsigned int ofs=2;  // Skip record number indicator

  while(ofs<=(*bytes_used)&&record[ofs]) {
    unsigned int shuffle = 1 + 1 + ((record[ofs]&1)?256:0) + record[ofs+1];

    if ((record[ofs]&0xfe) == type) {
      *len = ((record[ofs]&1)?256:0) + record[ofs+1];
      return &record[ofs+2];
    }

    ofs+=shuffle;    
  }
  return NULL;
}
 

Note that we have reserved the first two bytes of each record to have an immutable record number. We can't use the track and sector marker for this, because if the records get sorted, their unique identifier would change.  For the main contacts database we want those to remain in order and unchanging: Contacts can only be added, deleted or edited there.  It's only in the sorted lists that they will be rearranged, and where those record numbers will be used to point back to the original record, which is the one that will actually get edited.

We'll deal with those sorting and searching functions shortly. But first, lets actually build, store and retrieve actual contact records, and pre-set the record numbers on the disk image ready for use.

Lots of squashing the usual variety of bugs later, it looks like I can write contact records to the CONTACT0.D81 disk image:

00000000  01 01 3f 00 00 00 00 00  00 00 00 00 00 00 00 00  |..?.............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100  01 02 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  01 03 00 00 02 06 4a 65  72 72 79 00 04 09 57 69  |......Jerry...Wi|
00000210  6c 6c 69 61 6d 73 00 06  0d 57 69 6c 6c 69 61 6d  |lliams...William|
00000220  73 00 01 c7 de 70 00 00  00 00 00 00 00 00 00 00  |s....p..........|
00000230  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000300  01 04 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000310  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  01 05 00 00 02 08 59 6f  6c 61 6e 64 61 00 04 09  |......Yolanda...|
00000410  41 6e 64 65 72 73 6f 6e  00 06 0d 41 6e 64 65 72  |Anderson...Ander|
00000420  73 6f 6e 00 01 c7 de 70  00 00 00 00 00 00 00 00  |son....p........|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000500  01 06 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000510  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................| 

... well, the names are written, but we're writing the last name twice, instead of the phone number. But that's an easy fix.

Now that I have all the machinery for contacts, I should be able to do much the same for messages, so that I have the import direction complete. 

Importing messages is more complex, because we have to look up the contact from the incoming phone number. And to do that, we need our sorted versions of the CONTACT0.D81 file.  So I guess I need to make the sorting routines next.

We want the sort routine to be (a) reasonably fast/efficient; and (b) still able to run on a MEGA65 without attic RAM.  This is a bit annoying, because we don't have enough RAM to load a whole D81 into RAM to sort the records.  We can probably read 64KB or 128KB at a time, sort those sectors and write them back, and then step forward through the disk image sorting each portion.  If we make those portions overlap, and check if we changed the order of any sector in any portion, we can repeat the process until the list is sorted. Basically at the large-scale it's a bubble sort.  At the fine scale, we could get fancier than a bubble sort.

Okay, after poking about and using ChatGPT, I've settled on a quick sort for the inner sort, and then a merge sort of the resulting 10 sorted 80KB slabs. This requires only one full read of the input disk, a full write to a scratch disk image, then reading that scratch disk image track-at-a-time, and then writing linearly to the sorted disk image.

That's much less bad than my naive approach. It does increase the complexity a bit, but hopefully not too much.  I'm going to write some tests for this, because I can just smell that it's going to be a great breeding ground for esoteric bugs.

Once it is working, it will be great, though, because it exercises all sorts of things in this system, including sectorisation and desectorisation of data structures, searching for fields in records and goodness-knows-what else.

Making progress there. I do get records written into the sorted D81 -- but every record is the same. It looks like the records in the scratch D81 are sorted correctly for each slab. So it must be the merge that is wonky. Ok, found the problem: I wasn't fetching the record to actually be written, so it was just stamping out the old contents of the sector buffer each time.  But with that fixed, the darn thing seems to work!  Here's the contacts sorted by first and last names:

$ hexdump -C PHONE/SORT02-0.D81 | tail -42
*
000c7600  01 07 00 00 02 07 43 61  72 6d 65 6e 00 04 06 53  |......Carmen...S|
000c7610  6d 69 74 68 00 06 0d 2b  39 39 39 38 36 33 30 32  |mith...+99986302|
000c7620  39 34 35 00 00 00 00 00  00 00 00 00 00 00 00 00  |945.............|
000c7630  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7700  01 08 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7710  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7800  01 09 00 00 02 05 45 6d  6d 61 00 04 08 4a 6f 68  |......Emma...Joh|
000c7810  6e 73 6f 6e 00 06 0d 2b  39 39 39 37 33 30 31 34  |nson...+99973014|
000c7820  35 31 32 00 00 00 00 00  00 00 00 00 00 00 00 00  |512.............|
000c7830  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7900  01 0a 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7910  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7a00  01 03 00 00 02 06 4a 65  72 72 79 00 04 09 57 69  |......Jerry...Wi|
000c7a10  6c 6c 69 61 6d 73 00 06  0d 2b 39 39 39 32 30 39  |lliams...+999209|
000c7a20  31 35 30 36 30 00 00 00  00 00 00 00 00 00 00 00  |15060...........|
000c7a30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7b00  01 04 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7b10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7c00  01 0b 00 00 02 07 4e 69  63 6f 6c 65 00 04 07 48  |......Nicole...H|
000c7c10  6f 6c 6d 65 73 00 06 0d  2b 39 39 39 36 36 30 34  |olmes...+9996604|
000c7c20  39 33 37 32 00 00 00 00  00 00 00 00 00 00 00 00  |9372............|
000c7c30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7d00  01 0c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7d10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7e00  01 05 00 00 02 08 59 6f  6c 61 6e 64 61 00 04 09  |......Yolanda...|
000c7e10  41 6e 64 65 72 73 6f 6e  00 06 0d 2b 39 39 39 38  |Anderson...+9998|
000c7e20  35 34 32 33 30 39 38 00  00 00 00 00 00 00 00 00  |5423098.........|
000c7e30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7f00  01 06 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7f10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c8000
$ hexdump -C PHONE/SORT04-0.D81 | tail -42
*
000c7600  01 05 00 00 02 08 59 6f  6c 61 6e 64 61 00 04 09  |......Yolanda...|
000c7610  41 6e 64 65 72 73 6f 6e  00 06 0d 2b 39 39 39 38  |Anderson...+9998|
000c7620  35 34 32 33 30 39 38 00  00 00 00 00 00 00 00 00  |5423098.........|
000c7630  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7700  01 06 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7710  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7800  01 0b 00 00 02 07 4e 69  63 6f 6c 65 00 04 07 48  |......Nicole...H|
000c7810  6f 6c 6d 65 73 00 06 0d  2b 39 39 39 36 36 30 34  |olmes...+9996604|
000c7820  39 33 37 32 00 00 00 00  00 00 00 00 00 00 00 00  |9372............|
000c7830  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7900  01 0c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7910  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7a00  01 09 00 00 02 05 45 6d  6d 61 00 04 08 4a 6f 68  |......Emma...Joh|
000c7a10  6e 73 6f 6e 00 06 0d 2b  39 39 39 37 33 30 31 34  |nson...+99973014|
000c7a20  35 31 32 00 00 00 00 00  00 00 00 00 00 00 00 00  |512.............|
000c7a30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7b00  01 0a 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7b10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7c00  01 07 00 00 02 07 43 61  72 6d 65 6e 00 04 06 53  |......Carmen...S|
000c7c10  6d 69 74 68 00 06 0d 2b  39 39 39 38 36 33 30 32  |mith...+99986302|
000c7c20  39 34 35 00 00 00 00 00  00 00 00 00 00 00 00 00  |945.............|
000c7c30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7d00  01 08 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7d10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7e00  01 03 00 00 02 06 4a 65  72 72 79 00 04 09 57 69  |......Jerry...Wi|
000c7e10  6c 6c 69 61 6d 73 00 06  0d 2b 39 39 39 32 30 39  |lliams...+999209|
000c7e20  31 35 30 36 30 00 00 00  00 00 00 00 00 00 00 00  |15060...........|
000c7e30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000c7f00  01 04 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000c7f10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

I know I shouldn't be amazed, but it's still a really nice feeling, because while this code is running under Linux, it's been written using the faked MEGA65 C API, and should be able to be easily compiled for the MEGA65 when I get to that point.

But first, let's make our export utility, that will let us extract the contacts we've loaded in, so that we can make unit tests that populate and then check. 

Okay, that's now working -- and thanks to all the foundation work to date, wasn't particularly hard to do. So now I can run a command like this:

$ python3 src/telephony/sms-stim.py 5 10 ; \
   make src/telephony/linux/{import,export,provision} \
   && src/telephony/linux/provision \
   && src/telephony/linux/import stim.txt \
   && src/telephony/linux/export export.txt 

And then export.txt will end up containing something like this:

$ cat export.txt 
CONTACT:+99920915060:Jerry:Williams:0:
CONTACT:+99985423098:Yolanda:Anderson:0:
CONTACT:+99986302945:Carmen:Smith:0:
CONTACT:+99973014512:Emma:Johnson:0:
CONTACT:+99966049372:Nicole:Holmes:0: 

And they match! The only difference is that exporting contacts includes the number of unread messages for each contact. I've since updated the import program to allow setting those values, too.

So the next step is to import and export messages. Exporting will be fairly easy. Importing is a bit fiddlier.  We need to have our contact index by phone number working, so that the messages can be efficiently inserted. So let's work on making the indices.

As a reminder, the indices will index every two-byte pair in the field being indexed.  But since we have only 1,580 sectors available in an index D81, each of which can hold two index pages, we can't index all 256x256 combinations.  Our practical limit is 56x56 = 3,136 halves = 1,568 full sectors.  

Now, it would be nice if we could have had 64x64, as we could just mask bytes into the 0x20 -- 0x7F range, which would make indexing unicode UTF-8 sequences a bit nicer easier.  We might just be able to do it, if we accept the trade-off of having index pages spanning sectors. This is because each index is a 1-bit bitmap of 1,580 bits = 198 bytes.  790K / 198 bytes = 4,085, which is just under 64x64. Drat! But 799KB (less 8 bytes per KB for CBM DOS compatible sector links) is just over 4096x198 bytes. So if we limit ourselves to a single directory sector we can make it work.  It's a bit of a hack. 

The question is whether it's worth the hack? I'd have to modify the disk image provisioning stuff to handle it.  Index searches would require up to twice as many sector accesses, too. The more I think about it, I don't think it is worth it. Instead we can just use repeated subtraction to wrap all byte values into the modulo 56 space. For 8-bit values we only need to subtract at most 4 types, which feels reasonable.  The only niggling concern I have is if this results in unfortunate aliasing.  Both problems could be mitigated by using a custom byte mapping with a 256-byte lookup table. Probably requires barely more RAM than implementing the repeated subtraction.

An related topic is selecting between case sensitive and case-insensitive indexing. The norm is case-insensitive, so we'll just make our 256-byte mapping table handle that. 

I think the mapping table can just be 1:1 except for the upper/lower case chars. Forming sieves of each diphthong's bitmap should let us filter very efficiently, I think. I could get all fancy pants and add the bitmaps together so that we can get a sense of quality of match via number of matching diphthongs, that would also allow for ranking and thus accepting partial matches where no exact matches exist. That's outside of our core requirements, and would require a 1,580 byte structure when searching, which would be a similarity score of each message slot to the query text, which will be really nice.

And maybe map space, tab, carriage-return and line-feed all to the same value, so that the type of white-space doesn't matter so much? But I'm not even sure that will be necessary if I have this nice similarity score thing happening. 

So let's make that mapping table with just case folding, make the index builder, and see how it works.  For efficiency, we don't want to do random reads and writes as we go over each diphthong in a message. So we should probably build the bitmap that represents the diphthongs present in the message being indexed (requires only 1,580 bits), and then go linearly through the index reading and updating each index -- not just setting the bits for the present diphthongs, but making sure we clear those not present in the record.  This will make creating or modifying a contact or message take perhaps a second or two. But that feels acceptable. Otherwise we risk indices getting out of sync, which will make the search function mess up.  So we'll start with that, and optimise it if we need to.

I've refactored a bunch of code for this, in part to handle if the work area we have is no the current 128KB. This is because if a message or contact is being added or edited, we will want to have the display active. But we need space in the precious chip RAM for the unicode font glyphs. It might be that I can still keep enough space for those and a large enough work area for the indexes and sorting. If so, great. But I suspect I'll have to reduce it from 128KB to perhaps 40KB or even smaller. We'll see. But the main thing is that we're prepared for it now.

We will want to take considerable care to keep the indices up to date, because to build the index for all contacts or all messages in a thread requires a complete pass over the index D81 for each contact or message.  That can take a long time!  Later, I should make this much faster by using a substantial work area to generate the index for the contacts and messages in batches. This should happen in a separate helper program with no messages on the screen, so that we can build the index bitmaps for perhaps a few hundred records in one go, and then only need a few passes, instead of currently potentially requiring 1,680 passes if a thread or contact D81 were full... which could take a couple of hours.

I'll get it working in this inefficient way first, and then work on optimising it. 

But first I've gone through and refactored out all large buffers used by the different modules (eg sort vs indexing) that cannot call each other.  By "large" I mean anything at all, since our code + data size is limited to about 40KB with CC65.  But my initial target was the 508 record buffers that I need for sorting and indexing.  I also made all the header files reentrent, as the dependencies between them is starting to get a bit more complex.

It really would be nice if the TE0725's with HyperRAM were in stock, so that we could have the extra 8MB as Attic RAM like on the desktop MEGA65. This would let us do all index generation in a single pass, and deal with this whole efficiency problem that way. I'll implement it all to work without Attic RAM for now, but I'll then probably just make some #ifdefs to switch to much more efficient algorithms (and not using precious chip RAM for the 128KB work area) when Attic RAM is available.

But let's keep on task for now, to actually produce the contact/message thread indexes.

And now I've broken something. Sorting the contacts D81 is failing. Found it: Just a stupid error in the slab_read() refactor, where I had it reading the slab from the wrong disk image.

Okay, so something is being written to the indexes now when I create them. But something is broken.  Only very few index pages get set. And none of the index page bytes have more than a single bit set. All highly suspicious. Sure, I have only 5 contacts I'm indexing, but I still really expect more than 10 bits to be set in the index.  

Yup, it looks like only diphthongs 0 and 0x780 are being set, and are being set for each contact. Let's figure out what's happening there. But I'm also not convinced that even those are being written out correctly, since the 10 bits that are being set are not localised to the two index pages, but a single bit each in 10 index pages.

Okay, so first problem there was that I wasn't actually using the table to map characters to the modulo 56 number space. As a result totally invalid diphthongs were being generated.  I've fixed that now, but they aren't all being written.  Something is still going wonky, possibly in writing them out, rather than the code that is generating the set of diphthongs now.  This is what I see:

DEBUG: Diphthong 0x00b is in the indexed text.
DEBUG: Diphthong 0x26d is in the indexed text.
DEBUG: Diphthong 0x12a is in the indexed text.
DEBUG: Diphthong 0x402 is in the indexed text.
DEBUG: Diphthong 0x409 is in the indexed text.
DEBUG: Setting bit for diphthong 0x00b from record 1 at slab offset 0x00b00, slab 0
DEBUG: Setting bit for diphthong 0x12a from record 1 at slab offset 0x12a00, slab 0
DEBUG: Setting bit for diphthong 0x80b from record 1 at slab offset 0x08b00, slab 3
DEBUG: Setting bit for diphthong 0xa6d from record 1 at slab offset 0x06d00, slab 4
DEBUG: Setting bit for diphthong 0x100b from record 1 at slab offset 0x10b00, slab 6
DEBUG: Setting bit for diphthong 0x126d from record 1 at slab offset 0x0ed00, slab 7
DEBUG: Setting bit for diphthong 0x1402 from record 1 at slab offset 0x00200, slab 8
DEBUG: Setting bit for diphthong 0x1409 from record 1 at slab offset 0x00900, slab 8

Something funny is going on with the diphthong numbers it thinks are relevant for each index page. Found that bug. Now it's less bad looking, but it's still messed up somewhere:

DEBUG: Diphthong 0x00b is in the indexed text.
DEBUG: Diphthong 0x26d is in the indexed text.
DEBUG: Diphthong 0x12a is in the indexed text.
DEBUG: Diphthong 0x402 is in the indexed text.
DEBUG: Diphthong 0x409 is in the indexed text.
DEBUG: Setting bit for diphthong 0x000b from record 1 at slab offset 0x00b00, slab 0
DEBUG: Setting bit for diphthong 0x012a from record 1 at slab offset 0x12a00, slab 0
DEBUG: Setting bit for diphthong 0x026d from record 1 at slab offset 0x12d00, slab 1
DEBUG: Setting bit for diphthong 0x0402 from record 1 at slab offset 0x04200, slab 3
DEBUG: Setting bit for diphthong 0x0409 from record 1 at slab offset 0x04900, slab 3
DEBUG: Setting bit for diphthong 0x080b from record 1 at slab offset 0x08b00, slab 6
DEBUG: Setting bit for diphthong 0x092a from record 1 at slab offset 0x06a00, slab 7
DEBUG: Setting bit for diphthong 0x0a6d from record 1 at slab offset 0x06d00, slab 8
DEBUG: Setting bit for diphthong 0x0c02 from record 1 at slab offset 0x0c200, slab 9
DEBUG: Setting bit for diphthong 0x0c09 from record 1 at slab offset 0x0c900, slab 9 

We're now writing all 5 of the dipthongs, but we're duplicating them, with 2048 added to them, by the look of things.  Okay, that problem was using a char instead of a 16-bit type for the index into the diphthong bitmap.

The indexes written to disk now look plausible!  But I won't believe it until I've verified it.  I'll make a little search utility that will use the index to find matching records, and calculate their similarity scores, and then rank them. 

Searching using the index is pretty simple, we just have to count up the number of hits against each diphthong against each record.  Then we'll have a nice score fore each record. Sort those by descending score, and we'll have a ranked set of results. Alternatively, leave them unsorted for it to act as a filter.  We'll probably have an option to toggle that sort order.  But first, let's calculate those scores.

We also want search queries to be able to be quickly edited and have the results update in more or less real-time.  This includes both inserting and deleting characters.  We might also want to add or delete multiple at a time, e.g., for UTF-8 multi-byte sequences, so we should separate updating the query and query state from providing the optionally sorted search results.

The API so far is looking like this:

char search_query_init(void);
char search_query_release(void);
char search_query_append(unsigned char c);

I'll add functions for deleting characters at end or ranges of characters at specific positions within the query -- or inserting them -- as well as the function to ask it to update the query results, and select whether to sort the results, or leave them in their natural order.

Okay, I've added these now, too:

char search_query_delete_char(void);
char search_query_delete_range(unsigned int first, unsigned int last);

I still don't have the code to filter and sort, but let's start by testing what we have first, by making a linux command line tool for testing search. First step done: I can run a query from the command line utility and get the scores of the contacts (I'm searching by first name here):

$ make src/telephony/linux/search \
   && src/telephony/linux/search PHONE/CONTACT0.D81 PHONE/IDX02-0.D81 "Nicole"
make: 'src/telephony/linux/search' is up to date.
INFO: Disk image 'PHONE/CONTACT0.D81' mounted as drive 0
INFO: Disk image 'PHONE/IDX02-0.D81' mounted as drive 1
1:1:
2:2:
3:1:
4:1:
5:6:
INFO: Completed.
$ cat export2.txt 
CONTACT:+99920915060:Jerry:Williams:0:
CONTACT:+99985423098:Yolanda:Anderson:0:
CONTACT:+99986302945:Carmen:Smith:0:
CONTACT:+99973014512:Emma:Johnson:0:
CONTACT:+99966049372:Nicole:Holmes:0:

And it works! Contact 5 gets a score of 6, because it is a strong match to my query. Record 2 also gets a hit, because "Yolanda" contains the diphthong "ol" from our query.

However, it is still being case-sensitive for some reason. I can only assume I messed up my lookup table. Yup -- out by one error in position of the lower-case alphabet.  Now it works just fine with any case.

So now we can work on getting the filtered list of results, and then having them optionally sorted by score.  I've implemented these functions now:

char search_query_rerun(void);
char search_collate(void);
char search_sort_results_by_score(void);

So let's update our search program to show the matching records, and to ignore those with score == 1, since those don't actually match the query -- they're just valid. Okay, all done, and it works :)

$ make src/telephony/linux/search \
  && src/telephony/linux/search PHONE/CONTACT0.D81 PHONE/IDX02-0.D81 "Jerry"
make: 'src/telephony/linux/search' is up to date.
INFO: Disk image 'PHONE/CONTACT0.D81' mounted as drive 0
INFO: Disk image 'PHONE/IDX02-0.D81' mounted as drive 1
1:5:Jerry:Williams:+99920915060:0
INFO: Completed.
$ make src/telephony/linux/search \
  && src/telephony/linux/search PHONE/CONTACT0.D81 PHONE/IDX02-0.D81 "Nicole"
make: 'src/telephony/linux/search' is up to date.
INFO: Disk image 'PHONE/CONTACT0.D81' mounted as drive 0
INFO: Disk image 'PHONE/IDX02-0.D81' mounted as drive 1
5:6:Nicole:Holmes:+99966049372:0
2:2:Yolanda:Anderson:+99985423098:0
INFO: Completed. 

Okay, so exact and fuzzy matching are both working. So what's next? Probably importing the SMS messages, now that we have a way to find the corresponding contact by looking for the contact that corresponds to the originating phone number, and then updating the unread message count and the message thread index to support searching within (but not between) threads. 

This really shouldn't be hard now that we've got all this foundation work done.

Messages from unknown numbers we should just map to a hard-wired "UNKNOWN Numbers" contact, that possibly we declare to always occupy contact record 1 for efficiency and simplicity. In that case, we'll record the orginating number with the messages, so that if the user creates a contact for them later, we can find all the relevant messages and pre-populate the message thread for that contact.  Actually, we can just always store the originating number. 

A couple of hours later I have message importing working fairly well -- except that the search for the contact by telephone number is completely failing. This is weird, because I can search for the phone number using my command line tool, and it works just fine.  If I disable that, and just let it do an O(n) trawl through the contacts D81 to find the contact, then they do get stored (yet to confirm the thread MESSAGES.D81 really has the messages stored, but the unread message count goes up for SMS messages received :).

That problem turned out to be that the import program wasn't generating the contact index by phone number until the end --- so the index was legitimately empty when the search happened.

With that fixed, it now looks good.

Next step is to make a message thread search utility, that with no query string will show all messages in a thread with a contact (also selected by a search string), optionally sorted by relevance.

Okay, I have that working without the message thread search query, like this:

$ make src/telephony/linux/thread && src/telephony/linux/thread "Nicole"
make: 'src/telephony/linux/thread' is up to date.
CONTACT:Nicole:Holmes:+99966049372:2
MESSAGETX:+99966049372:398185456:🍳🍿 Quasi nisi quidem, quis veniam sed numquam ipsam quo hic amet molestiae? Veritatis cupiditate ullam nihil et tenetur doloribus, accusantium:
MESSAGERX:+99966049372:398196160:Dolores excepturi facilis atque quas labore rerum, libero voluptates modi, dolorum delectus dolores praesentium rerum dignissimos 😭🥺:
MESSAGERX:+99966049372:398235394:Beatae consequuntur similique rerum et, est nostrum MEGAphone quo quidem facilis in corporis 🦄:
MESSAGETX:+99966049372:398298378:Impedit dolorem ut neque quo placeat eaque necessitatibus culpa blanditiis, cupiditate velit minima ducimus deleniti esse.:
INFO: Returning conversation only for best matching contact.

The problem is that the message thread index seems to be bonkers.  It has bits set in it, but there are bits for messages that don't exist. So I need to figure out what's going wrong there.  The first part of the problem was the physical sector markers that we have in the CONTACTS disk images. But they were being set in all D81s that were being provisioned.  With that fixed, we no longer have any spurious bits set that don't correspond to real messages.

So why is the query resulting in no matches? The search utility that is really made for contacts, rather than message threads does find hits. So I guess this means the problem is more likely in threads.c. Modifying search so that it can show the body text of messages that match, it looks like the index contents are all fine.

Found and fixed -- another simple PEBCAC: For some reason I had lost the line that actually added the query text into the search.  With that fixed, it looks like it's working now:

$ src/telephony/linux/thread "Nicole" "MEGA"
CONTACT:Nicole:Holmes:+99966049372:2
MESSAGERX:+99966049372:398236805:Beatae consequuntur similique rerum et, est nostrum MEGAphone quo quidem facilis in corporis 🦄:
INFO: Returning conversation only for best matching contact. 

And look at that: We've found a matching contact and message with the string in it :)

So, I think that's actually the last of the ingredients I need to actually start getting this all ready to run natively.  I'll have to adapt the code, and start adding the UI stuff, like actually rendering messages and contacts.  But with the stimulator, I can populate a fairly extensive set of contacts and messages for each, so that there's something to see in the various views. 

Anyway, this post has dragged on more than long enough, so I'll stop it right now, and start work on getting this transitioned to native MEGA65 running to display contacts etc.

MEGAphone Call, SMS and Contacts Management

Managing contacts on a system that has 384KB of RAM is not as simple as it first sounds. This is compounded by the limited support for direct FAT file system access in the Hypervisor.

My current plan is to use a D81 disk image to store each contact in a single 512 byte sector.  As the disk images are 8,192,000 bytes = 800KiB, this means we can have 1,600 contacts -- less a bunch for track 40 that we will keep reserved for a valid 1581 directory listing, that will basically identify it as a MEGAphone contacts disk image.  That will leave 790KiB = room for 1,580 contacts.

To keep the contacts file a valid 1581 file (as well as supporting direct sector access), the first two bytes of each contact sector will be a standard link to the next sector. Actually, each 512 byte sector has to have two of these sector pointers, one at offset 0 and the other at offset 256, because the 1581 uses 256 byte logical sectors for commonality with the 1541/1571. That leaves us with 508 bytes per contact.

Now, what exactly we do with all 508 bytes is up for grabs.  But what we can do, is decide on how to store the name and primary telephone number of the contact, since those are the two most important fields.  Things like an avatar we'll leave for now as out of scope, as while desirable, they are not essential for basic operation.

For flexibility, we'll use a single byte to indicate the field type, followed by a length field. We'll use the LSB of the field type as MSB of the length field, to allow for fields to consume all 508 bytes.

0x00 - End of contact record 

0x02 - Contact Name

0x04 - Contact Second Name (surname) 

0x06 - Primary number 

0x08 - Disk image name for SMS conversation file

0x0A - Number of unread SMS messages (length = 2, to allow for upto 65,535) unread messages.

The contact disk image should be sorted by telephone number for quickly finding a contact based on incoming telephone number (for handling SMS arrivals).

A secondary contact disk image should be maintained as a subordinate copy, re-sorted by name. And a 3rd copy sorted by second name.  Whenever a contact is modified in the primary contact file, these subordinate contact indices should be updated.  Similarly whenever a contact is added or deleted from the primary contact list.

This implies that we need a sort function for a contacts disk image, that sorts based on the supplied field type.  As the MEGAphone won't have the 8MB HyperRAM in the initial build, we have to be able to do this sort in chip RAM.  In practice, this means that we'll probably read 64KB of sectors in at a time, sort those, and then write them back, advance 32KB, and repeat. This will implement a kind of batched of bubble-sort.  Multiple such passes will be required, until no further changes occur during a complete pass.  We could do better in terms of big-O time cost, but this algorithm will be small and simple, and we can easily improve it down the track.

Now, we indicated having a disk image for each SMS conversation,  We'll do a similar approach to the contact disk images, by storing the most recent 1,580 messages, each in their own 512 byte sector.  The message format should indicate if it's an in-bound or out-bound message, the time and date of the message, followed by the message body. We'll use a similar type + length + data format:

0x00 - End of message record

0x02 - Message is in-bound (from contact) or outbound (1 byte length)

0x04 - Time and date of message (ASCII representation)

0x06 - Message body, UTF-8.

Now, for searching for contacts and messages within a conversation, we'll implement a horribly crude index, where we have a bitmap of 1,600 bits = 200 bytes for each paired combination of 56 characters, i.e., 56x56 = 3,136 256 byte sectors = 1,568 512 byte physical sectors, just fitting in the 1,580 sectors available in the D81.  For the 56 characters, we will allow A-Z (case insensitive), 0-9, some punctuation, and German extra characters for now (ö, ä, ü and ß), given the initial market focuses will be English and German speaking.  We'll also reserve one of those characters to mean "any other character," so that at least partial filtering will be possible, regardless of the language, provided it uses a reasonable fraction of latin letters. It's a far from perfect solution, but it will allow for fast searching, without using a lot of RAM or complex algorithms.

Basically when searching for a string like "potato", all we have to do is generate each consecutive character pairing, i.e., "po", "ot", "ta", "at", and "to", retrieve the bitmap for each and logically AND them together. Any 1 bits at the end mean that each of those two-character pairings are present in that message.  This will likely be good enough, and result in a fairly low false-positive rate. It will also be tolerant to some spelling errors.  We can slightly improve it by instead of ANDing, count the number of 1s in each message, and then return a search result list ordered by how good the match is. A bir crude, but I suspect it will be surprisingly effective.

We can use this kind of index on both the contact list, and one for each of the SMS message threads with each contact.  That keeps code compact and simple.

When we receive or send an SMS, we need to look at the next available sector in the contact conversation (and possibly create a new contact if the number hasn't been seen before, I need to think about that), and put the message there, with the in-bound or out-bound marker set.  Then if it's an inbound message, we need to update the unseen message count in the contact list, and then update that in the contact index disk images as well (ideally with a targeted update, rather than requiring a full re-sort that would take several seconds, probably.)

We should probably also have somewhere in the contacts disk image where we keep track of the whole unseen message count status, for quickly determining what we should show on the display to report unseen messages. This could be kept in some fake directory entries, which would also make it easy to query from 3rd-party programs.

If the thread for a contact fills, then we should either automatically rotate the conversation through, aging out old messages, create a 2nd disk image for the contact, prompt the user or otherwise. We should also keep track of what the most recent message position in the disk image is. This could be done in a fake directory entry. 

Okay, that all sounds a bit complex, but the reality is that it's about as simple as we can make a functional SMS and contact handling system.

It will let us make code to ask for the most recent n messages, which will make it easier for us to update a scrolling display of messages, without having to keep them all in RAM at the same time. That will need to work backwards, since by default we want to show the most recent message at the bottom, and be able to scroll backwards in time, without having to get all crazy-pants with algorithms. It might be that we allow scrolling only message-by-message to keep this simple. Probably okay to begin with. 

This means our SMS message chain renderer also needs to start from the bottom of the screen and work its way backwards up the display.  Again, not too hard to do. It's just a matter of implementing it.

Now, for managing piles of contacts with out disk searching going too slow, I'll probably use a directory structure, where we have PHONE/THREADS/00-FF/00-FF/ allowing for 64K contacts, each in their own directory, with one or more D81s holding conversations, and whatever else we decide to put there (like avatar images).  Then we'd also have PHONE/CONTACT0.D81, CONTACTS/IDX02-0.D81 (for sorted by first name index) and PHONE/IDX04-0.D81 (for sorted by second name index).

Now, one unsolved issue is messages received from numbers that are not yet contacts.  We could setup a new contact for each automatically, but as we have index issues beyond 1,580 contacts (although we could have CONTACT0, CONTACT1 etc D81s, to allow for growing beyond 1,580 contacts), it probably makes sense to just put all messages/calls from unknown senders into a common thread for now, with the sending number indicated in a specific data field.

That also reminds me, we will be treating missed calls as a special kind of SMS message, likewise whenever you call someone, it will be logged as a virtual outbound SMS.

Corrupt indices we will deal with by just rebuilding them.

Corrupt contacts we will just live with. On SD card they shouldn't happen, but if they do, it should only mess up fields in the record, and we can just let the user edit it, and make it correct when written back.

Deleting contacts or messages just leaves an empty record in the disk image, which the sort function will clean up. 

Now, we also need to store the current state of the telephony interface for when we get interrupted by an incoming call, incoming SMS or other event that requires us to switch which part of the telephony software is running (we are really breaking the software down into a bunch of smaller programs, so that each can fit in 64KB using the CC65 compiler (which really means about 40KB of code, which with CC65 is not necessarily that much). On returning from a helper program, we should be able to restore the state of what we were doing, e.g., showing any active call status, ringer status etc, restoring a message from draft status (we should store those in a reserved sector of the contact thread if it's a message to a contact, but that won't work if the recipient isn't a contact). There are probably other things, too.

To manage this, except for draft messages to contacts which we can put in a reserved sector in the thread D81 for each contact, we should probably just have a STATE.D81 where we progressively allocate sectors for each bit of state. Those sectors should be chained into the directory listing with appropriately named files, so that they can be easily modified by 3rd party programs -- so long as their track and sectors don't change on the disk, as we want to bypass having to use the CBDOS file system code (partly so that we can reclaim the 128KB of chip RAM where the "ROM" usually sits).  We'll mark the files as locked, so that they are nominally read-only, and include a warning in the directory entry that files must retain their track and sector numbers, and should only be read by 3rd party software if using the file system, instead of direct sector access.

When messages are received from the cellular modem (e.g., incoming call, incoming SMS, call terminated etc), then these should be written verbatim into a message buffer in STATE.D81 (to keep common cellular modem monitoring code simple), state should be stashed and then processed by the appropriate helper program.  Now, there is an issue here where a denial of service attack could be mounted against a user, by having people continually ring and/or text them, so that the thing is endlessly occupied saving and restoring state and switching programs.  To mitigate this, we should probably process RING messages from within each program, and, ideally, messages from the same contact that is currently being displayed, too. But everything else can be batched up and processed periodically. 

Or, we could have the program that does all this processing persistently held in RAM (or stored in the flash filesystem, which is also super-fast!), and then basically bank switched (or DMA swapped, or equivalent) into place, so that the latency and duration of activity is greatly reduced.  There are limits to this, though, because whenever we have to switch mounted disk images, there will be a considerable delay, as the hypervisor has to do FAT file system searches.  So we really want to avoid doing that too often.  It might be that we limit ourselves to doing a process of SMS messages addressed to other contacts/numbers to once every 10 seconds or so, or whenever the unprocessed message queue reaches 50% full, to give us time to drain it, without losing messages.

We'll reserve a healthy slab of STATE.D81 for unprocessed messages, perhaps 128KB, which would be enough for 256 unprocessed messages. 

And I think that covers the whole back-of-house stuff that we need for tracking SMS and phone calls. 

So let's try to turn this into a set of requirements (which expand on the requirements we already have documented in the repository, and are subordinate to them): 

Requirement: Linux program that can create the various core D81 files (e.g., STATE.D81).

Requirement: Linux program that sets up the directory structure on the SD card for the MEGAphone (PHONE/*).

Requirement: Code in dialer to save/restore state in STATE.D81

Requirement: Code in each UI program to save/restore state in STATE.D81 and any relevant contact thread.

Requirement: Code in each UI program to trigger a switch to the message queue processor program after a given timeout and/or the unprocessed queue has reached the 50% full level.

Requirement: Function to create a contact (including creating the D81 file(s) required).

Requirement: Function to modify a contact.

Requirement: Copy contact disk to index disk.

Requirement: Sort index disk by specified key.

Requirement: Create or open thread disk image.

Requirement: Insert message into thread disk image.

Requirement: Retrieve message from thread.

Requirement: Update unread message count for contact.

Requirement: Update unread message status for phone.

Requirement: Log non-contact calls/SMS as pseudo-contact "unknown" 

Requirement: Display an SMS / call log message (either in-bound or outbound, using appropriate formatting to differentiate)

Requirement: Display a screen full of SMS / call log messages.

Requirement: Text entry for composing an SMS.

Requirement: Dialer entry for telephone number.

Requirement: Contact list. Select to open message list.

Requirement: Exit message list back to contact list.

Desirable: Allow inserting unicode/emoji in contact fields. 

Desirable: Delete old messages from thread.

Desirable: Unicode/Emoji entry for SMS composition. 

Desirable: Contact list search function by typing a fragment of a name or number.

Desirable: Message thread search function by typing a fragment of a message.

Desirable: Highlight sections of messages that match search query.

Desirable: Update search index based on a single contact or message.

Desirable: Rebuild search index for all contacts/messages.

Desirable: Associate avatar/emoji with contacts.

Desirable: Display contact avatar/emoji on incoming call.

Desirable: Dispaly contact avatar/emoji during call.

Desirable: Allow SMS composition during calls.

Desirable: Allow pasting location into SMS message (including during call).

Desirable: Allow showing last known direction and distance to contact based on last shared location (including during a call).

That's all probably enough now, that I can start digging in to implement it all. 

The above requirements specification also addresses the following milestones in the NLnet funded project:

Milestone 1.4 Telephony software User-Interface: Requirements Specification

Milestone 1.6 Text Messaging software: User-Interface: Requirements Specification

Accessing Shared Resources from the MEGA65 System Partition

In my previous post, I implemented a facility for accessing the brand new Shared Resources area of the MEGA65's System Partition.

This facility has been created to provide rapid access to content anywhere in large files --- initially unicode pre-rendered fonts for the MEGAphone.  Using a traditional file-system would require tracking file-system state, and traversing file-system structures.   This would add latency for time-critical operations, like retrieving individual character glyphs when rendering SMS messages.

This area works by implementing a simple extent-based file list in the system partition.  The shres and shresls tools exist for linux in the mega65-tools repository to populate and query the contents of this area on a MEGA65-formatted SD card.   

Only the latest development version of MEGA65 FDISK will format an SD card to include a shared resource area.

test_897.c in mega65-tools has unit tests for this functionality, including reporting if your HYPPO version is too old, or there doesn't seem to be a shared resources area defined in your system partition.

So now we need to tie all this together is some code that could eventually go in mega65-libc that provides a simple API for accessing it. I'm thinking something like:

char shopen(char *resource_name,unsigned long long required_flags, struct shared_resource *file_handle) -- open  

shread(unsigned char *ptr, unsigned int count, struct shared_resource *file_handle) -- Read a slab of bytes from the opened resource from the current offset.

shseek(struct shared_resource *,unsigned long long offset, unsigned char whence) -- Seek to arbitrary point in the file.

unsigned int shdopen() -- open directory handle to shared resource area

sddread(unsigned long long required_flags, unsigned int *directory_handle, struct shared_resource *dirent) -- read next directory entry that matches.  The resource is implicitly an opened resource (since "opening" is absurdly light weight, the way I'm doing it).

The shared_resource struct will simply contain the filename, flags, starting sector in the shared resource area, its length, and current seek position (initially 0):

#define MAX_RES_NAME_LEN 256
#define SEEK_SET 1
#define SEEK_CUR 2
#define SEEK_END 3 
struct shared_resource {
   char name[MAX_RES_NAME_LEN];
   unsigned long long flags;
   unsigned long long length;
   unsigned long long first_sector;
   unsigned long long position;
}; 

I'm tracking this via https://github.com/MEGA65/mega65-libc/issues/70.

To test this API, I'm just going to start writing a wrapper program in the megaphone-modular repository in src/telephony. 

Okay, so in the process of setting that up, I'm discovering that CC65 mainline doesn't have support for the 45GS02 instruction set, so I can't use LDQ and STQ directly.  I guess I'll just implement those as the underlying instruction sequences.

Next challenge is that the SD card keeps getting stuck busy.  One of the subtleties involved in this is that the SD card interface _does_ lockup if you ask it to read a sector while it's already reading a sector. We should probably fix that, but for now I have to live with it.

To deal with this, I've made the shared resource API functions check the SD card status, and fail with an error if it's busy. So it should never be calling a SHRES Hypervisor trap when the SD card is busy (remember that the SHRES Hypervisor trap does the sector read request, so this is vital).

char do_shres_trap(unsigned long arg)
{
  // Fail if SD card is busy
  if (PEEK(0xD680)&0x03) {
    printf("SD card is busy\n");
    return 1;
  }

  printf("SD card was idle\n");
  
  shres_regs[0] = (arg>>0)&0xff;
  shres_regs[1] = (arg>>8)&0xff;
  shres_regs[2] = (arg>>16)&0xff;
  shres_regs[3] = (arg>>24)&0xff;
  shres_trap();

  printf("Trap C=%d\n",shres_regs[4]&1);
  
  // Check trap response in P
  return (shres_regs[4]&0x01) ^0x01;

}

Basically the wrapper is the only way I call the trap, and it should reliably fail if the SD card interface is already busy.

So what's going wrong here?

Okay, so it looks like it's trying to read an invalid sector from the SD card for some reason now.  The call to shdopen() always asks for sector 0 of the shared resource area:

 

shared_resource_dir shdopen()
{
  char i;
  if (do_shres_trap(0)) return 0xffff;
 

So provided that shres_regs is getting set properly in the wrapper it calls, it should be working.

I _think_ there might be something funny where the HYPPO trap is not adding the SHRES start offset to the $D681-4 SD card registers properly.  

readsharedresourcetrap:
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    ldq syspart_resources_area_start
    adcq $d681
    stq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
        ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
        jmp return_from_trap_with_success

Ah!  There is indeed a logical error here. It should look like this:

readsharedresourcetrap:
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    adcq syspart_resources_area_start
    stq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
        ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
        jmp return_from_trap_with_success
 

Yay! With that fix, I can now iterate over the (slightly random) set of files I put in the shared resource area of my SD card:

So perhaps I'll next make it try to cat the README.md file to the screen, just as a proof-of-concept --- and make it use the API, forcing me to implement all of the API.

Here's the resulting program:

struct shared_resource dirent;
unsigned long required_flags = SHRES_FLAG_FONT | SHRES_FLAG_16x16 | SHRES_FLAG_UNICODE;

unsigned char buffer[128];

unsigned char i;

char filename_ascii[]={'r','e','a','d','m','e','.',0x6d,0x64,0};

void main(void)
{
  shared_resource_dir d;

  mega65_io_enable();

  // Make sure SD card is idle
  if (PEEK(0xD680)&0x03) {
    POKE(0xD680,0x00);
    POKE(0xD680,0x01);
    while(PEEK(0xD680)&0x3) continue;
    usleep(500000L);
  }

  if (shopen(filename_ascii,0,&dirent)) {
    printf("ERROR: Failed to open README.md\n");
    return;
  }

  {
    unsigned int b = 0;
    printf("Found text file\n");
    while(b = shread(buffer,128,&dirent)) {
      for(i=0;i<b;i++) {
    if (buffer[i]==0x0a) putchar(0x0d); else putchar(buffer[i]);
      }
      
    }
  }

  return;
}

And it works! And I can even use shseek() to print just a bit of the text file.

  {
    unsigned int b = 0;
    printf("Found README.md\n");
    shseek(&dirent,-200,SEEK_END);
    while(b = shread(buffer,128,&dirent)) {
      for(i=0;i<b;i++) {
    if (buffer[i]==0x0a) putchar(0x0d); else putchar(buffer[i]);
      }
      
    }
  }
 

Running this we can get something like this:

 In short, it works. So now it's time to make a pull-request into mega65-libc for this new API, and then we're done, and I can go back to working on making the actual unicode pre-rendered fonts for the MEGAphone telephony software! 

 

Embedding Shared Resources in the MEGA65 System Partition

In another blog post I'm writing at the moment, I'm exploring how I will do the UI for the new MEGAphone work.  Part of that involves me figuring out how I can possibly support the full Unicode gammut -- including emojis, that are not more or less a required part of SMS communications.

In that post, I considered putting them into files on the FAT32 file system, or stashing them in a set of specially formatted D81 or D65 disk images.  But all of those approaches require more than 1 SD card sector read to fetch a random glyph, and in some cases many such requests.

So I came up with the idea of putting linear slabs of shared resources into the system partition, and having a Hypervisor call that lets a user program ask for an arbitrary 512 byte SD card sector of that data.  For fonts in the MEGAphone software, this would allow the program to work out and store the sector offset within the shared resources for the font(s) it needs to use, and then just adding an offset to that to retrieve the SD card sector with the required font glyph to load into chip RAM.

In this blog post I'm going to design and implement this facility in the hypervisor, and make a test program that will let me check that it's all working.  From there, I'll work on some kind of utility that will let me modify the contents of the system partition to install the fonts.  That might end up being a Linux-based utility that works directly on the SD card for speed --- since a full Unicode font will be about 300MB in the pre-rendered format that I'll be using (more about that in the other blog post, when it's done).

I'm going to track the addition of this functionality into HYPPO via Issue #897.

So let's start by making the system partition code in the Hypervisor read and store the starting sector and size of this slab.

Now I have to remember how I made it possible to update the hypervisor using m65 -k.  From memory this can be done safely from the MEGA65's READY prompt, as that means that it's in user-space, and thus it's safe to overwrite the hypervisor.

... except that that doesn't seem to do it.

Okay, so because the Hypervisor RAM is only writeable when the CPU is in Hypervisor mode, we can't do it from the READY prompt any more. But we also need it to happen in some situation where the Hypervisor isn't really running any code, since otherwise the CPU will go off into la-la land, and bork things.

I am using an old version of m65, though.  Maybe in newer versions -k looks after this, e.g., by setting the CPU to single-step mode, then causing the CPU to start a Hypervisor trap, and then stopping the CPU once in Hypervisor mode, writing the new version of the Hypervisor code, then resetting and resuming the CPU, so that it boots the new HYPPO fresh.  If not, that would be the way to do it.

I'm tracking this via Issue #222, and pull request #223.

Now with m65 recompiled with that fix, I can quickly and easily test HYPPO builds. That's already made it _way_ more comfortable and convenient to find and fix a bug in my initial implementation of the of the shared resource thing.

So now I can implement the HYYPO trap that tries to read the sector.  Because we are going to use A,X,Y and Z as the Q pseudo-register to make it easy for a user program to access, we will have to use up a whole trap address, but that's okay.

So far, the trap code looks like this:
 

readsharedresourcetrap:
    stq $d681
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    ldq syspart_resources_area_start
    adcq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
        ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
    clc
    rts
bad_syspart_resource_sector_request:    
    sec
    rts
 

In principle, that should be all we need. 

The next step is to make a unit test for this, checking that it does cause a sector to get read, and that requests for out-of-range sectors fails.

I'll use the unit test framework that we have in mega65-tools.

Those tests use CC65, which I can't remember it's call convention for 32-bit args.  So I'll just stash the 32-bit resource sector number into some convenient memory location that it doesn't use.  Maybe $7F0-$7F3, i.e., the end of the screen RAM.

The test will then make the HYPPO call, and store the register values from HYPPO back where it read them from. Like this:

 

        .export _test_resource_read

        .p4510

.segment    "CODE"

.proc    _test_resource_read: near
    lda $7f0
    ldx $7f1
    ldy $7f2
    ldz $7f3
    sta $d644         ; Trigger Hypervisor trap
    nop            ; CPU delay slot required after any hypervisor trap
    sta $7f4
    stx $7f5
    sty $7f6
    stz $7f7
    php
    pla
    sta $7f8
    rts
.endproc

 

Pulling it all together, we can now easily run tests like this:

 

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
/home/paul/Projects/mega65/mega65-tools/cc65/bin/cl65 --config src/tests/tests_cl65.cfg -I src/mega65-libc/cc65/include -O -o src/tests/test_897.prg --mapfile src/tests/test_897.map --listing src/tests/test_897.list src/tests/test_897_asm.s src/tests/test_897.c src/mega65-libc/cc65/libmega65.a
2025-06-07T07:10:52.445Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-07T07:10:52.454Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T07:10:52.454Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T07:10:52.660Z NOTE loading file 'src/tests/test_897.prg'
2025-06-07T07:10:52.841Z NOTE running
2025-06-07T07:10:52.841Z NOTE Entering unit test mode. Waiting for test results.
2025-06-07T07:10:52.841Z NOTE System model: UNKNOWN MODEL $06
2025-06-07T07:10:52.841Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-07T07:10:52.841Z NOTE System ROM version: M65 V920395
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z  FAIL (Issue#0897, Test #000 - HYPPO RESOURCE READ TEST FAILED)
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:10:52.845Z NOTE terminating after completion of unit test
 

For those who haven't seen the unit test framework, it's quite nifty: The unit test running on the real hardware communicates test results back via the serial monitor interface to the m65 program, allowing it to directly report the test results.

Because I haven't implemented everything yet, the test claims to fail.

So now let's fix the implementation of our trap -- it should return from the Hypervisor by jumping to one of the return routines:

readsharedresourcetrap:
    stq $d681
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    ldq syspart_resources_area_start
    adcq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
    ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
    jmp return_from_trap_with_success
    
bad_syspart_resource_sector_request:
    ;; Return "illegal value" if trying to read beyond end of region
    lda #dos_errorcode_illegal_value
    jmp return_from_trap_with_failure

So now if we ask for an illegal sector number, we should get this illegal value error, which we know has the value 0x11 from the source.

So let's now update our test to check whether the trap is defined or not, and whether reading sector 0 results in success or an error:

  // Try reading sector 0 of resource area
  resource_sector = 0;
  *(unsigned long *)0x7f0 = resource_sector;
  
  if (test_resource_read() == 0) {
    unit_test_ok("");
  }
  else {
    unit_test_fail("hyppo call helper failed");
  }

  return_value = *(unsigned long *)0x7f4;

  carry = PEEK(0x7f8)&1;

  if (!carry) {
    switch(PEEK(0x7f4)) {
    case 0xff: unit_test_fail("hyppo returned 'trap not implemented'"); break;
    case 0x11: // dos_errorcode_illegal_value
      unit_test_fail("reading resource sector 0 failed -- no resource area in system partition?");
      break;
    default: {
      snprintf(message,80,"hyppo resource read failed with unknown error $%02x",PEEK(0x7f4));
      unit_test_fail(message);
    }
    }
    unit_test_report(ISSUE_NUM, 0, TEST_DONEALL);
    return;
  }
 

This routine now checks whether the Hypervisor trap is implemented or not, and then whether reading sector 0 works.  We need both of those to succeed, if we are going to able able to test that it works to actually read system resources.

With the fixed m65 -k option to easily load a HICKUP.M65 file containing an updated Hypervisor, I can now easily load the version that supports this new hypervisor call:

paul@bob:~/Projects/mega65/mega65-core$ make bin/HICKUP.M65 && m65 -k bin/HICKUP.M65 

And then execute the test, and see the result:

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
...
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z  FAIL (Issue#0897, Test #001 - READING RESOURCE SECTOR 0 FAILED -- NO RESOURCE AREA IN SYSTEM PARTITION?)
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:31:16.102Z NOTE terminating after completion of unit test
 

And look at that --- we can see it now passes the check for the hypervisor trap being implemented (Test #000), but fails reading the first sector of the shared resource area (Test #001). This is exactly what we expect, since the SD card in this MEGA65 doesn't (yet) contain a shared resource area.

In fact, I haven't even made the utility to create them.  So let's tackle that next.

Currently the MEGA65 FDISK utility is the thing that's responsible for creating the system partition.  So I'll probably just tweak that to by default create a reasonable size system resource partition, perhaps 1GB or so, but maybe also have the option to select a different size.

I'm tracking that issue here: https://github.com/MEGA65/mega65-fdisk/issues/28

The FDISK program is really quite old and crusty now, but it does it's job.  There is an outstanding issue to totally refactor it, so while I will try to not add to the mess, I'm not going to go overboard with making this new feature overly sophisticated.

Okay, done a first cut at that. It will try to allocate half the system partition for shared resources. It also increases the target SYSPART size from 2GiB to 4GiB, thus allowing 2GiB for shared resources by default.

Now to backup my SD card, and try reformatting it!

Okay, so after a bit of procrastination, it's time to just pull the SD card, copy it all off, and put it back in, and run MEGA65 FDISK on it.

 

So formatting with FDISK looked promising -- it was reserving 4GB for the system partition, which is a change I made.

But then when I tried to boot with it, it was all borked up.

Basically the FAT file system seems to be messed up.  But if I power on with the new HICKUP, it does see that we have $400000 sectors reserved for resources, which equates to 2GiB, so that's promising!

Okay, so why on earth is the FAT file system all messed up, then?

First thing to try: Downgrade to FDISK prior to the changes I've made, and see if it formats up fine with that.

Nope. Same thing. So probably not something that I've done.

Okay, so after a second go, including copying all my files back on, it's fine.

Who knows what went wrong.

The main thing is it is booting again.

Although, oddly it's running the intro disk, even though I don't have a default disk image to mount at boot set.

Hmmm... And now the new HICKUP is showing 0 sectors reserved for shared resources. Did I accidentally format it with the old version of FDISK? Yup --- looks like that was the case.

After I copied my files back into place, it was all fine. 

And, the test to read sector 0 of the shared resource area works now:

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
...

2025-06-07T13:30:30.590Z NOTE 2025-06-07T13:30:30.590Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.590Z NOTE 2025-06-07T13:30:30.590Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.592Z NOTE 2025-06-07T13:30:30.592Z  PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ TEST PASSED)
2025-06-07T13:30:30.592Z NOTE 2025-06-07T13:30:30.592Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.592Z NOTE terminating after completion of unit test
 

So now I can work on making the test more complete, by doing a binary search of the address space to determine the exact size of the shared resource partition. 

 Like this:

  resource_sector = 0xffffffffUL;
  for(i=31;i>=0;i--) {
    *(unsigned long *)0x7f0 = resource_sector;

    if (test_resource_read() != 0) unit_test_fail("hyppo trap failed");
    carry = PEEK(0x7f8)&1;

    if (!carry) {
      resource_sector -= (1UL <<i);
    }

  }

  if (resource_sector==0xffffffffUL || (!resource_sector)) {
    unit_test_fail("failed to determine size of shared resource area");
  } else {
    snprintf(message,80,"determined shared resource area is $%08lx sectors",
         resource_sector + 1);
    unit_test_fail(message);
  }
  
  printf("Shared resource area = $%08lx sectors.\n",resource_sector+1);
 

 And it works, correctly measuring the size!

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
/home/paul/Projects/mega65/mega65-tools/cc65/bin/cl65 --config src/tests/tests_cl65.cfg -I src/mega65-libc/cc65/include -O -o src/tests/test_897.prg --mapfile src/tests/test_897.map --listing src/tests/test_897.list src/tests/test_897_asm.s src/tests/test_897.c src/mega65-libc/cc65/libmega65.a
2025-06-07T13:41:32.282Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-07T13:41:32.296Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T13:41:32.296Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T13:41:32.502Z NOTE loading file 'src/tests/test_897.prg'
2025-06-07T13:41:32.691Z NOTE running
2025-06-07T13:41:32.691Z NOTE Entering unit test mode. Waiting for test results.
2025-06-07T13:41:32.691Z NOTE System model: UNKNOWN MODEL $06
2025-06-07T13:41:32.691Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-07T13:41:32.691Z NOTE System ROM version: M65 V920395
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z  PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ SECTOR 0 TEST PASSED)
2025-06-07T13:41:32.695Z NOTE 2025-06-07T13:41:32.695Z  PASS (Issue#0897, Test #002 - DETERMINED SHARED RESOURCE AREA IS $00400000 SECTORS)
2025-06-07T13:41:32.696Z NOTE 2025-06-07T13:41:32.696Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.696Z NOTE terminating after completion of unit test  

 Okay, so now the last thing to test is that it actually causes the SD card to read the sector.

First, do the SD card registers get the correct sector number written into them? I'll test this by just making sure that they don't end up with all zeroes. Anyway, nope, they don't. Found and fixed -- the bug was in the hypervisor code (the interested reader can look at the code snippet for the hypervisor trap above and see if they can spot it --- leave a comment if you find it :)

So then the next step is to check that the SD sector buffer actually changes contents.  That worked fine, without further changes (although I had to make sure the SD card was not busy, before making the HYPPO call, because requesting a sector read while the SD card is busy causes it to lock up).

Next up, then, is to come up with a simple file system for the shared resource area. It needs to be super simple to parse.  I'm thinking it could be as simple as the first sector containing version information and the number of resources in the area, followed by one sector per file, that contains the name of the file/resource, and the start sector within the resource area, and the length of the resource in both sectors and bytes.  After the final resource directory sector, we have a completely empty sector, so that a program scanning it can just look for that empty sector and know to stop, without even having to track the resource count, in case that's easier to implement in a given situation.

So let's make a utility that can build the shared resource area from a set of files and their names.  Since it's just boring boiler-plate kind of code, I used ChatGPT to do it faster. Not sure if it actually was any faster to write. But we can now do things like this:

paul@bob:~/Projects/mega65/mega65-tools$ rm bin/shresls ; make bin/shres bin/shresls && bin/shres foop mega65-screen-000000.png,31  README.md,23 && bin/shresls foop
make: 'bin/shres' is up to date.
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md
MEGA65 Shared Resource File: foop
Declared resources: 2

Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md

SHA1                                     Bytes      Check      Name
--------------------------------------------------------------------------------
40ce6cfcaaf14d11a8c3ffa71fd4726087613eee  4242       MATCH      mega65-screen-000000.png
3b31809d69f814d2173484855072333db65190cb  6436       MATCH      README.md
paul@bob:~/Projects/mega65/mega65-tools$ 

Next step is to make a tool to extract or insert a shared resource area onto/off of an actual SD card.  My plan here is to make the existing tool I've made above to work out if it's writing to an SD card, in which case, it will find the SYSPART in there, and write the resources directly into there.

Okay, so I think I have writing the resources to SD card working, as well as updating shresls so that it can read and check them on a real disk image, too:

paul@bob:~/Projects/mega65/mega65-tools$ rm bin/shresls ; make bin/shres bin/shresls && sudo bin/shres /dev/sdb mega65-screen-000000.png,31  README.md,23
make: 'bin/shres' is up to date.
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
INFO: Attempting to open shared resource file or disk image '/dev/sdb'
INFO: Found MEGA65 SYSPART at sector 0x014d7ffe
INFO: Found MEGA65 SYSPART shared resource area of 2048 MiB.
DEBUG: area_start=0x00029b0ffc00, area_length=0x80000000
Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md
 

paul@bob:~/Projects/mega65/mega65-tools$ make bin/shresls && sudo bin/shresls /dev/sdb
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
INFO: Attempting to open shared resource file or disk image '/dev/sdb'
INFO: Found MEGA65 SYSPART at sector 0x014d7ffe
INFO: Found MEGA65 SYSPART shared resource area of 2048 MiB at sector 2048 of SYSPART.
MEGA65 Shared Resource File: /dev/sdb
Declared resources: 2

Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md

SHA1                                     Bytes      Check      Name
--------------------------------------------------------------------------------
40ce6cfcaaf14d11a8c3ffa71fd4726087613eee  4242       MATCH      mega65-screen-000000.png
3b31809d69f814d2173484855072333db65190cb  6436       MATCH      README.md

So now I just need to make some tests for scanning the shared resource list, and trying to read them.  Actually, all we probably need is to check for the magic string in the shared resource area. This is because for that to be found, it means that the SD card access is working, as well as the sector address calculation.

And it works!

mega65-tools$ make src/tests/test_897.prg  && m65 -4 -r -u src/tests/test_897.prg 
make: 'src/tests/test_897.prg' is up to date.
2025-06-13T07:15:45.034Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-13T07:15:45.048Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-13T07:15:45.048Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-13T07:15:45.254Z NOTE loading file 'src/tests/test_897.prg'
2025-06-13T07:15:45.447Z NOTE running
2025-06-13T07:15:45.447Z NOTE Entering unit test mode. Waiting for test results.
2025-06-13T07:15:45.447Z NOTE System model: UNKNOWN MODEL $06
2025-06-13T07:15:45.447Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-13T07:15:45.447Z NOTE System ROM version: M65 V920395
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z  PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ SECTOR 0 TEST PASSED)
2025-06-13T07:15:45.452Z NOTE 2025-06-13T07:15:45.452Z  PASS (Issue#0897, Test #002 - DETERMINED SHARED RESOURCE AREA IS $00400000 SECTORS)
2025-06-13T07:15:45.455Z NOTE 2025-06-13T07:15:45.455Z  PASS (Issue#0897, Test #003 - SD CARD BUSY CLEARED ON RESET)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #004 - SD CARD REGISTERS WRITTEN TO)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #005 - SD CARD BUSY CLEARED)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #006 - SD CARD BUFFER CONTENTS CHANGE)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #007 - READ MAGIC STRING FROM SHARED RESOURCE AREA)
2025-06-13T07:15:45.457Z NOTE 2025-06-13T07:15:45.457Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.457Z NOTE terminating after completion of unit test
 

Okay, so now I have all the ingredients here for this to work -- HYPPO support for the shared resource area, patched MEGA65 FDISK to create the section in the system partition, linux-based tools to populate and interrogate the shared rescource area quickly and easily, and unit tests that provide proof-of-concept for accessing this area, and checking that the loaded HYPPO has support for it.

So I'm going to leave it at that for now, and I'll document writing a library to open and read from shared resources in a separate blog post. 

I'll also pursue getting the HYPPO changes rolled into mega65-core development branch, and updated FDISK into mega65-fdisk development branch.