Friday, 18 July 2025

MEGAphone Call, SMS and Contacts Management

Managing contacts on a system that has 384KB of RAM is not as simple as it first sounds. This is compounded by the limited support for direct FAT file system access in the Hypervisor.

My current plan is to use a D81 disk image to store each contact in a single 512 byte sector.  As the disk images are 8,192,000 bytes = 800KiB, this means we can have 1,600 contacts -- less a bunch for track 40 that we will keep reserved for a valid 1581 directory listing, that will basically identify it as a MEGAphone contacts disk image.  That will leave 790KiB = room for 1,580 contacts.

To keep the contacts file a valid 1581 file (as well as supporting direct sector access), the first two bytes of each contact sector will be a standard link to the next sector. Actually, each 512 byte sector has to have two of these sector pointers, one at offset 0 and the other at offset 256, because the 1581 uses 256 byte logical sectors for commonality with the 1541/1571. That leaves us with 508 bytes per contact.

Now, what exactly we do with all 508 bytes is up for grabs.  But what we can do, is decide on how to store the name and primary telephone number of the contact, since those are the two most important fields.  Things like an avatar we'll leave for now as out of scope, as while desirable, they are not essential for basic operation.

For flexibility, we'll use a single byte to indicate the field type, followed by a length field. We'll use the LSB of the field type as MSB of the length field, to allow for fields to consume all 508 bytes.

0x00 - End of contact record 

0x02 - Contact Name

0x04 - Contact Second Name (surname) 

0x06 - Primary number 

0x08 - Disk image name for SMS conversation file

0x0A - Number of unread SMS messages (length = 2, to allow for upto 65,535) unread messages.

The contact disk image should be sorted by telephone number for quickly finding a contact based on incoming telephone number (for handling SMS arrivals).

A secondary contact disk image should be maintained as a subordinate copy, re-sorted by name. And a 3rd copy sorted by second name.  Whenever a contact is modified in the primary contact file, these subordinate contact indices should be updated.  Similarly whenever a contact is added or deleted from the primary contact list.

This implies that we need a sort function for a contacts disk image, that sorts based on the supplied field type.  As the MEGAphone won't have the 8MB HyperRAM in the initial build, we have to be able to do this sort in chip RAM.  In practice, this means that we'll probably read 64KB of sectors in at a time, sort those, and then write them back, advance 32KB, and repeat. This will implement a kind of batched of bubble-sort.  Multiple such passes will be required, until no further changes occur during a complete pass.  We could do better in terms of big-O time cost, but this algorithm will be small and simple, and we can easily improve it down the track.

Now, we indicated having a disk image for each SMS conversation,  We'll do a similar approach to the contact disk images, by storing the most recent 1,580 messages, each in their own 512 byte sector.  The message format should indicate if it's an in-bound or out-bound message, the time and date of the message, followed by the message body. We'll use a similar type + length + data format:

0x00 - End of message record

0x02 - Message is in-bound (from contact) or outbound (1 byte length)

0x04 - Time and date of message (ASCII representation)

0x06 - Message body, UTF-8.

Now, for searching for contacts and messages within a conversation, we'll implement a horribly crude index, where we have a bitmap of 1,600 bits = 200 bytes for each paired combination of 56 characters, i.e., 56x56 = 3,136 256 byte sectors = 1,568 512 byte physical sectors, just fitting in the 1,580 sectors available in the D81.  For the 56 characters, we will allow A-Z (case insensitive), 0-9, some punctuation, and German extra characters for now (ö, ä, ü and ß), given the initial market focuses will be English and German speaking.  We'll also reserve one of those characters to mean "any other character," so that at least partial filtering will be possible, regardless of the language, provided it uses a reasonable fraction of latin letters. It's a far from perfect solution, but it will allow for fast searching, without using a lot of RAM or complex algorithms.

Basically when searching for a string like "potato", all we have to do is generate each consecutive character pairing, i.e., "po", "ot", "ta", "at", and "to", retrieve the bitmap for each and logically AND them together. Any 1 bits at the end mean that each of those two-character pairings are present in that message.  This will likely be good enough, and result in a fairly low false-positive rate. It will also be tolerant to some spelling errors.  We can slightly improve it by instead of ANDing, count the number of 1s in each message, and then return a search result list ordered by how good the match is. A bir crude, but I suspect it will be surprisingly effective.

We can use this kind of index on both the contact list, and one for each of the SMS message threads with each contact.  That keeps code compact and simple.

When we receive or send an SMS, we need to look at the next available sector in the contact conversation (and possibly create a new contact if the number hasn't been seen before, I need to think about that), and put the message there, with the in-bound or out-bound marker set.  Then if it's an inbound message, we need to update the unseen message count in the contact list, and then update that in the contact index disk images as well (ideally with a targeted update, rather than requiring a full re-sort that would take several seconds, probably.)

We should probably also have somewhere in the contacts disk image where we keep track of the whole unseen message count status, for quickly determining what we should show on the display to report unseen messages. This could be kept in some fake directory entries, which would also make it easy to query from 3rd-party programs.

If the thread for a contact fills, then we should either automatically rotate the conversation through, aging out old messages, create a 2nd disk image for the contact, prompt the user or otherwise. We should also keep track of what the most recent message position in the disk image is. This could be done in a fake directory entry. 

Okay, that all sounds a bit complex, but the reality is that it's about as simple as we can make a functional SMS and contact handling system.

It will let us make code to ask for the most recent n messages, which will make it easier for us to update a scrolling display of messages, without having to keep them all in RAM at the same time. That will need to work backwards, since by default we want to show the most recent message at the bottom, and be able to scroll backwards in time, without having to get all crazy-pants with algorithms. It might be that we allow scrolling only message-by-message to keep this simple. Probably okay to begin with. 

This means our SMS message chain renderer also needs to start from the bottom of the screen and work its way backwards up the display.  Again, not too hard to do. It's just a matter of implementing it.

Now, for managing piles of contacts with out disk searching going too slow, I'll probably use a directory structure, where we have PHONE/THREADS/00-FF/00-FF/ allowing for 64K contacts, each in their own directory, with one or more D81s holding conversations, and whatever else we decide to put there (like avatar images).  Then we'd also have PHONE/CONTACT0.D81, CONTACTS/IDX02-0.D81 (for sorted by first name index) and PHONE/IDX04-0.D81 (for sorted by second name index).

Now, one unsolved issue is messages received from numbers that are not yet contacts.  We could setup a new contact for each automatically, but as we have index issues beyond 1,580 contacts (although we could have CONTACT0, CONTACT1 etc D81s, to allow for growing beyond 1,580 contacts), it probably makes sense to just put all messages/calls from unknown senders into a common thread for now, with the sending number indicated in a specific data field.

That also reminds me, we will be treating missed calls as a special kind of SMS message, likewise whenever you call someone, it will be logged as a virtual outbound SMS.

Corrupt indices we will deal with by just rebuilding them.

Corrupt contacts we will just live with. On SD card they shouldn't happen, but if they do, it should only mess up fields in the record, and we can just let the user edit it, and make it correct when written back.

Deleting contacts or messages just leaves an empty record in the disk image, which the sort function will clean up. 

Now, we also need to store the current state of the telephony interface for when we get interrupted by an incoming call, incoming SMS or other event that requires us to switch which part of the telephony software is running (we are really breaking the software down into a bunch of smaller programs, so that each can fit in 64KB using the CC65 compiler (which really means about 40KB of code, which with CC65 is not necessarily that much). On returning from a helper program, we should be able to restore the state of what we were doing, e.g., showing any active call status, ringer status etc, restoring a message from draft status (we should store those in a reserved sector of the contact thread if it's a message to a contact, but that won't work if the recipient isn't a contact). There are probably other things, too.

To manage this, except for draft messages to contacts which we can put in a reserved sector in the thread D81 for each contact, we should probably just have a STATE.D81 where we progressively allocate sectors for each bit of state. Those sectors should be chained into the directory listing with appropriately named files, so that they can be easily modified by 3rd party programs -- so long as their track and sectors don't change on the disk, as we want to bypass having to use the CBDOS file system code (partly so that we can reclaim the 128KB of chip RAM where the "ROM" usually sits).  We'll mark the files as locked, so that they are nominally read-only, and include a warning in the directory entry that files must retain their track and sector numbers, and should only be read by 3rd party software if using the file system, instead of direct sector access.

When messages are received from the cellular modem (e.g., incoming call, incoming SMS, call terminated etc), then these should be written verbatim into a message buffer in STATE.D81 (to keep common cellular modem monitoring code simple), state should be stashed and then processed by the appropriate helper program.  Now, there is an issue here where a denial of service attack could be mounted against a user, by having people continually ring and/or text them, so that the thing is endlessly occupied saving and restoring state and switching programs.  To mitigate this, we should probably process RING messages from within each program, and, ideally, messages from the same contact that is currently being displayed, too. But everything else can be batched up and processed periodically. 

Or, we could have the program that does all this processing persistently held in RAM (or stored in the flash filesystem, which is also super-fast!), and then basically bank switched (or DMA swapped, or equivalent) into place, so that the latency and duration of activity is greatly reduced.  There are limits to this, though, because whenever we have to switch mounted disk images, there will be a considerable delay, as the hypervisor has to do FAT file system searches.  So we really want to avoid doing that too often.  It might be that we limit ourselves to doing a process of SMS messages addressed to other contacts/numbers to once every 10 seconds or so, or whenever the unprocessed message queue reaches 50% full, to give us time to drain it, without losing messages.

We'll reserve a healthy slab of STATE.D81 for unprocessed messages, perhaps 128KB, which would be enough for 256 unprocessed messages. 

And I think that covers the whole back-of-house stuff that we need for tracking SMS and phone calls. 

So let's try to turn this into a set of requirements (which expand on the requirements we already have documented in the repository, and are subordinate to them): 

Requirement: Linux program that can create the various core D81 files (e.g., STATE.D81).

Requirement: Linux program that sets up the directory structure on the SD card for the MEGAphone (PHONE/*).

Requirement: Code in dialer to save/restore state in STATE.D81

Requirement: Code in each UI program to save/restore state in STATE.D81 and any relevant contact thread.

Requirement: Code in each UI program to trigger a switch to the message queue processor program after a given timeout and/or the unprocessed queue has reached the 50% full level.

Requirement: Function to create a contact (including creating the D81 file(s) required).

Requirement: Function to modify a contact.

Requirement: Copy contact disk to index disk.

Requirement: Sort index disk by specified key.

Requirement: Create or open thread disk image.

Requirement: Insert message into thread disk image.

Requirement: Retrieve message from thread.

Requirement: Update unread message count for contact.

Requirement: Update unread message status for phone.

Requirement: Log non-contact calls/SMS as pseudo-contact "unknown" 

Requirement: Display an SMS / call log message (either in-bound or outbound, using appropriate formatting to differentiate)

Requirement: Display a screen full of SMS / call log messages.

Requirement: Text entry for composing an SMS.

Requirement: Dialer entry for telephone number.

Requirement: Contact list. Select to open message list.

Requirement: Exit message list back to contact list.

Desirable: Allow inserting unicode/emoji in contact fields. 

Desirable: Delete old messages from thread.

Desirable: Unicode/Emoji entry for SMS composition. 

Desirable: Contact list search function by typing a fragment of a name or number.

Desirable: Message thread search function by typing a fragment of a message.

Desirable: Highlight sections of messages that match search query.

Desirable: Update search index based on a single contact or message.

Desirable: Rebuild search index for all contacts/messages.

Desirable: Associate avatar/emoji with contacts.

Desirable: Display contact avatar/emoji on incoming call.

Desirable: Dispaly contact avatar/emoji during call.

Desirable: Allow SMS composition during calls.

Desirable: Allow pasting location into SMS message (including during call).

Desirable: Allow showing last known direction and distance to contact based on last shared location (including during a call).

That's all probably enough now, that I can start digging in to implement it all. 

The above requirements specification also addresses the following milestones in the NLnet funded project:

Milestone 1.4 Telephony software User-Interface: Requirements Specification

Milestone 1.6 Text Messaging software: User-Interface: Requirements Specification

Saturday, 14 June 2025

Accessing Shared Resources from the MEGA65 System Partition

In my previous post, I implemented a facility for accessing the brand new Shared Resources area of the MEGA65's System Partition.

This facility has been created to provide rapid access to content anywhere in large files --- initially unicode pre-rendered fonts for the MEGAphone.  Using a traditional file-system would require tracking file-system state, and traversing file-system structures.   This would add latency for time-critical operations, like retrieving individual character glyphs when rendering SMS messages.

This area works by implementing a simple extent-based file list in the system partition.  The shres and shresls tools exist for linux in the mega65-tools repository to populate and query the contents of this area on a MEGA65-formatted SD card.   

Only the latest development version of MEGA65 FDISK will format an SD card to include a shared resource area.

test_897.c in mega65-tools has unit tests for this functionality, including reporting if your HYPPO version is too old, or there doesn't seem to be a shared resources area defined in your system partition.

So now we need to tie all this together is some code that could eventually go in mega65-libc that provides a simple API for accessing it. I'm thinking something like:

char shopen(char *resource_name,unsigned long long required_flags, struct shared_resource *file_handle) -- open  

shread(unsigned char *ptr, unsigned int count, struct shared_resource *file_handle) -- Read a slab of bytes from the opened resource from the current offset.

shseek(struct shared_resource *,unsigned long long offset, unsigned char whence) -- Seek to arbitrary point in the file.

unsigned int shdopen() -- open directory handle to shared resource area

sddread(unsigned long long required_flags, unsigned int *directory_handle, struct shared_resource *dirent) -- read next directory entry that matches.  The resource is implicitly an opened resource (since "opening" is absurdly light weight, the way I'm doing it).

The shared_resource struct will simply contain the filename, flags, starting sector in the shared resource area, its length, and current seek position (initially 0):

#define MAX_RES_NAME_LEN 256
#define SEEK_SET 1
#define SEEK_CUR 2
#define SEEK_END 3 
struct shared_resource {
   char name[MAX_RES_NAME_LEN];
   unsigned long long flags;
   unsigned long long length;
   unsigned long long first_sector;
   unsigned long long position;
};
 

I'm tracking this via https://github.com/MEGA65/mega65-libc/issues/70.

To test this API, I'm just going to start writing a wrapper program in the megaphone-modular repository in src/telephony. 

Okay, so in the process of setting that up, I'm discovering that CC65 mainline doesn't have support for the 45GS02 instruction set, so I can't use LDQ and STQ directly.  I guess I'll just implement those as the underlying instruction sequences.

Next challenge is that the SD card keeps getting stuck busy.  One of the subtleties involved in this is that the SD card interface _does_ lockup if you ask it to read a sector while it's already reading a sector. We should probably fix that, but for now I have to live with it.

To deal with this, I've made the shared resource API functions check the SD card status, and fail with an error if it's busy. So it should never be calling a SHRES Hypervisor trap when the SD card is busy (remember that the SHRES Hypervisor trap does the sector read request, so this is vital).

char do_shres_trap(unsigned long arg)
{
  // Fail if SD card is busy
  if (PEEK(0xD680)&0x03) {
    printf("SD card is busy\n");
    return 1;
  }

  printf("SD card was idle\n");
  
  shres_regs[0] = (arg>>0)&0xff;
  shres_regs[1] = (arg>>8)&0xff;
  shres_regs[2] = (arg>>16)&0xff;
  shres_regs[3] = (arg>>24)&0xff;
  shres_trap();

  printf("Trap C=%d\n",shres_regs[4]&1);
  
  // Check trap response in P
  return (shres_regs[4]&0x01) ^0x01;

}

Basically the wrapper is the only way I call the trap, and it should reliably fail if the SD card interface is already busy.

So what's going wrong here?

Okay, so it looks like it's trying to read an invalid sector from the SD card for some reason now.  The call to shdopen() always asks for sector 0 of the shared resource area:

 

shared_resource_dir shdopen()
{
  char i;
  if (do_shres_trap(0)) return 0xffff;
 

So provided that shres_regs is getting set properly in the wrapper it calls, it should be working.

I _think_ there might be something funny where the HYPPO trap is not adding the SHRES start offset to the $D681-4 SD card registers properly.  

readsharedresourcetrap:
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    ldq syspart_resources_area_start
    adcq $d681
    stq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
        ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
        jmp return_from_trap_with_success

Ah!  There is indeed a logical error here. It should look like this:

readsharedresourcetrap:
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    adcq syspart_resources_area_start
    stq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
        ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
        jmp return_from_trap_with_success

 

Yay! With that fix, I can now iterate over the (slightly random) set of files I put in the shared resource area of my SD card:

So perhaps I'll next make it try to cat the README.md file to the screen, just as a proof-of-concept --- and make it use the API, forcing me to implement all of the API.

Here's the resulting program:

struct shared_resource dirent;
unsigned long required_flags = SHRES_FLAG_FONT | SHRES_FLAG_16x16 | SHRES_FLAG_UNICODE;

unsigned char buffer[128];

unsigned char i;

char filename_ascii[]={'r','e','a','d','m','e','.',0x6d,0x64,0};

void main(void)
{
  shared_resource_dir d;

  mega65_io_enable();

  // Make sure SD card is idle
  if (PEEK(0xD680)&0x03) {
    POKE(0xD680,0x00);
    POKE(0xD680,0x01);
    while(PEEK(0xD680)&0x3) continue;
    usleep(500000L);
  }

  if (shopen(filename_ascii,0,&dirent)) {
    printf("ERROR: Failed to open README.md\n");
    return;
  }

  {
    unsigned int b = 0;
    printf("Found text file\n");
    while(b = shread(buffer,128,&dirent)) {
      for(i=0;i<b;i++) {
    if (buffer[i]==0x0a) putchar(0x0d); else putchar(buffer[i]);
      }
      
    }
  }

  return;
}

And it works! And I can even use shseek() to print just a bit of the text file.

  {
    unsigned int b = 0;
    printf("Found README.md\n");
    shseek(&dirent,-200,SEEK_END);

    while(b = shread(buffer,128,&dirent)) {
      for(i=0;i<b;i++) {
    if (buffer[i]==0x0a) putchar(0x0d); else putchar(buffer[i]);
      }
      
    }
  }

 

Running this we can get something like this:


 In short, it works. So now it's time to make a pull-request into mega65-libc for this new API, and then we're done, and I can go back to working on making the actual unicode pre-rendered fonts for the MEGAphone telephony software! 

 

Friday, 13 June 2025

Embedding Shared Resources in the MEGA65 System Partition

In another blog post I'm writing at the moment, I'm exploring how I will do the UI for the new MEGAphone work.  Part of that involves me figuring out how I can possibly support the full Unicode gammut -- including emojis, that are not more or less a required part of SMS communications.

In that post, I considered putting them into files on the FAT32 file system, or stashing them in a set of specially formatted D81 or D65 disk images.  But all of those approaches require more than 1 SD card sector read to fetch a random glyph, and in some cases many such requests.

So I came up with the idea of putting linear slabs of shared resources into the system partition, and having a Hypervisor call that lets a user program ask for an arbitrary 512 byte SD card sector of that data.  For fonts in the MEGAphone software, this would allow the program to work out and store the sector offset within the shared resources for the font(s) it needs to use, and then just adding an offset to that to retrieve the SD card sector with the required font glyph to load into chip RAM.

In this blog post I'm going to design and implement this facility in the hypervisor, and make a test program that will let me check that it's all working.  From there, I'll work on some kind of utility that will let me modify the contents of the system partition to install the fonts.  That might end up being a Linux-based utility that works directly on the SD card for speed --- since a full Unicode font will be about 300MB in the pre-rendered format that I'll be using (more about that in the other blog post, when it's done).

I'm going to track the addition of this functionality into HYPPO via Issue #897.

So let's start by making the system partition code in the Hypervisor read and store the starting sector and size of this slab.

Now I have to remember how I made it possible to update the hypervisor using m65 -k.  From memory this can be done safely from the MEGA65's READY prompt, as that means that it's in user-space, and thus it's safe to overwrite the hypervisor.

... except that that doesn't seem to do it.

Okay, so because the Hypervisor RAM is only writeable when the CPU is in Hypervisor mode, we can't do it from the READY prompt any more. But we also need it to happen in some situation where the Hypervisor isn't really running any code, since otherwise the CPU will go off into la-la land, and bork things.

I am using an old version of m65, though.  Maybe in newer versions -k looks after this, e.g., by setting the CPU to single-step mode, then causing the CPU to start a Hypervisor trap, and then stopping the CPU once in Hypervisor mode, writing the new version of the Hypervisor code, then resetting and resuming the CPU, so that it boots the new HYPPO fresh.  If not, that would be the way to do it.

I'm tracking this via Issue #222, and pull request #223.

Now with m65 recompiled with that fix, I can quickly and easily test HYPPO builds. That's already made it _way_ more comfortable and convenient to find and fix a bug in my initial implementation of the of the shared resource thing.

So now I can implement the HYYPO trap that tries to read the sector.  Because we are going to use A,X,Y and Z as the Q pseudo-register to make it easy for a user program to access, we will have to use up a whole trap address, but that's okay.

So far, the trap code looks like this:
 

readsharedresourcetrap:
    stq $d681
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    ldq syspart_resources_area_start
    adcq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
        ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
    clc
    rts
bad_syspart_resource_sector_request:    
    sec
    rts
 

In principle, that should be all we need. 

The next step is to make a unit test for this, checking that it does cause a sector to get read, and that requests for out-of-range sectors fails.

I'll use the unit test framework that we have in mega65-tools.

Those tests use CC65, which I can't remember it's call convention for 32-bit args.  So I'll just stash the 32-bit resource sector number into some convenient memory location that it doesn't use.  Maybe $7F0-$7F3, i.e., the end of the screen RAM.

The test will then make the HYPPO call, and store the register values from HYPPO back where it read them from. Like this:

 

        .export _test_resource_read

        .p4510

.segment    "CODE"

.proc    _test_resource_read: near
    lda $7f0
    ldx $7f1
    ldy $7f2
    ldz $7f3
    sta $d644         ; Trigger Hypervisor trap
    nop            ; CPU delay slot required after any hypervisor trap
    sta $7f4
    stx $7f5
    sty $7f6
    stz $7f7
    php
    pla
    sta $7f8
    rts
.endproc

 

Pulling it all together, we can now easily run tests like this:

 

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
/home/paul/Projects/mega65/mega65-tools/cc65/bin/cl65 --config src/tests/tests_cl65.cfg -I src/mega65-libc/cc65/include -O -o src/tests/test_897.prg --mapfile src/tests/test_897.map --listing src/tests/test_897.list src/tests/test_897_asm.s src/tests/test_897.c src/mega65-libc/cc65/libmega65.a
2025-06-07T07:10:52.445Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-07T07:10:52.454Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T07:10:52.454Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T07:10:52.660Z NOTE loading file 'src/tests/test_897.prg'
2025-06-07T07:10:52.841Z NOTE running
2025-06-07T07:10:52.841Z NOTE Entering unit test mode. Waiting for test results.
2025-06-07T07:10:52.841Z NOTE System model: UNKNOWN MODEL $06
2025-06-07T07:10:52.841Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-07T07:10:52.841Z NOTE System ROM version: M65 V920395
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z  FAIL (Issue#0897, Test #000 - HYPPO RESOURCE READ TEST FAILED)
2025-06-07T07:10:52.845Z NOTE 2025-06-07T07:10:52.845Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:10:52.845Z NOTE terminating after completion of unit test

 

For those who haven't seen the unit test framework, it's quite nifty: The unit test running on the real hardware communicates test results back via the serial monitor interface to the m65 program, allowing it to directly report the test results.

Because I haven't implemented everything yet, the test claims to fail.

So now let's fix the implementation of our trap -- it should return from the Hypervisor by jumping to one of the return routines:

readsharedresourcetrap:
    stq $d681
    cpq syspart_resources_area_size
    bcs bad_syspart_resource_sector_request
    ldq syspart_resources_area_start
    adcq $d681
    ;; Ask SD card to read the sector.
    lda #$02
    sta $d680
    ;; Note that we don't wait for the request to finish -- that's
    ;; up to the end-user to do, so that we don't waste space here
    ;; in the hypervisor, where space is at an absolute premium.
    jmp return_from_trap_with_success
    
bad_syspart_resource_sector_request:
    ;; Return "illegal value" if trying to read beyond end of region
    lda #dos_errorcode_illegal_value
    jmp return_from_trap_with_failure

So now if we ask for an illegal sector number, we should get this illegal value error, which we know has the value 0x11 from the source.

So let's now update our test to check whether the trap is defined or not, and whether reading sector 0 results in success or an error:

  // Try reading sector 0 of resource area
  resource_sector = 0;
  *(unsigned long *)0x7f0 = resource_sector;
  
  if (test_resource_read() == 0) {
    unit_test_ok("");
  }
  else {
    unit_test_fail("hyppo call helper failed");
  }

  return_value = *(unsigned long *)0x7f4;

  carry = PEEK(0x7f8)&1;

  if (!carry) {
    switch(PEEK(0x7f4)) {
    case 0xff: unit_test_fail("hyppo returned 'trap not implemented'"); break;
    case 0x11: // dos_errorcode_illegal_value
      unit_test_fail("reading resource sector 0 failed -- no resource area in system partition?");
      break;
    default: {
      snprintf(message,80,"hyppo resource read failed with unknown error $%02x",PEEK(0x7f4));
      unit_test_fail(message);
    }
    }
    unit_test_report(ISSUE_NUM, 0, TEST_DONEALL);
    return;
  }

 

This routine now checks whether the Hypervisor trap is implemented or not, and then whether reading sector 0 works.  We need both of those to succeed, if we are going to able able to test that it works to actually read system resources.

With the fixed m65 -k option to easily load a HICKUP.M65 file containing an updated Hypervisor, I can now easily load the version that supports this new hypervisor call:

paul@bob:~/Projects/mega65/mega65-core$ make bin/HICKUP.M65 && m65 -k bin/HICKUP.M65 

And then execute the test, and see the result:

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
...
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z  FAIL (Issue#0897, Test #001 - READING RESOURCE SECTOR 0 FAILED -- NO RESOURCE AREA IN SYSTEM PARTITION?)
2025-06-07T07:31:16.102Z NOTE 2025-06-07T07:31:16.102Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T07:31:16.102Z NOTE terminating after completion of unit test
 

And look at that --- we can see it now passes the check for the hypervisor trap being implemented (Test #000), but fails reading the first sector of the shared resource area (Test #001). This is exactly what we expect, since the SD card in this MEGA65 doesn't (yet) contain a shared resource area.

In fact, I haven't even made the utility to create them.  So let's tackle that next.

Currently the MEGA65 FDISK utility is the thing that's responsible for creating the system partition.  So I'll probably just tweak that to by default create a reasonable size system resource partition, perhaps 1GB or so, but maybe also have the option to select a different size.

I'm tracking that issue here: https://github.com/MEGA65/mega65-fdisk/issues/28

The FDISK program is really quite old and crusty now, but it does it's job.  There is an outstanding issue to totally refactor it, so while I will try to not add to the mess, I'm not going to go overboard with making this new feature overly sophisticated.

Okay, done a first cut at that. It will try to allocate half the system partition for shared resources. It also increases the target SYSPART size from 2GiB to 4GiB, thus allowing 2GiB for shared resources by default.

Now to backup my SD card, and try reformatting it!

Okay, so after a bit of procrastination, it's time to just pull the SD card, copy it all off, and put it back in, and run MEGA65 FDISK on it.

 

So formatting with FDISK looked promising -- it was reserving 4GB for the system partition, which is a change I made.

But then when I tried to boot with it, it was all borked up.

Basically the FAT file system seems to be messed up.  But if I power on with the new HICKUP, it does see that we have $400000 sectors reserved for resources, which equates to 2GiB, so that's promising!


Okay, so why on earth is the FAT file system all messed up, then?

First thing to try: Downgrade to FDISK prior to the changes I've made, and see if it formats up fine with that.

Nope. Same thing. So probably not something that I've done.

Okay, so after a second go, including copying all my files back on, it's fine.

Who knows what went wrong.

The main thing is it is booting again.

Although, oddly it's running the intro disk, even though I don't have a default disk image to mount at boot set.

Hmmm... And now the new HICKUP is showing 0 sectors reserved for shared resources. Did I accidentally format it with the old version of FDISK? Yup --- looks like that was the case.

After I copied my files back into place, it was all fine. 

And, the test to read sector 0 of the shared resource area works now:

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
...

2025-06-07T13:30:30.590Z NOTE 2025-06-07T13:30:30.590Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.590Z NOTE 2025-06-07T13:30:30.590Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.592Z NOTE 2025-06-07T13:30:30.592Z  PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ TEST PASSED)
2025-06-07T13:30:30.592Z NOTE 2025-06-07T13:30:30.592Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:30:30.592Z NOTE terminating after completion of unit test

 

So now I can work on making the test more complete, by doing a binary search of the address space to determine the exact size of the shared resource partition. 

 Like this:


  resource_sector = 0xffffffffUL;
  for(i=31;i>=0;i--) {
    *(unsigned long *)0x7f0 = resource_sector;

    if (test_resource_read() != 0) unit_test_fail("hyppo trap failed");
    carry = PEEK(0x7f8)&1;

    if (!carry) {
      resource_sector -= (1UL <<i);
    }

  }

  if (resource_sector==0xffffffffUL || (!resource_sector)) {
    unit_test_fail("failed to determine size of shared resource area");
  } else {
    snprintf(message,80,"determined shared resource area is $%08lx sectors",
         resource_sector + 1);
    unit_test_fail(message);
  }
  
  printf("Shared resource area = $%08lx sectors.\n",resource_sector+1);
 

 And it works, correctly measuring the size!

paul@bob:~/Projects/mega65/mega65-tools$ make src/tests/test_897.prg && m65 -u -4 -r src/tests/test_897.prg 
/home/paul/Projects/mega65/mega65-tools/cc65/bin/cl65 --config src/tests/tests_cl65.cfg -I src/mega65-libc/cc65/include -O -o src/tests/test_897.prg --mapfile src/tests/test_897.map --listing src/tests/test_897.list src/tests/test_897_asm.s src/tests/test_897.c src/mega65-libc/cc65/libmega65.a
2025-06-07T13:41:32.282Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-07T13:41:32.296Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T13:41:32.296Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-07T13:41:32.502Z NOTE loading file 'src/tests/test_897.prg'
2025-06-07T13:41:32.691Z NOTE running
2025-06-07T13:41:32.691Z NOTE Entering unit test mode. Waiting for test results.
2025-06-07T13:41:32.691Z NOTE System model: UNKNOWN MODEL $06
2025-06-07T13:41:32.691Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-07T13:41:32.691Z NOTE System ROM version: M65 V920395
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.694Z NOTE 2025-06-07T13:41:32.694Z  PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ SECTOR 0 TEST PASSED)
2025-06-07T13:41:32.695Z NOTE 2025-06-07T13:41:32.695Z  PASS (Issue#0897, Test #002 - DETERMINED SHARED RESOURCE AREA IS $00400000 SECTORS)
2025-06-07T13:41:32.696Z NOTE 2025-06-07T13:41:32.696Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-07T13:41:32.696Z NOTE terminating after completion of unit test
  

 Okay, so now the last thing to test is that it actually causes the SD card to read the sector.

First, do the SD card registers get the correct sector number written into them? I'll test this by just making sure that they don't end up with all zeroes. Anyway, nope, they don't. Found and fixed -- the bug was in the hypervisor code (the interested reader can look at the code snippet for the hypervisor trap above and see if they can spot it --- leave a comment if you find it :)

So then the next step is to check that the SD sector buffer actually changes contents.  That worked fine, without further changes (although I had to make sure the SD card was not busy, before making the HYPPO call, because requesting a sector read while the SD card is busy causes it to lock up).

Next up, then, is to come up with a simple file system for the shared resource area. It needs to be super simple to parse.  I'm thinking it could be as simple as the first sector containing version information and the number of resources in the area, followed by one sector per file, that contains the name of the file/resource, and the start sector within the resource area, and the length of the resource in both sectors and bytes.  After the final resource directory sector, we have a completely empty sector, so that a program scanning it can just look for that empty sector and know to stop, without even having to track the resource count, in case that's easier to implement in a given situation.

So let's make a utility that can build the shared resource area from a set of files and their names.  Since it's just boring boiler-plate kind of code, I used ChatGPT to do it faster. Not sure if it actually was any faster to write. But we can now do things like this:

paul@bob:~/Projects/mega65/mega65-tools$ rm bin/shresls ; make bin/shres bin/shresls && bin/shres foop mega65-screen-000000.png,31  README.md,23 && bin/shresls foop
make: 'bin/shres' is up to date.
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md
MEGA65 Shared Resource File: foop
Declared resources: 2

Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md

SHA1                                     Bytes      Check      Name
--------------------------------------------------------------------------------
40ce6cfcaaf14d11a8c3ffa71fd4726087613eee  4242       MATCH      mega65-screen-000000.png
3b31809d69f814d2173484855072333db65190cb  6436       MATCH      README.md
paul@bob:~/Projects/mega65/mega65-tools$ 


Next step is to make a tool to extract or insert a shared resource area onto/off of an actual SD card.  My plan here is to make the existing tool I've made above to work out if it's writing to an SD card, in which case, it will find the SYSPART in there, and write the resources directly into there.

Okay, so I think I have writing the resources to SD card working, as well as updating shresls so that it can read and check them on a real disk image, too:

paul@bob:~/Projects/mega65/mega65-tools$ rm bin/shresls ; make bin/shres bin/shresls && sudo bin/shres /dev/sdb mega65-screen-000000.png,31  README.md,23
make: 'bin/shres' is up to date.
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
INFO: Attempting to open shared resource file or disk image '/dev/sdb'
INFO: Found MEGA65 SYSPART at sector 0x014d7ffe
INFO: Found MEGA65 SYSPART shared resource area of 2048 MiB.
DEBUG: area_start=0x00029b0ffc00, area_length=0x80000000
Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md
 

paul@bob:~/Projects/mega65/mega65-tools$ make bin/shresls && sudo bin/shresls /dev/sdb
gcc -Wall -g -std=gnu99 `pkg-config --cflags-only-I --libs-only-L libusb-1.0 libpng` -mno-sse3 -march=x86-64 -o bin/shresls src/tools/shresls.c -lssl -lcrypto
INFO: Attempting to open shared resource file or disk image '/dev/sdb'
INFO: Found MEGA65 SYSPART at sector 0x014d7ffe
INFO: Found MEGA65 SYSPART shared resource area of 2048 MiB at sector 2048 of SYSPART.
MEGA65 Shared Resource File: /dev/sdb
Declared resources: 2

Resource Table (2 entries):
Idx  Start   Sectors  Bytes      Flags      Name
---- ------- -------- ---------- ---------- -------------------------
0    4       9        4242       0x80000000 mega65-screen-000000.png
1    13      13       6436       0x00800000 README.md

SHA1                                     Bytes      Check      Name
--------------------------------------------------------------------------------
40ce6cfcaaf14d11a8c3ffa71fd4726087613eee  4242       MATCH      mega65-screen-000000.png
3b31809d69f814d2173484855072333db65190cb  6436       MATCH      README.md

So now I just need to make some tests for scanning the shared resource list, and trying to read them.  Actually, all we probably need is to check for the magic string in the shared resource area. This is because for that to be found, it means that the SD card access is working, as well as the sector address calculation.

And it works!

mega65-tools$ make src/tests/test_897.prg  && m65 -4 -r -u src/tests/test_897.prg 
make: 'src/tests/test_897.prg' is up to date.
2025-06-13T07:15:45.034Z NOTE MEGA65 Cross-Development Tool 20250531.20-222fix-ee98359
2025-06-13T07:15:45.048Z NOTE #0: /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-13T07:15:45.048Z NOTE selecting device /dev/ttyUSB0 (Arrow; 0403:6010; undefined; 03636093)
2025-06-13T07:15:45.254Z NOTE loading file 'src/tests/test_897.prg'
2025-06-13T07:15:45.447Z NOTE running
2025-06-13T07:15:45.447Z NOTE Entering unit test mode. Waiting for test results.
2025-06-13T07:15:45.447Z NOTE System model: UNKNOWN MODEL $06
2025-06-13T07:15:45.447Z NOTE System CORE version: 726-f...-disk,20240428.10,3439a13
2025-06-13T07:15:45.447Z NOTE System ROM version: M65 V920395
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z START (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z  PASS (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.451Z NOTE 2025-06-13T07:15:45.451Z  PASS (Issue#0897, Test #001 - HYPPO RESOURCE READ SECTOR 0 TEST PASSED)
2025-06-13T07:15:45.452Z NOTE 2025-06-13T07:15:45.452Z  PASS (Issue#0897, Test #002 - DETERMINED SHARED RESOURCE AREA IS $00400000 SECTORS)
2025-06-13T07:15:45.455Z NOTE 2025-06-13T07:15:45.455Z  PASS (Issue#0897, Test #003 - SD CARD BUSY CLEARED ON RESET)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #004 - SD CARD REGISTERS WRITTEN TO)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #005 - SD CARD BUSY CLEARED)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #006 - SD CARD BUFFER CONTENTS CHANGE)
2025-06-13T07:15:45.456Z NOTE 2025-06-13T07:15:45.456Z  PASS (Issue#0897, Test #007 - READ MAGIC STRING FROM SHARED RESOURCE AREA)
2025-06-13T07:15:45.457Z NOTE 2025-06-13T07:15:45.457Z  DONE (Issue#0897, Test #000 - HYPPO SYSTEM RESOURCE ACCESS)
2025-06-13T07:15:45.457Z NOTE terminating after completion of unit test

 

Okay, so now I have all the ingredients here for this to work -- HYPPO support for the shared resource area, patched MEGA65 FDISK to create the section in the system partition, linux-based tools to populate and interrogate the shared rescource area quickly and easily, and unit tests that provide proof-of-concept for accessing this area, and checking that the loaded HYPPO has support for it.

So I'm going to leave it at that for now, and I'll document writing a library to open and read from shared resources in a separate blog post. 

I'll also pursue getting the HYPPO changes rolled into mega65-core development branch, and updated FDISK into mega65-fdisk development branch. 

 

 

 


 

Sunday, 23 March 2025

MEGAphone Bluetooth module

The next module on the list is something that can do BlueTooth to talk to a headset. For security reasons, I'd prefer something that is BlueTooth only, if I can find it. The problem is that the ESP32 is probably otherwise the obvious choice, but I'm not totally sure that it can be trusted. But then, there is lots of alternative firmware available for it, and at the end of the day, if you are using wireless to talk to a headset, then you have already compromised your own security.

But let's look at the options...

1. Requirements Analysis

The requirements relating to this module are:

Requirement 2.1.1, Bluetooth host module that can connect to headsets and other audio devices. Must support UART or I2C control, and audio interfaces that we can easily implement from the FPGA, such as I2S audio.

Desirable 2.1.2, Bluetooth host module support for keyboards and other HID peripherals. 

Note that 2.1.2 was previously listed as a requirement, but it is not required in the milestones, and so I have allowed it to be relaxed to a desirable. I'll document this in the repository.

2. Key Component Selection

 I do have a BM63 eval kit laying around, which I had originally intended for this use-case, but I believe it is only able to act as a sink (headset/speaker), not a source (phone).  But chatgpt is trying to tell me that it can do both -- and I fear it is hallucinating this, as much as I'd like to believe that it could do it.

So the IS2083 IC can be both source or sink. But I think that's in the BM83, not the BM63. 

Confirmed -- I need the BM83, not BM63.

This all feels vaguely familiar, like I explored this in the past. I'm staring at the BM63 eval kit I have here, and wondering if I didn't get a BM83 kit as well at some point. It would be nice if I did, as they are A$500+ each. 

Ah, it _is_ a BM83 kit -- its just my aging eyes in the dim light here in the middle of the night! Hurrah!

So I can immediately start setting that up, and trying to make it pair with a BT headset, and see how it goes.

I'm starting to read through the documentation, so that I can do some basic testing of this module to make sure it works -- I want it to pair with a headset and pretend to start sending audio to it, and also be able to pretend to start a voice call. I should be able to hear the difference in noise in the ear piece of the old Bose QC35s I have here to tell the difference.

The BM83 supports an "AT command set". Now, this is _not_ a modem-style Hayes-AT command set, but rather an Audio Transceiver Command Set, which is binary in format.

But looking at the commands, it looks like everything I should need is there.  Time for sleep now, but I'll start setting up the board in the morning.

Okay, so now I'm reading my way through the BM83 Evaluation Board User's Guide, with the intention of powering it up and talking to it via UART. That board has a PIC32 microcontroller on it, which I don't want to use, so I may need to figure out which of the many switches need to be set correctly to run the BM83 in stand-alone UART controlled mode.

Following the instructions in section 4, I can put it into pairing mode and make it pair with my phone -- thus confirming that the module is superficially working, which is great.  Except that after pairing, the phone refuses to actually connect with it. But it's a cheap android phone, so it could be that. Also it shows up on my phone under "Other devices" which might mean that it doesn't recognise the set of profiles its offering. After some more pressing the MFB button in semi-random ways, I have gotten it to pair, and it is showing support for HD-Audio AAC, Telephone calls and audio media, which is great. The buttons to control the volume etc, however, are still not causing anything to happen.  But I don't care too much about that right now, as I intend for it to be used primarily in the reverse direction: as the phone, rather than the headset.

Switching SW300 on lets me seemingly switch the BM83 eval board into "test mode" where I can talk directly to it via the USB UART.

Then I can use the isupdate.exe tool from Microchip to talk to it.

I had a frustrating half hour where I could make it connect, and it would even identify it as BT5511_002 in a little box, but then fail to connect.  After some googling around, I found this post, with this sage piece of advice:

Put the TTY into raw mode:

stty -F /dev/ttyACM0 raw
 

With that, I was able to flash the main firmware, the DSP firmware and the application firmware (you have to do them all -- there is a way to "rehex" them to make a single file for convenience, but I didn't do that) with SW300 set to on, and then turn it off again after flashing everything.

This also confirms that SW300 should be off for normal operation.  I did find that if I press RESET_N and then long-press (~1 sec) the MFB button, that the BM83 does output something that looks like the kind of AA + checksum packet format we expect:

AA 00 03 30 00 00 CD
AA 00 03 2D 05 01 CA

So let's try to understand how those should be decoded.

Except I can't even get it to do that reliably now.  I suspect the PIC32 MCU on the eval board might still be doing things and getting in the way.

Let's try to get to the bottom of this.

We want "host mode", where the BM83 expects an MCU to be connected via UART. That's good, because we have the host firmware loaded.

But the eval board will be expecting to have the PIC32 act as the MCU in this case. We need to disable that. I'm finding the documenation for this evaluation board quite frustrating, to be honest. For example, there is a big fat JP303 connector with 40 signals brought out onto it -- but it isn't even mentioned in the user's guide. And it's not like its a different PCB revision that lacks it -- you can see the big 2x20 0.1" header at the top of the PCB in this screenshot, and how there is carefully nothing pointing at it. JP303, which is silkscreened next to it isn't event mentioned once in the manual.

Fortunately despite this seemingly intentional ignorance of it, it has various signals identified on it in silkscreen. Those of interest to us are:

HCI_RXD

HCI_TXD

Indeed HCI_TXD follows the UART TX line from the USB UART adaptor. So my bits are getting that far at least.  And HCI_RXD really seems to be connected to the BM83, because I see those responses that I've been seeing when pressing the MFB ("Multi-Function Button", by the way).  The MFB pin is also routed out onto this JP303.

A bit more digging indicates that S400 selects bewteen onboard PIC32 and an external microcontroller. It was already set to external, which is what I want. So most probably the PIC32 isn't actually being a problem.

So why doesn't the firmware talk to me in any more sensible way?

The first assumption is that the "application" firmware that I loaded on isn't the right one, or doesn't support the UART packetised commands.

So I have perhaps found a clue on this page. It talks about running the config utility to enable the BM83's UART. Maybe RX on the UART has to be enabled? But I find it a bit odd that the UART can send, but not receive.

The datasheet does seem to indicate that this GUI config tool is required to configure host mode. The problem is that that is windows-only software, and I haven't figured how to get it running under Wine yet. Or rather, I can run it, but it so far makes no sense at all. It doesn't seem to communicate with the module but rather allows loading and saving of configurations, without a great deal of clarity as to how to actually drive it.

To add to the fun, there is corrupted copy of the documentation for this config tool in the SDK:

./Tools/Config Tool
./Tools/Config Tool/~$208x_Config_GUI_Tool Release Note.docx
./Tools/Config Tool/is208x_config_gui_tool v1.3.23.exe
./Tools/Config Tool/~$BT5511_Config_Parameter_Table_R0.xlsx
./Tools/Config Tool/is208x_config_default_table.ini
./Tools/Config Tool/IS208x_Config_Parameter_Table_R0.ihlp 

So I know there is some sort of documentation for it, but it's not included.

There is some instructions on use in:

https://ww1.microchip.com/downloads/en/DeviceDoc/BM83-Bluetooth-Audio-Development-Board-User-Guide-DS50002902B.pdf

So let's see what it explains.

Ok, so the tool is quite a bit newer than in the documentation. And the order of the options has changed.

But broadly speaking, it can be followed, and when you save it, it writes a .HEX file that can then be flashed onto the device using the isupdate.exe program.

After a lot of poking around, it looks like the MSPKv2_1.03.0406_SPP firmware is missing many of the UART commands that we'd need. But potentially they are in the v1.3 era. But finding that old firmware is proving non-trivial.  It's not helping that the Microchip website is under maintenance for a few hours right now.

Actually, it looks like the firmware I downloaded fresh is not the latest according to this post. There should be a 2v1.3.5 instad of the 2v1.2.4 that I have. Now to try to find a link where I can download it...

I've since posted a question on the Microchip forum and emailed my local FAE for their distributor to see if I can get some help with this, as I'm just spinning my wheels, when I have much more that I need to get on with. 

I'm going to work on the assumption that there will be some solution to this problem at some point, and since the module does pair with things, that it fundamentally works.  In short, I can get on with designing a module to carry the BM83.

3. Schematic Layout

So for this one, we really just need to host the BM83, and work out which of its 50 pins we actually care about. So let's take a look:

Okay, so that's our pin out. Let's now work out which ones we definitely need, which we definitely don't need, and if there are any we aren't sure about.
 
BM83 Module Pin Description
Pin Number Pin Name Required? Description
1DR1YES (I)I2S interface: digital left/right data
2RFS1YES (I/O)I2S interface: digital left/right clock
3SCLK1YES (I/O)I2S interface: bit clock
4DT1YES (O)I2S interface: digital left/right data
5MCLK1YES (O)I2S interface: master clock
6AOHPRNOR-channel analog headphone output
7AOHPMNOHeadphone common mode output/sense input
8AOHPLNOL-channel analog headphone output
9MICN2NOMIC 2 mono differential analog negative input
10MICP2NOMIC 2 mono differential analog positive input
11AIRNOR-channel single-ended analog input
12AILNOL-channel single-ended analog input
13MICN1NOMIC 1 mono differential analog negative input
14MICP1NOMIC 1 mono differential analog positive input
15MICBIASNOElectric microphone biasing voltage
16GNDYES (P)Ground reference
17DMIC_CLKNODigital MIC clock output
18DMIC1_RNODigital MIC right input
19DMIC1_LNODigital MIC left input
20P3_2MAYBE (I/O)GPIO (default: AUX_IN DETECT)
21P2_6MAYBE (I/O)General purpose I/O port
22ADAP_INNO (P)5V adapter input / USB DFU / battery charge
23BAT_INYES (P)Power supply input (3.2V–4.2V)
24SYS_PWRNOSystem power output (internal use only)
25VDD_IOYES (P)I/O power supply (internal use)
26PWR (MFB)YES (I)Multi-function push button
27SK1_AMB_DETMAYBE (I)Temperature sense channel 1
28SK2_KEY_ADMAYBE (I)Temperature sense channel 2
29P8_6 / UART_RXDYES (I/O)GPIO / UART RX data
30P8_5 / UART_TXDYES (I/O)GPIO / UART TX data
31P3_4 / UART_RTSYES (I/O)GPIO / UART RTS / Mode select
32LED1MAYBE (I)LED driver 1
33P0_2MAYBE (I/O)GPIO (default: play/pause)
34LED2MAYBE (I)LED driver 2
35P0_6MAYBE (I/O)GPIO
36DMNO (I/O)USB D-
37DPNO (I/O)USB D+
38P0_3MAYBE (I/O)GPIO (default: reverse)
39P2_7MAYBE (I/O)GPIO (default: volume up)
40P0_5MAYBE (I/O)GPIO (default: volume down)
41P1_6 / PWM1MAYBE (I/O)GPIO / PWM1 output
42P2_3MAYBE (I/O)GPIO
43RST_NYES (I)System reset (active-low)
44P0_1MAYBE (I/O)GPIO (default: forward)
45P0_7MAYBE (I/O)GPIO
46P1_2 / TDI_CPU / SCLMAYBE (I/O)GPIO / Debug / I2C SCL
47P1_3 / TCK_CPU / SDAMAYBE (I/O)GPIO / Debug / I2C SDA
48P3_7 / UART_CTSMAYBE (I/O)GPIO / UART CTS
49P0_0 / UART_TX_INDYES (I/O)GPIO / Codec reset (Embedded) / TX_IND (Host)
50GNDYES (P)Ground reference

So that's 15 yeses and 18 maybes. That would make 33 total, if we include everything that we could feasibly want (given we are only using this to send and receive I2S audio. I am a bit concerned about whether the BM83 can convey bidirectional audio from a call via it's digital I2S interface, but we can live without that for this initial version, as the microphone array on the MEGAphone unit itself will be fine to use for many use-cases. For true hands-free operation, e.g., a headset in a bike helmet, like I use from time to time, we'd need it though. But as said, we will defer that for now.
 
Back to our 33 pins, that means that we need at least a 2x17 module, which would be 1.7 inches long.  The BM33 is 22mm from the bottom to where the PCB antenna starts, i.e., ~0.9 inches. So if we want all those extra pins, we will end up using more board realestate than we'd like.  So let's prune out the bluetooth fuction buttons, e.g., reverse, volume up, down, forward, play (5 pins), the temperature sensor (2 pins), the stray GPIOs (5 pins), and that leaves us with 21 pins, which would fit just about perfectly -- and still leave us with the most likely pins we'd need. The updated table would look like this: 
 
BM83 Module Pin Description
Pin Number Pin Name Required? Description
1DR1YES (I)I2S interface: digital left/right data
2RFS1YES (I/O)I2S interface: digital left/right clock
3SCLK1YES (I/O)I2S interface: bit clock
4DT1YES (O)I2S interface: digital left/right data
5MCLK1YES (O)I2S interface: master clock
6AOHPRNOR-channel analog headphone output
7AOHPMNOHeadphone common mode output/sense input
8AOHPLNOL-channel analog headphone output
9MICN2NOMIC 2 mono differential analog negative input
10MICP2NOMIC 2 mono differential analog positive input
11AIRNOR-channel single-ended analog input
12AILNOL-channel single-ended analog input
13MICN1NOMIC 1 mono differential analog negative input
14MICP1NOMIC 1 mono differential analog positive input
15MICBIASNOElectric microphone biasing voltage
16GNDYES (P)Ground reference
17DMIC_CLKNODigital MIC clock output
18DMIC1_RNODigital MIC right input
19DMIC1_LNODigital MIC left input
20P3_2NOGPIO (default: AUX_IN DETECT)
21P2_6NOGeneral purpose I/O port
22ADAP_INNO (P)5V adapter input / USB DFU / battery charge
23BAT_INYES (P)Power supply input (3.2V–4.2V)
24SYS_PWRNOSystem power output (internal use only)
25VDD_IOYES (P)I/O power supply (internal use)
26PWR (MFB)YES (I)Multi-function push button
27SK1_AMB_DETNOTemperature sense channel 1
28SK2_KEY_ADNOTemperature sense channel 2
29P8_6 / UART_RXDYES (I/O)GPIO / UART RX data
30P8_5 / UART_TXDYES (I/O)GPIO / UART TX data
31P3_4 / UART_RTSYES (I/O)GPIO / UART RTS / Mode select
32LED1MAYBE (I)LED driver 1
33P0_2NOGPIO (default: play/pause)
34LED2MAYBE (I)LED driver 2
35P0_6NOGPIO
36DMNO (I/O)USB D-
37DPNO (I/O)USB D+
38P0_3NOGPIO (default: reverse)
39P2_7NOGPIO (default: volume up)
40P0_5NOGPIO (default: volume down)
41P1_6 / PWM1NOGPIO / PWM1 output
42P2_3NOGPIO
43RST_NYES (I)System reset (active-low)
44P0_1NOGPIO (default: forward)
45P0_7NOGPIO
46P1_2 / TDI_CPU / SCLMAYBE (I/O)GPIO / Debug / I2C SCL
47P1_3 / TCK_CPU / SDAMAYBE (I/O)GPIO / Debug / I2C SDA
48P3_7 / UART_CTSMAYBE (I/O)GPIO / UART CTS
49P0_0 / UART_TX_INDYES (I/O)GPIO / Codec reset (Embedded) / TX_IND (Host)
50GNDYES (P)Ground reference

So let's filter out all the NOs and see how it looks:
BM83 Module Pin Description
Pin Number Pin Name Required? Description
1DR1YES (I)I2S interface: digital left/right data
2RFS1YES (I/O)I2S interface: digital left/right clock
3SCLK1YES (I/O)I2S interface: bit clock
4DT1YES (O)I2S interface: digital left/right data
5MCLK1YES (O)I2S interface: master clock
16GNDYES (P)Ground reference
23BAT_INYES (P)Power supply input (3.2V–4.2V)
25VDD_IOYES (P)I/O power supply (internal use)
26PWR (MFB)YES (I)Multi-function push button
27SK1_AMB_DETMAYBE (I)Temperature sense channel 1
28SK2_KEY_ADMAYBE (I)Temperature sense channel 2
29P8_6 / UART_RXDYES (I/O)GPIO / UART RX data
30P8_5 / UART_TXDYES (I/O)GPIO / UART TX data
31P3_4 / UART_RTSYES (I/O)GPIO / UART RTS / Mode select
32LED1MAYBE (I)LED driver 1
34LED2MAYBE (I)LED driver 2
43RST_NYES (I)System reset (active-low)
46P1_2 / TDI_CPU / SCLMAYBE (I/O)GPIO / Debug / I2C SCL
47P1_3 / TCK_CPU / SDAMAYBE (I/O)GPIO / Debug / I2C SDA
48P3_7 / UART_CTSMAYBE (I/O)GPIO / UART CTS
49P0_0 / UART_TX_INDYES (I/O)GPIO / Codec reset (Embedded) / TX_IND (Host)
50GNDYES (P)Ground reference

Okay, so I relented, and added the temperature sensor back in, because I think it might be good to know the internal temperature of the unit, and I'd miscounted pins, so this still leaves us at just 22 pins, which feels acceptable. 

Ah, there are also two GND pads, so that saves us a pin. Also, the VCC_IO pin isn't needed either, apparently, according to the footprint that I'm using. The footprint is annoying in that it has a keep-out zone for copper surrounding the GND pads on the rear of the PCB! I've just modified that so that it allows vias, so that I can still connect those pads to GND.

The end result is a very simple schematic:

Note that we feed 3.3V to the modules VBAT input. This is so that we can still switch power on and off to this module, rather than running it directly from the battery.

4. PCB Layout

The PCB layout is also trivial, with the pin assignments on the module being mapped so that all routing happens on the front, except for the power and ground which are routed on the rear.
 

5. Requirements Cross-Check

Requirement 2.1.1, Bluetooth host module that can connect to headsets and other audio devices. Must support UART or I2C control, and audio interfaces that we can easily implement from the FPGA, such as I2S audio.

Met -- in that we have the Bluetooth module, and the necessary interfaces for I2C and I2S and UART.

Desirable 2.1.2, Bluetooth host module support for keyboards and other HID peripherals. 

Met if and only if the BM83 can be configured to talk to HID devices. This remains unknown, but as this is only a desirable rather than a requirement, we are fine.

In short, we have this module finished!

 


 
 
 
 

Sunday, 9 March 2025

MEGAphone Fabrication of the first module PCBs

I've been busy working on the schematic and PCB design for a bunch of the modules for the MEGAphone lately.  The first batch of those I had sent off for fabrication by PCBWay, and they have finally arrived. So let's have a look at what we have:

Well, that's a bit hard to see what's going on, so let's look at each in turn. My lighting here isn't that great right now, but it should be enough that we can identify each PCB from it's packing label.

First we have the audio jack PCB:

Then the high-efficiency DC:DC converter:
The Audio Codec board:
SIM and SD-Card carrier:
Cellular modem carrier:
MPPT Battery Charger board:
Low-current DC:DC converter module:
And finally, the MEMS microphone carrier board.

 So let's look at where this gets us to in terms of the milestones:

Milestone 2.1.2 High-efficiency DC-DC converter module to power the various sub-systems: Schematic -- done -- see https://c65gs.blogspot.com/2025/02/megaphone-2a-dcdc-converter-module.html and https://github.com/MEGA65/megaphone-modular/tree/main/modules/high-efficiency_dc-dc_converter

Milestone 2.1.3 High-efficiency DC-DC converter module to power the various sub-systems: PCB Layout -- done -- see https://c65gs.blogspot.com/2025/02/megaphone-2a-dcdc-converter-module.html and https://github.com/MEGA65/megaphone-modular/tree/main/modules/high-efficiency_dc-dc_converter

Milestone 2.1.4 High-efficiency DC-DC converter module to power the various sub-systems: PCB Fabrication -- done -- see the photos above.

Milestone 2.3.2 Battery management and energy harvesting module to allow USB-C, integrated solar and external 12/24V vehicle battery power sources, and efficient management of the integrated rechareable battery: Schematic -- done -- see https://c65gs.blogspot.com/2025/03/megaphone-battery-management-and-energy.html, https://github.com/MEGA65/megaphone-modular/tree/main/modules/battery-connector, https://github.com/MEGA65/megaphone-modular/tree/main/modules/input-power-selector, https://github.com/MEGA65/megaphone-modular/tree/main/modules/usb-c-power-in, https://github.com/MEGA65/megaphone-modular/tree/main/modules/solar-and-dc-connectors, and https://github.com/gardners/mppt-charge-controller/tree/8d80fb90b56f2cbbb3eda396a8fb4dab4cb2aabd

Milestone 2.3.3 Battery management and energy harvesting module to allow USB-C, integrated solar and external 12/24V vehicle battery power sources, and efficient management of the integrated rechareable battery: PCB Layout -- done -- see https://c65gs.blogspot.com/2025/03/megaphone-battery-management-and-energy.html, https://github.com/MEGA65/megaphone-modular/tree/main/modules/battery-connector, https://github.com/MEGA65/megaphone-modular/tree/main/modules/input-power-selector, https://github.com/MEGA65/megaphone-modular/tree/main/modules/usb-c-power-in, https://github.com/MEGA65/megaphone-modular/tree/main/modules/solar-and-dc-connectors, and https://github.com/gardners/mppt-charge-controller/tree/8d80fb90b56f2cbbb3eda396a8fb4dab4cb2aabd

Milestone 2.3.4 Battery management and energy harvesting module to allow USB-C, integrated solar and external 12/24V vehicle battery power sources, and efficient management of the integrated rechareable battery: Fabrication -- done -- see photos above

Milestone 2.4.2 Cellular modem communications module, including SIM card module: Schematic -- done -- see https://c65gs.blogspot.com/2025/02/megaphone-sim-card-sd-card-module.html, https://c65gs.blogspot.com/2025/02/megaphone-cellular-modem-module.html, https://github.com/MEGA65/megaphone-modular/tree/main/modules/cellular-modem, https://github.com/MEGA65/megaphone-modular/tree/main/modules/sim-and-sd-card

Milestone 2.4.3 Cellular modem communications module, including SIM card module: PCB Layout -- done -- see https://c65gs.blogspot.com/2025/02/megaphone-sim-card-sd-card-module.html, https://c65gs.blogspot.com/2025/02/megaphone-cellular-modem-module.html, https://github.com/MEGA65/megaphone-modular/tree/main/modules/cellular-modem, https://github.com/MEGA65/megaphone-modular/tree/main/modules/sim-and-sd-card

Milestone 2.4.4 Cellular modem communications module, including SIM card module: Fabrication -- done -- see photos above

Milestone 2.5.2 Internal microphone and speaker module: Schematics -- done -- see https://c65gs.blogspot.com/2025/02/megaphone-mems-microphone-module.html, https://c65gs.blogspot.com/2025/02/megaphone-audio-codec-speaker-driver.html, https://github.com/MEGA65/megaphone-modular/tree/main/modules/mems-microphone, https://github.com/MEGA65/megaphone-modular/tree/main/modules/audio-codec

Milestone 2.5.3 Internal microphone and speaker module: PCB Layout -- done -- see https://c65gs.blogspot.com/2025/02/megaphone-mems-microphone-module.html, https://c65gs.blogspot.com/2025/02/megaphone-audio-codec-speaker-driver.html, https://github.com/MEGA65/megaphone-modular/tree/main/modules/mems-microphone, https://github.com/MEGA65/megaphone-modular/tree/main/modules/audio-codec

Milestone 2.5.4 Internal microphone and speaker module: Fabrication -- done -- see photos above

Let's see how this looks on the block-diagram, to get a sense of the overall progress represented by this work:



Not bad -- and as we can see, some of these modules are reused in multiple places, so we have knocked off most of the orange boxes, too.

The main difference between what was planned and how I have progressed is that I've tended to further break down some of the modules, to make their designs simpler.

From here, I'll decide whether the next step is to assemble these modules, or whether I'll do the PCB design for more of the remaining modules on the schedule. Probably a mix of the two.  Certainly I can prepare the bill of materials for each of the modules I've already designed, and order those parts ready for assembly, since they will take a week or so to arrive, in any case.