Yes, LLMs are all the craze right now, for a variety of reasons.
One of our great community members made the TinyBrain little LLM that can run natively on the MEGA65. It's only 325K weights, but that's enough for it to run a little model trained specifically on answering short questions about the MEGA65's architecture.
That got me thinking.
How big / useful an LLM could we run on the MEGA65, and could we set it up in a way that helps people learn to use and code on the MEGA65 -- all while being native and off-line?
And I think the answer is yes -- especially for the R6 and newer boards, because they have 64MB of SDRAM as well as the 8MB HyperRAM.
And I have to pause right here, because this would be the first architectural feature we're contemplating for the MEGA65 core, that would be officially supported, and have a real difference between the R3 (that could run perhaps 5M-10M weight models) and the R6 (that could run perhaps 50M-100M weight models, and run 50Mish weight models without messing up HyperRAM contents).
So first up, this is just me trying an experiment so far -- we'll put it through our architectural governance processes. But let me explain why I think this is feasible and reasonable.
First up, the SDRAM would be used only for this LLM helper -- that would be totally distinct from conventional software running on the MEGA65. And it would in fact make it more dangerous for developers to sneakily or accidentially use the unofficial diagnostic access mode to the SDRAM already present in the R6 MEGA65 core.
Second, we'd still have a fall-back for R3: The model would just be smaller. So it would be graceful degradation.
Okay, lots of words there. Blah blah LLM blah. So let me tell you the workflow that I'm aiming for this to support, with a user-story.
User Story #1: Bob the BASIC Programmer
Bob has a busy like running his business teaching chickens card tricks, and so has little time available to learn BASIC or other programming languages. But he enjoys trying. Unfortunately, his chicken training business is highly classified, so there's no internet available in the secure lunch room where he keeps his portable MEGA65 in it's nice blue tough-case.
Bob sits down after a grueling session of Canasta with a particularly troublesome chicken, and just wants to make some pretty graphics on the MEGA65. He types in a few lines of BASIC, gets a syntax error, presses the HELP key, but doesn't really understand what it's telling him. But thanks to his MEGA65 having the SSNAIL LLM engine and a pre-loaded models on the SDCARD, he can now long press the HELP key.
When he does that, the MEGA65's HYPPO Hypervisor does the same thing it does when you hold RESTORE to freeze. But instead of loading the normal freezer, it loads the SSNAIL Triager, which examines the contents of the screen, and gives Bob the recommendation to use the "BASIC Programming Assistant Model".
Bob accepts this suggestion, and if the model wasn't already loaded into the SDRAM of his R6 MEGA65, it gets loaded now, and gets automatically given the error message and the line of BASIC that triggered it, and Bob is prompted to ask for help, with some suggested pre-sets, such as "how do I fix this error?", but also allowing him to write free-form prompts, if he so wishes.
Bob accepts the suggestion, the model processes the prompt and begins emitting it's (hopefully useful) suggestions to Bob at several tokens per second, giving Bob time to read as it goes along.
Once Bob is happy with the output, he can then exit the SSNAIL helper, returning to his BASIC programming session where he can manually make the changes to his program based on the advice he received. If he forgot what it told him, he can long press HELP again, and choose the option to look at recent chat logs.
---
Okay -- so not a key thing here: The LLM helper won't write your program for you. Totally intentional. You can ask it to write some lines of code, or fix existing ones, sure. But then you have to type it in manually yourself, so that -- hopefully -- you learn and remember as you go along.
I think this is a much healthier approach for a retro-computing community. And of course it doesn't need internet access, no one in the NSA can log your MEGA65 programming activities, and all the rest.
And the Triager can potentially offer different models, e.g., general usage, exploring the MEGA65 software library, assembly or graphics programming help etc. Basically by having a bundle of models that are quite specialised, ~50M -- 100M weight models are potentially quite useful.
And of course people can make models that do whatever other fun stuff they want -- like generate pixel art for sprites or whatever.
I think it has the potential to be fun and useful and fairly healthy mentally.
And, I think, it can have pretty good performance.
I've been using Claude to help me shake down bugs in both the SDRAM and HyperRAM controllers, and add in a new "LUMP" channel to them -- Linear Unorganised Memory Port -- that provides maximum memory bandwidth -- about 320MB/sec on the SDRAM. Combine that with INT4 weight quantisation, and that means potentially 640M weights per second. With sufficient pixie dust, hand-waving and a healthy dollop of hope -- this should mean single to double-digits tokens per second if we have the VHDL implement a nice streaming tensor processor thing. Basically a tiny AI optimised GPU that runs in parallel to the CPU.
The current progress on this should be able to do ~500M multiply-accumulates per second running at 162 or 324MHz internally, with a lot of pipelining. It really is a little GPU for our wonderful 8-bit machine. Still horribly slow by modern standards (hence the name), but still potentially really useful. The design of the SSNAIL GPU will also allow you to use it from regular MEGA65 programs, e.g., to do fast matrix operations for graphics. We'll see how far I can generalise it for that purpose, but in principal, that's a possibility.
So that's the plan.
So far I've got Claude to build the simple SSNAIL GPU thing, and a software implementation and a bunch of tools to convert GGUF model files to SSNAIL model files, and confirm that they can be used and generate identical results to the GGUF files running under OLLAMA or similar.
It's not done too badly, but it does use way too many LUTs right now, so I'm working with it to find the best impact places to reduce that. Same for the HyperRAM and SDRAM controller. A nice side-effect is that the HyperRAM controller will end up faster than before, and a bunch of cache and other weird bugs in it have been shaken out.
I could use Fable or one of the pay-to-win frontier models, but I'm too cheap for that, and I want more intellectual ownership of the result, too. And I'm finding that I'm still learning a lot along the way.
What do you need to actually run an LLM?
How can we architect it to be as efficient as possible?
How the heck can I work out where all the LUTs are being eaten?
That last one, I realised I could just ask the LLM to make me the magic TCL and shell script to ask Vivado to give me a breakdown of resources that a module is using. I knew theoretically that this could be done in Vivado, but not the specifics. But I don't need to. That's a good thing about LLMs.
So now we have scripts/vivado_module_report.sh that can generate a resource usage report for any module in the design after you try to build a bitstream. Super nifty. I've got a build running with some of the optimisations to resource usage the LLM recommended to shrink the SSNAIL module and the over-bloated HyperRAM caching stuff back down. Once the build finishes, I'll be able to check them easily. Noice.
Anyway, I've got it to build a working bitstream again, and the HyperRAM cache now seems rock-solid. SDRAM cache consistency checks are still failing in hyperramtest.prg. I'll have to dig into why. But that worries me less than if HyperRAM were failing, since the CPU isn't supposed to have direct SDRAM access, anyway.
And the performance of the HyperRAM is now much better:
The SDRAM is a bit faster for the operations that involve reading from it, which makes sense, because it does have a bit lower latency.
Ah, the auto-advance of the presented cache line wasn't on. It doesn't seem to help the HyperRAM any, but look at this for copy from SDRAM to chip RAM now: It's only marginally slower than a Chip-to-Chip copy!
Digging through with LLM found that the pre-fetching on the HyperRAM controller is causing problems with our reduced cache size there that was implemented to stop the HyperRAM consuming an absurd amount of resources. Disabling that gets a result that's almost as fast as the SDRAM, and certainly gets that juicy 16MB/sec copy from Slow to Chip RAM:
Now it's just tracking down the SDRAM data-eye training regression that's crept in. It really shouldn't have happened, but clearly it has. This is one of the annoying things about coding with an LLM -- bizarre regressions for bizarre reasons. That often turn out to be horrible architectural decisions it made at some earlier point.
What looked like a cache coherence problem is actually just that the SDRAM can't reliably read back data at the moment. Or isn't reliably writing it. Or both.
Claude Opus is still way better than ChatGPT for VHDL -- it's ploughing through the remaining timing closure problems, and will hopefully get it finished in the next day or so (I don't trust it to actually go and run Vivado alone).
Back at it on the following weekend, and in theory timing closure is more or less there after lots of iterations and pipelining and goodness knows what by Claude. It seems to have made enough unit tests that I have some confident that it might actually work.
Found a bug in BASIC65 in the process of running the tests.
And then there were lots more bugs to fix, but in the end, it now passes all the on-machine tests that Claude wrote for it:
So in theory, at least, I should be able to load a .SSNAIL model and the little toy chat client it wrote. Except that it's not as finished as I thought. Claude's trying to flesh that out for me now, and I'm also getting it to further automate the generation of the .SSNAIL model files based on the set of sensible available models.
Some more poking around and getting Claude to fix the toy client, and well, it does _something_, and does it at 12.3 TPS with a 20M model! Not bad. Too bad that the token stream it outputs is completely random.
You will have to excuse me for having a bout of maniacal laughter about now. Because this is a MEGA65. With no internet connection. No external support processor. Just what we've crammed into the FPGA, and is accessed via a pile of PEEK and POKE style registers.
And the bloody thing just wrote a story (for certain values of story) from my prompt at 175.8 TPS. Not 0.1758 TPS. Nor 1.758 or even a respectable 17.58 TPS. But 175.8 TPS. It wrote it faster than I could read it.
I am going to have to step away for another bout of maniacal laughter.
The monster lives!
Ok, time to find one of the bigger models that works. TinyStories 20M with a huge 1K context window! That didn't work. There's something wrong with the LLAMA2 F16 ones, I think. But this one with a 1K context window works, and gets a healthy 30 TPS:
Okay, so they seem to work, although clearly a temperature of 4 is a bit high ;)
What I want next, is to get the smaller ones of the Falcon H1 tiny family of models running, because they're actually "chatty". This requires implementing a few more operations in our SSNAIL LLM accelerator, which I'll tackle in due course. But that will be for another post!
For the curious, the code for the chat client and model conversion tools is here: https://github.com/MEGA65/SSNAIL and the SSNAIL LLM accelerator is currently on the 949 feature branch of the MEGA65-Core repo.
No comments:
Post a Comment