Friday, December 22, 2017

Automatic VIC-II Compatibility Regression Testing - Part 1

One of the challenges of implementing the MEGA65 is to be sure that we have compatibility with the VIC-II and other custom chips in the C64.  This gets more complex as we fix bugs and add missing functions, as we might accidentally introduce a regression. 

This is of course why test cases are very useful, and there are a bunch of test cases as part of the excellent VICE emulator suite. However, from the ones I have looked at, they are rather specialised, and there is no trivial way to run them all, and get informed when one or more of them fails.  (It is possible that I have the wrong end of the stick here, and that this can be done with VICE, and I would welcome correction on this point.)

I have decided to fix this state of affairs, and write a single program that runs through a large number of function and compatibility tests, and provides meaningful failure messages, so that whenever we build a bitstream, we can check to make sure that nothing has gone backwards.

My expectation is that this program will grow in fits and starts according to our needs, and may end up being a set of automatically chaining test programs, similar to the 6502 test suite for the C64, that has programs that test a specific instruction, and then load the test for the next instruction when done.

Anyway, I have begun implementing the first tests to help track down a problem with sprite positioning in the new 800x600 video modes.  It starts by checking sprite to sprite collision and then sprite-to-character collision, in such a way that it can work out where sprites are being drawn,relative to the text display.

The program purposely is C64 compatible, so that I can make sure that the tests run the same on a real machine (or in VICE for the simpler stuff, until I get an SD-IEC or Ultimate 1541).  Here is what it looks like running in VICE so far:
The test that fails might actually be wrong -- I am currently researching to find out of a real C64 detects sprite collisions that happen behind the borders. My gut feeling is that it probably does, but I am not yet sure.

Anyway, it is a starting point, and I will post an update when it is somewhat more matured.