Wednesday, 7 September 2022

MEGAphone R4 PCB Bring-up Part 1

I finally have the new MEGAphone R4 PCB in my hands, and just a few weeks left to do the hardware bring-up to meet the final milestone of the NLnet grant supporting this -- so it's all hands on deck now to get this done.

First up, here is what the R4 PCB looks like after Goran did initial testing and debugging a few issues with the schematic.

First, we have the front-side, where the buttons, speaker, thumbwheels, and of course, LCD panel will go.  Down the right hand side you might be able to see all the power isolation switches.  There are 6 of those, that we will explain later as we go through, which can cut power hard to most subsystems.

You will see that Goran has had to make a few modifications based on his initial testing:

Those will all get fixed in the R4A PCB layout.

Finally, we have the rear-side, with most of the interesting bits:

Taking a tour clock-wise around this board from the top-left corner, we have:

1. The HDMI connector (grey with the orange bit of tape) for digital video output.

2. Outline showing where a Raspberry Pi Compute Module 4 can be attached.  This is so that if you want to run Android or other "big" things, you can safely do that in secure compartment, i.e., the Pi, that has no control over the rest of the system.  The display from the Pi will be combined using the MEGA65's GenLock functionality, to allow the MEGA65 display to overlay over the top of the Pi's display, complete with transparency.  That last bit of the red channel of the C65's palette will finally see it's use!

3. To the right of that is the connector to hold an OrangeCrab FPGA module.  This module is an over-kill for what it needs to do, but with COVID supply chain dramas, we had to run with what we could get.  The Crab listens to the serial ports from the cellular modems and other communications bays, and can then activate power to the main FPGA to wake it up if something interesting happens. Like someone calls you or sends you an SMS.  This is super important for keeping the idle power consumption of this FPGA-powered phone down low enough to give it a long battery life.

4. To the right of that is an 18-pin header that is designed to accept an ESP32 to provide WiFi and Bluetooth support to the MEGAphone.

5. Then we have that port that every mobile phone should include: A standard Atari/Commodore-compatible joystick port! No more being limited to touch based games for us, nor to single-player games when someone else is using the D-pad and press-buttons on the front. You will be able to connect your favourite Commodore 64 joystick for authentic game play while under way.  Also handy to use as a clicker when using the MEGAphone to make presentations using the digitial video output.

6. Then because we are not Apple, we include a 3.5mm headphone/microphone jack.

7. In the top-right corner we have the MEGA65-compatible power pack input. This can take from 6V to about 20V, and doesn't care if it is a mains powered power pack, or a solar panel, with its integrated Maximum Power Point Tracker (MPPT). This is to support the integrated solar panel that we plan to put on the rear of the MEGAphone, potentially allowing infinite battery life. In the future, I would like to also add a USB-C power input, not just because the EU requires it, but also because it is common and convenient. But that might have to wait for the R5 PCB.

8. Coming down the right hand side, we have the big green connector for the battery.  For testing I am using a 10A 3.2V LiFePO4 prismatic cell, which is also the cell type I intend to include in the prototype MEGAphone R4 units that I will build up. This is what the cell looks like:

9. Below that we can see the two silver SIM / SD Card holders. One SIM card for each of the two cellular modem bays that live in this corner of the PCB.  Only one of the SD card slots is connected to the main FPGA.

10. Very near the middle of the PCB, there are two 16-pin connectors. These are designed to accept RFD900 / 3D Radio style UHF packet radios or other utility modules that folks might come up with.  Having a place for these is to allow the MEGAphone to one day act as a completely stand-alone mobile communications device, being both energy independent (thanks to the power efficiency, large battery and integrated solar cell) and communications infrastructure independent through mesh/ad-hoc wireless networking.  

11. Then along the left hand side we have the two black headers for the main FPGA module, a Trenz Electronics TE0725.

There are other various small bits hiding around the board which we will meet as we progressively bring up the various sub-systems. But that's enough to get us started.

The first step is getting the board to power up, and to be able to talk to the OrangeCrab FPGA board, since that controls the power supply to every other module on the board.  This uses a Lattice ECP5 FPGA, which has the nice advantage of being fully supported by the yosys open-source FPGA development tool chain. Not only is it free (in both senses), but it's also really fast.  The Xilinx Vivado suite takes well over a minute on my i7 to synthesise the simplest design. With yosys, its less than 4 seconds. This really helps for rapid development.

Speaking of which, one nuisance of the OrangeCrab, is that to flash it via the USB port, you have to drop it into "DFU mode". This is done by powering it off, and then holding down the single press-button on the board while you power it back on.  For testing the MEGAphone, this would mean disconnecting both the USB cable (which carries power) and also the battery and any power pack.  This is a lot of fiddling, and could wear out connectors over time, as well as just being a pain.  The PCB design also requires the battery to be present for the OrangeCrab to boot up, so I can't even optimise it by just having the power pack, but not battery connected.  It can run from just USB power, but then the other sub-systems of the phone won't be able to be powered up.

On Xilinx FPGAs, there is a way you can tell the FPGA to "reboot", which we use to support multiple "cores" on the MEGA65.  As the DFU Bootloader is just a "core" on the OrangeCrab we should be able to do the same. After quite a bit of hunting, I discovered that the SB_WARMBOOT primitive is the apparent way to do this on Lattice FPGAs.  However my Verilog is quite scratchy, and this is what yosys prefers over VHDL, and I couldn't get it to recognise this primitive.  So with Goran's help I discovered that while the PROGRAMN pin of the OrangeCrab, which does a hard reset of the FPGA, isn't connected to any other pin to make it easy to drive, it can still be driven from FPGA logic internally.  So with a single line of Verilog, I was able to make it so that if you press the push button at any time, it will reboot the FPGA into the DFU mode:

   // Reset when btn0 is pressed
   assign rst_n = usr_btn;   

This now means that I can super-conveniently just press that button when I want to re-flash the OrangeCrab FPGA with a new bitstream.  So time to load it onto the board, hookup the battery, provide some DC on the power jack, and get moving.

We have LEDs on the crab lit, and the green LED near the joystick port indicates that the board is charging the battery (quite slowly, because its only powered by the micro-USB of the Crab right now, which I think is limited to the old 500mW USB 1.1 standard).  

I can see the voltage on the battery slowly climb, confirming that the charge circuit is working.  If I disconnect the battery, the charging LED goes out, and the fully charged LED below it lights instead.  So all up, the battery charger is doing what it should. 

There is one issue with it that needs to be cared for, though, which is that as LiIon charger, it will aim for 4.2V, which is the absolute upper limit of the LiFePO4 chemistry.  The 1% tolerance of this could take it over the edge.  So we will need some sort of means to disable the charger before it over-charges the battery.  The charger is a CN3791, which doesn't have any mechanism to disable the charger nor to tweak the upper voltage limit.

We do feed the CN3791 via a LM74610-Q1 reverse-polarity protector. It might be possible to kludge that to disconnect the battery from the charger if the battery voltage climbs to high, but that would also disconnect the battery from the whole device, which is rather sub-optimal, shall we say, to treat a full battery like a completely empty one...

I'll have to have a think about how we solve this. It might be possible to add a diode to cause a small voltage drop at the right place in the circuit. Or better, we find a CN3791 equivalent that is made for LiFePO4 characteristics. Like the CN3801 from the same manufacturer, that is identical, except that it is for 3.625V, not 4.2V max. So I'll order a couple of those, and switch it out when they arrive.  Sounds like a perfect solution. It's only the paying $20 for postage on a $0.50 part that is making me reluctant. But I've got over myself and ordered some. It will be nice to put it on, and then not worry about frying my nice LiFePO4 batteries.

I'll try charging from a solar panel later, when I have time to hook up a lead on the panel. Spring has only just started here, so I'll also have to wait for a suitable patch of sunshine. It's a lower priority than getting the rest of the system powering up.

So, on to getting other things to power up, we have a direct 5V enable line, that is used to provide power to the microphones. This is under the control of the Crab, and was easy to confirm working. But the rest of the sub-systems have their power enable lines controlled via an I2C IO expander.  The Crab needs to talk I2C to the expander to set the appropriate lines to output, and set them high.

I've done plenty of I2C communications before, although I do always find it frustrating to get rock-solid.  But I can just lift the I2C controller from the MEGA65, if I trust the vhdl2vl VHDL to Verilog translator. Or I can use the I2C controller in Verilog that Goran has already put in for testing.  I'll give that a go in the first instance, and use my oscilloscope to probe that things are doing what they should.

The IO expanders are PCA9535s. This is the connections for the one we care about right now:

 

There is an error in the schematic, in that we have pin 3 labeled RESET when it should be A2. But that's not a big problem. I've filed an issue for us to fix that.  The main thing is we can work out the I2C address for this device connected this way: The top 4 bits are fixed to 0100, then we have 100 for A2--A0, so the 7-bit address will be $24, i.e., $48 for write and $49 for read.

I have also just noticed that this IO expander is powered from VCC_FPGA, which can get turned off. It should instead be powered from the same power source as the Crab, so that it stays on.  I've noted that in an issue as well.

Except it doesn't seem to be quite so simple... With the crab removed and only the battery, I see 3.46V on VCC pin of U14, even though the battery is showing 3.26V. Unless we have a "negative resistor" on the board, something funny is going on.

First, let's trace VCC_FPGA:

 

We can see that it comes from Power Supply PS1, and that the PS_ENABLE signal that enables it, that this depends on the position of switch S4, which is the 3rd switch from the top on the right side of the board: If the switch is in one position, then VCC_FPGA is definitely on. If it is in the other position, then the FPGA_Power_Off signal can be used to disable VCC_FPGA.

Most of the power cut switches disable power to a subsystem, but we made this one the other way around, so that we couldn't end up with a dead board in case the SN74HC74 didn't behave the way we expected.  This is because we are using it in a slightly odd way: By pulling /PRE high, we are requiring /INT to go low to activate power to the FPGA.  On the other hand, we can use the FPGA_Power_Off signal to reset the flip-flop, thus cutting power to the FPGA. 

FPGA_Power_Off is a direct output from the Crab, effectively allowing the Crab to ask for the main FPGA to be turned off.  The simplest source of the /INT signal is the second button on S6, i.e., the right-hand side of the two press buttons on the bottom of the device.  This gives us a nice hardware power-on switch.  So that all makes sense -- its just the mystery of why we connected the IO expanders that control other power systems to VCC_FPGA, as this will require the main FPGA to be powered on (and consuming power) for us to be able to turn off the other sub-systems. 

With that issue filed, let's get back to getting the I2C communications happening, so that we can test enabling and disabling the various power sub-systems.  The key enabling step here is that I need to be able to monitor the I2C bus with my oscilloscope, so that I can see what is being communicated.  The only place I can really get to these pins is on the connector of the OrangeCrab.  The question is how I can get my probes to stay on those pins, as I have only the tail of the hole-through header, which aren't really conducive to holding oscilloscope probes.  If I had thought ahead, I should have just had a 2-pin header with the SCL and SDA lines on it, so that I could do this part more easily.

One of my colleagues at work makes beautiful masterpieces from clothes pegs to hold probes in all manner of strange positions. I might be able to do the same here.  

And the proof of the pudding: I can see the I2C traffic on my oscilloscope and I can reach to press the reset button the crab board without much risk of disturbing the setup.

 

So now to overhaul the I2C traffic generation so that it queries and sets the registers of the IO expander.

I was able to fairly quickly get it to repeatedly attempt to read multiple registers. I can see the bytes are being acknowledged on the oscilloscope.  I am seeing the same two byte values repeatedly, even when I ask for more than two bytes. The values are 0x00 and 0x7F, which based on the schematic snipped above, means that the various EN_* lines to enable power supplies are all low, as is LCD_STBY and DTR_M1, and the non-connected bit 7 of the second port. This sounds at least plausible.

I know for some of these IO expanders that this is a feature, that allows you to reduce the latency when monitoring the registers that correspond to the inputs.  So I am having a read through the datasheet now, to see if that is the case.

Table 8-11 on page 22 of the datasheet seems to suggest that it does repeatedly just read a pair of registers, rather than sequencing through the 8 registers. It seems that this behaviour is controlled by whether or not the register address is supplied for the read.  Anyway, it explains what I am seeing, and also how I can work around it.

Next step is to make the sequence write to all 8 registers, so that I can set the state of the power rails. The power rails are:

Port 0 Bit 0 - Cellular Modem Bay 1
Port 0 Bit 1 - Cellular Modem Bay 2
Port 0 Bit 2 - Packet Radio Connectors (both)
Port 0 Bit 3 - ESP32 / WiFi Module slot
Port 0 Bit 4 - LCD Panel
Port 0 Bit 5 - Audio amplifier for the speakers
Port 1 Bit 5 - Audio amplifier for headphones

The other bits can all be safely set to zero for now.

For testing, I will make the loop set both ports to output, non-inverted, and change which power rails are being enabled every 0.5 seconds based on the counter loop.  This will let me easily check if they are all doing their thing.

To check if power rails are on, the MEGAphone has red LEDs on the front side that should illuminate when the rail is turned on, and the INDICATE button is pressed. That's the button to the left of the soft "power on" button.  However what I am seeing is that LED D2 is permanently on, whether or not I press the indicate button, and that nothing ever turns on when I press the illuminate button.

LED D2 staying on is okay, actually, because it is one of the LEDs that is under direct control of the Xilinx FPGA, which I don't have plugged in, so it is floating, which is allowing the FET for the LED to activate, and thus the LED is illuminated.

However, the other LEDs should turn on with activate.  Time to back-track and check if the port pins on U14 are being toggled... which they don't appear to be.

I think I will modify the I2C transaction chain to both write and then read the registers, so that I can see if the writes are taking effect or not. I can see the writes on the bus, but they aren't actually setting the registers. Actually, this reminds me that I once had a problem with the I2C stuff not finalising a write transaction problem -- the same thing could be happening here. Yes, that was indeed the problem.  

Now that I can write to a register, I will now rejig it to setup the registers correctly to allow control of the power rails as I originally planned. The registers are:

0 - Read Port 0
1 - Read Port 1
2 - Write to Port 0
3 - Write to Port 1
4 - Polarity inversion Port 0
5 - Polarity inversion Port 1
6 - Data Direction Port 0 (1s= inputs, 0s= outputs)
7 - Data Direction Port 0 (1s= inputs, 0s= outputs)

So we need to write to ports 2 through 7, putting setting the power rail control bits to output, and writing some sensible values into the others:

2 - Bits 0 -- 5 = power rail enables (I'll make those cycle through on a counter)
3 - Bit 5 = power rail enable for headphones amplifier (I'll also make that cycle)
4 - All zeroes to disable inversion of bits
5 - All zeroes to disable inversion of bits
6 - 0x3f = 1s in the bottom 6 bits that are power rail enables.
7- 0x20 = 1 in bit 5 for the power rail enable there.

Except that if I try to write to all the ports at the same time, it doesn't work.  It might be that writing to ports 6 and 7 overrides writing to the actual port. Anyway, I will have to break it up into several single register write transactions.  This isn't a big deal. It's just a little bit fiddly, because my I2C control state machine counts the number of times the i2c_busy flag advances, which means that at the end of a multiple action transaction that the state machine won't know to advance.

I'll start by confirming which register write is causing the write to the port 0 output register to fail. Hopefully I can set both registers 2 and 3 in one transaction -- which I can.  So let's see how many more I can do... Looks like writing to registers 4 and 5 stops it working.

Reading the datasheet there is this ambiguous description of writes:

"The eight registers within the PCA9535 are configured to operate as four register pairs. The four pairs are Input Ports, Output Ports, Polarity Inversions, and Configurations. After sending data to one register, the next data byte is sent to the other register in the pair (see Figure 8-8 and Figure 8-9). For example, if the first byte is sent to Output Port 1 (register 3), the next byte is stored in Output Port 0 (register 2).
There is no limitation on the number of data bytes sent in one write transmission. In this way, each 8-bit register may be updated independently of the other registers."

I think this means that you can only write to one register pair at a time, and that if you write more than 2 bytes in that transaction, that it just writes over the same two registers again and again. While a bit annoying, I can easily work around that.

It looks like I now have it all working, in that I can write to the registers and read back the updated port values. However, I am still not seeing the pins on the IO expander toggling.  But its bed time now, so I will have to investigate that tomorrow.  It's at least some progress, though.

Tomorrow has come, and I had an idea overnight as to what the problem would likely be:  I was clearly talking I2C to one of these IO expanders, but the pins on the one in question weren't toggling, so I checked whether I had the correct address, which was indeed wrong.  It should have been $24, as calculated earlier in this post, but for some reason in the code, I had $25 there.

Now I can toggle power rails, but there seems to be some funny effects going on.  To debug those, I really am going to need to have a serial interface to command the Crab to turn particular rails on and off, so that I can see what is happening.  Basically turning some rails on seems to be turning on or off others, possibly due to back-flow of current among the sub-systems.  But that can wait for The Next Blog-Post(tm)!

Saturday, 18 June 2022

TEI0004 JTAG Adaptor adaptors can now be ordered

So in the last blog post I designed a simple adaptor to allow use of the (at time of writing) still in stock TEI0004 JTAG adaptor with the MEGA65*, in place of the not currently in stock TE0790 JTAG adaptors.

Well, I have made up the first dozen of them, and they work. I can communicate with the serial monitor interface of the MEGA65, and I can push bitstreams to the MEGA65 over it as well. 

You can even stack them like LEGO (tm etc)!

This means it can be used to support development workflows that use the serial monitor interface. It can also be used to de-brick your MEGA65 if you mess up flash slot 0.

Note that these adaptors are _not_ supported by Vivado, but they _are_ supported by the MEGA65 m65 and m65connect tools on Linux and Mac, and for the serial monitor interface on Windows. 

The JTAG interface also technically works on Windows, but is not currently well supported by m65 and m65connect, so in practice may not work in that mode. It might work for you, but it might not. We are not promising that it does right now, let's put it that way. 

But if you are on Windows and need the JTAG interface, e.g., for pushing a bitstream to de-brick a MEGA65, or just try out other bitstreams without having to flash them, they will work 100% in a Linux VM.

My 11 year old son and I have decided to use this as a good opportunity for him to learn about the process of producing and selling a product, including working out all the costs, setting an appropriate price etc, so he is offering a batch of 50 of these for sale.

So if you would like to order one, they will be AU$15.95 or 9,99€ plus postage from Australia, depending on your preferred currency.  Payments will be accepted via EFT transfer from Australian bank accounts in Australian dollars or via IBAN/Direct Transfer/Sofortüberweisung in Euros to a German bank account -- i.e., no currency exchange fees will be involved.

Postage within Australia is $9.30 or $12.50 if you would like it Express Post.  Postage to most of Europe, including Germany and the UK is 10€ for slow-and-cheap or 18€ for express via Australia Post/EMS. Post should be constant for up to 3 or 4 adaptors in a single shipment. If you would like to order more, let me know, and I can get a more exact quote.

Finally, if you would like to order one (or more to save postage with friends), please visit the MEGA65 Discord Server and send me a private message.

But don't forget to check stock and order a TEI0004 from Trenz Electronic first, so that you know you will be able to make use of the adaptor!

Of course, if you would prefer to make your own, I'm happy to share the design with anyone who would like it.

Sunday, 5 June 2022

Using a TE0004 JTAG adapter in place of a TE0790 JTAG adapter

The MEGA65 has three FPGAs: One from Lattice in the keyboard, one from Xilinx as the main FPGA on the mother board, and one from Altera/Intel also on the motherboard.  The Altera FPGA is programmed using a Trenz TEI0004-02 JTAG adapter on J17, while the Xilinx on is flashed using a TE0790 on JB1.  The reason for these two adapters is a bit complex and historical, and influenced by which adapter works easily with the Xilinx and the Altera tools.  But regardless of how we got here, it's our starting position.

Why this matters, is that some MEGA65 owners would really like to get a JTAG adapter connected to their MEGA65 for flashing and interacting with the main Xilinx FPGA.  And this is where the problem comes in: Because of Chipaggeddon, Trenz will likely not have any more of the TE0790 adapters available for more than a year.  BUT they do have the TEI0004-02's in stock.  

Thus we would like to find a way to use the TEI0004-02 to control the Xilinx FPGA. We know that it won't work from in the Xilinx Vivado software, but it will work from the m65 command line tool, and also from m65connect on OSX and Linux.  Windows may be a bit tricky, because we don't have a rock-solid way to talk to the USB JTAG adapters from in Windows, except to use Vivado.  But even if Windows folks have to spin up a virtual machine running Linux, that will still in many cases be better than waiting a year or more. So, in short, there is good reason to do this little project.

First up, we need to get the pinout of JB1 and J17 from the MEGA65 schematics, and also making reference to the datasheet for the TEI0004 and TE0790 adapters.

Let's start with the TEI0004's pinout for its 10-pin header:

Pin 1 - JTAG TCK (output from adapter)
Pin 2 - GND
Pin 3 - TDO (input to adapter)
Pin 4 - Reference I/O-voltage from target board for JTAG and UART
Pin 5 - TMS (output from adapter)
Pin 6 - Reserved Output (May be used as Processor Reset in future software releases)
Pin 7 - UART RX (input to adapter)
Pin 8 - UART TX (output from adapter)
Pin 9 - TDI (output from adapter)
Pin 10 - GND

The TE0790 is a bit more complex, because it has a little CPLD that allows the pinout to be reassigned dynamically. So we need to know which profile it uses normally.  For this, its easiest to work backwards from the MEGA65 schematics, where the pin assignments will be fixed.

Pin 1 - GND
Pin 2 - GND
Pin 3 - UART RX
Pin 4 - TCK
Pin 5 - 3.3V (but not connected, as I discover later...)
Pin 6 - 3.3V
Pin 7 - UART TX
Pin 8 - TDO
Pin 9 - Not Connected
Pin 10 - TDI
Pin 11 - Not Connected
Pin 12 - TMS

So, as we already knew should be the case, all the required signals are there: We just need to connect them to one another.  For testing, I will just use a packet of header jumpers.  If that works, we can design ourselves a little bitty PCB that will do the adaption.

So what we want is:

TEI0004     TE0790
      1 TCK 4
      2 GND 2
      3 TDO 8
      4 VCC 6
      5 TMS 12
      6 ---
      7 RX  3
      8 TX  7
      9 TDI 10
     10 GND 1

So that all looks good.  I have an 8-pin header cable ready.  To make my life easier, I will record which colour I am using for which signal:

TEI0004     TE0790
      1 TCK 4         BROWN
      2 GND 2         RED
      3 TDO 8         ORANGE
      4 VCC 6         YELLOW
      5 TMS 12        GREEN
      7 RX  3         BLUE
      8 TX  7         PURPLE
      9 TDI 10        GREY

I have removed the not-connected pin, and the duplicate GND lines in the process.

Testing that out, it looks like the adapter powers up, and I can even see the output of the serial monitor interface from the MEGA65, which is good.  But JTAG communications don't work, nor does writing to the UART interface, so there is something odd going on.

In fact, if I even try to talk JTAG over it, then the machine goes all weird, resetting itself, and seeming to put random junk in memory.

This all suggests to me that I have likely got one of the UART and JTAG lines crossed... Now to figure out which.  Perhaps the easiest approach here is to find the UART line, since it is the uart line from the host computer to the adapter, so I should be able to type some characters into a terminal, and see the correct line waggle. We are expecting it to be pin 7 or 8 on the TEI0004, on the blue and purple lines. 

Without even going that far, I can see that the purple wire (UART TX to the MEGA65) is near GND, rather than floating at 3.3V. This is being caused by the TEI0004. So I am guessing I have the wrong pin... but double-checking everything seems to indicate that I do have the correct pin.  The MEGA65 main-board has a 10K pull-down on the UART TX line, but that should not be so strong that the TEI0004 can't drive the line high.

I think I have figured it out: I was using pin 5 on JB1 for VCC, but this is not connected on the MEGA65. VCC is instead on pin 6. With it on pin 5, the VIO reference voltage would have been near zero volts, and thus we saw the problems we were seeing.  

(Now, the astute reader might have noticed that I have pin 6 and not pin 5 in the table above, that is because I purposely changed it there after writing this, so that if anyone happened to glance at that to use it as a reference for themselves, that they wouldn't accidentally use the wrong wiring.)

... and funilly enough, with the correct voltage, both the UART and JTAG interface are now working: I can even do m65 -b mega65.bit to push a new bitstream to the MEGA65, or use m65 -S to take a screen-shot.  In short, we have it all working :)

So now let's design a simple little adapter PCB, so that folks can do this, without having to have a bodge cable.

Done! and sent off to PCBWay for prototyping!   

So it should look like in the following images when done:

From underneath, we have the 12-pin connector that goes onto the MEGA65's TE0790 header JB1. The big hole is supposed to line up with the 3mm hole in the MEGA65's PCB, so that if you want to secure it in place semi-permanently, this is possible.  It also provides a nice visual indicator that you have it the right way around.

From above, we have the 10-pin header to accept the TEI0004.

And a couple of different angles, just so you get an idea of the whole thing.  It is quite obviously very simple, hence why I was able to design it in about an hour, even though I had forgotten how to drive KiCad:

The particularly observant among you might have noticed that the TE0790 header is not lined up properly with the other two in the images above. This is a side-effect of the footprint I was using in KiCad having the pin numbers in the wrong order if I put it on the correct side of the board. So I designed it looking like this, i.e., with both on the same side, but with the pin positions actually correct. Of course, for assembly it doesn't make any difference at all:

 



Now its the annoying wait... PCBWay do have nice updates though as it goes through. Because the board is so small, I chose the largest batch size that would still give 24 hour build time (presumably because it fits on a single panel).  So right now, it is already showing:


So while we wait for the boards and the connectors to arrive, thanks to AmokPhase101 we have mock-up images that will help you to see how it will fit together, and so that if you get one (or make your own), you can easily fit it into place:

First, we have the upper right area of the MEGA65's mainboard, with the JB1 connector where the TE0790 JTAG adapter would normally sit. This is the 2x6 pin male header just to the left of the screw, and just below the microSD card slot, and above and right of the word "TEAM" in "MOULDS TEAM".

Second, we can see what it would look like with the adaptor in place: We see the pins of the adapter poking through where JB1 is beneath it, and the 2x5 pin male header that can accept the TEI0004:
Finally, we can put the TEI00004 onto that connector, with the micro USB connector facing towards the front of the computer:

So that's everything, really, until the PCBs and connectors turn up, probably in a week's time, and I can test one out.  I'm hoping that with it being so simple, I haven't screwed up one of the tracks or flipped pin numbers around or done anything else equally stupid (but that are all so easy to do with PCB prototyping).  But those (mis)adventures will have to wait until the parts arrive...



Monday, 2 May 2022

The "Chipageddon" parts shortage is weird

As many of you are aware, the MEGA65 has been delayed because of the whole supply chain disruption stuff that we have all come to know with COVID, and not at all helped by Russia deciding to invade its next-door neighbour.

We managed to source all the electronic parts for the first batch of MEGA65 months ago, and thought that we were all in the clear.

Then we found out that the power supplies were delayed for a bit. But not as much as the cardboard for the boxes.

Well, the cardboard turned up at the factory and was busy getting printed, but was delayed for a few more days because of problems getting a truck driver and the wooden palettes on which to stack the printed boxes.

The truck driver thing I knew was an issue already, with Germany going backwards by about 15,000 truck drivers a year, in part because they stopped teaching people to drive trucks in their year of national service, and not many people are willing to fork out around 10,000€ to get their Class-E driver's license, to qualify them for a job that is pretty poorly paid, and has notoriously bad working conditions.  As a result there are about 10,000 less qualified drivers created each year, which amounts to a good 2/3 of the short-fall.

I take a bit of interest in the truck-driver thing, because I have the Australian equivalent of a Class-E license (A "Heavy Combination" license for the curious, which means I can drive semi-trailers, but not b-doubles or road-trains), and am mindful of truck driving as a Career Of Last Resort(tm), which is at least handy to have as a short-term solution should the need ever arise. And the key points there are short-term and last-resort.  It's fun to drive 80 tonnes of truck around for a few days, but to have to do it day-in, day-out, and not see your family for days at a time, all the time, is not hugely appealing to most folks. So why would you invest 10K in getting qualified for such an unappealing job?

But anyway, the truck driver wasn't as much of a problem as the packing palettes are turning out to be. This is because the highly specialised and efficient German palette making machines are designed to use a specific high-grade nail made in ... you guessed, it, Russia. The other options for importing finished palettes for Germany are typically, Russia, Ukraine and Belarus. Oh dear.  

Meanwhile, we are also seeing other less obvious supply chain issues.  For example, I needed to order some parts for the prototyping of the next iteration of the hand-held version of the MEGA65, and had to search all over the place to find a supplier with the combined microSD / SIM card holders we are using to save space. I did eventually find a wholesaler in Japan who had them, so that I wouldn't have to wait the 72 week lead time (lead times of up to 100 weeks are now becoming quite common).  The catch was the minimum order quantity (MoQ) was 244, when I only needed 8 pieces.  Fortunately they are less than 0,20€ each, so even multiplied out, this would mean about 50€.

I placed my order, and got an email a few days later saying that the MoQ for this scarce part had increased from 244, up to a complete reel of 1,200.  Needless to say this struck me as rather odd: Because of a parts shortage, I was being forced to buy hundreds times more parts than I needed.  It also happened with some other parts like resistors and capacitors, where we had to order a reel of 2,000, when we only needed 20 or 50 pieces. So even though I only needed this many:


 

I ended up with this many:

 

So what is going on here? I think there are a few factors at play: One, the wholesalers are simply too busy to be bothered cutting a few parts of a reel, like they might have done before. Two, they don't want the risk of being stranded with most of a reel. This is especially true for weird parts like the ones I was ordering.  I think there is a good chance that they just wanted to get rid of them, and free up their warehouse space for something that people are more likely to need.

Anyway, back to the MEGA65, we got our palettes, and I believe that the cartons should be arriving at Trenz more or less any day now, and the shipping of the first batch of ~400 units should be starting perhaps as early as the end of this week. Exciting times, despite all of the dysfunction in supply chains at the moment.

Thursday, 24 March 2022

Solving the last of the digital video output glitching, at least I think so

First, as many will have noticed, we are now in March 2022, the month when we hope that the first 400 MEGA65s will ship. At the time of writing, we are still waiting to hear that the cardboard packaging and related materials have arrived at Trenz to go with the assembled PCBs, keyboards and cases, to make the complete units.  We still don't know exactly when they will arrive, but are waiting expectantly to hear news on this any day now -- as we are sure many of you are!

MEGA65 R3A Production board under test, touched up using the "my crappy phone's auto-focus is about as reliable as its auto-correct feature" filter.

But in the meantime...

As past readers will know, we have access to an old N5998A HDMI protocol analyser that we use to debug the HDMI and DVI compatible audio-video output on the MEGA65.  Recently we have seen some of the production boards having glitchy digital video output, and have set about fixing this in the VHDL.  However, in the process, we have found that in many cases the HDMI protocol analyser fails to recognise the video output, even though it displays fine on a monitor.

Also, in some cases where the protocol analyser does recognise the display, it reports "pixel errors", which is the problem we are trying to fix, however it doesn't give us any information about what those pixel errors are -- but that information would be tremendously helpful for us in debugging the signal. For example, are some of the DVI data words being shifted, are there stuck bits, or are some words being reversed or replaced with the lyrics to a lost shakespearian sonnet?  We really want to know, and we know that the N5998A's capture files seem to be raw bit-level captures of the three data plus one clock channel, and thus should contain the information that we need.  The trick is figuring out the file format.

The files seem to be in fact a completely raw capture with no header at all. The data looks like this:

00000000: 3f c4 00 3f c1 03 00 00 80 0f fc 80 01 0c 00 00    ?D@?AC@@@O|@AL@@
00000010: 3f c4 00 3f c1 03 00 00 80 04 00 80 01 05 00 00    ?D@?AC@@@D@@AE@@
00000020: 3f cf fc 3f c1 0a 00 00 3f c4 00 3f c1 03 00 00    ?O|?AJ@@?D@?AC@@
00000030: 80 0f fc 80 01 0c 00 00 3f c4 00 3f c1 03 00 00    @O|@AL@@?D@?AC@@
00000040: 80 0f fc 80 01 0c 00 00 3f c4 00 3f c1 03 00 00    @O|@AL@@?D@?AC@@
00000050: 80 0f fc 80 01 0c 00 00 3f c4 00 3f c1 03 00 00    @O|@AL@@?D@?AC@@
00000060: 3f c4 00 3f c1 03 00 00 80 0f fc 80 01 0c 00 00    ?D@?AC@@@O|@AL@@
00000070: 3f c4 00 3f c1 03 00 00 80 0f fc 80 01 0c 00 00    ?D@?AC@@@O|@AL@@
00000080: 3f c4 00 3f c1 03 00 00 80 0f fc 80 01 0c 00 00    ?D@?AC@@@O|@AL@@
00000090: 3f c4 00 3f c1 03 00 00 3f cf fc 3f c1 0a 00 00    ?D@?AC@@?O|?AJ@@
000000a0: 80 04 00 80 01 05 00 00 3f c4 00 3f c1 03 00 00    @D@@AE@@?D@?AC@@
000000b0: 80 0f fc 80 01 0c 00 00 3f c4 00 3f c1 03 00 00    @O|@AL@@?D@?AC@@
000000c0: 80 0f fc 80 01 0c 00 00 3f c4 00 3f c1 03 00 00    @O|@AL@@?D@?AC@@

We can see that there seems to be six bytes of data, followed by two zero bytes, in a repeating pattern. 

We know how HDMI/DVI pixel encoding works, e.g., from the description here:

https://en.wikipedia.org/wiki/Transition-minimized_differential_signaling

The relevant part being:

The method is a form of 8b/10b encoding but using a code-set that differs from the original IBM form. A two-stage process converts an input of 8 bits into a 10 bit code with particular desirable properties. In the first stage, the first bit is untransformed and each subsequent bit is either XOR or XNOR transformed against the previous bit. The encoder chooses between XOR and XNOR by determining which will result in the fewest transitions; the ninth bit encodes which operation was used. In the second stage, the first eight bits are optionally inverted to even out the balance of ones and zeros and therefore the sustained average DC level; the tenth bit encodes whether this inversion took place.  

There are only 460 such unique codes that are used.

Also of relevance are the four special code words used to indicate H and V sync:

0010101011
0010101010
1101010100
1101010101

Those are interesting in that one of them, the one that encodes horizontal sync, should occur every raster line, i.e. every ~864 ticks of the 27MHz pixel clock, a pattern that we should be able to fairly readily find.

There are also 16 more that can be used to HDMI data islands.

So a quick bit of a histogram of the frequency of each two-byte combination should give us a hint as to whether each two bytes encodes one of the R,G and B channels respectively, as I suspect that they might.

I cooked up a quick program to count the number of unique tokens:

$ make hist && ./hist 0.cap
gcc -Wall -g -o hist hist.c
Read 65536000 8-byte rows.
Saw 116 unique 2-byte tokens.

Given that there can be 65536 unique values, the highly constrained set of values gives me hope that they might be raw HDMI/DVI 8/10 bit data words.

In fact, if I separate each of the 2 byte tokens and treat them as separate channels, we see that they each have quite different populations:

(The binary values are least significant bit first, i.e., the most significant bit is on the right hand side.)

CHANNEL 0:
Read 65536000 8-byte rows.
  $0255 : 1010101001000000 : 16634
  $0440 : 0000001000100000 : 70011
  $0480 : 0000000100100000 : 11761792
  $0567 : 1110011010100000 : 46
  $0667 : 1110011001100000 : 46
  $0967 : 1110011010010000 : 1308
  $09B9 : 1001110110010000 : 116
  $0A4F : 1111001001010000 : 2156
  $0A67 : 1110011001010000 : 72541
  $0AB9 : 1001110101010000 : 2213
  $0B67 : 1110011011010000 : 5417
  $0D55 : 1010101010110000 : 9649169
  $0DD5 : 1010101110110000 : 483395
  $0F80 : 0000000111110000 : 9801503
  $85B1 : 1000110110100001 : 510
  $86B1 : 1000110101100001 : 508
  $89B1 : 1000110110010001 : 1455
  $8A63 : 1100011001010001 : 173
  $8AB1 : 1000110101010001 : 572465
  $8AB8 : 0001110101010001 : 24457
  $8BB1 : 1000110111010001 : 12142
  $C2AA : 0101010101000011 : 197898
  $C43F : 1111110000100011 : 16172474
  $C458 : 0001101000100011 : 12130
  $C4B0 : 0000110100100011 : 92075
  $C558 : 0001101010100011 : 91
  $C5B0 : 0000110110100011 : 1021
  $C658 : 0001101001100011 : 46
  $C6B0 : 0000110101100011 : 508
  $C958 : 0001101010010011 : 560
  $C9B0 : 0000110110010011 : 965
  $CA58 : 0001101001010011 : 24081
  $CAB0 : 0000110101010011 : 188469
  $CB58 : 0001101011010011 : 174
  $CBB0 : 0000110111010011 : 1986
  $CD2A : 0101010010110011 : 38720
  $CDAA : 0101010110110011 : 3724825
  $CF3F : 1111110011110011 : 12601920
Saw 38 unique 2-byte tokens.
CHANNEL 1:
Read 65536000 8-byte rows.
  $2AAC : 0011010101010100 : 214532
  $3F00 : 0000000011111100 : 16172474
  $3FFC : 0011111111111100 : 12601920
  $4000 : 0000000000000010 : 70011
  $4CCC : 0011001100110010 : 104205
  $5C38 : 0001110000111010 : 1108
  $5C90 : 0000100100111010 : 556
  $63C4 : 0010001111000110 : 1113
  $8000 : 0000000000000001 : 11761792
  $80FC : 0011111100000001 : 9801503
  $9870 : 0000111000011001 : 1108
  $988C : 0011000100011001 : 2818
  $A70C : 0011000011100101 : 11453
  $A770 : 0000111011100101 : 872912
  $A78C : 0011000111100101 : 1586
  $A790 : 0000100111100101 : 1082
  $B00C : 0011000000001101 : 2297
  $B070 : 0000111000001101 : 11452
  $B970 : 0000111010011101 : 1083
  $B990 : 0000100110011101 : 4331
  $B9C4 : 0010001110011101 : 555
  $D550 : 0000101010101011 : 13896109
Saw 22 unique 2-byte tokens.
CHANNEL 2:
$06C1 : 1000001101100000 : 328
  $0801 : 1000000000010000 : 483395
  $0AC1 : 1000001101010000 : 12601920
  $0C01 : 1000000000110000 : 9801503
  $0C41 : 1000001000110000 : 508
  $1481 : 1000000100101000 : 46
  $1CC1 : 1000001100111000 : 6520
  $1D01 : 1000000010111000 : 38720
  $1D81 : 1000000110111000 : 512
  $1FC1 : 1000001111111000 : 971
  $2601 : 1000000001100100 : 4933
  $3A01 : 1000000001011100 : 23719
  $3AC1 : 1000001101011100 : 348
  $4241 : 1000001001000010 : 46
  $43C1 : 1000001111000010 : 484
  $4B41 : 1000001011010010 : 508
  $4C01 : 1000000000110010 : 86
  $4DC1 : 1000001110110010 : 92075
  $52C1 : 1000001101001010 : 4932
  $5301 : 1000000011001010 : 565945
  $5501 : 1000000010101010 : 232
  $55C1 : 1000001110101010 : 438
  $5A01 : 1000000001011010 : 24457
  $5EC1 : 1000001101111010 : 965
  $6941 : 1000001010010110 : 510
  $6D01 : 1000000010110110 : 174
  $6E01 : 1000000001110110 : 484
  $7101 : 1000000010001110 : 2156
  $7401 : 1000000000101110 : 995
  $8501 : 1000000010100001 : 70011
  $8601 : 1000000001100001 : 1998
  $8801 : 1000000000010001 : 9649169
  $8901 : 1000000010010001 : 67261
  $9201 : 1000000001001001 : 186988
  $9B01 : 1000000011011001 : 46
  $9D01 : 1000000010111001 : 3724825
  $9F41 : 1000001011111001 : 46
  $A401 : 1000000000100101 : 1867
  $A7C1 : 1000001111100101 : 116
  $AA01 : 1000000001010101 : 87
  $B9C1 : 1000001110011101 : 2119
  $BC01 : 1000000000111101 : 173
  $C501 : 1000000010100011 : 1986
  $C581 : 1000000110100011 : 45
  $DB01 : 1000000011011011 : 2213
  $DE81 : 1000000101111011 : 510
  $EBC1 : 1000001111010111 : 276
  $EEC1 : 1000001101110111 : 16634
  $EFC1 : 1000001111110111 : 178
  $F001 : 1000000000001111 : 6520
  $F301 : 1000000011001111 : 509
  $F341 : 1000001011001111 : 46
  $F5C1 : 1000001110101111 : 12130
Saw 55 unique 2-byte tokens.
CHANNEL 3:
Read 65536000 8-byte rows.
  $0000 : 0000000000000000 : 65536000
Saw 1 unique 2-byte tokens.

So we see that in fact each of four two-byte tokens are drawn from completely separate populations, with the total number of unique tokens equalling the sum of the number of unique tokens within each of the four channels.  This is a good hint that we are looking at in fact four channels of data.

Note also that I have picked the byte order and bit order within the bytes quite arbitrarily -- and either or both might be wrong.

Channel 2 is particularly interesting, as there are six bits that have a fixed value, leaving exactly 10 bits that take differing values... and an HDMI/DVI data word is 10 bits long.

Now, looking at more of the file, I can see a general structure that repeats every 863 x 8 bytes. This is significant, because the raster lines in this video mode are 863 cycles in duration.  This is further circumstancial evidence that each 8 byte block corresponds to one cycle of time, and thus should contain the 3x10 bits of data plus 10 bits of clock. Hmmm... That adds up to 40 bits = 5 bytes, so we have one extra byte of something, presumably, or more if the clock is not recorded explicitly. The clock is interesting, because it should be 5 bits of 1 followed by 5 bits of 0, which should be easy to spot.

Also looking, I can see that there are 64 repetitions of:

aa cd 50 d5 01 9d 00 00

every raster line. That's the number of horizontal sync pulses, so we should expect at least the DVI channel 0 to have the HSYNC control word in it, i.e., one of those values that has lots of 1010101 in it.  Plus the clock should have 1111100000, so let's turn those bytes into binary, and see what we can see, recognising that we might need to reverse the byte and/or bit orders:

10101010 11001101 01010000 11010101 00000001 10011101

Well, that's reassuring that we can see plenty of 1010101 looking stuff in there, e.g., in the first and fourth bytes, in particular. What I can't see, though, is any way to have the 1111100000 clock pattern, so perhaps it isn't recorded.  This would make sense, as it would kind of be consumed during the decoding in the analyser, and implied.  That then leaves the mystery of why we have 48 bits being used to represent 3 x 10 bit values, when 30 would have been enough.

We then see 67 lots of:

55 0d 50 d5 01 88 00 00

01010101 00001101 01010000 11010101 00000001 10001000 

This looks to me, like 12 bits is reserved for each of the 10 bit words,and that they are laid out quite naturally, as we can see by lining those up on top of each other:

10101010 11001101 01010000 11010101 00000001 10011101
01010101 00001101 01010000 11010101 00000001 10001000

We see the different control word in channel 0 in the left most two bytes, while those in the other two channels remain unchanged after the end of the HYSNC period.  The values in the channel 0 control word are only valid, if the bit order were reversed over the 10 bits, to yield:

1101010101 xx 0010101011 xx 0010101011 xx ????????????

I would like to be really sure that this pattern holds, which I can check by testing for any of the xx bits being non-zero anywhere in the file. If not, then we will assume that they aren't used -- which holds true.  

So what on earth is in those last 2 bytes, then, if contains only 2 bits of channel data?

It holds to a specific pattern, where only the last 8 bits actually change.  My guess is that it is some kind of checksum or CRC, and I think I am right: Remember those two 8 byte vectors we were looking at?

55 0d 50 d5 01 88 00 00
aa cd 50 d5 01 9d 00 00

0x55+0x0d+0x50+0xd5=0x0188

0xaa+0xcd+0x50+0xd5=0x29c

If we assume that the fifth byte is simply required to have "000001" in its lower bits, then that leaves the last byte as a simple 8-bit checksum minus 0x100, with internal carry, i.e., the sum 0x29c becomes 0x19c becomes 0x9c + 1 = 0x9d.

I can try to verify that this algorithm is correct, by testing it on every single 8-byte vector. Well, it's almost right, in that it comes out to within 1 of the correct value every time. The internal carry might be wrong, perhaps, and there might be some other fancy adjustment being made.

Anyway, I'm not going to get too worried about it, since it is pretty clear that it is some kind of checksum.  This all means that we have a pretty clear idea of the format of the file, and can move on to making a simple decoder for it.    

It's the weekend again, and I have made some more progress, by beginning to create a little program that accepts in one of these capture files, and begins probing and testing the captured log.  The reason for this is to debug an issue with glitchy/absent DVI/HDMI video on some of the R3A production boards, which we know is due to manufacture variation of the FPGA part itself, as previously discussed.

So my plan is to make a capture from an R3 board that has good video output on a given bitstream, and then use the same bitstream on an R3A board that doesn't show video in the same circumstances. I will then implement the various tests -- starting from the low-level signalling, which is where I suspect the problem will be, and working my way up.

One nuisance of the capture format of the HDMI analyser is that it doesn't include an indication of the clock frequency, so there isn't a hard and fast way to automatically recognise the video mode, or be sure that it is at the right pixel clock. To work around this, I have already implemented logic that figures out the HYSNC and VSYNC timing, and then compares that to the list of official HDTV modes, and computes a least-squares error to find out which mode it most probably is.  This allows for some slop in the mode implementation, which is not uncommon (the MEGA65 in PAL for example uses 863 cycles per raster instead of 864 cycles per raster, so that it is divisible by 3).

In trying to remember exactly how the DVI/HDMI SYNC marking works, I was reminded of this series of blogs by someone else who made a basic HDMI analyser from an FPGA:

https://warmcat.com/2015/10/20/hdmi-capture-and-analysis-fpga-project.html

While not directly applicable to what we are trying to do here (as we already have a box that grabs the HDMI/DVI signals), it is an interesting read through, and has a hint for something that I will try to fix on the MEGA65 video out -- the HSYNC and VSYNC pulses should be synchronised, which they are not currently on the MEGA65.  That can be fixed easily enough in the VHDL.

But back to the HDMI analyser software so that we can compare good and bad output from different boards, I'll now implement the SYNC detection by tracking the SYNC values, and updating them when we see differing SYNC control word values on the blue channel.  

That now works, and I can see the video mode parameters being correctly inferred:

$ make hist && ./hist good-r3.cap
make: „hist“ ist bereits aktuell.
DEBUG: 459 unique pixel values
DEBUG: Read 13107200 records.
ERROR: 2603320 invalid DVI 10-bit words observed:
       10790x $013 (0000010011)
       22150x $043 (0001000011)
       25085x $20f (1000001111)
       45970x $270 (1001110000)
       40350x $2bc (1010111100)
       2458975x $2ec (1011101100)
DEBUG: Saw control word counts: 29122 144 15490 120
DEBUG: Saw most frequent lengths of: 14(x14013) 799(x96) 64(x14625) 64(x120)
DEBUG: HSYNC+ intervals = 863 863 863
DEBUG: HSYNC- intervals = 863 863 863
DEBUG: VSYNC+ intervals = 539375 539375 539375
DEBUG: VSYNC- intervals = 539375 539375 539375
INFO: Raster lines are 863 cycles long.
INFO: Frames 539375 cycles long.
INFO: Frames consist of 625 raster lines
DEBUG: Most frequent VSYNC low/high length = 65535(x24) 4315(x24)
DEBUG: VSYNC low -> 65535 / 863 = 75.94,  VSYNC high -> 4315 / 863 = 5.00
INFO: VSYNC pulse lasts 5 rasters, polarity is POSITIVE
INFO: File contains 24 frames (first is possibly partial, and a last partial frame may also be present)
DEBUG: Most frequent HSYNC low/high length = 799(x15166) 64(x15167)
INFO: HSYNC duration is 64 cycles
INFO: Mode most closely matches 720x576 50Hz non-interlaced (mode error = 1)
INFO: The mode differs from the expected mode in the following ways:
      h_total: saw 863, expected 864

You can also see in the 2nd-last INFO line, that the video mode detection magic has correctly inferred the video mode, using that least-square error algorithm I mentioned. The error is only 1, because the only divergence between the actual and model video modes is the duration of the raster lines differing by 1 cycle.  

The algorithm isn't perfect, if a mode differs in multiple ways from a standard mode, but it should in most cases give a good indication, and can act as a sanity check if you have fed in a different mode to that expected -- something that the fancy HDMI analyser software was blind of, and would instead just complain about every divergence between what was seen and expected, without suggesting that you check that the source was set to the correct mode.

We can also see that there are some unexpected control words (in italics).  Interestingly, this is on a board that is producing a valid picture. So I need to figure out what is going on there: Are we producing nonsense codes systematically, or is my decoder doing something odd.  The fact that there are very large numbers of them suggests to me that it is some systematic problem. For example, am I incorrectly calculating the codes for the different pixel values in either the VHDL, or in this analyser program -- either could cause what we are seeing. Or is it a problem with the transmission of the codes over the digital serial interface that is DVI/HDMI?

In the end I solved that problem by finding a complete list of the codes from one of Hamster (Mike Field)'s great FPGA projects, and slurping those in.

With that in place, I now get no errors of bad TMDS code words, which is great. 

Even better, when I ran the revised version of the program over the capture from one of the boards that was showing glitchy video, it showed that the video mode was not the PAL video mode that it should have been:

$ make && ./hist bad-r3a.cap
make: Für das Ziel „all“ ist nichts zu tun.
DEBUG: 460 unique pixel values
DEBUG: Flip test passed.
DEBUG: Read 13107200 records.
DEBUG: Saw control word counts: 1938 14476 162 28505
DEBUG: Saw most frequent lengths of: 176(x637) 64(x12843) 799(x75) 14(x12358)
DEBUG: HSYNC+ intervals = 863 863 863
DEBUG: HSYNC- intervals = 863 863 863
DEBUG: VSYNC+ intervals = 1771 12900 4423
DEBUG: VSYNC- intervals = 2133 12694 4640
INFO: Raster lines are 863 cycles long.
ERROR: Frames are not of consistent length.
INFO: Frames consist of 0 raster lines
ERROR: Frames are not an integer number of rasters long.
DEBUG: Most frequent VSYNC low/high length = 176(x63) 504(x5)
DEBUG: VSYNC low -> 176 / 863 = 0.20,  VSYNC high -> 504 / 863 = 0.58
ERROR: VSYNC pulses don't seem to be a multiple of the raster length.
INFO: VSYNC pulse lasts 0 rasters, polarity is UNKNOWN
INFO: File contains 63 frames (first is possibly partial, and a last partial frame may also be present)
DEBUG: Most frequent HSYNC low/high length = 64(x13335) 799(x12467)
INFO: HSYNC duration is 64 cycles
INFO: Mode most closely matches 720x480 60Hz interlaced (mode error = 274641)
INFO: The mode differs from the expected mode in the following ways:
      h_total: saw 863, expected 858
      v_total: saw 0, expected 524
      hsync_len: saw 64, expected 62
      vsync_len: saw 0, expected 6

Basically it thinks the screen is zero raster lines long. Something very odd is going on. Well, we knew that, because we weren't seeing a picture. But now we are getting some info on what is going on.

This led me down a fruitful path of investigation that revealed that the RESET line was glitching, with the glitch coming from the MAX10 2nd FPGA on the MEGA65 main board.  Further looking revealed that the communications from the MAX10 to the main FPGA is a basically totally confusing itself.  The MAX10 is also responsible for reading the dip switches -- one of which toggles between DVI and HDMI.     

All that means that the HDMI encoder was thinking it had to switch between HDMI and DVI all the time, which was confusing it no end. Especially since this could happen in the middle of a pixel.

So now I have something concrete to fix.

For now, I will just disable the MAX10 communications, and see if I can't get something stable.

The HDMI test bitstream with MAX10 disconnected is stable, provided all the HDMI features are turned off, i.e., sending pure DVI signalling.  So I will turn on one of the packets, and then try to play spot the difference between the two.

So, in fact, I don't need to do that, because I have confirmed the glitching on the reset line is the entire problem.  It took a bit of fiddling to fix the glitching, because the MAX10 is using its internal oscillator which drifts with temperature to anywhere between 55MHz and 116MHz.  

Essentially the problem I had was that I was counting how long the incoming 41MHz clock from the Xilinx to the MAX10 FPGA on the communications path holds, to detect the sync (which is a long hold) among the clock pulses (short holds).  41MHz < 55MHz, so it should be fine, right? Nope, because each _half_ of the clock is effectively an 81MHz signal -- which is more than 55MHz, so if the MAX10's internal clock was running slow, it could sample on the same half of the clock multiple times running, and thus mistakenly believe that it is in a sync pulse.  

To solve this, I halve the 41MHz signal in the MAX10 FPGA, and then check that. With a bit of fiddling of the constants, that had it fixed. I could have fixed it in the Xilinx FPGA, but then we would have had to modify our release bitstream and test it again, which I still really wanted to avoid. And fortunately I was able to do this. In fact, I solved these last problems during the live stream I did on this topic:





Thursday, 10 February 2022

MEGA65 PCB Population

As we get closer to the first 400 units being sent out, we thought you might like to see how our manufacturing partner in Germany, Trenz Electronic GmbH, went about populating the PCBs of the MEGA65.

Trenz designed the MEGA65's PCB and are uniquely placed to do the manufacturing and assembly -- and as with the rest of the key parts of the MEGA65 PCB, these are all manufactured in the German-speaking parts of Europe. The keyboards are also made in Germany by GMK, and the cases in Austria by Hintsteiner. As an example of this partnership, you can read about it on Hintsteiner's blog (in German, sorry, but Google Translate is pretty good these days).

It's worth stressing just how fantastic it is to work with these companies. They all place quality is an extremely high priority, and really take pride in their work. Their close proximity to the core MEGA65 team in Germany is also very helpful, and the lack of any language or cultural barrier also just makes work quicker, easier and more precise. Without their generous support, the MEGA65 would not be where it is today -- which is of course not very far off from being shipped out (more on that in a bit)!

In fact, it has turned out to be critical to the project that we have taken this quality-first approach, because this pesky pandemic has made it very hard for small projects to get the attention of the low-cost Chinese manufacturers, and even to get the components required to build boards.  We really feel for some of the other retro computing projects out there at the moment that are struggling to get the FPGAs and even components like resistors that are in very short supply right now.

But back to the PCB, the MEGA65's mainboard is a multilayer PCB with the big fat FPGA at the heart of the machine.  Just designing a board that can fan out many hundreds of pins from a chip to the various places on the board that they are required is non-trivial. Then to manufacture, you have to be able to place the solder paste for the reflow precisely, and then place the components in their correct locations using a pick-and-place machine, often called a "chicken-picker" for the way that they alternately "peck" at the component rolls and mainboard. Here you can see the head over the mainboard:

These chicken-picker machines are quite large and expensive machines, especially the ones that are able to place hundreds or thousands of components per hour. The parts themselves come from tape reels, which automatically rotate as each component is taken, allowing the whole process to be driven automatically. You can see the reels on the left of the machine here, as the shuttle moves to collect the components there, before moving back to the right where the board is hidden behind the structure from this angle:


The chicken-picker machines also typically have a camera that helps them to visually check that the "chicken head" is in the absolute correct location, and that everything is going right, which you can see here dwelling on some components near the MEGA65 logo on the board:

 

You can see all of this in action in the following video that we have made with Trenz to show how the PCBs get processed in the chicken-picker:

Otherwise, behind the scenes we continue to work on a bunch of improvements to the MEGA65 core, including among others, improving our already pretty-good SID to sound even better, improving international character input with the keyboard, lots of other little improvements of various kinds, continuing to improve the BASIC65 ROM and KERNAL to support nice things like direct access to the SD card, D64 disk image mounting, fixing 40 year old bugs in Commodore's DOS among others. To join the fun in real-time, maybe join our discord server.