I still don't like passing signals between clock domains

I am still working on the DDR memory interface for the C65GS.

It feels like it should have been finished ages ago.  However, as I have gone along I have been progressively reminded of various perils of passing signals between different clock domains.

The problem all comes about because the DDR memory controller operates at something like 145MHz, which is some fraction of the DDR bus speed required for the DDR RAM on the Nexys4DDR board.  The key point is that this is not the same as the 48MHz that the CPU runs at.

This means that you can't just pass signals, and assume that they will be read correctly. For example, if you want to ask the DDR controller to read something, you need to make sure that all the address lines have settled before they get sampled by the DDR controller.  The same goes for asking for it to write to memory.  Then there are similar sorts of fun getting results back in the other direction.

To simplify the problem, I have used a simple dual-port memory, i.e., one side can read and the other side can write, so that the DDR controller can provide data to the CPU.  This has the useful side-effect that I can easily construct a cache for the DDR RAM, so that the CPU doesn't have to wait for it so often.  This is what I have done, creating an 8KB cache, that operates with something like 4 wait states.

After lots of frustrating fiddling, I now have the DDR memory interface working most of the time. Some reads and some writes get missed, which suggests that there is still some cross-domain timing glitches that are causing grief.   One in every several hundred memory writes doesn't stick. Inserting some more latches between the CPU and the DDR controller will hopefully settle that down.  Alternatively, I could skip adding extra delay, and instead make the process of writing to DDR memory include a verification step, and to repeat the write if it doesn't work.  This is probably preferable.

There are also some remaining niggly intermittent problems that look to me like the DDR memory is not communicating with the FPGA correctly, i.e., the temperature-compensated termination calibrations that the DDR memory chip does might not be being done often enough or something.  When this happens every other byte is often $00, and writing to any address seems to get ignored.  This last problem is the most annoying, because I don't really have any idea of how to fix it, nor a way to verify that it has been properly corrected.  I would feel very bad if people were buying these Nexys boards only to find that they have frequent problems running the C65GS bitstream on them.  I'll start looking into this once I have the other memory interface problems solved.