Found the source of the erratic behavior and improved it

CAREFULLY INSTRUMENTED ANALYZER SETUP

Armed with an error latch that was set on when the byte transfer from the Arduino ended without having completed the read from the memory interface, plus a shadow of that latch properly transitioned across clock domains to the signals driven by the memory interface (ramclock is generated), I set up all the signals of the user interface with the memory module. 

FOUND THE CASE THAT PRODUCED THE STALL

When the error latch went on and the shadow latch appeared a couple of cycles later, I immediately saw the condition that would clearly and obviously cause a stall. The user interface required that the app_en signal remain asserted while app_rdy is false otherwise the request would be lost. It must remain asserted until the clock cycle when both the enable signal and the ready signal are true. At that point it is accepted and we can drop the enable. 

I saw that the app_rdy signal dropped to false, during a refresh cycle for the DRAM, at the same cycle when my logic asserted app_en. I then dropped app_en although the ready signal was still false. It appeared that if the ready signal was already false a cycle before I was to assert enable, I would handle it correctly and of course if the interface was ready when I was asserting it worked properly. The failure occurs if ready just happens to drop just as I prepare to assert enable. 

This is very timing dependent and indeterministic to my logic, as the times when the memory interface would pursue refresh cycles was buried inaccessibly in the interface IP. Things had to align just right (or just wrong) to fail, something that happened often enough to fail during a 321 word transaction of reads but not so often that every read would fail. Exactly the kind of erratic situation I knew was the cause of the bad behavior. 

DEFECT IN MY STATE MACHINE HANDLING OF MEMORY INTERFACE BUSY

My state machine for reading and writing memory would check the ready signal and if it were true, advance to a next state which asserted the enable signal for one cycle. This worked fine if ready were true during the whole process, and worked properly if ready was false during the testing, but if ready were to drop to false at the start of the next state, when I was raising enable, things would go wrong. 

This is due to the change times of the various signals. Inputs to the state machine such as app_rdy which determined the next state to enter at the clock edge might change at that clock edge just as we moved to the next state. 

I should have tested the ready signal in the same cycle where I was asserting the enable signal. This was a stylistic approach which led to the error in handling the case where ready drops just as I asserted enable. 

CORRECTION WAS EASY

I made changes to the state machine, such that it didn't raise the app_en (or app_wdf_en for writing) until a certain state where it checked to see if it needed to keep the enable high due to a false ready, or it could drop enable and move on to further states to complete the read or write operation. 

QUICK TEST SHOWS THE STALL CONDITION HAS DISAPPEARED

I reran my testing and the error latch never went on. Further, the state machine didn't stall and returned to idle once the unload transaction was complete. Looking at the data returned, however, shows that this is still not operating correctly.

Can't trust the DRAM memory interface - considering radical restructuring to use on chip static ram

STILL DON'T HAVE CAUSE OF ERRATIC BEHAVIOR NAILED DOWN, BUT . . .

I can instrument internal logic analyzer cores for a relatively small number of signals at a time and can only record 8K cycles on the core. Secondarily, each clock domain requires a separate analyzer core and they aren't easy to trigger 'simultaneously'. Third, signals in the SPI link domain can't be traced by an analyzer core because it needs a constant rather than intermittent clock signal.

Thus each time I have a new suspicion I have to resynthesize, set up the testbed and then capture only a small sample in time. The failure is so likely that I can't get through a single unload of 321 words, but not deterministic thus it fails on different words and perhaps in different ways on each test. If it always failed on a given word of the sector I could set better triggering for the analyzer cores.

Grossly, however, I seem to have stalling of the state machines and only return the last good value for all the subsequent transactions. My current suspicion is that it is triggered by a refresh cycle of the DRAM at exactly the worst moment. 

The delay could be 32 clock cycles, which when added to the delays transiting through FIFOs across clock domains, can add up to the major SPI link state machine having moved beyond the point where it needed the RAM data. I don't have proof that this is happening, although I will likely continue to construct tests where I might observe the smoking gun.

What I do know, however, is that were I to have a memory with a known and consistent access time that fits inside the state machine steps for the SPI link, I could have a reliable upload. Thus, if I can't find and fix the cause of erratic behavior, I might shift to a deterministic and reliable method to avoid said erratic conditions.

STATIC RAM ON FPGA CHIP CAN OFFER DETERMINISTIC READ AND WRITE

FPGA chips have static ram available onboard. It comes as both block RAM and distributed RAM. The block ram are sections of SRAM that are embedded in the chip and available to the designer. The look up tables and flip flops that are usually employed to create logic circuits can also configured as SRAM, and these are distributed among the LUTS of the FPGA. 

Block RAM use has essentially zero impact on the amount of logic that can be instantiated on the FPGA chip, since it is distinct areas of the chip that are not involved in generalized logic. Each chip has a fixed capacity of block RAM - in the case of the board I am using, 1,658,880 bits that is organized in words of up to 18 bits wide. 

Distributed RAM, on the other hand, takes up LUTS that otherwise would be available to form logic circuits. The more memory you instantiate, the less logic you can create. There are only 3, 650 LUTs, the basic building block of an FPGA, for my chip. Each LUT used as distributed ram instantiates 16 bits. An entire cartridge would require 521,304 LUTs and even a single cylinder would consume a large fraction of the available LUT capacity. 

SIZE CHALLENGE AS ENTIRE CARTRIDGE IMAGE CAN'T FIT ON THIS FPGA CHIP

One cylinder of the 2315 disk has eight sectors of 321 words, each 16 bits, thus it takes only 41,088 bits to hold that cylinder. The problem is when you look at the entire cartridge, all 203 cylinders of it, which would take five times the capacity of the block RAM to hold in its entirety. Distributed RAM provides little additional capacity. 

The erratic DDR3 DRAM, on the other hand, is 256 MB, far more than it needed for a cartridge. This is the reason I selected the DRAM initially to hold the cartridge image while the virtual drive was operating. 

CONSIDERING USING BLOCK RAM FOR ONE CYLINDER AT A TIME, DRAM FOR REST

If I have an entire cylinder in the block RAM, then the Unload transaction up to the Arduino will be deterministic and reliable. The disk drive controller reading and writing through the head electronics would also be satisfied easily and reliably from this cylinder buffer. 

When moving to a different cylinder, the current contents (potentially updated if writes from the CPU have take place) would be written to the DRAM and then the contents of that new cylinder would be read from DRAM and written to the block RAM.

SOME CHALLENGES TO CONSIDER WITH THE DUAL MEMORY APPROACH

The time it would take to dump 321 words from block RAM to DRAM, then load new block RAM contents from DRAM, may take longer than the time a real disk drive would take to perform a single cylinder seek. The minimum seek time is 15 milliseconds, a relative eternity to the FPGA operating at with 10 or 20 ns cycles, which provides about 2,336 cycles per word to do both a read and a write. 

The consequence of not meeting that timing would be that my virtual 2315 will no longer be timing accurate on short seeks. Even worse, the drive controller signals that the access is complete via a single shot timer, not some signal from the drive, thus the CPU will be justified to begin reading or writing before our slower dump/restore has completed. 

Another issue results from the current SPI link protocol, where the Arduino specifies the particular sector (including cylinder) where it wants to load or upload as part of each transaction. Thus, it might be commanded to seek to a new cylinder as part of the first two words of the transaction, but expect to receive words almost instantly on word 3 which is far too quick for the swap to occur.

It is conceivable that I could implement a reverse feedback signal to the Arduino that would hold it in mid word of a transaction until the swap completed. This is the major problem because I don't really have a constraint with the timing issues raised at the start of this section.

THIS CONCEPT ON BEING EXPANDED

It appears I can keep up with the disk drive seeks rather easily, so my only issue is in holding off the Arduino Unload or Load transactions. I am looking at various ways to handle this elegantly.

Update on debugging the SPI link between Virtual 2315 FPGA and Arduino sides

EVIDENCE INDICATES THIS IS ERRATIC AND TIMING DEPENDENT

I can run the test transactions from the Arduino to the FPGA multiple times and I see it failing at different points. I load a fixed pattern where each word of the sector has its word number as the content - 1, 2, 3 etc. I then fetch the sector content up to the Arduino and report where the returned value does not match the word number. 

I will find that two broad cases. In one case, after the FPGA hits some unknown state it will return gibberish that is constant for every word and every transfer over the SPI link. The second and more meaningful case is where it begins with agreement for some number of words and then the value returned is a fixed one at some word value for all subsequent words. 

The interesting observation for the second case is that the word number where it stops sending the proper value will change from test run to test run. It might be on the third word, it might be on the 50th word, but it will occur for certain sometime during the 320 words of a unload transaction. 

This tells me that I don't have a rare situation like a metastable signal or cross clock domain problem, it is a large window that is certain to hit a transfer sometime during a signal transaction. This is good, in that it should be easier to find than a very infrequent issue. However, it has not been obvious to me so far. 

MANY FIXES IN ATTEMPT TO TIGHTEN UP RESISTANCE TO TIMING VARIATIONS

Because this was clearly a timing issue that varied from run to run, I focused on timing between state machines and in all signals crossing clock domains. I had put synchronizers on all external signals coming into the FPGA. I even put on a synchronizer plus debouncer/hysteresis for the key signal that bracketed each two byte word of the SPI transaction. 

In my refactoring I put in a tightly interlocked set of signals to keep state machines in sync. One raises a trigger for the other but won't drop that trigger until the response signal is seen. The driven state machine will raise a response signal when it sees the trigger and won't drop the response until it sees the trigger go away. 

CURRENTLY LOOKING AT THE RAM STATE MACHINE AS IT IS LOCKING UP

I have recently found that the central memory access state machine the one that drives the memory interface IP that in turn controls the DDR3 DRAM, will end up stuck on some state other than its rest or idle state. That aligns with the symptoms, in that when it locks up it will stop responding with incrementing word values or it will not return even the first - thus the mismatch values I saw in the Arduino. 

When the first error case occurs, no meaningful match for any word, the value being returned is consistently the first value that was received to declare this as an unload transaction. That is, the value F8 09 which is the code for unload (B11111) and the value for the targeted sector number for my test which is B00000001001 and thus the outbound link remains frozen with the first value returned back to the Arduino. 

Normally, we send the value x0000 as we are receiving the first word defining the command, we then send the command value back in the second word of the transaction as we are receiving the inverse of the command. Error checking verifies that x07F6 is the valid inverse of the command word xF809 and we proceed to reach RAM and send up the contents for the next 321 words. Being stuck, we see xF809 coming back. 

In the second case, we do fetch the RAM locations properly for a while, sending that value up the link, but then we are frozen so the upward bound link keeps sending the last properly fetched value all the way to the end. 

Hurricane Nicole now in the rear view mirror; tweaking simulation and nailing the reset and startup sequences with refactored design

HURRICANE NICOLE VISITED UNEXPECTEDLY

A rare November hurricane formed with little warning and was upon us midweek. It reached Category 1 intensity, windspeeds around 70 mph and made landfall approximately 40 miles south of me. The experience was similar to Ian, which while it had been more intense when it first made landfall in the gulf side of the state, was down to Cat 1 by the time it passed over us last month. 

Zero water or damage to the workshop and its computers, zero damage to my condo. Water flying at the windows with gusts to 85 mph finds its way through even the best sealing, such that I had maybe two quarts of water puddling on the tile along the ocean side windows. A few towels soaked that all up and all was well. 

The beach in front of my building did get pounded. Lots of erosion. My building has a very high seawall so that even with a full moon, high tide and six foot storm surge, no water made it up to the ground level or the garages. The sand down on the beach was vacuumed away, however. There is a plan to dredge up sand and reestablish the beaches all along the coast here, although not instantly. 

Also the crashing waves, coming sideways due to the rotating hurricane winds, did smash up most of the wood stairways that lead down to the beach. All 12 of the public access walkways in our town, for example, were damaged and impassible. My building used to have a stairway, too, but all we have now is a 'diving platform' looking down to the sand below. 

IMPROVED MY SIMULATION WITH RANDOM TIME DELAYS

Since the major issue to validate is how my logic performs with the SPI link which is driven by the Arduino completely unrelated to any of my clocks or logic, I made use of a random number generator for the testbench which varies the timing of the bytes which I present to the FPGA simulating the Arduino. 

NAILED DOWN STARTUP/RESET TIMING WITH SIMULATION

Using the post-implementation simulation, modeling the actual structure of logic cells and routing from my design, I was able to spot and repair some weaknesses in the relative timing of starting various state machines and the initialization of the memory interface and FIFO IP that I am using. 

Completely refactoring the SPI link logic

STATE MACHINES DEPEND ON TIMING OF INCOMING WORDS TO ADVANCE

Several of the state machines driving the SPI link depend on the SlaveSelect line which is active for each two byte word being transmitted. Alternatively it used an SPIbusy signal which in turn was driven by SlaveSelect. In both cases, the state machine first sits waiting while the transmission/reception of a word is underway then advances to snag the output when SlaveSelect turns off. 

I suspect that there are times when I have SlaveSelect already active but I am first waiting for it to turn off, or vice versa, because of the relative timing of the Arduino driven SPI signals and what I am doing inside the logic in the FPGA. That certainly aligns with the symptoms I see, where the SPI machine is out of sync with the words being sent by the Arduino or one of the state machines stalls. 

REFACTORING IS MY SOLUTION WHEN I AM ENCOUNTERING FLAKY BEHAVIOR

If I spend enough time fighting with erratic behavior, it is time to look at the problem again and refactor the design. I try to come at the required behavior in a different way, focusing especially on interlocking or other means of ensuring that various state machines work together as intended.

SPI LINK LOGIC BEING REDESIGNED

It is now time to refactor all the state machine gear. I have evolved it several times, in some cases because the way the Arduino worked was different than I expected and in some cases due to defects or poor approaches I found. The longer you layer fixes atop some code, the worst it tends to get. Refactoring lets me redesign with the benefit of all the correct information about the Arduino and all the experience I gained working on the logic.

Erratic results cast suspicion towards testbed itself or subtle issue

ERRATIC RESULTS UNDER TESTING

The last few test runs have produced puzzling results. I saw mostly correct results going up the SPI link, but the first value sent was incorrect, then we were off by one for the next 285 words of the 321 word sector, then it repeated the 285th value over and over until the end. The SPI state machine did not complete nor reset.

With one run, I saw garbage values again and the integrated logic analyzer showed that the state machines for driving the SPI link froze after the first word, acting as if the SlaveSelect line was never asserted again by the Arduino master. The way that my state machines are set up, as soon as SlaveSelect is asserted we start over pumping out the first byte of the word to the SPI link module, but that wasn't happening. 

STALL IMPLIES THAT SLAVE SELECT FAILS TO ACTIVATE

SlaveSelect is a signal that I set and reset from my Arduino code for every word that is exchanged over the SPI link. That is, I assert select, exchange two bytes, then drop the selection line, thus delineating every word on the link. An overall signal, SPItransaction, is asserted to start a multiword transaction and dropped at the end of the 325 word exchange. 

A signal from the Arduino Mega 2560, with 5V logic levels, is converted by my level shifter MOS transistors to the 3.3V levels of the FPGA board. My Arduino itself produces both 5V and 3.3V to power the two sides of the level shifter. Previous oscilloscope probes showed very good swings of logic levels, so that when the +5 dropped to near 0 on the Arduino side I would have the +3.3V level on the FPGA side drop near zero. 

WIRING OR LEVELS OR METASTABLE OR SOMETHING ELSE

I don't have the scope here where I am testing, but that is the next step in investigating this weird behavior. A number of possibilities exist:

• The Arduino output may not drive low enough to produce an asserted low level at the FPGA
• The level shifter may be misbehaving
• Resistance in my makeshift wiring and connections may be producing invalid logic levels at the FPGA input
• The FPGA may reach a metastable invalid state if SlaveSelect changes near a clock edge
• Timing issues in the routing on the FPGA chip may produce state machine errors or other logical 'farts'

I have synchronizers on every exterior signal, including SlaveSelect, which should have reduced the change of a metastable state to extremely low odds, especially to occur as often as it seems to be.

I can use an oscilloscope to validate the voltages appearing at the FPGA input pins, ruling out the first few potential causes or directing me to corrective action. 

If the issue is timing, I will have an extra frustrating road ahead. The timing report shows continual failure to meet timing, driven by error messages about inability to place the clock buffer and clock generating resources in the same portion of the FPGA chip. It forces me to override the conditions. I am loathe to allow this but the microscopic detail level necessary to work on this, particularly as it involves Intellectual Property (the memory interface) that I didn't write and which is in Verilog - a language I don't know. 

RAM retrieving data but not yet transferred up SPI link - good progress

LOGIC ANALYZER CORES MONITOR READ AND WRITE OF MEMORY INTERFACE

My two integrated logic analyzer cores, one operating at the 4:1 speed of the memory interface used to clock in requests and grab data from memory, the other operating at my general logic frequency, were useful in spotting the state of signals as my logic dealt with SPI link requests to load and unload data from a target sector of the virtual cartridge image as held in the DDR3 RAM on the FPGA board.

I was able to see that the data was properly written into the memory interface and that information came out later when reading the same addresses. I will need this facility both to feed the SPI link during unload operations for virtual cartridges but also to feed the signals into the disk drive controller when we are simulating the head signals if it were a real cartridge spinning on the drive. 

CONTENTS OF READ DATA BUS FROM INTERFACE IS FLEETING

From the analyzer I could see that we only had valid data from the memory interface for the two clock cycles when the app_rd_data_valid signal is asserted, telling us we have good data. It then reverts back to the wrong data. Thus the timing of when we latch in the app_rd_data bus is critical to successfully getting memory contents out to the functions that need them. 

ADJUSTING TIMING TO CAPTURE THE DATA IN ORDER TO FEED IT TO SPI LINK

The fix seems pretty straightforward, so I will implement it and enter a new round of testing. Ideally, we will grab and hold the memory contents, pass it properly to the SPI link state machine, which will properly load it into the SPI slave link module itself where it will be properly clocked up to the Arduino.