Hurricane Nicole now in the rear view mirror; tweaking simulation and nailing the reset and startup sequences with refactored design

HURRICANE NICOLE VISITED UNEXPECTEDLY

A rare November hurricane formed with little warning and was upon us midweek. It reached Category 1 intensity, windspeeds around 70 mph and made landfall approximately 40 miles south of me. The experience was similar to Ian, which while it had been more intense when it first made landfall in the gulf side of the state, was down to Cat 1 by the time it passed over us last month. 

Zero water or damage to the workshop and its computers, zero damage to my condo. Water flying at the windows with gusts to 85 mph finds its way through even the best sealing, such that I had maybe two quarts of water puddling on the tile along the ocean side windows. A few towels soaked that all up and all was well. 

The beach in front of my building did get pounded. Lots of erosion. My building has a very high seawall so that even with a full moon, high tide and six foot storm surge, no water made it up to the ground level or the garages. The sand down on the beach was vacuumed away, however. There is a plan to dredge up sand and reestablish the beaches all along the coast here, although not instantly. 

Also the crashing waves, coming sideways due to the rotating hurricane winds, did smash up most of the wood stairways that lead down to the beach. All 12 of the public access walkways in our town, for example, were damaged and impassible. My building used to have a stairway, too, but all we have now is a 'diving platform' looking down to the sand below. 

IMPROVED MY SIMULATION WITH RANDOM TIME DELAYS

Since the major issue to validate is how my logic performs with the SPI link which is driven by the Arduino completely unrelated to any of my clocks or logic, I made use of a random number generator for the testbench which varies the timing of the bytes which I present to the FPGA simulating the Arduino. 

NAILED DOWN STARTUP/RESET TIMING WITH SIMULATION

Using the post-implementation simulation, modeling the actual structure of logic cells and routing from my design, I was able to spot and repair some weaknesses in the relative timing of starting various state machines and the initialization of the memory interface and FIFO IP that I am using. 

Completely refactoring the SPI link logic

STATE MACHINES DEPEND ON TIMING OF INCOMING WORDS TO ADVANCE

Several of the state machines driving the SPI link depend on the SlaveSelect line which is active for each two byte word being transmitted. Alternatively it used an SPIbusy signal which in turn was driven by SlaveSelect. In both cases, the state machine first sits waiting while the transmission/reception of a word is underway then advances to snag the output when SlaveSelect turns off. 

I suspect that there are times when I have SlaveSelect already active but I am first waiting for it to turn off, or vice versa, because of the relative timing of the Arduino driven SPI signals and what I am doing inside the logic in the FPGA. That certainly aligns with the symptoms I see, where the SPI machine is out of sync with the words being sent by the Arduino or one of the state machines stalls. 

REFACTORING IS MY SOLUTION WHEN I AM ENCOUNTERING FLAKY BEHAVIOR

If I spend enough time fighting with erratic behavior, it is time to look at the problem again and refactor the design. I try to come at the required behavior in a different way, focusing especially on interlocking or other means of ensuring that various state machines work together as intended.

SPI LINK LOGIC BEING REDESIGNED

It is now time to refactor all the state machine gear. I have evolved it several times, in some cases because the way the Arduino worked was different than I expected and in some cases due to defects or poor approaches I found. The longer you layer fixes atop some code, the worst it tends to get. Refactoring lets me redesign with the benefit of all the correct information about the Arduino and all the experience I gained working on the logic.

Erratic results cast suspicion towards testbed itself or subtle issue

ERRATIC RESULTS UNDER TESTING

The last few test runs have produced puzzling results. I saw mostly correct results going up the SPI link, but the first value sent was incorrect, then we were off by one for the next 285 words of the 321 word sector, then it repeated the 285th value over and over until the end. The SPI state machine did not complete nor reset.

With one run, I saw garbage values again and the integrated logic analyzer showed that the state machines for driving the SPI link froze after the first word, acting as if the SlaveSelect line was never asserted again by the Arduino master. The way that my state machines are set up, as soon as SlaveSelect is asserted we start over pumping out the first byte of the word to the SPI link module, but that wasn't happening. 

STALL IMPLIES THAT SLAVE SELECT FAILS TO ACTIVATE

SlaveSelect is a signal that I set and reset from my Arduino code for every word that is exchanged over the SPI link. That is, I assert select, exchange two bytes, then drop the selection line, thus delineating every word on the link. An overall signal, SPItransaction, is asserted to start a multiword transaction and dropped at the end of the 325 word exchange. 

A signal from the Arduino Mega 2560, with 5V logic levels, is converted by my level shifter MOS transistors to the 3.3V levels of the FPGA board. My Arduino itself produces both 5V and 3.3V to power the two sides of the level shifter. Previous oscilloscope probes showed very good swings of logic levels, so that when the +5 dropped to near 0 on the Arduino side I would have the +3.3V level on the FPGA side drop near zero. 

WIRING OR LEVELS OR METASTABLE OR SOMETHING ELSE

I don't have the scope here where I am testing, but that is the next step in investigating this weird behavior. A number of possibilities exist:

• The Arduino output may not drive low enough to produce an asserted low level at the FPGA
• The level shifter may be misbehaving
• Resistance in my makeshift wiring and connections may be producing invalid logic levels at the FPGA input
• The FPGA may reach a metastable invalid state if SlaveSelect changes near a clock edge
• Timing issues in the routing on the FPGA chip may produce state machine errors or other logical 'farts'

I have synchronizers on every exterior signal, including SlaveSelect, which should have reduced the change of a metastable state to extremely low odds, especially to occur as often as it seems to be.

I can use an oscilloscope to validate the voltages appearing at the FPGA input pins, ruling out the first few potential causes or directing me to corrective action. 

If the issue is timing, I will have an extra frustrating road ahead. The timing report shows continual failure to meet timing, driven by error messages about inability to place the clock buffer and clock generating resources in the same portion of the FPGA chip. It forces me to override the conditions. I am loathe to allow this but the microscopic detail level necessary to work on this, particularly as it involves Intellectual Property (the memory interface) that I didn't write and which is in Verilog - a language I don't know. 

RAM retrieving data but not yet transferred up SPI link - good progress

LOGIC ANALYZER CORES MONITOR READ AND WRITE OF MEMORY INTERFACE

My two integrated logic analyzer cores, one operating at the 4:1 speed of the memory interface used to clock in requests and grab data from memory, the other operating at my general logic frequency, were useful in spotting the state of signals as my logic dealt with SPI link requests to load and unload data from a target sector of the virtual cartridge image as held in the DDR3 RAM on the FPGA board.

I was able to see that the data was properly written into the memory interface and that information came out later when reading the same addresses. I will need this facility both to feed the SPI link during unload operations for virtual cartridges but also to feed the signals into the disk drive controller when we are simulating the head signals if it were a real cartridge spinning on the drive. 

CONTENTS OF READ DATA BUS FROM INTERFACE IS FLEETING

From the analyzer I could see that we only had valid data from the memory interface for the two clock cycles when the app_rd_data_valid signal is asserted, telling us we have good data. It then reverts back to the wrong data. Thus the timing of when we latch in the app_rd_data bus is critical to successfully getting memory contents out to the functions that need them. 

ADJUSTING TIMING TO CAPTURE THE DATA IN ORDER TO FEED IT TO SPI LINK

The fix seems pretty straightforward, so I will implement it and enter a new round of testing. Ideally, we will grab and hold the memory contents, pass it properly to the SPI link state machine, which will properly load it into the SPI slave link module itself where it will be properly clocked up to the Arduino. 

Bizarre clock domain discovered by logic analyzer core - digging deep

 CROSS CLOCK DOMAIN AND RESET SIGNAL DESIGN COMPLICATIONS

My design has multiple clock domains which brings with it the challenge of synchronizing signals going across domains. The board has a 100MHz hardware clock (also 12MHz but I didn't use that) which generates various clocks for the RAM controller and another (50MHz) used for my general logic. The controller gets 100MHz and 200Mhz, then it will generate a ramclk of 25MHz for driving my memory interface logic. Finally, the Serial Peripheral Interface (SPI) link has its own 4MHz clock which runs intermittently. 

We therefore have five active clock domains driving logic, plus two hardware clock domains one of which generates most of the others. Even if two of the domains would be at the same frequency, they are not in phase nor do they have aligned clock edges. 

Any external signal such as the SPI data lines but also all the signals from the 1130 disk drive, should be synchronized as they might otherwise be changing right near a logic clock edge leading to metastable state errors. As such I had a pool of synchronizers to make sure very signal in a particular clock domain changes only at clock edges. 

Too, I needed to reset various state machines and elements in a proper sequence, thus there are reset signals generated in steps - original, a FIFO clearing state, and a reset for the logic running under the ram clocks. In that ballet of startup steps, I had an issue which resulted in my main ram handling state machine stalling. This didn't occur in the regular simulation, but when I did a functional simulation with the post-synthesis design, I was able to dig out the issue previously. The fix was easy even if finding it was not. This was several rounds of testing ago, but interesting to understand the wrinkles involved in this sort of project.

INTEGRATED LOGIC ANALYZERS ASSOCIATE SIGNALS WITH CLOCK DOMAINS

When I select signals to watch with the integrated logic analyzer cores, the Vivado tool chain will determine the clock domain and build an analyzer core for each clock domain which has signals to monitor. I decided to watch the SPIbyteout bus signal which is the eight bits that are sent to the SPI protocol module to shift out to the Arduino. This was the last point outside the SPI module logic and thus I could monitor to see whether I was passing the RAM values properly.

STRANGE ASSOCIATION OF SIGNAL WITH CLOCK DOMAIN

The toolchain built a third logic core for the 100 MHz clock domain. That is only passed into the memory interface module and not involved in any of my logic. There is no way that the bus value I want to monitor should be tied to that clock domain. This suggests some subtle error which is the root cause of my difficulties but it is a very opaque sign. 

The logic that is generating SPIbyteout is clocked by the 50MHz clock domain but also tests an unsynchronized input from the SPI clock domain (SlaveSelect). That is a flaw that I have to correct, altering the logic to remove any reference to SlaveSelect or at least synchronizing it properly. In spite of the issue I see, I cannot imagine how that produces the errant clock domain assignment by Vivado. 

Data returned from the response FIFO correctly, not getting up to Arduino properly

NEXT SET OF RUNS SHOW THAT MY RESPONSE FIFO IS WORKING PROPERLY

I set the internal logic analyzer core to detect when I was reading the sixth word of the sector, by triggering on the return pattern 0006 and then watching all the related signals such as the SPI state machine.

The value 0006 is clearly detected and the state machine moves forward, thus my problem lies somewhere past the FIFO. It may be in the timing of passing the data word to the SPI outbound routine, in the way that the message is encoded on the SPI link itself, or on how the Arduino routine is detecting the results. 

I will reimplement with new watched signals to view the handoff from FIFO to the SPI out routine and inspect the word passed to the SPI logic. It would be wonderful if I could directly monitor the SPI link from a logic analyzer core, but the SPI clock is not continuous thus I can't start a logic core that is driven by the SPI clock. All signals in that clock domain, MISO, MOSI and SCLK itself, are thus inaccessible by the internal analyzer cores. 

RAM writing and reading good, data not getting back to SPI link state machine correctly

INTEGRATED LOGIC ANALYZER CAPTURED RAM OUTPUT

My load transaction stored the ascending integers in the word addresses of the sector, e.g. 0001 for word 1 and 0002 for word 2. The unload transaction read the same sector back and shipped the value back on the SPI link.

While the upstream data appeared to be gibberish, the values coming from the memory interface was indeed the same ascending integers as I had written and they were properly set up in the data in field for the FIFO that would transport that result back to to normal FPGA clock domain for use by my SPI link. 

THE ISSUE IS SOMEWHERE IN THE RETURN OF THE DATA

The data is set up properly in the FIFO that transports results. I will now focus on watching the FIFO operate and judge the correctness of the data returned on the regular clock side of that FIFO. If that is good, I could still have a problem capturing that into the SPI link state machine for the outgoing byte that is sent up to the Arduino. 

I reconfigured the logic analyzers to focus upon that data I need to watch for the response FIFO and my SPI link state machine. This does cross clock domains, which means I can't watch both the data entering the FIFO and the data entering the FIFO in the same logic analyzer core. I can, however, trigger on the FIFO empty flag to at least watch each result being presented in my FPGA logic clock domain. 

QUICK RESCUE OF SOME HARDWARE

I learned of a mainframe and some related hardware that was at risk of being scrapped as a person nearby had to clear out a storage unit. He was no longer interested in restoring the system and rents for the storage space had doubled suddenly. 

Today I had a small moving business I worked with previously meet me at the storage unit and haul those boxes to my workshop before the end of October. It is an IBM Z9 BC mainframe, almost 1,700 pounds which was a beast to get rolling up and down ramps for transport, plus a 3490E tape system and some ancillary parts. The tape unit was no walk in the park either, just easy in comparison to the behemoth. 

IBM Z9 Business Class mainframe

IBM 3490E 36 track tape cartridge drives