Adding diagnostic output from FPGA via Neopixel arrays

NEOPIXEL ARRAYS

Neopixels are tricolor LEDs with an integrated controller, wired in a long string. The single data line that connects all of the LEDs allows a string of pulses that address every LED so that one can control any or all of them by sending the proper pulse stream. 

Each of the three colored LEDs inside a single Neopixel takes an eight bit brightness value, thus we are shifting a 24 bit quantity into the Neopixel to set the brightness of the three colors. When we have a long string of Neopixels, we begin shifting out the 24 bit value for the last Neopixel, followed by the next to last and so forth until the last value we shift is for the first Neopixel in the string. 

The bits flow through each Neopixel and out to the next, thus the first Neopixel has seen the bits for all its successors flow through it. The final Neopixel only sees the 24 bits intended for it, while the next to last sees 48 bits. 

I bought arrays of Neopixels in a grid of 16 by 16 positions, these are addressed as 256 linear units. They can be chained, thus I will start with two arrays that consist of 512 addressable Neopixels. 

DRIVING THE LIGHTS

The wire that runs through our 512 Neopixels is clocked at 800KHz sending out 24 bit values at the rate of about 33,333 positions per second. The string of pulses begins with a minimum low interval of 50 microseconds to reset the devices then the pulses go out. The proportion of on versus off time for each of the bit positions encodes whether that bit is a 1 or a 0. 

Each bit position is 1.25 microseconds long. The full 24 bits takes 30 microseconds and my string of 512 Neopixels will therefore consume 15.41 milliseconds to set their values including the initial quiet period. If the chain is updated as fast as possible we will have a refresh rate of almost 65 per second. 

The single data line that drives the chain of Neopixels requires 5V logic levels, thus we will be passing it through a level shifter from the 3.3V of the FPGA. The Neopixels are also fed +5V and ground to feed the LEDs themselves.

USEFUL ONLY FOR STATIC OR ONE TIME LATCHED SIGNALS

Some debugging requires me to capture fleeting signals or to see the time relationship. These would be quite difficult to capture via the light arrays, but any static, long lasting or latched signals can be routed to the arrays for display. As an example, I can use one position for each state of a finite state machine I want to monitor. If the FSM sticks in some position I will immediately see which one by the light that is illuminated. 

I can also track interesting values such as the cylinder and track active for the drive. I may latch the last address on the SPI link as another idea for helpful data to show on the array. I only give up one output pin to drive all 512 lights, a good tradeoff even with the limitation to slow changing or static information. 

Aha - spotted a defect in my logic that was stalling the SPI link state machine

SPOTTED A FLAWED APPROACH IN MY STATE MACHINE

I should have realized that since I saw the two RAM access controlling commands - RAM to drive and RAM to SPI - working successfully but the load and unload commands stalling, the error had in parts of the SPI link state machine that weren't involved in the two good transaction types.

It was then that I spotted my error. I was triggering requests to read or write RAM and then waiting for a signal that was generated by the RAM state machine, but the done signal is in the faster clock domain unique to the RAM and not valid in the general logic clock domain. 

My simulation worked okay but there must have been just enough phase difference in the real world that it never caught the go-ahead signal. That pulse was one cycle long in the 100MHz clock domain but if it wasn't high at the beginning of a 50MHz clock cycle we could miss it. Apparently we did miss it consistently. 

Imagine that our clock rises a few nanoseconds before the RAM clock rises. When we look at the rising edge of our, the done signal from RAM has not yet been emitted. It goes high while we are in our 50 ns clock period, but we only look at our rising edge. The RAM clock advances twice as fast thus it has already dropped the done signal before we get to our next clock edge in our clock domain. Signal missed entirely. 

TESTING THE IMPROVED LOGIC

The solution was to emit a signal in our clock domain - from the FIFO that fetches the responses from RAM. That FIFO is loaded under the RAMs clock and thus is synchronized with the done signal I originally used - it works fine. The output of the FIFO is running under our slower general logic clock and the state machine for that side of the FIFO can emit a new type of done signal in the proper clock domain. 

The SPI link state machine now watches for the new done signal and will see it because it has a common clock domain. This should avoid the deadlocked condition we experienced earlier. 

While I was looking over the logic, I came up with an improved method of pulling the two bytes off the SPI link for each word, recoded it into two state machines, one for each direction on the link, and was happy with the simulation results.

I set up to test in the real world once again and this time I found it stalling only during the unload transactions. Will be digging into this.

First word x0009 - unload transaction

Green is RAM fetch completion before transmitting first word

FINISHED BUILDING THE LEVEL SHIFTER PCB AND CABLE TO WIRE WRAP ADAPTER

Level shifter board

Ribbon cable to wire wrap adapter

Watching state machines and hunting for defects

INDICATIONS ADDED TO SEE STATE MACHINES NOT AT IDLE

I set up a set of indications to display on the four LEDs for each of the state machines, illuminated when it is not at its idle or starting position. That way I can quickly find any machine that is stalled or active when it shouldn't be. 

I also set up the red color of one of the multicolor LEDs to light when ANY of the state machines are not at idle, a very immediate visual cue that something odd is happening. While it is processing transactions it should turn red, reflecting the activity, but then turn off in quiet times. 

OBSERVATIONS DURING TESTING

I could see that the main SPI transaction state machine was not returning to idle. This made no sense as every state where it can wait includes a top priority test that if the SPItransaction signal is off, it returns to idle. I will be instrumenting this more thoroughly as this is consistent with the flawed behavior I was seeing - one attempt at accessing RAM but no more than the first was attempted. 

Obviously I will be studying this one closely for any situation that might cause it to lock up, but also planning on the best instrumentation I can install to locate the problem. I have some ideas on how I can accomplish this without having to make multiple 25 minute runs reprogramming the FPGA board in between investigations. 

PCBS ARRIVED AND ONE NIT DISCOVERED

My PCBs arrived and look great. There is an extraneous REF comment on the silkscreen next to the four mounting holes, but that is irrelevant to its functionality. When I began looking closely at my connectors and cables for the links to the 1130 drive electronics I realized that I had bought IDE connectors not generic 40 pin connectors.

This is bad because there is a blank pin position on an IDE cable, right where I intended to run a signal. Both the IDE cable and the connector I bought to anchor the wire wrap lines has this missing pin. Fortunately the connectors for the PCB itself are full 40 pin units.

I have new cables coming in two days which resolve the blank position issue, but I do have to work around the missing pin on the anchor board for the wire wrap leads. It appears that I can press in a pin and solder that onto the board, restoring the connector to full functionality, but it will depend on what pins I can scrounge up. 

Finished PCB

Digging into stall in state machines driving RAM access; switch cover mounted; header for backplane wiring prepped

BACKPLANE WIRING HEADER PREPPED

The connection for the Virtual 2315 Cartridge Facility to the disk drive electronics inside the 1130 is by wire wrap to the backplane on the drive, with those signals routed over an IDE ribbon cable to the level shifter board I designed. 

Wire wrapping uses individual 30 gauge wires which I will solder to a header with an IDE connector. I found a gender changer for IDE that uses a small PCB between the two connectors. By unsoldering and removing one of the connectors, I would have solder pads to attach the wire wrap lines. 

I completed the removal of one connector and cleaned the pads, ready to attach wires. I do need to design and install a mount to hold this firmly near the backplane, otherwise someone yanking on the IDE cable might rip the wires off the backplane pins or bend the pins. 

SWITCH COVER MOUNTED ON VIRTUAL/REAL MODE SWITCH

I installed the red safety switch over the DPDT switch that converts the disk drive between real and virtual mode. To protect the disk cartridges and heads, we want to stay in Virtual mode unless we are exceedingly certain that we want to risk the actual heads flying on a platter. 

In real mode, the pick signal from the electronics activates the solenoid, lowering the heads onto the spinning disk surface and via a microswitch indicating to the electronics that the load is complete. In virtual mode, the pick signal is instead sent to the Arduino in my facility and the Arduino tells the electronics when the heads are 'loaded', even though they don't actually move down in virtual mode since the solenoid is not connected.

LOOP TIME TO EXPOSE SIGNALS FOR SCOPE MONITORING IS 25 MINUTES

The FPGA board has only a few exposed input-output pins that are not used by my logic, to which I can route various state information to be displayed on a scope. Currently I have access to only five of them, thus every time I have a new area I need to examine, I have to select five signals or signal states, then run the Vivado toolchain to reprogram the FPGA board. 

The longest step by far is the routing process. The tool synthesizes my VHDL into logic elements, it places these on locations across the board, then generates the signal routing between all the elements to complete the circuits I have designed. Routing hundreds of thousands of gates is a slow process, well over twenty minutes on a pretty fast 'gaming' laptop. 

This means that each time I watch the current set of diagnostic signals and form a hypothesis, another group of signals must be monitored to progress onward with debugging. That means another 25 minutes staring at the wall, waiting on the toolchain. 

PROCESS TO DRIVE RAM ACCESS IS COMPLEX

The memory controller for the DDR3 RAM on this board operates with multiple clocks at 100 and 200 MHz, actually slow compared to the speed this memory can attain. It produces a 50MHz clock that drives all the rest of the logic on the FPGA.

However, this means that signals are changing in different clock domains, an issue as they may be switching right at some boundary where a logic gate is being clocked. To deal with this, clock domain crossing techniques are necessary. In this case, I am using FIFO logic built into the FPGA chip which operates with different clocks on each side of the queue. 

Thus I need state machines to push a request into the request FIFO, another to see that a request entered and pull it out under the memory controller clock. A state machine sees that request and drives the signals to the memory controller to accomplish the read or write. 

The results of the memory access appear in the controller clock domain, so they must be pushed into a response FIFO by yet another state machine. The output side of the response FIFO, running in the FPGA logic clock domain, sees a response, pulls it and puts the data (for a read) on the appropriate register for further processing. 

These must all run appropriately and not deadlock, smoothly handing requests through to the memory controller and pulling out the responses. This all seems to work in simulation, but I wasn't able to accurately model the different clock domains and transitions for my simulation, as it works in sim but not in real life. 

SEEING ONLY ONE PASS THROUGH RAM STATE MACHINES

What I have observed so far is that we get one request for RAM access, which passes in as a request, the memory controller signals completion by the ram_done signal, and the request FIFO is reset to its idle position. 

However, there is only one request for RAM. We never move on to send the next read or write and are stalled somewhere. I am going to have to figure out a way to instrument the various state machines to my diagnostic outputs to see which state machine(s) are stalled. 

SPI link debugged, transactions working; next up is testing whether RAM is properly written and read back

FINAL TESTS IDENTIFIED PROBLEM

As I though the issue was resynchronizing after the first transaction had ended. I spotted the issue which wasn't caught by the particular signal timing I had set up in the simulation. The fix was obvious and quickly made.

RESOLUTION TESTED WITH DRIVE ON AND OFF COMMANDS

My test setup sent a repeating sequence of transactions that flipped the FPGA between the SCspi and SCdrive states, that is with the RAM access controlled by the SPI link or by the drive emulation side of the logic. It was observable by the color of one of the color LEDs, where the blue color would be on when the RAM was set to SCspi. 

Indeed, it nicely switched between the two states, every time I started it up. I am happy that the link is working well and moved on to check out the access to RAM from the SPI link side. 

TEST SENDING DUMMY DATA TO RAM AND READBACK

I set up a routine that, during startup of the Arduino, will send a defined pattern to the RAM, a different pattern to a different location, then read back the original sector and verify it. If that works we know that we are accessing RAM, uniquely addressing it and the Load and Unload transactions are working at least to some level. 

The Arduino code that is testing the data coming back from the unload is indicating a flaw. The FPGA side is happy with the format of the transaction, thus not indicating any error. However, the Arduino side is giving the indication that the data was not what was expected. 

I looked superficially at the MISO and MOSI lines during the transactions - the pattern I saw coming back from the Unload transaction appeared to match what I had sent during the first Load transaction and not what was sent in the second Load. I was only looking at one word near the end of the transaction and therefore may have missed an error with the first or 321st words, or it may not really have matched.

I am back at my home working out better diagnostic information. I suspect that I will have to use the USB serial link to my laptop to view better information about what is coming back on the link. I will also switch the five diagnostic outputs of the FPGA board to signals that will show me key memory controller signals. 

Virtual/Real switch being wired into drive

More battles with SPI link debugging; built bracket to hold the Virtual/Real mode switch for the disk drive

TWO STEPS BACKWARDS IN TESTING

Because of obligations I had elsewhere I only got to the shop late in the day to run the tests with my new and improved diagnostic outputs. When last I tested, the first transaction received was successfully executed, switching the FPGA RAM access to the drive from the SPI link. However, it seemed to hang up at that time and didn't indicate the subsequent transactions, nor execute them.

I thought I had made changes that would ensure resynchronization to avoid the observed behavior. When I hooked up and tested, however, I no longer saw the transaction command code latching at all. My diagnostics showed that the SPI transaction state machine was not reaching various states I had instrumented, nor was the state machine that assembled the two transmitted bytes into a 16 bit word. 

CAUSE WAS A MINOR TYPO, DISCOVERED AFTER INVESTIGATING SUBTLE MESSAGES

Because I mistyped the state of the SPI machine that would trigger the byte assembly machine to start, they were deadlocked. This is why neither reached the states I was monitoring. When I looked at the warning messages from Vivado when it was synthesizing my logic, I found messages that it was trimming SPI link FSM One-Hot register bits 18 to 2 from the design, as well as most of the FSM One Hot Register bits from the byte assembly state machine.

FSM is Finite State Machine, and it is implemented by setting bits in a register. The One-Hot method assigns one bit to each possible state of the machine, which speeds up testing to see if the machine is in any given state as only one bit is interrogated. 

This message was subtly telling me that my state machine only had a few states it could ever reach, therefore there was no need to implement hardware to hold the other one-hot bits. It doesn't say "Your FSM can never reach all the states", it refers to FSM One-Hot Register bits.

I did agree that it was going to deadlock and verified that in a simulation. I then corrected the typo, simulated to successful outcome, then synthesized the logic once again. When it was complete, I looked VERY carefully over the hundreds of somewhat obscure warning messages to be sure that nothing like this was occurring elsewhere in the logic. 

Next test session I hope to have more success. I have prepared a test routine so that if I can alternate flipping the FPGA between drive and SPI access to RAM, I can move on to write data into RAM in a few locations and verify that one of them is read back as written. 

MODE SWITCH FOR DISK DRIVE MOUNTED ON BRACKET BEHIND THE DRIVE

Most of the connections I am making to the internal disk drive of the IBM 1130 are passive, they simply detect what various signals are doing. There are two that are passive as long as I don't assert them by pulling them low - the raw read pulses I produce and the write error halt I inject if errors occur during capture of written data from the CPU to the drive.

However, there are some active changes that must be made for my Virtual 2315 Cartridge Facility to work. I must block the signal from the drive electronics that activates the head loading solenoid. I must instead route that signal (pick) to the Arduino. The microswitch that detects when the heads have physically been lowered onto the platter must be unwired from the drive electronics and instead the Arduino must cause the drive electronics to believe the microswitch has activated. 

I want the drive to be capable of working in either mode - Virtual or Real - thus there is a toggle switch with those two settings. In the Real mode, it connects the pick signal to the head loading solenoid and leaves the microswitch hooked to the drive electronics. The heads will really load down onto the platter in Real mode. In Virtual mode, the active changes in the prior paragraph are implemented so that the Arduino sees the pick signal and it simulates the microswitch closing. 

I am using an aircraft style toggle switch, one with a red plastic guard that protects against inadvertently switching the switch to Real mode. The operator must lift the red guard to throw the switch, otherwise it stays in Virtual mode. I chose this scheme because of the risk of head crash if the heads are actually down on the platter surface. 

I formed a bracket to hold the switch in a convenient location just behind the drive, then drilled holes in the machine frame for the mounting screws. I have begun wiring the switch to the drive side - the solenoid, pick signal line, microswitch and microswitch signal line to the electronics. I have not yet connected the wires that will go to the Arduino as they have to be connected to the level shifter board which is still in fabrication in China. 

SPI link testing continues with some success; Wiring the Virtual/Real switch into the IBM 1130

DIAGNOSTIC INFORMATION I WILL WATCH

The SPI state machine moves through a number of states from the first to the 325th word being exchanged. To verify that it is reaching the end properly I emitted some signals that can be displayed on the scope alongside the end of transaction (SPItransaction going low). 

The state machine reaches a state SBcheckA when it has finished with the 321st data word and is preparing to send and receive the checksum of those data words. It then passes through SBtest where it verifies if an error free transaction has been received. Near the very end it reaches SBflagB state. 

I am also monitoring the command it believes it has received. One line goes high when it sees the bit pattern SCdrive that sets the RAM access to the drive side. Another line is the default state SCspi for when the RAM access should be connected to the SPI link. If an error condition is detected during the transaction, I would expect to see the command switch from SCspi to SCdrive for a while then revert before the end of the transaction. 

TESTING RESULTS AND OBSERVATIONS

I worked on various issues including not setting the command state to SCspi at power on reset, something that would have no impact on the system in real operation but needed repair to have a clean design. With the new instrumentation I discovered a small flaw, corrected it, and was pleased to see that the FPGA board switched the RAM over to drive mode from SPI mode - a successful transaction completion.

It appears however that something gets wedged after that one transaction, as I never latch in the subsequent command to switch back to SPI mode, while I can see the properly formatted transaction being received. I changed the diagnostic outputs and watched the state machine working well for the first transaction and it should be resetting to the idle state when the SPItransaction signal is dropped. 

Next up is a careful review of all the state machines and how they reset. It may be that I need to control how they move out of reset to avoid whatever is taking place to lock it up. I will also produce some new diagnostic output signals to help me find and resolve the issue causing this behavior. 

SWITCH BEING WIRED TO SWITCH DRIVE BETWEEN REAL AND VIRTUAL MODE

My modification to support the Virtual 2315 Cartridge Facility is a switch that reroutes signals that ran to the Head Load solenoid and the Heads Loaded microswitch. In the Real mode, signals are wired as usual, but in Virtual mode, the solenoid is out of the loop. 

For Virtual mode, the signal that would activate the solenoid is instead routed up to the Arduino inside my facility. The Heads Loaded signal line to the drive electronics is disconnected from the microswitch and instead routed to the Arduino. This lets the Arduino see that the drive electronics is requesting the heads to be down on the platter surface and the Arduino can lie to the electronics, telling it that the heads have indeed come down. 

I bought some final connectors and a mounting plate to install the switch in position on the frame of the IBM 1130 behind the disk drive. The remainder of the wiring and the physical installation will take place over the next few days.