Thursday, February 28, 2019

Debugging a Core Memory Shield for an Arduino

I built a kit for a 32 bit core memory, fitting as a shield onto an Arduino. This was built and sold by Jussi KilpelainenMy kit wasn't working properly so I dug into the unit to figure out what was malfunctioning.

The design consisted of 16 driver transistors to switch the ends of the 4 X and 4 Y addressing lines, connecting one end to ground and the other end to 5V. By routing one set of address lines in the reverse direction, those drivers could uniquely flip any of 32 cores to the on or off state.

A sense wire woven through the 32 cores in a diagonal pattern balanced out the pulse coming from the X and Y address lines, improving the signal to noise ratio for detecting the pulse if a core were to flip state. These wire ends when into two comparators (since the pulse could be positive or negative depending on the particular sense wire path) to produce an output only when a pulse above a certain threshold is detected.

The two comparator signals are ORed together to produce a 1 condition when a core was seen to flip. The Arduino program communicates with a user over serial I/O to get requests for specific core operations. It sets up the address for the desired core, toggles another line to indicate read or write, and then pulses an enable line.

A CPLD device decodes the binary core number and the read/write line into outputs connected to the bases of the 16 driver transistors. Thus, it will route a current in the intended direction on one pair of X and Y lines.

Read core works by routing the X and Y current to flip the addressed core to the zero orientation.  If the core was already in that orientation, no pulse is seen on the sense output. If, however, it was previously in the one orientation, the flip in state generates a sense output. Reading, in other words, is destructive, setting the core to zero in the process of testing its state.

Writing into a core is done by routing the X and Y current in the opposite direction, to flip that core to the one orientation. The output of the sense amp is ignored. In most core memories, the X and Y current is sent to the core and a third wire, the inhibit line, will have a counter current to block the flip if we want the core to be set to zero. No inhibit current, we get a one.

This particular kit does not do that. To write a bit, it sends the X and Y current in the direction requested, thus either flips it to one or does a read to flip it to zero. Sense output is still ignored. The logic equations that were programmed into the CPLD determines the direction of current based on reading or writing and the five bit binary address that indicates core 0 to 31.

The CPLD formulae are pretty simple. They look at the binary address, the read/write signal and the enable line and yield a + or a 0 output for each transistor. As an example, to address bit 31 requires a current running between terminal P3 and P12 (the leftmost column of the 4 columns in the core array). It also requires a current running between terminals P10 and P21, the two row of the array.

Because of the special trick that implements 32 cores with a 4 x 4 addressing, the wire from P21 is connected to P20, the fifth row of the core array, whose other end is P27. Thus, if we fire up P10 and P27 in one direction, the current in the top row is reversed in the fifth row.

Cores are placed on the intersection of a row and column wire, oriented at a 45 degree angle. The orientation alternates, so that the core at the left column and top row is tilting to the left while the core at the left column and fifth row is tilting to the left again, but rows 2, 4, 6 and 8 are tilting to right.

Wiring of 32 core memory array with driver circuits
. To address core 31 with current flowing down through the column and from left to right across the row, two pairs of transistors must fire. To get the horizontal current correct, it is necessary to inject +3.3V into P27 and sink it to ground at P10, by firing Q15 and Q4. To get the vertical current correct, we have to inject 3.3V into P3 and sink P12 to ground by firing Q5 and Q10.
Currents combining to flip core 31 in the alpha direction
If we say that this flips the core in direction alpha, then to flip it to the reverse state beta we have to inject 3.3V in P12 and sink it to ground in P3 by firing Q9 and Q6, while injecting 3.3V into P10 and sinking it to ground at P27, firing Q3 and Q16.
Opposite current flow to flip core 31 in beta directions
Note that cores 31 (top left corner) and 27 (fifth row of left column) are both affected by our currents flowing through the column and row. Since we sent current down the column and from left to right in the top row, the current flowing in the fifth row is from right to left, or counter. The two currents flowing in core 27 are in opposite directions, thus they don't combine to flip the core while the same currents flow in the same direction for core 31, flipping it.
Reversed current directions through cores in top and fifth row
I hooked up the logic analyzer to the bases of Q5, Q6, Q9 and Q10 to watch the conduction when I issue a read or write for cores in the left column (addresses 24 to 31). Nothing showed up - except the enable pulse.

Since the logic analyzer was digital but I was monitoring the base terminal of transistors whose level might not be right for the analyzer, I moved over to my oscilloscope. I should be seeing at a bare minimum 0.3V movement from +3.3V for Q5 and 0 for Q6. That is what I saw.

The driving transistors Q5 and Q6 are hooked up through a bridge rectifier, although you only saw half of it in my diagrams above. The other half is connected to the number two column, thus Q5 and Q6 will energize both of the left two columns. Only the left of those two will have its current sunk or fed by Q9 and Q10, while Q11 and Q12 handle the second from the left.

This does mean that I will see Q5 or Q6 fire for addresses in either of the left two columns, that is for addresses from 16 to 31 inclusive. This is clearly visible on the scope.

However, I did see something interesting when I watched Q5 and Q6. When I do a read of any address, I see Q6 fire to sink current from P3 and presumably Q9 is firing to source 3.3V to the bottom of the column at P12. Soon thereafter, I see Q5 fire to deliver 3.3V from the top.

Looking over the Arduino program and the sense pulses being detected, I understand why. The logic of the program sends current in the beta orientation, which tells us this is the orientation that represents a zero bit value.
Core orientation to represent a zero value
It appears that the sense wire sees an output pulse, indicating that the core was originally in the alpha or one orientation. The program reverses the current to write the one value back into the core.
Setting core to one value orientation
The pulse looks weaker when driven by Q5 and sunk by Q10 than when it is resetting to zero driving with Q9 and Q6 sinking. I watched pairs with the scope to see if this was an essential characteristic or some flaw. It seems to be by design, or at least is symmetric for both directions of current flow.

Next up I move on to all the other pairs of drive lines to verify that all the transistors are emitting pulses appropriately. However, I notice that whether I write a zero or a one value into the core, the Arduino software sees the sense line pulse being output. Clearly this is wrong. That is, I should not see a flip for a succession of writes of the same value. I also don't see the enable line toggling like it should, another wrongness.

First I checked the voltage bias on the two comparators for the sense circuit - the should put a 20mv bias holding each of the comparators off, but when a pulse arrives on the sense wire that exceeds that value, it would flip the comparator output on.

The voltages were 20mv and when monitoring the outputs, I saw the blip of the comparator firing under appropriate conditions. That is, I would write a 0 to the core and then read - no pulse. Write a 1 to the core and then read - pulse. In fact I monitored the pulses even on the writes, seeing that writing the same bit value twice gave no pulse on the second try, but whenever it changed value, one or the other of the comparators gave me a pulse.

This is fairly good news, as it shows that at least for core 31 which I was addressing, it was reliably writing and the sense output from the comparators were correct as well. I then moved on to monitor the output of the CPLD which is a flipflop that gets set by a comparator pulse and is reset at the start of each core access (by the rising edge of the enable input).

There is an LED wired to the CPLD which is turned on whenever the enable input goes high - that works properly, another confirmation that some of the CPLD is working properly. However, there are two test outputs, one of whom gives me a pulse whenever enable input goes high, but the other should give me the OR of both comparators.

Instead, it floats high and barely dips down when pulses are arriving. This is incorrect behavior and indicates some flaw inside the CPLD, either hardware damage or loss of programming. I moved on to the read flipflop output which is delivered up to the Arduino to provide the results of reading a core. It is fixed at high, never going down even when the Enable input rises where it should be temporarily reset. It acts as if the flipflop function inside is frozen on.

This is consistent with the application software view, which reports a 1 as the output of every bit. I have to get a replacement CPLD (Lattice ISPMach4A 32/32 in a PLCC 44 form). The VHDL is available but I needed to download the Lattice toolchain, synthesize the design, then work out the JTAG loading method. I should have the new CPLD chip by the end of the week.

The software is putatively installed but when I try to synthesize I get very basic error messages. Doing some research clarified that this won't run on any 64 bit operating system, so I have to move it over to an older desktop system just to get it to run. After getting a new license and installing it on that system, I tried again to synthesize the CPLD code.

Same error again, but this time I simply set it to run as administrator and success was at hand. A bit confusing at first as the generation of the vendor neutral programming file (.svf format) requires use of a different program and wasn't run from the project navigator. Soon, however, I had my coremem.svf file ready to program the device once it arrives.

I have a USB-JTAG programming cable somewhere, so all I need to do is insert the new CPLD in the socket, plug the cable onto the JTAG header on my shield, run Lattice's ispVM and the chip should be programmed. I will do some simple testing to ensure it went well, then resume full testing of my core memory operation.

In the meantime, I removed the Arduino to see if the CPLD output of the flipflop that records a sense wire pulse might be stuck on. It was off by itself, thus I turned my attention to the two inputs on the CPLD, one each from the threshold comparators, to see if one of them was jammed on or false triggering.

With the Arduino back in place, I monitored the enable pulse as a trigger and watched the two comparator output pins - they were good. I then put the probe on the CPLD inputs that are linked to the comparators. They seemed good but somehow I was not seeing constant 1 values in the serial terminal any more!

For a while, the board was now working properly. I did some pattern writes to all 32 cores and found oddities. Writing all zeroes gave me all zeroes on read, repeatedly. However, writing all ones would then read back consistently as an odd pattern groups of bits on but groups of mainly four bits at zero.

I decided it was time to put the scope on each pair of driver transistors involved in the core, making sure that a source of 3.3V and a sink to ground worked properly when a write of the associated value is attempted. I hooked up the first pair, began adjusting the scope to trigger on the low voltage swing of the base, but then disaster struck. The board was back to reading 1 for everything regardless of what I wrote.

Something is flaky here, hopefully the CPLD since I am expecting to flash another one when the various parts arrive. Still, I could proceed with the pairwise test of all drivers for the eventuality that the board goes back to normal operation.

Their are 16 driver transistors, thus 8 pairwise tests I had to accomplish. These all worked properly when addressing their related row or column. I checked for dual pulses in the intended pair, but also saw some correct cases where one or the other fired.

This is a valid case since the drivers use some diode magic. A transistor might drive +3.3V to both of the left two columns (1+2), but the other transistors that sink the current is hooked to columns 1+3 and 2+4. Thus, if the sink transistor for 2+4 fires while the source transistor for 1+2 fires, there is current only flowing through column 2, since the circuit is incomplete on columns 1 and 4 even through transistors are switching.

I moved on to looking at the current cross the four current limiting resistors. These handle rows 0+2+6+8 and rows 1+3+5+7 for one pair and columns 1+3 and 2+4 for the other pair. To watch these, I put probes on each side of the resistor and set up math on the scope to show me the differential in voltage.

I was able to see current flow in both directions by suitably setting the address and the bit value to write. There were four resistors on the board, R22, R20, R21 and R19. I observed proper current pulses across all four resistors, in both directions. This tells me that the row and column wires are properly connected.

At this point, I have to conclude that all my addressing/read/write drivers are doing the proper thing and should be storing any bit pattern I want into the core array. Any difficulty must be in the sense circuit and CPLD latch that records the state of the core after a destructive read.

As a cross check, I put the scope leads on the two comparator outputs and wrote both 0 and 1 values into all 32 cores, watching for core flip signals. Writing the same value twice should see no output on the second try, but the second of each dissimilar pair should always show a blip. If this worked, I would have isolated the problem to the far side of the comparator, e.g. it can only be in the CPLD or the Arduino itself.

The comparators worked exactly as I expected them to, but somehow the CPLD is seeing them as on. The outputs stay flat and low but the CPLD latch output is not behaving as it should. The VHDL defining its function has it reset to off with the rising edge of the ENABLE signal, which also gates the X and Y driver transistors to send current to the core in one or the other direction depending on desired data value.

What I see is that the latch drops by about 400mv for 1 us, in a parabolic dip, then flatlines right back at full on level of 3.3V. Thus it appears to always have read a value of one, since we write a 0 value and watch the sense to see if it flips, thus having been one before.

Top is the CPLD flipflop output at 200mv per division, bottom is the comparator output
The first scope image is plausible, but this one shows the flipflop latching with no input
That made no change at all to the behavior. All eyes are back on the CPLD. To make this definitive, I used some jumper cables to isolate the Arduino from the Shield, rather than plugging them together. That allows me to watch pin 9 without it being attached to the Arduino at all.

No change at all. The CPLD is sitting at 3.3V with the teeny dip when the enable pulse arrives, rather than working correctly. The new one should arrive in a few days and I can replace it. In the interim, I can breadboard together a flipflop to replace the portion of the CPLD which is clearly defective.

Our ENABLE line is hooked up as the clock of the 74LS74 D flipflop, with a 0 wired to the D input. Since this is positive edge triggered, it will reset the flipflop whenever the ENABLE pulse rising edge arrives.

The two comparator outputs are connected to a 74LS02 NOR gate, whose output acts as the asynchronous PRESET for the D flipflop. Thus, whenever a pulse arrives from the NOR gate, it will set the D flipflop off. It will remain that way in perpetuity until our next rising edge of an ENABLE line.

My temporary flipflop circuit to bypass defective CPLD logic
The two horizontal red lines on the top will be connected to the outputs of the two comparators on the shield. The horizontal orange line is connected to the ENABLE signal. The horizontal gray line is the output which is fed to pin 9 of the Arduino. Of course, +5V and ground must be connected for operation.

Experimental setup to substitute my breadboard for part of the CPLD
I hooked this all up and tested successfully. The core memory works perfectly now, confirming that it is a CPLD failure which blocks the shield from working by itself. Writing any pattern to the cores yields the correct output pattern and the various testing regimes built into the Arduino code prove it is solid.
Writing and successful readback of several patterns

My replacement CPLD arrived yesterday, but the programming cable won't be here until later today. Since it will be slam dunk to program the replacement chip and insert it into the socket on my shield, I will declare victory and post this entry now.

Replacement CPLD chip ready for programming and insertion into the shield
I did discover that the ISPLever Classic toolchain assigned the input and output signals to different pads that were used by Jussi, so I have to figure out how to constrain or guide the tool to do what I want. Once that is done, there may be further hiccups forcing ISPvm to load the programming onto the chip, but none of these are insurmountable obstacles.

2 comments:

  1. now there's something I never thought I'd see... Mag Core Memory interface on an Arduino Shield :-O

    ReplyDelete
  2. https://www.tindie.com/products/kilpelaj/core-memory-shield-for-arduino/

    ReplyDelete