Beginning testing of new PCB for 1130 MRAM board

NEW PCB READY FOR TESTING

I had spent many hours checking continuity for all signals on the board and eliminating the chance of any shorts to adjacent pins. On the testbench I verified that all the outputs are producing correct pulses and the parity generation produces the proper pulses. The new PCB was installed on the 1130 system where I will do the final testing. 

MANUAL LOAD AND DISPLAY TESTS

Using the rotary mode switch, I could set it to LOAD and push the Prog Start button to load the pattern in the console entry switches (CES) into the memory location set by the Instruction Address Register (IAR). Turning the mode switch to DISPLAY and pushing the Prog Start button will read memory and display its output on the Storage Buffer Register (SBR) lights. 

To set the IAR to a particular address, I set the address in the IAR, set the mode switch to LOAD and push the Load IAR button on the console. This way I could put multiple patterns into memory.

Using this, I verified that I could set patterns which would be read back correctly. Changing the CES and storing different patterns first, I went back and displayed the locations to verify that what I stored was what was returned, including correct parity. 

I did not yet set up the oscilloscope to verify that I wasn't experiencing any spurious retriggering of the read or write timer outputs. That was the main issue I was experiencing with the prior PCB. Off visiting family for a few days and handling other obligations but will get to this soon.

FILLING STORAGE WITH A CHOSEN PATTERN

Turning the rotary mode switch to RUN restored normal function. I then opened the top cover to access the six customer engineer toggle switches. Turning on the Storage Load toggle switch set up the machine to loop through memory repeatedly storing the value set in the CES into all memory words. I pushed the Start button to begin the filling process and used the rotary mode switch to halt the operation by momentarily switching it away from RUN.

I then turned the rotary mode switch back to DISPLAY and checked some locations. The location is set in the CES, Load IAR is pushed, then Start is pushed to read back the stored value. 

Some patterns led to a parity check stop. If I am still seeing spurious read pulses updating the 1130 Storage Buffer Register at the wrong time, that might be causing this issue. Alternately I might have some issue on the board that I didn't catch on the test bench. Using the scope will help me understand what is happening. 

Worse as far as I am concerned, it appears that during these stops location 0 is overwritten with the contents that were being fetched from the address with the error. I will need to understand this failure mode and block it - that might require an update to the design. 

Replacing some chips on 1130MRAM board based on testing

TOGGLING STORAGE READ AND OBSERVING THE SENSE OUTPUTS

I have the board set up with a switch that will drive a rising edge on the +Storage Read line, causing my board to emit a sense pulse for any data or parity bit whose value is a 1. The output has a pullup resistor to +3.3V and a sense pulse consists of a short pulse to ground for less than 100 nanoseconds, occurring roughly 800 ns after the rising edge of the triggering signal. 

SUSPICIOUS RESULTS

When the memory chip was not yet installed, most sense bits appeared to be a 1 and produced pulses, but not bits 2, 3, 4, 13 and 14. The two parity bits produced pulses when odd parity existed on the eight bits they covered - those were correct based on which bits were generating pulses. 

SEVERAL CHIPS COULD BE THE CAUSE

I removed the buffer chips to see if they were holding down the five erroneous bit positions. This is a tri-state buffer that will drive the data line on the memory chip when we are doing a write, but floats to let the memory chip produce an output during all other times. With the chip removed, all the data bits appeared to be a 0, thus it was the tristate outputs of the buffer chip that were sensed as a 1. 

The data outputs feed into a NAND gate which produces the sense pulse when the data bit is 1 and the timer chain drives the output pulse at the correct time in a read. One more chip is connected to the data lines, an XOR chip that is used to calculate the odd parity and produce the parity bit value. 

Thus, the cause for bits 2, 3, 4, 13 and 14 to act differently could have been the NAND gate, the XOR gate, the buffer chip or the memory chip. The memory chip driving an output during a read, the other three chips having a bad input that pulls down the data line. 

REMOVING VARIOUS CHIPS TO NARROW DOWN THE CAUSE

I desoldered various chips to help identify which chips needed replacement. With the memory chip uninstalled but the others there, I observed the problems. When I yanked the buffer chip from one location, the problem bits from that side seemed to go away. When I soldered down the memory chip but had the two buffer chips removed, the results were still mixed. 

There were some differences from the prior condition, but not all bits acting the same either 1 or 0. That could be the contents of data in the chip that are driving those results, but it could also be the XOR chip inputs causing the problem or the NAND outputs.

I had changed the NAND chip that produced bits 12, 13, 14 and 15, but still had the same results in the earlier tests before the memory chip was installed. However, I could have an issue with the chips that output 2, 3, and 4 - two different chips as one handles 0-3 and the other covers 4-7. 

ORDERING REPLACEMENT CHIPS

I have spare NAND chips on hand, but no spares for the buffer nor the XOR chips. I placed orders with Digikey so that I can swap in known good chips in these positions and continue my testing until I am satisfied that the sense outputs are working properly on the test bench. Only then will I move over to the 1130 and test with this cabled into the system. 

Finishing new PCB for 1130 MRAM, waiting on memory chip

CONNECTOR PINS INSTALLED AND MORE CHECKING ON THE PCB

I use square profile gold plated pins, soldered onto the PCB with the correct spacing, to form the SLT connector into which the three cables from the 1130 system will plug. A spare cable socket held under the PCB lets me orient the pins correctly as I insert them and solder them down. 

Everything but the memory chip is installed

I did more checking of the solder joints to the chips and connectivity of the key circuits. Using an external testing box I inserted a rising edge on the +Storage Read and +Storage Write pins, which allowed me to watch for the proper timer output pulses. They looked good.

Without the MRAM memory chip installed, the NAND gates that produce the sense bit outputs will see every bit of a memory word as a logic 1, thus each sense bit should produce an output pulse when I drive a rising edge on +Storage Read. Using the oscilloscope, I verified that this occurred as intended. I found a couple of anomalies to investigate. 

It also allowed me to verify that the parity generation circuitry would produce a parity bit value of 1 for both halves of the word, since each halfword had an even number of 1 bits (8) and the 1130 uses odd parity checking. 

WAITING FOR THE MRAM MEMORY CHIP FROM DIGIKEY

The shipment from Digikey is expected to arrive late Friday. I will be away visiting family (again) for a few days but by the end of the weekend I can get this soldered onto the new board. 

Building new version of the PCB for 1130 MRAM

USING SOLDER PASTE AND HOT AIR GUN FOR ALL SOLDERING

Because so many of the chips are tiny footprints with very narrow pin spacing, I decided to use solder paste and the hot air station to solder them down. I mixed a bit of flux with the paste, which reduces some of the splatter after soldering. 

As the paste is turned into liquid solder, it forms balls. When the balls touch an exposed pad on the board, it flows onto the pad and other balls accrete to this structure. However, if a ball does not bump into a pad or other balls, it remains as a loose very small ball on the surface of the board. I had to clean these away otherwise they might move between pins to create intermittent shorts. 

I controlled the amount of splatter by applying the solder paste/flux mix carefully to reduce how much is away from the pads. I then dropped the chip into place with the pins aligned over the pad pattern. The hot air rework station as a control for the air volume, which I lowered because otherwise small parts like resistors could be blown out of place before the solder liquified. 

I used by PCBite with its small probes to test each pin for connectivity after a chip was soldered down. I touched up high on the pin with one probe and touched the far end of the trace with the other probe and listed for the beep of the continuity tester. I also touched adjacent pins to check for shorts. This was a slow process with all the chips and pins, but important to be certain that the board is correctly assembled. If a pin didn't have a good connection, I put on a dab of solder paste/flux and heated it until the connection was good while remaining free of shorts. 

First set of chips soldered down, circled in yellow

Another group of chips installed, circled in yellow

NEED TO ORDER A COUPLE OF CHIPS BEFORE FINISHING

I found that the main MRAM memory chip had developed a fault and one of the sense output NAND gates had developed a broken pin. I placed an order with Digikey and should have the parts before the week is out, when I can finish the board.

CONTINUING WITH NON CHIP COMPONENTS

I began to install all the resistors and capacitors onto the board. I used the solder paste and hot air station, even though these were easy enough to solder with an iron. In addition, there are a couple of connectors and all the pins that form the SLT connectors where the cables T1, T3 and T4 are attached. 

More work on retriggering issue with 1130MRAM

GOOD SUGGESTION FROM A BLOG READER

One of the blog readers, Merlin Skinner-Oakes, suggested that I add a low pass filter to the output signals that are causing the ground bounce. These are the eighteen sense bit outputs, open collector gates which pull the line to ground and sink around 8ma for every bit in the word or parity that has a value of 1. The more 1 bits in a word, the worst the retriggering. 

I had to choose a chip that could sink 8ma on every output, would be gated by the pulse from the second read timer chip, has an open collector (drain) output, and would operate properly at 3.3V. The selection was slim. The chip I chose has a very fast edge, compared to the SLT logic in the 1960s era IBM 1130 system. 

The initial values to test were 100 ohms and 100 pF, to see if slowing the fall time of the signals would lessen the problems. This establishes an RC time constant of 10 nanoseconds, thus the falling edge is spread over very approximately this time instead of falling very steeply. I can adjust these component values as necessary to fine tune if the initial results seem promising. 

BODGING THE CURRENT BOARD TO TEST THE FIX

The lines run directly from the surface mount output chips to the SLT connector cable pins. I will need to break the connection between the chip output pin and the cable connector pin in order to insert a resistor in line and a capacitor to ground. I would need to cut the traces on the top layer coming out of each chip, then run bodge wires from the relevant pins to my RC filter components.

This would be very messy to do with eighteen output pins and bodge wires to 36 discrete components. Instead I will test this with the minimum changes by altering only the two parity bit outputs and using a word of all zeroes. That will only produce pulses on the two parity bit lines, as these must have a 1 value to achieve odd parity. 

First two bodged RC filters on parity outputs

I just had to cut two top traces coming out of chip U11 to disconnect output pins 3 and 6 from the connections up to the connector cable pins. I then tacked on two wires at pins 3 and 6, connected them to an RC pair each, hooked the other end of the resistors to the T4 connector cable pins L9 and L10 and hooked the other end of the capacitors to ground. 

I had many frustrating setbacks with this work - it was extremely easy to form a solder bridge across the surface mount chip pins. When wicking the solder away, it was easy to break the link to the pad invisibly, so that a connection appeared good but was not. Solder could form shorts underneath the chip, not visible to the microscope. I even tacked a wire onto the wrong pin in one case. 

Each time I had to pull the board and do work at the soldering/rework station. Every manipulation threatens to break off tacked wires. I persevered until I had the board ready for its testing. 

TESTING RESULT

I had to store a word of all zeroes first, then perform a display of the word to see whether the spurious retriggering and ground bounce is suppressed by the RC filters I added to the outputs for the two parity bits P1 and P2. 

Initially I had more problems with solder joints as I worked on the chip to add the bodge wiring - one of the parity bits wasn't working due to such an issue. I finally got all signals connected properly with no shorts.

I was not seeing the parity check bits in the IBM 1130 being set by the sense pulses after they passed through the RC filter. The circuit that sets the flip flops depends on the falling edge of the signal to discharge energy in a capacitor. Previously with a prior output chip I found that the flip flop wasn't being set because the output chip couldn't sink the 8ma of current to cause the edge pulse in the 1130. 

In fact, the resistance of the RC filter lowered the current enough to either prevent the flipflop from setting or cause it to only set sometimes. I had to cut those out of the circuit to restore the setting action. 

GROUND ISSUES IN THE PCB ENTER THE CROSSHAIRS AGAIN

I did more measurements of the ground voltages on the PCB and found ringing of more than 500 millivolts up and down. They are coincident with the retriggering of the timers. If I can stomp this out, the rest should work properly. 

I removed the FET transistor to gain access to the big ground pad underneath, then soldered an 18 gauge stranded wire to it which is connected to the IBM 1130 ground bus on the other side. With this in place, the ringing was substantially lower, as you can see from the scope output below:

Yellow is the +Storage Read signal, purple is the P1 parity bit, blue is the P2 parity bit and green is the ground where I monitored it. The bounce was cut down more than 50% and the result is no retriggering. I am convinced that it is time to make an updated version of the PCB with special attention to ground to minimize the ringing/bouncing. 

1130 MRAM - quick check after removing large buffer capacitors

PREVIOUSLY ADDED CAPACITORS WHEN SUPPLY SAG SUSPECTED

A while back I worried that the rails on the PCB were sagging during the heavy output pulses from the eighteen sense amplifier pulse generating gates. The design I originally used, informed by the datasheet for the LDL1117 voltage regulator, was a 4.7 uF capacitor across the regulator 3.3V output, as well as .1uF decoupling capacitors at every chip.

I tacked on a 470 uF capacitor to provide substantial additional buffering, but the spurious retriggering of the timer chips continued to occur. At my last session, I added an additional 46,000 uF across the 470uF and 4.7uF capacitors. The symptoms continued. 

POSSIBILITY THAT EXCESSIVELY LARGE CAPACITANCE LEADS TO SUPPLY OSCILLATION

However, it is a known issue with switching regulators that too much buffer capacitance can lead to the regulator oscillating on momentary demands because it is too slow to notice the energy being pulled from the capacitor, then has difficulties recharging the energy into the capacitor. 

While the LDL1117 is not a switching device, instead it is a linear regulator, which is why I initially disregarded the risk of excess buffer capacitance. However, it regulates with an op amp and oscillation is certainly possible if I have imbalanced the chip too much. 

REMOVED BOTH EXTRA BUFFER CAPACITORS AND RECHECKED 3.3V AND GROUND

I de-soldered the two capacitors from the PCB, reverting to the recommended 4.7uF capacitor across the output of the regulator chip. I then used the same measurements with the oscilloscope that I had yesterday, looking at AC variations on the 3.3V output of the regulator and also between the ground on the PCB and the ground out at an SLT slot on gate B compartment A1 of the 1130 system. 

RESULTS OF RECHECKING

The 3.3V rail is much more stable with the extra capacitors out of the circuit. Unfortunately I still have the spurious retriggering. 

What is still happening is a big bounce between ground on the regulator of the PCB and the ground pin of an SLT slot in the 1130. 

ADDED A SECOND GROUND WIRE FROM THE 1130 TO THE PCB

The PCB is connected to the IBM 1130 ground terminal block by an 18 gauge stranded wire. That is the same wire type used to connect the grounds on all the SLT backplanes from this terminal block. However, I hooked up a second 18 gauge stranded wire between the ground terminal block and the regulator ground connection. No difference in the observed bounce. 

GROUND BOUNCE TRACE OF 1130 SLOT PIN MAY LIE DUE TO CONNECTION METHOD

I don't have scope probe tips that slide over the .025" pins on the rear of the SLT backplane (official name is SLT board). I have used jumper wires, male-female type, to fit over the SLT pin and have the scope probe hook grab the other end. In other words, I don't have a great low resistance direct connection to the probe. 

GROUND BOUNCE MAY BE DUE TO REGULATOR CONNECTION TO PCB GROUND

The ground plane of the PCB is an inner layer, but the connections are on the top layer. Vias carry the current down to the inner layer. I thought I had plenty to ensure excellent conductivity. Since I have a helper wire from the regulator ground where my new wire attaches all the way to the read timer chip grounds, the retriggering doesn't fully make sense yet. 

OBSERVATIONS

When I combine the images of the ground bounce relative to the 1130 SLT logic with the +Storage Read signal coming into the PCB and the pulse retriggering, the wiggles in the control signal seem directly correlated with the ground bounce as seen at the 1130 SLT board. When the data returned has more one bits, the retriggering pulses continue longer and the ground bounces with them.

A cause appears to be the NAND gates sinking the current from the 1130 side to flip on the bits in the B Register. The 1130 side has a capacitor that is charged up to sensitize it, then when our PCB sinks that to ground, the edge causes the flip flop in the B register to turn on. 

Four feet of 18 gauge stranded wire should drop about 7 millivolts with that current draw, not hundreds of mV. Two parallel such wires should reduce the max bounce further. 

Still not sure of exactly what is causing this thus not certain how to correct it, but getting closer. 

1130MRAM smoking gun found

SCOPE ON 3.3V AND GROUND, TRIGGERED BY THE ANOMALY

I set the scope on the yellow trace and triggered on the first fall of the +Storage Read signal, which will occur during the glitch that surrounds the spurious retriggering in the read timer chain. I wanted to look extremely closely at the 3.3V line at the time of the anomaly, watching very closely for any variation that suggests bouncing on the ground or 3.3V lines that feed all the chips.

I set up the green trace to a wire plugged into a ground pin in gate B compartment A1, in AC mode with 200mV per division sensitivity. The blue trace was hooked to the +3.3V side of a huge capacitor I added to bolster the 3.3V supply stability, also on 200mV per division in AC mode. The purple lead was connected to the output of the second read timer to watch for retriggering and thus duplicate pulses. 

Because I suspected that I might have some bounce in the supply, I had added a 46,000 uF electrolytic in parallel with the 470uF filter capacitor I previously added. Even with all that buffer capacitance I was seeing the retriggering occurring. 

When I looked at the 3.3V output at the buffer capacitor, I saw a horrifying bounce of 800mV up and down. 

Watching the 3.3V supply bounce along with the second read timer output pulse, the correlation is obvious. 

Watching the ground pin on the IBM 1130 to compare its level to the ground at the huge filter capacitor on my board, I saw about a 700mV +/- bounce as well. 

This is a classic analog issue, lying beneath the digital abstractions of logic gates. There is likely some kind of resonance or ringing at play here, looking to be around a 1-2 ns cycle duration. My suspicions are going to be aimed initially at the power traces to all the chips, but I will also look into the chip bypass capacitance values I chose and its interplay with the board and trace capacitance, resistance and inductance. 

I suppose I will also route the traces much more carefully to round every bend and avoid any spots that could produce reflections, since this board designed for 2.5MHz signal rates is having more difficulties that I had anticipated. 

It seems inescapable that I will have to buy another round of PCBs to fix all of this, incurring the cost and delay.