Checking out the low ESR cause for the 1130 MRAM ringing

QUICK VERIFICATION OF THIS THEORY OF A ROOT CAUSE

I soldered a  ohm resistor in series with the 4.7 uF ceramic capacitor. I then tried the board on the 1130 again and observed the operation. 

It failed in the same way as before. Perhaps the regulator itself is bad or I still don't have the Equivalent Series Resistance (ESR) in the stable range. 

I used a quick and dirty hack was to use an external 3.3V power supply after lifting the LD1117 from the circuit. I used heavy stranded wires between the supply and the board. This would let me know immediately if oscillation of the regulator is the root cause of my problems. 

And . . . it is not. The exact same failure mode occurred using a bench supply to power the logic. Arrgh. 

Time for a plan C. I will think on this a while and figure out something else to do that might resolve this problem. I think it is down to three possibilities:

1. The timer chips are either bizarre or defective, but that isn't likely since I see spurious repeats on the write timer chain as well as the read timer chain. 
2. Something in the 1130 side is causing this, some way that I am not understanding. 
3. Something on the PCB is causing this, maybe a combination of the decoupling capacitors and traces, again in a way I am not understanding.

That means my plan C has to be more of a desperation move where I change parts of the design just to see if a different set of parts or different approach avoids the problem. 

Oscillation in LD1117 voltage regulator appears to be the root cause

OUTPUT CAPACITOR ON REGULATOR IS CRITICAL TO STABILITY

This type of linear voltage regulator requires an external capacitor to achieve stability in its feedback loop, otherwise it can oscillate. The data sheet recommended a 4.7 uF ceramic capacitor of XR5 or XR7 dielectric type, which is what I used. 

Almost every power supply component is best with the lowest equivalent series resistance (ESR). However, I see reports that too low an ESR can lead to oscillation with this regulator. 

DATASHEET FOR THE REGULATOR IS NOT VERY HELPFUL

The data sheet recommended 4.7uF ceramic with XR5 or XR7, however those dielectrics are know for very low ESR. There is no discussion of having a high enough ESR, other than the charts buried near the end of the sheet. 

The data sheet shows how the regulator behaves with load variations but the chart provided has a rise time of 5 microseconds, much much slower than the rise time of the 80-100 ns pulses. Thus I don't know how this behaves with higher speed circuits driven by the regulator. 

Two charts show the stability region for the output capacitor, one for 5V output and the other for 1.2V output. My regulator is 3.3V, but I guess I can interpolate between the two charts since the manufacturer doesn't provide a specific datasheet for my part. It shows ESR at 100KHz frequency but my operating frequency is up over 10MHz. 

The bottom bound at 100 Khz seems to be an ESR of roughly.015 ohms so the next question is what ESR does my output capacitor exhibit at my actual operating frequency.

MY CHOSEN CAPACITOR HAS AN ESR THAT IS VERY LOW

The ESR is way, way below that level from approximately14KHz all the way up to roughly 70MHz. Thus any signal changes in that range risk oscillation of the regulator, exactly what I am seeing. Yikes. The smoking gun. 

SOME POSSIBLE FIXES FOR THE PROBLEM

I could choose a different voltage regulator part that doesn't have this issue, as long as it will drop in on the footprint of the existing PCB. I found a great part, Microchip Technology MIC39100-3.3 but its footprint is incompatible. It is SOT-223 just like the other one, but the middle pin and tab are ground on this one and output on the old part. It would require a new PCB version to work properly. 

The data sheet, however, tells me that a low ESR ceramic capacitor, like the one recommended for the old regulator, would also cause instability in this regulator. Instead, the recommend a tantalum of perhaps 10 uF as long as the ESR is 2 ohms or less. The datasheet was much, much more helpful than the prior regulator datasheet. 

Higher than 2 ohms, instability. Fractional ohms like my ceramic XR7 also unstable. However, the correct part, in the goldilocks zone of ESR, should work properly. 

I could source a 4.7uF or larger capacitor in the same 0805 footprint with a much higher ESR. In fact the same 10uF tantalum as for the MIC39100-3.3 should work properly on the existing LD1117 regulator chip.  

Avoiding changes to the PCB is desirable. First, it saves the delay waiting for a new board. Second, it avoids the labor of moving all the components over from one board to another. Third, it avoids all the connectivity testing I have to do verifying the SMD soldering. 

My first task is to do some testing to verify that it was the regulator oscillating, by adding a series resistor to the decoupling capacitor temporarily, or by removing the regulator chip and using a solid external power supply. Based on that, I can then try a higher ESR capacitor with the LD1117 and validate its good behavior. 

Same problem with 1130 MRAM in spite of new board

SCOPE SHOWS SPURIOUS RETRIGGERING DURING READS

The scope was set to trigger on the +STORAGE READ signal (yellow trace) activating. I displayed the sense output for the P1 parity bit (green trace) and P2 parity bit (purple trace), as well as the sense pulse for bit 15 (blue trace). Depending on the number of one bits in the stored data word - sense pulses are only generated when the bit value is 1 - I saw various amounts of retriggering occurring. 

The time chain produces a pulse about 800 nanoseconds after the +STORAGE READ rising edge. That is what gates sense output pulses when the bit value is 1. Retriggering means that the timing chain produces additional pulses 800 nanoseconds after the first one. Those force the Storage Buffer Register to have 1 bits in every position where sense output pulse arrives, but the retriggered pulses will corrupt the register during unrelated cycles, causing the processor to work incorrectly. 

My first display when I powered up was a word that had six bits of the word set to 1, thus generating six sense output pulses plus pulses on parity bits P1 and P2 for a total of eight. The scope showed me three retriggers, similar to the defective behavior I was seeing in the past.

Data value b0000111100000110 

I then stored a word of b0000000000000001 which has only one of the data bits with a 1 value. Parity bit P1 is a 1 because the left eight bits need to have an odd number of ones, while parity bit P2 is off because the right eight bits already have an odd number of ones. No retriggering occurred as you can see in the scope output below:

You can see that bit 15 (blue trace) has a pulse because the bit value is one, and the parity bits P1 and P2 are 1 and 0 respectively. No retriggering.

I then added a second bit so that the data word was b0000000000000011 producing sense pulses on bits 14 and 15. Both halves of the word have an even number of one bits, so both parity bits should be a one. That is what I observed, but now there is one retriggered pulse making the system misbehave.

We observe that the more sense pulses produced, the more spurious retriggering occurs. I moved the blue trace to the +3.3V rail of the PCB and watched it in AC mode so that I saw disturbances on the rail. The same case as above, the one with one retrigger, shows the same pattern on the 3.3V rail in the trace below:

The 3.3V line rings 1 to 1.5V peak to peak, with the peaks right at the time when the trigger and retrigger take place. I went back to the original case with six data bits of 1 and both parity bits at 1, but watched the 3.3V rail. 

Now there are four rings rather than two, which are associated with the three spurious retriggers. I did a major rework of the PCB to address the voltage and ground rails to eliminate this kind of behavior. A ground and a 3.3V layer plus large traces and multiple vias carrying current from the power layers to the chip connections. I used multiple large stranded wires for the ground connection to the 1130. Decoupling capacitors were placed on the bottom of the PCB to place them as close to the VCC/ground pins as possible. 

Each sense output pulse sinks about 8 ma of current from the 1130 system, thus the worst case of a word of all one bits would generate 18 pulses for a sink of 144 ma over a duration of about 100 nanoseconds. Each of the chips that produces the sense pulse handles four bits, thus that chip sinks 32ma of current from the 1130. 

The chip is an open drain NAND gate which should not involve the 3.3V rail at all in the current being sunk - that should flow through the transistor to ground and back to the 1130. Unless there is something obscure inside the circuitry of the NAND gate that causes parasitic power consumption from VCC when the open drain transistor is sinking power from the 1130, this doesn't seem like a likely cause. 

When I test the output pulses with a pullup resistor on the workbench, there is no bounce but when I hook the PCB to the IBM 1130, there is all the current flowing through the transistors. I don't see the rail ringing nor the retriggering on the workbench, only on the real machine, but that is probably because the current flowing on the workbench is much lower. 

I am focusing on investigating ringing caused by the design of the PCB and circuit. It is possible to have the decoupling capacitors on a board create ringing if they reach a resonance at the frequency of the sense pulses. This would happen if the inductance, resistance and capacitance of the capacitors and the traces have values that line up to create strong resonant peaks. 

If that is indeed what is happening, there are ways I can resolve this by changing component values or tweaking other aspects of the design. Since this is a new PCB version where I made very substantial changes to the traces, including rounding every change in direction to reduce reflections. the trace impedance values have to be pretty different, yet the ringing is the same. This suggest that it is the decoupling capacitors or the voltage regulator that is the root cause.

Beginning testing of new PCB for 1130 MRAM board

NEW PCB READY FOR TESTING

I had spent many hours checking continuity for all signals on the board and eliminating the chance of any shorts to adjacent pins. On the testbench I verified that all the outputs are producing correct pulses and the parity generation produces the proper pulses. The new PCB was installed on the 1130 system where I will do the final testing. 

MANUAL LOAD AND DISPLAY TESTS

Using the rotary mode switch, I could set it to LOAD and push the Prog Start button to load the pattern in the console entry switches (CES) into the memory location set by the Instruction Address Register (IAR). Turning the mode switch to DISPLAY and pushing the Prog Start button will read memory and display its output on the Storage Buffer Register (SBR) lights. 

To set the IAR to a particular address, I set the address in the IAR, set the mode switch to LOAD and push the Load IAR button on the console. This way I could put multiple patterns into memory.

Using this, I verified that I could set patterns which would be read back correctly. Changing the CES and storing different patterns first, I went back and displayed the locations to verify that what I stored was what was returned, including correct parity. 

I did not yet set up the oscilloscope to verify that I wasn't experiencing any spurious retriggering of the read or write timer outputs. That was the main issue I was experiencing with the prior PCB. Off visiting family for a few days and handling other obligations but will get to this soon.

FILLING STORAGE WITH A CHOSEN PATTERN

Turning the rotary mode switch to RUN restored normal function. I then opened the top cover to access the six customer engineer toggle switches. Turning on the Storage Load toggle switch set up the machine to loop through memory repeatedly storing the value set in the CES into all memory words. I pushed the Start button to begin the filling process and used the rotary mode switch to halt the operation by momentarily switching it away from RUN.

I then turned the rotary mode switch back to DISPLAY and checked some locations. The location is set in the CES, Load IAR is pushed, then Start is pushed to read back the stored value. 

Some patterns led to a parity check stop. If I am still seeing spurious read pulses updating the 1130 Storage Buffer Register at the wrong time, that might be causing this issue. Alternately I might have some issue on the board that I didn't catch on the test bench. Using the scope will help me understand what is happening. 

Worse as far as I am concerned, it appears that during these stops location 0 is overwritten with the contents that were being fetched from the address with the error. I will need to understand this failure mode and block it - that might require an update to the design. 

Replacing some chips on 1130MRAM board based on testing

TOGGLING STORAGE READ AND OBSERVING THE SENSE OUTPUTS

I have the board set up with a switch that will drive a rising edge on the +Storage Read line, causing my board to emit a sense pulse for any data or parity bit whose value is a 1. The output has a pullup resistor to +3.3V and a sense pulse consists of a short pulse to ground for less than 100 nanoseconds, occurring roughly 800 ns after the rising edge of the triggering signal. 

SUSPICIOUS RESULTS

When the memory chip was not yet installed, most sense bits appeared to be a 1 and produced pulses, but not bits 2, 3, 4, 13 and 14. The two parity bits produced pulses when odd parity existed on the eight bits they covered - those were correct based on which bits were generating pulses. 

SEVERAL CHIPS COULD BE THE CAUSE

I removed the buffer chips to see if they were holding down the five erroneous bit positions. This is a tri-state buffer that will drive the data line on the memory chip when we are doing a write, but floats to let the memory chip produce an output during all other times. With the chip removed, all the data bits appeared to be a 0, thus it was the tristate outputs of the buffer chip that were sensed as a 1. 

The data outputs feed into a NAND gate which produces the sense pulse when the data bit is 1 and the timer chain drives the output pulse at the correct time in a read. One more chip is connected to the data lines, an XOR chip that is used to calculate the odd parity and produce the parity bit value. 

Thus, the cause for bits 2, 3, 4, 13 and 14 to act differently could have been the NAND gate, the XOR gate, the buffer chip or the memory chip. The memory chip driving an output during a read, the other three chips having a bad input that pulls down the data line. 

REMOVING VARIOUS CHIPS TO NARROW DOWN THE CAUSE

I desoldered various chips to help identify which chips needed replacement. With the memory chip uninstalled but the others there, I observed the problems. When I yanked the buffer chip from one location, the problem bits from that side seemed to go away. When I soldered down the memory chip but had the two buffer chips removed, the results were still mixed. 

There were some differences from the prior condition, but not all bits acting the same either 1 or 0. That could be the contents of data in the chip that are driving those results, but it could also be the XOR chip inputs causing the problem or the NAND outputs.

I had changed the NAND chip that produced bits 12, 13, 14 and 15, but still had the same results in the earlier tests before the memory chip was installed. However, I could have an issue with the chips that output 2, 3, and 4 - two different chips as one handles 0-3 and the other covers 4-7. 

ORDERING REPLACEMENT CHIPS

I have spare NAND chips on hand, but no spares for the buffer nor the XOR chips. I placed orders with Digikey so that I can swap in known good chips in these positions and continue my testing until I am satisfied that the sense outputs are working properly on the test bench. Only then will I move over to the 1130 and test with this cabled into the system. 

Finishing new PCB for 1130 MRAM, waiting on memory chip

CONNECTOR PINS INSTALLED AND MORE CHECKING ON THE PCB

I use square profile gold plated pins, soldered onto the PCB with the correct spacing, to form the SLT connector into which the three cables from the 1130 system will plug. A spare cable socket held under the PCB lets me orient the pins correctly as I insert them and solder them down. 

Everything but the memory chip is installed

I did more checking of the solder joints to the chips and connectivity of the key circuits. Using an external testing box I inserted a rising edge on the +Storage Read and +Storage Write pins, which allowed me to watch for the proper timer output pulses. They looked good.

Without the MRAM memory chip installed, the NAND gates that produce the sense bit outputs will see every bit of a memory word as a logic 1, thus each sense bit should produce an output pulse when I drive a rising edge on +Storage Read. Using the oscilloscope, I verified that this occurred as intended. I found a couple of anomalies to investigate. 

It also allowed me to verify that the parity generation circuitry would produce a parity bit value of 1 for both halves of the word, since each halfword had an even number of 1 bits (8) and the 1130 uses odd parity checking. 

WAITING FOR THE MRAM MEMORY CHIP FROM DIGIKEY

The shipment from Digikey is expected to arrive late Friday. I will be away visiting family (again) for a few days but by the end of the weekend I can get this soldered onto the new board. 

Building new version of the PCB for 1130 MRAM

USING SOLDER PASTE AND HOT AIR GUN FOR ALL SOLDERING

Because so many of the chips are tiny footprints with very narrow pin spacing, I decided to use solder paste and the hot air station to solder them down. I mixed a bit of flux with the paste, which reduces some of the splatter after soldering. 

As the paste is turned into liquid solder, it forms balls. When the balls touch an exposed pad on the board, it flows onto the pad and other balls accrete to this structure. However, if a ball does not bump into a pad or other balls, it remains as a loose very small ball on the surface of the board. I had to clean these away otherwise they might move between pins to create intermittent shorts. 

I controlled the amount of splatter by applying the solder paste/flux mix carefully to reduce how much is away from the pads. I then dropped the chip into place with the pins aligned over the pad pattern. The hot air rework station as a control for the air volume, which I lowered because otherwise small parts like resistors could be blown out of place before the solder liquified. 

I used by PCBite with its small probes to test each pin for connectivity after a chip was soldered down. I touched up high on the pin with one probe and touched the far end of the trace with the other probe and listed for the beep of the continuity tester. I also touched adjacent pins to check for shorts. This was a slow process with all the chips and pins, but important to be certain that the board is correctly assembled. If a pin didn't have a good connection, I put on a dab of solder paste/flux and heated it until the connection was good while remaining free of shorts. 

First set of chips soldered down, circled in yellow

Another group of chips installed, circled in yellow

NEED TO ORDER A COUPLE OF CHIPS BEFORE FINISHING

I found that the main MRAM memory chip had developed a fault and one of the sense output NAND gates had developed a broken pin. I placed an order with Digikey and should have the parts before the week is out, when I can finish the board.

CONTINUING WITH NON CHIP COMPONENTS

I began to install all the resistors and capacitors onto the board. I used the solder paste and hot air station, even though these were easy enough to solder with an iron. In addition, there are a couple of connectors and all the pins that form the SLT connectors where the cables T1, T3 and T4 are attached.