Debugging a Core Memory Shield for an Arduino

I built a kit for a 32 bit core memory, fitting as a shield onto an Arduino. This was built and sold by Jussi Kilpelainen. My kit wasn't working properly so I dug into the unit to figure out what was malfunctioning.

The design consisted of 16 driver transistors to switch the ends of the 4 X and 4 Y addressing lines, connecting one end to ground and the other end to 5V. By routing one set of address lines in the reverse direction, those drivers could uniquely flip any of 32 cores to the on or off state.

A sense wire woven through the 32 cores in a diagonal pattern balanced out the pulse coming from the X and Y address lines, improving the signal to noise ratio for detecting the pulse if a core were to flip state. These wire ends when into two comparators (since the pulse could be positive or negative depending on the particular sense wire path) to produce an output only when a pulse above a certain threshold is detected.

The two comparator signals are ORed together to produce a 1 condition when a core was seen to flip. The Arduino program communicates with a user over serial I/O to get requests for specific core operations. It sets up the address for the desired core, toggles another line to indicate read or write, and then pulses an enable line.

A CPLD device decodes the binary core number and the read/write line into outputs connected to the bases of the 16 driver transistors. Thus, it will route a current in the intended direction on one pair of X and Y lines.

Read core works by routing the X and Y current to flip the addressed core to the zero orientation.  If the core was already in that orientation, no pulse is seen on the sense output. If, however, it was previously in the one orientation, the flip in state generates a sense output. Reading, in other words, is destructive, setting the core to zero in the process of testing its state.

Writing into a core is done by routing the X and Y current in the opposite direction, to flip that core to the one orientation. The output of the sense amp is ignored. In most core memories, the X and Y current is sent to the core and a third wire, the inhibit line, will have a counter current to block the flip if we want the core to be set to zero. No inhibit current, we get a one.

This particular kit does not do that. To write a bit, it sends the X and Y current in the direction requested, thus either flips it to one or does a read to flip it to zero. Sense output is still ignored. The logic equations that were programmed into the CPLD determines the direction of current based on reading or writing and the five bit binary address that indicates core 0 to 31.

The CPLD formulae are pretty simple. They look at the binary address, the read/write signal and the enable line and yield a + or a 0 output for each transistor. As an example, to address bit 31 requires a current running between terminal P3 and P12 (the leftmost column of the 4 columns in the core array). It also requires a current running between terminals P10 and P21, the two row of the array.

Because of the special trick that implements 32 cores with a 4 x 4 addressing, the wire from P21 is connected to P20, the fifth row of the core array, whose other end is P27. Thus, if we fire up P10 and P27 in one direction, the current in the top row is reversed in the fifth row.

Cores are placed on the intersection of a row and column wire, oriented at a 45 degree angle. The orientation alternates, so that the core at the left column and top row is tilting to the left while the core at the left column and fifth row is tilting to the left again, but rows 2, 4, 6 and 8 are tilting to right.

Wiring of 32 core memory array with driver circuits

. To address core 31 with current flowing down through the column and from left to right across the row, two pairs of transistors must fire. To get the horizontal current correct, it is necessary to inject +3.3V into P27 and sink it to ground at P10, by firing Q15 and Q4. To get the vertical current correct, we have to inject 3.3V into P3 and sink P12 to ground by firing Q5 and Q10.

Currents combining to flip core 31 in the alpha direction

If we say that this flips the core in direction alpha, then to flip it to the reverse state beta we have to inject 3.3V in P12 and sink it to ground in P3 by firing Q9 and Q6, while injecting 3.3V into P10 and sinking it to ground at P27, firing Q3 and Q16.

Opposite current flow to flip core 31 in beta directions

Note that cores 31 (top left corner) and 27 (fifth row of left column) are both affected by our currents flowing through the column and row. Since we sent current down the column and from left to right in the top row, the current flowing in the fifth row is from right to left, or counter. The two currents flowing in core 27 are in opposite directions, thus they don't combine to flip the core while the same currents flow in the same direction for core 31, flipping it.

Reversed current directions through cores in top and fifth row

I hooked up the logic analyzer to the bases of Q5, Q6, Q9 and Q10 to watch the conduction when I issue a read or write for cores in the left column (addresses 24 to 31). Nothing showed up - except the enable pulse.

Since the logic analyzer was digital but I was monitoring the base terminal of transistors whose level might not be right for the analyzer, I moved over to my oscilloscope. I should be seeing at a bare minimum 0.3V movement from +3.3V for Q5 and 0 for Q6. That is what I saw.

The driving transistors Q5 and Q6 are hooked up through a bridge rectifier, although you only saw half of it in my diagrams above. The other half is connected to the number two column, thus Q5 and Q6 will energize both of the left two columns. Only the left of those two will have its current sunk or fed by Q9 and Q10, while Q11 and Q12 handle the second from the left.

This does mean that I will see Q5 or Q6 fire for addresses in either of the left two columns, that is for addresses from 16 to 31 inclusive. This is clearly visible on the scope.

However, I did see something interesting when I watched Q5 and Q6. When I do a read of any address, I see Q6 fire to sink current from P3 and presumably Q9 is firing to source 3.3V to the bottom of the column at P12. Soon thereafter, I see Q5 fire to deliver 3.3V from the top.

Looking over the Arduino program and the sense pulses being detected, I understand why. The logic of the program sends current in the beta orientation, which tells us this is the orientation that represents a zero bit value.

Core orientation to represent a zero value

It appears that the sense wire sees an output pulse, indicating that the core was originally in the alpha or one orientation. The program reverses the current to write the one value back into the core.

Setting core to one value orientation

The pulse looks weaker when driven by Q5 and sunk by Q10 than when it is resetting to zero driving with Q9 and Q6 sinking. I watched pairs with the scope to see if this was an essential characteristic or some flaw. It seems to be by design, or at least is symmetric for both directions of current flow.

Next up I move on to all the other pairs of drive lines to verify that all the transistors are emitting pulses appropriately. However, I notice that whether I write a zero or a one value into the core, the Arduino software sees the sense line pulse being output. Clearly this is wrong. That is, I should not see a flip for a succession of writes of the same value. I also don't see the enable line toggling like it should, another wrongness.

First I checked the voltage bias on the two comparators for the sense circuit - the should put a 20mv bias holding each of the comparators off, but when a pulse arrives on the sense wire that exceeds that value, it would flip the comparator output on.

The voltages were 20mv and when monitoring the outputs, I saw the blip of the comparator firing under appropriate conditions. That is, I would write a 0 to the core and then read - no pulse. Write a 1 to the core and then read - pulse. In fact I monitored the pulses even on the writes, seeing that writing the same bit value twice gave no pulse on the second try, but whenever it changed value, one or the other of the comparators gave me a pulse.

This is fairly good news, as it shows that at least for core 31 which I was addressing, it was reliably writing and the sense output from the comparators were correct as well. I then moved on to monitor the output of the CPLD which is a flipflop that gets set by a comparator pulse and is reset at the start of each core access (by the rising edge of the enable input).

There is an LED wired to the CPLD which is turned on whenever the enable input goes high - that works properly, another confirmation that some of the CPLD is working properly. However, there are two test outputs, one of whom gives me a pulse whenever enable input goes high, but the other should give me the OR of both comparators.

Instead, it floats high and barely dips down when pulses are arriving. This is incorrect behavior and indicates some flaw inside the CPLD, either hardware damage or loss of programming. I moved on to the read flipflop output which is delivered up to the Arduino to provide the results of reading a core. It is fixed at high, never going down even when the Enable input rises where it should be temporarily reset. It acts as if the flipflop function inside is frozen on.

This is consistent with the application software view, which reports a 1 as the output of every bit. I have to get a replacement CPLD (Lattice ISPMach4A 32/32 in a PLCC 44 form). The VHDL is available but I needed to download the Lattice toolchain, synthesize the design, then work out the JTAG loading method. I should have the new CPLD chip by the end of the week.

The software is putatively installed but when I try to synthesize I get very basic error messages. Doing some research clarified that this won't run on any 64 bit operating system, so I have to move it over to an older desktop system just to get it to run. After getting a new license and installing it on that system, I tried again to synthesize the CPLD code.

Same error again, but this time I simply set it to run as administrator and success was at hand. A bit confusing at first as the generation of the vendor neutral programming file (.svf format) requires use of a different program and wasn't run from the project navigator. Soon, however, I had my coremem.svf file ready to program the device once it arrives.

I have a USB-JTAG programming cable somewhere, so all I need to do is insert the new CPLD in the socket, plug the cable onto the JTAG header on my shield, run Lattice's ispVM and the chip should be programmed. I will do some simple testing to ensure it went well, then resume full testing of my core memory operation.

In the meantime, I removed the Arduino to see if the CPLD output of the flipflop that records a sense wire pulse might be stuck on. It was off by itself, thus I turned my attention to the two inputs on the CPLD, one each from the threshold comparators, to see if one of them was jammed on or false triggering.

With the Arduino back in place, I monitored the enable pulse as a trigger and watched the two comparator output pins - they were good. I then put the probe on the CPLD inputs that are linked to the comparators. They seemed good but somehow I was not seeing constant 1 values in the serial terminal any more!

For a while, the board was now working properly. I did some pattern writes to all 32 cores and found oddities. Writing all zeroes gave me all zeroes on read, repeatedly. However, writing all ones would then read back consistently as an odd pattern groups of bits on but groups of mainly four bits at zero.

I decided it was time to put the scope on each pair of driver transistors involved in the core, making sure that a source of 3.3V and a sink to ground worked properly when a write of the associated value is attempted. I hooked up the first pair, began adjusting the scope to trigger on the low voltage swing of the base, but then disaster struck. The board was back to reading 1 for everything regardless of what I wrote.

Something is flaky here, hopefully the CPLD since I am expecting to flash another one when the various parts arrive. Still, I could proceed with the pairwise test of all drivers for the eventuality that the board goes back to normal operation.

Their are 16 driver transistors, thus 8 pairwise tests I had to accomplish. These all worked properly when addressing their related row or column. I checked for dual pulses in the intended pair, but also saw some correct cases where one or the other fired.

This is a valid case since the drivers use some diode magic. A transistor might drive +3.3V to both of the left two columns (1+2), but the other transistors that sink the current is hooked to columns 1+3 and 2+4. Thus, if the sink transistor for 2+4 fires while the source transistor for 1+2 fires, there is current only flowing through column 2, since the circuit is incomplete on columns 1 and 4 even through transistors are switching.

I moved on to looking at the current cross the four current limiting resistors. These handle rows 0+2+6+8 and rows 1+3+5+7 for one pair and columns 1+3 and 2+4 for the other pair. To watch these, I put probes on each side of the resistor and set up math on the scope to show me the differential in voltage.

I was able to see current flow in both directions by suitably setting the address and the bit value to write. There were four resistors on the board, R22, R20, R21 and R19. I observed proper current pulses across all four resistors, in both directions. This tells me that the row and column wires are properly connected.

At this point, I have to conclude that all my addressing/read/write drivers are doing the proper thing and should be storing any bit pattern I want into the core array. Any difficulty must be in the sense circuit and CPLD latch that records the state of the core after a destructive read.

As a cross check, I put the scope leads on the two comparator outputs and wrote both 0 and 1 values into all 32 cores, watching for core flip signals. Writing the same value twice should see no output on the second try, but the second of each dissimilar pair should always show a blip. If this worked, I would have isolated the problem to the far side of the comparator, e.g. it can only be in the CPLD or the Arduino itself.

The comparators worked exactly as I expected them to, but somehow the CPLD is seeing them as on. The outputs stay flat and low but the CPLD latch output is not behaving as it should. The VHDL defining its function has it reset to off with the rising edge of the ENABLE signal, which also gates the X and Y driver transistors to send current to the core in one or the other direction depending on desired data value.

What I see is that the latch drops by about 400mv for 1 us, in a parabolic dip, then flatlines right back at full on level of 3.3V. Thus it appears to always have read a value of one, since we write a 0 value and watch the sense to see if it flips, thus having been one before.

Top is the CPLD flipflop output at 200mv per division, bottom is the comparator output

The first scope image is plausible, but this one shows the flipflop latching with no input

That made no change at all to the behavior. All eyes are back on the CPLD. To make this definitive, I used some jumper cables to isolate the Arduino from the Shield, rather than plugging them together. That allows me to watch pin 9 without it being attached to the Arduino at all.

No change at all. The CPLD is sitting at 3.3V with the teeny dip when the enable pulse arrives, rather than working correctly. The new one should arrive in a few days and I can replace it. In the interim, I can breadboard together a flipflop to replace the portion of the CPLD which is clearly defective.

Our ENABLE line is hooked up as the clock of the 74LS74 D flipflop, with a 0 wired to the D input. Since this is positive edge triggered, it will reset the flipflop whenever the ENABLE pulse rising edge arrives.

The two comparator outputs are connected to a 74LS02 NOR gate, whose output acts as the asynchronous PRESET for the D flipflop. Thus, whenever a pulse arrives from the NOR gate, it will set the D flipflop off. It will remain that way in perpetuity until our next rising edge of an ENABLE line.

My temporary flipflop circuit to bypass defective CPLD logic

The two horizontal red lines on the top will be connected to the outputs of the two comparators on the shield. The horizontal orange line is connected to the ENABLE signal. The horizontal gray line is the output which is fed to pin 9 of the Arduino. Of course, +5V and ground must be connected for operation.

Experimental setup to substitute my breadboard for part of the CPLD

I hooked this all up and tested successfully. The core memory works perfectly now, confirming that it is a CPLD failure which blocks the shield from working by itself. Writing any pattern to the cores yields the correct output pattern and the various testing regimes built into the Arduino code prove it is solid.

Writing and successful readback of several patterns

My replacement CPLD arrived yesterday, but the programming cable won't be here until later today. Since it will be slam dunk to program the replacement chip and insert it into the socket on my shield, I will declare victory and post this entry now.

Replacement CPLD chip ready for programming and insertion into the shield

I did discover that the ISPLever Classic toolchain assigned the input and output signals to different pads that were used by Jussi, so I have to figure out how to constrain or guide the tool to do what I want. Once that is done, there may be further hiccups forcing ISPvm to load the programming onto the chip, but none of these are insurmountable obstacles.

Building cables, steady progress on the display, fixing the core rope simulator module

DSKY SUBSTITUTE FOR THE AGC

Display

One repair to the PCB remained - installing a jumper from a nearby via that has connectivity down to the Arduino, carrying the signal over to the four current limiting resistors for the COMP ACTY LEDs.

At first it appeared the only way to make this work is to drill through a side wall of at least one of the light dams - 3D printed plastic shapes that are epoxied onto the face of the PCB. However, there is a very teeny crack under the dam near the LEDs in question, and another under the indicator side light dam.

If I could find a small enough wire that can still carry 20ma, able to be forced under the two light dams, then I could make the repair without drilling. And, in fact, I found such a wire, routed it under the dams and made the connections to the resistors on one side and the live via which is still connected down to Arduino pin 30 where I control the COMP ACTY indication.

Jumper in place to repair connectivity of COMP ACTY drive signal to LEDs

I received some smoked tinted gray plexiglass that I ordered as a replacement for the spray paint frosting, which I was unable to get applied in an even but thin coat to allow enough light through. In the interim I experimented with another option, some grey tinted plastic (antistatic bags that come with electronic components).

I was happy enough with the brightness, grey color imparted to the panel and the only area I still needed to work on was the diffusion or frosting for the legends (COMP ACTY, PROG, VERB and NOUN). I did a test with some diffusion paper under the gray bag under a clear acrylic panel and was reasonably happy with this.

Testing of diffusion and coloring for DSKY - no diffuser on upper right legends - panels loose on top

It is possible that I could make a mask and spray paint some frosting on the rear of the acrylic panel, just in the areas where the text shows through. This will depend on how I feel about the display as it sits now. I am on the fence with this, I could either wrap commit to the diffuser paper, gray antistatic bag material and acrylic, or work further on diffusing so the individual LEDs are no longer visible.

My tinted plexi came in and was just perfect for the role, even better than the antistatic bag material. I took this over to Tap plastic to be cut into panels of the proper size. After checking out the stack once again, I decided that this side is ready for assembly.

Another area where I am dissatisfied is the light dam for the indicator panel. The openings do not properly line up with the lights, text legends I created and the top cardboard mask. I took the dimensions I used to cut the light mask and updated the 3D printing design.

With that done, I have uploaded the design to the 3D service. In the meantime, I had to safely remove the existing light dam which was epoxied in place.  That wasn't as difficult as I had expected, so now the PCB is ready for when the new part arrives.

Because I used real electroluminescent wires to form the three continuous lines that separate the R1, R2 and R3 rows of digits, it needs a high voltage supply to operate. I have such a driver module, which is small enough to fit in my enclosure, but it is fed with 9V from a battery which is operationally unacceptable.

I also need these lines to switch off when the AGC is in standby mode, I decided to power these from the BPLUSSW (+14V switched line) coming from the computer. When the computer is in standby, no power for the EL driver, otherwise they light up. I only need to drop the voltage down from 14 to 9 for the driver module.

An adjustable buck converter is just what I need. I found a nice small board with a regulated, adjustable converter, only a few tens of mw power loss for the conversion; these should arrive on Tuesday and I will have the board wired into the display PCB soon thereafter for testing.

Buck converter board to drop +14V from AGC down to 9V for EL driver

Cables

I began to put together the cables for the DSKY substitute. One short one will run between the display and keyboard PCBs. Three cables run out with AGC inputs, AGC outputs and miscellaneous AGC signals, all to be connected to power supplies or the Apollo Guidance Computer itself. Finally, a small power cable runs out to supply the two boards with 5V for their operation.

I had bought cable supplies so that I could begin to fabricate the end that fits onto the DSKY printed circuit boards. I have a dual row 2x12 connector, a single 8 pin connector and two single row 10 pin connectors on the display board, plus a barrel shaped power connector.

The display to keyboard cable has a single row of 10 pins on each end, but due to a previously confessed error in the design of the display PCB, one of those lines is intended to deliver 5V to the keyboard but instead feeds 28V on the display side.

I had to modify the cable so that pin 9 is connected on the keyboard but routed away from the connector on the display board and hooked to a source of +5V. A large via was located into which I soldered a header socket for a single pin to provide 5V. I also glued  the hole on the display connector so that 28V cannot get on to the wires if the plug is reversed by accident.

Keyboard to Display cable for DSKY

The next cable to build was a 2x12 ribbon cable, which I could leverage from PC components. It carries 28V from the spacecraft supply (through the AGC) and 23 signals that come to the DSKY from the AGC. This fit nicely; thus I could proceed to build the others.

AGC inputs cable (2x12)

I moved on to the 10 conductor cable, which carries the 8 output signals from the DSKY to the AGC, as well as providing a common ground. This was also pretty easy to build up, using superglue to accrete the jumper pin housings together until I had a block of 10. With both the keyboard cable and this one, I use a black wire at the ground end to help maintain proper orientation. I cut the tip off the unused pin 9 and put glue in the PCB connector.

AGC outputs and ground cable

The last of the signal cables is a 8 conductor ribbon cable which brings two loosely defined AGC signals to the DSKY - LGC Warn and AGC Warn - then loops two pairs of lines which run through the DSKY in the spacecraft but have no role in what is displayed. I used glue to accrete jumper pin housings to form these six lines, plus there is one inactive spare.

The last pin is the +14V signal, BPLUSSW, which the AGC switches off when in standby and on when it is running. That signal triggers blanking logic on the DSKY to darken the panel. When I have no BPLUSSW voltage, it shuts off the three horizontal lines, blanks the legends PROG, VERB and NOUN, turns off the digital displays and signs, and blocks the COMP ACTY from lighting. This causes the EL side of the display to go dark.

Having the cables built is only part of the challenge. I still have to determine how I will route these out of the DSKY enclosure - what kind of hole and on what face of the box. Next, I need some kind of junction board or box that we can use to connect from these cable wires to the wires coming from the AGC.

CORE ROPE SIMULATOR MODULE REPAIR

As Ken continues to test his driver for the Core Rope Simulator modules, which plug into the AGC and feed results in lieu of plugging in the six actual core rope modules. This provides the 36K of read only memory that contains the programming for Apollo missions.

Ken's driver board will store all the captured images we have of various CM and LM mission software, allowing us to select one to be accessed by the AGC.

Raytheon built this using a technology that was popular for a short while, mainly the early 1970s, called a DipStik. This is a plastic enclosure that mounts six dual-inline (DIP) integrated circuit chips on the inside and has solder tabs on the top and bottom for connection to the rest of the circuit.

These are not the most reliable of sockets for the chips. it presses the chip legs between contacts on the base and top of the DipStik, with screws at both ends to tighten the top onto the base. The plastic bows up in the middle, however, making contacts on the middle two chips erratic or open.

It had a bit of springy metal that was intended to press the chips down on the top contacts, but that doesn't solve the continuity issues on the base contacts. Worse, the springy metal disintegrated on all our DipStiks, a sample of about a dozen units. After we picked the fragments out, the bowing was the next major issue, one we still must resolve.

While Ken was opening and closing one DipStik unit, the screw snapped off in the base. This was highly surprising since he was using mild finger pressure, not even a screwdriver applied to turn it. Marc removed the fragment from the base and then disassembled the top to unfasten the broken screw top.

It was an unusual shape, with a groove milled for a circlip style set of restraining tangs, then a conical shape down to the threaded end. Marc did careful measurements, found a similar style screw with a larger diameter, and then milled it all down on his lathe and put on threads with a die. It was a good match except it is a more silver color and has a Phillips head top rather than a slotted top, but worked just great.

Suffered minor failures in display PCB, requiring exhaustive testing, rework and wasting time

DSKY SUBSTITUTE FOR THE AGC

Display

With the acrylic panels frosted with too thick a layer for good viewing of the display underneath, I had to either clean off a panel for repainting or buy some new ones. I chose to buy new ones, picked them up and now need to experiment to get a very thin but still even layer of frosting.

My attempts with spray paint are not coming out well. To get an even looking coating requires so much paint mist that the surface is too opaque. I have ordered some smoked tinted acrylic to see if it will give me the effect I want.

Meanwhile, for some reason, the COMP ACTY LEDs stopped illuminating. At first I though this was a programming problem, but then realized something had gone wrong with the board. I began debugging the path from code to the LEDs themselves. I could light the LEDs manually and had continuity to the +5V supply and to the net connecting all four current limiting resistors.

Moving backwards, I had to determine if the path from the resistors back to the Arduino connector was intact. This is the problem! Somehow I have no connectivity where I did before. I can follow the signal line vertically on the bottom of the board from the Arduino connector up to a via that routes the signal up to the top layer.

From that via, it runs across horizontally and directly to the four current limiting resistors. There is connectivity up from the Arduino to the via but nothing from that point to the resistors. This line runs under the light dams which I glued down, making me thing that the pressure of gluing caused the failure.

If this line has failed, what else might not work properly anymore? This is a serious problem that I must resolve since the input and output circuits for the AGC might have been compromised and the DSKY rendered unusable. I need to carefully and thoroughly test out the board again.

To check this out, I produced several test sheet with all the pin connections to allow me to test everything. I would check the AGC inputs, then the AGC outputs, then the circuit that blanks the display when +14V from the AGC is not present, and then some miscellaneous circuits

I tested 23 input circuits, verifying that the related Arduino pin jumps between +5 and ground as the inputs are pulled down to ground through a 2K resistor while feeding 28V to the circuits. All responded correctly.

I tested the output circuits that would be fed by the keyboard PCB, ensuring that each would toggle the relays on the display board and pull the relevant output pin down to ground through 2K ohms or float high. These too worked flawlessly. I was getting a bit less discouraged at this point.

I know that the seven segment displays, the signs, and the 14 indicator side lights are working correctly because I can see them. Similarly, I know the VERB, NOUN, and PROG legend lights are working properly. I haven't yet wired the electroluminescent wire drive board and my yet to be built power converter to the +14V supply from the AGC, but that testing will be deferred until I I complete the board and installation.

I then tested the 14V input from the AGC to see that the VERB, NOUN, and PROG legend lights turn on and off with the blanking. To do this, I needed two power supplies, one at 5V and one at 14V.
My first try wasn't successful.

The way this section should work, the 14V input is first dropped to 5V with a pair of voltage divider resistors, then that is delivered to pin 38 of the Arduino for sensing. The Arduino logically reverses the input from pin 38, driving the output on pin 37.

Thus, when 14V is present, the output of pin 37 is zero and when the AGC drops 14V, the Arduino should deliver 5V on pin 37. The relay circuit that powers the VERB, NOUN and PROG LEDs is driven by a transistor that fires if its input drops near zero and releases the relay when the input jumps up near 5V.

I first checked continuity to the voltage divider pair from the 14V input pin and then continuity of the output of the divider to pin 38 of the Arduino socket. The first part was good and when I apply +14V to this divider circuit I get the +5V I expect from the output. However, I have no connection down to the Arduino pin 38 from this divider.

This signal runs somewhere near the area where I lost the COMP ACTY signal continuity. I had a via in the path of the voltage divider output down to the Arduino, so I did more continuity testing. The path is good from the via down to the Arduino connector, but not between the voltage divider output and the via.

This can be fixed with a small bit of wire added as a rework. I will do this at the same time as I add a wire to bridge my COMP ACTY LED resistors to its good via. The challenge is to find wire small enough on the scale of the vias and pads where I am soldering. Normal hookup wire is gigantic by comparison.

I was able to make the first jumper installation, for the 14V voltage divider line to the Arduino and verified its operation. Next up was the tougher connection to the COMP ACTY LEDs since these run underneath the existing epoxied light dams.

Jumper installed from 14V divider network over to working via

Repair must run from circled via on left, to circled pad on right, yellow indicates connection

While I can reach the left side via from either the top or the bottom, there is no path into the opening on the right for the COMP ACTY LEDs. This poses a major problem. I will probably have to drill an opening, which puts even more stress on an area of the board where I have just had two failures. I will have to work on this later in the week.

Now that the 14V sense works and the Arduino could do its job, the next step was to check the continuity from pin 37 to the transistor driver circuit, validate all that circuitry and the path out to the relay coil. Too, I had to check that the relay contacts will switch +5V to the switched LED power plane.

I found a missing base resistor for the driver to the relay. I replaced it and the circuit works perfectly when I drive it with 0V, forcing on the relay and illuminating the VERB, NOUN and PROG legend lights.

I did take note of a design risk I took, where I was using the digital I/O pin of the Arduino to directly sink the current from the four LEDs. My original concept had higher value resistors which would have kept the current from the LEDs down to 5ma, thus with four I would match the 20ma capacity of an Arduino.

However, in matching perceived intensity of the LEDs to the 7-segment displays, I dropped the resistance to 680 ohms. That increased the draw of each LED to 6.5ma, bringing the group of four up to 26ma. Thus, there is a risk that the ATMEL chip on the Arduino could be damaged such that the I/O pin no longer worked.

I will change out the resistors for 1K values just to avoid future damage. Unfortunately, light dams and the mask are already glued in place, making this a more challenging microscopic repair task. After some time working away at them, I was able to get the four 680 ohm resistors removed and the pads prepared.

Old resistors removed and pads prepared for install of new 1K parts

The original pads were set up for 1206 sized surface mount resistors, but I made use of much smaller 0603 components to give me good access to the pads in the narrow canyon in which I was working. These parts are able to dissipate 100mw but in this circuit we will have about a fifth of that flowing, and the COMP ACTY light is intermittent anyway.

After installation, I tested with the VOM to be sure I had good continuity. All the remains is to install the jumper to restore the connection between these resistors and the Arduino. That would return this light to operation.

As part of the extensive circuit testing, I checked all the signal lines and pin assignments on the connectors. I discovered an egregious flaw that might require rework of the board to correct. The supply voltage I connected to the cable down to the keyboard PCB is dangerously wrong. That board requires +5V to operate, but for some reason I had routed +28V on the line. The first time I cabled the two up and tested, I would have blown out my PCB.

I  believe the easier solution for the mismatched power supply is to custom build the cable so that it brings pin 9 of the cable over to a source of +5V instead of using the 28V supplied on pin 9 of the connector. This will correct the problem and allow safe interconnections. I will glue over the opening to pin 9 on the connector to insure that the 28V is never fed to the cable if it is accidentally plugged in reversed.

Improving the display for the DSKY substitute

DSKY SUBSTITUTE FOR THE AGC

Display

With the EL wires in place, I then glued down the main light dam, which has to clear the EL wire segments  It was important to get the top surface parallel to the PCB so that it can fit flat against the bottom of the acrylic panel. I had to cut out a section of the dam the EL cabling runs, to keep it as flat to the PCB as possible.

Light dams being glued in place

Finishing off the EL display side consisted of some rice paper diffusers to be attached to the light mask and that mask in turn be glued precisely in place atop the light dam I did a final test of the firmware to ensure that the sandwich of parts was aligned properly. It was.

I turned to the indicator panel side, where I have a much simpler arrangement. There is a light dam that separates the lights into the fourteen segments, one per indicator/warning. Atop that goes the acrylic panel itself.

Mike noticed that virtually all the existing replicas and simulations have the color wrong for the two lowest lights in the left column of the indicator panel, as had I. It was presumed that all seven lights on the left column are white and all seven on the right column are yellow.

That caused people to implement the PRIO DISP and NO DAP in white, but the NASA documentation we recently retrieved shows that these were in fact yellow. The yellow conditions such as GIMBAL LOCK represent warnings that need to be heeded.

This makes sense for NO DAP, since lack of digital autopilot mode is a serious condition and should be a warning (yellow) rather than simply an indication (white). PRIO DISP warned the astronaut that the original data they were displaying had been interrupted by a higher priority piece of information.

I had already glued the light dam over the indicator panel side of the PCB, but believed I could get in with my hot air rework to make a swap and reach the pads with the soldering tool. I gave it a try.

It appears that the presence of the light dam so near to the LED is deflecting the hot air. I was able to finally pry away three of the LEDs but the fourth (there are two per indicator light) just wouldn't come loose. I finally had to pull the light dam off the board in order to make the substitution.

The new LEDs were soldered in place but glowed too dimly. The two types of LEDs, for white and yellow, required different current limiting resistors and I forgot to switch them over. White needed 1K but the yellow required 680 ohm to reach the same apparent brightness. I removed the old and put in the proper resistors to complete the repair. All is good now, so the light dam had to be epoxied back in place.

My tests with the frosted panels in place weren't good. The frosting is too heavy, reducing the light output and causing unacceptable blurring of the digits and lines. I took the acrylic out and tried to remove the layer of paint on the back, in preparation for attempting a much lighter coating.

I don't have the proper solvents on hand to remove the coating. In any case, I might just buy some more acrylic panels cut to the proper size and work with them. I am done for today.

Mask in place for EL side of the DSKY

Gluing work on DSKY substitute

DSKY SUBSTITUTE FOR THE AGC

Keyboard

I bought some gel superglue, to give me enough setup time to maneuver the cardboard shims into place in the honeycomb. I had designed the shims and had them laser cut to ensure repeatable and exact sizes for all of them.

Laser cut cardboard to serve as shims inside the keyboard

Working with tweezers and the superglue, I carefully placed each shim in place and provided some side pressure on the key plunger until the glue began to set up. Soon I had all the interior shims installed. They keys no longer wobble around - this should provide the smooth movement I expect from the keyboard.

Shims inserted, but dependent on side support to completely support the keyplungers

Because of the design of the enclosure, forced by the narrow margins between keys and the edge of the DSKY, one side of the cell holding the keys on the left, right and bottom sides are formed with milled relief on the aluminum enclosure. I then superglued the remaining shims to the enclosure itself, thus providing the needed side support for those keys.

The last area needing support is the top of the upper row of five keys. My honeycomb is open at the top, thus I need to make and glue some supports to give the final side support. I am still designing that top support and haven't glued it on yet. It is not straightforward, due to a post that supports the faceplate sitting just above the top of the honeycomb.

Display

I ran the special version of the DSKY firmware with scripted events, allowing me to align the light dam for the plus/minus signs and glue it into place. That is the anchor around which I organized all the rest of the layers on the EL panel side.

The next step was to install the electroluminescent wires, using the light dam as the end stop for each wire segment. I lined everything up and epoxied the wires down. Cleaning up required insulating the wiring the joins the three segments and gluing down the cabling.

Glued down light dam for plus/minus sign plus EL wires

Preparing to add text decals and glue up the DSKY display, still working on keyboard

EXAMINING REAL DSKY AND CHECKING ANOTHER ERASABLE MEMORY MODULE

We were given access by the Computer History Museum to the Apollo Guidance Computer and DSKY that they have on public display. The computer is the first block II built, used at MIT to check out each change they made to hardware. The DSKY appeared to be an early block II DSKY, since the indicator panel had three lights (AUTO, HOLD and FREE) that were removed before the versions that were placed in all the spacecraft.

The plan for the DSKY was to record the serial numbers from the plate on the back, which was not accessible while the DSKY was mounted on the wall at the museum. The number matches a slightly later DSKY design, which should not have had the three removed indications I discussed above. Mike will be investigating all the paperwork to try to understand how this hybrid version came about.

I had the chance to work the keyboard, validating the amount of movement and the resistance of the keys when pushed. The actual DSKY feels indistinguishable from the keyboard action I achieved in my substitute, which made me happy.

Seeing the DSKY reinforced how deep is the real item. The enclosure I am fitting into is less than a third as deep. In some ways, that is not a problem because I can use higher density PCBs and components instead of the banks of crystal can sized relays in the real item. However, my interior is pretty crowded anyhow.

We removed the erasable memory module (2K words of traditional core memory) to test it electrically with a modern resistance meter (low applied voltage and current). We were able to confirm that all the addressing, inhibit and sense lines behaved as expected, with all the diodes operating properly in one direction.

That does confirm that our module from the AGC we are restoring does have an open inhibit line for bit 16. It is a final proof for what was already quite overwhelming evidence. There is a note on the memory module from the museum's AGC that indicates that bit 13 of a specific word address drops bits.

Thus, even if this otherwise perfect memory module were to be inserted into the AGC, we would have to carefully check all the software we want to run (e.g. Luminary) to understand the impact of bit 13 of that word becoming zero. At this time there is no expectation that we would be able to plug in the module, but we will do the advance research just to be prepared.

We then sat down to plan out means to test the memory drive electronics (current drivers, switches and sense amp modules) before we insert our memory module into the tray. We will design and build a PCB, populate it with some of the recreated connectors that are being made for us by a sponsor, and use it as a testing bed.

DSKY SUBSTITUTE FOR THE AGC

Keyboard

Marc and I worked out a method to glue cardboard shims into the honeycomb after the plungers are inserted in each cell. This lessens wobble and erratic movement of the key as it is moved up and down by user presses. He did a test on one key by hand cutting and maneuvering cardboard pieces, but to complete the other 18 keys would be too slow and labor intensive.

Thus, I designed the cardboard pieces to be laser-cut at Ponoko.com as well as refining my cardboard masks for the top of the displays. The parts arrived on Tuesday Feb 12th and I did a test fit. Now to do the careful gluing for all 19 keys. .

When the honeycomb is completed, with its shims, I will need to etch out some space on the bottom edge of the honeycomb to clear small features on the surface of the PCB, before I install the coil springs and join the board to the honeycomb assembly.

Display

I refined my ideas for the light masks that sit under the acrylic panels on the DSKY. The new designs were shipped off to be laser-cut; these also arrived on Tuesday. After I satisfied myself with how everything fits together and looks while lit up, I can start gluing up the displays.

For the EL panel side, that involves gluing down the electroluminescent wires, gluing the sign digit light dam, gluing on the diffuser sheet, gluing down the main light dam, then gluing on the light mask at the top.

Quick and dirty test of stack, but still have mylar protective fil which is blurring the output slightly.

I had frosted the back of my acrylic panes, which I tested quickly on the board to see how the light dams, masks and frosting worked out. I was satisfied with the results - they are not perfect but good enough for me to complete the EL panel side. The indicator side is not sufficiently diffused to hide the two discrete LEDs that light up the space. While this is not great, I will proceed anyway.

Illumination behind the PROG indicator space - still have mylar film over acrylic panel

Paint, light blocking materials and other cosmetic work on DSKY substitue

DSKY SUBSTITUTE FOR THE AGC

Display

My cardboard light masks came back from the laser cutting service. I had to make a few additions but they do fit properly. I am now ready to begin gluing up the sandwich of materials on the PCB face. First are the electroluminescent wires, then the black light dam. Next is the cutout for the plus/minus signs. Third would be the diffuser paper over the signs. Finally, the light masks are glued down.

I have also frosted the rear of both acrylic panels. White for the indicator panel and light gray for the EL panel side. These are now ready to have the labeling installed on the front with dry rub-on decals.

Frosted acrylic on left, plus cardboard light dam

I primed and painted the plastic covers that will hold the acrylic panels in place in the faceplate. Once the labeling is complete I can glue the covers over the panels and install them. The covers look fine, as does the case.

Case and faceplate

I cleaned, primed and painted the case and faceplate in the light gray I chose. I did not try to exactly match the Apollo spacecraft color, simply to produce a plausible gray that the brain will associate with pictures of the installed DSKYs.

Painted enclosure and panel covers

I still need to cut openings for cables that will run between the DSKY, Apollo Guidance Computer and external power supplies. Decision time is near - cables through holes or connector installed on side of case?

Creating masks and text for DSKY substitute display panels

DSKY SUBSTITUTE FOR THE AGC

Display

I created the mirror image files for the DSKY display and used the printer at CHM to produce several copies of the dry rub transfer decals which I will use to layer on the acrylic panels. The indicator panel is simply black text which will be backed with some frosted white paint. The EL panel has six white dots, a blue line, and black text to apply, requiring three sets of decals for the various colors. It will then be backed with frosted gray paint.

Three color layers for decals on EL acrylic panel

Multiple copies of text for indicator acrylic panel

The system I use to create the decals has the color layer on the top of the text/image, which means that it is not facing the surface where you adhere the decal. This posed a challenge for my original intent to build the panel by applying decals to the underside of an acrylic panel, since only the underside of the decal will be visible by the user.

I have found that black mylar is good enough even when viewed from the underside of the decal, but I was concerned about how the blue line and white dots would appear. I realized I needed to place these on the top surface of the acrylic, instead of underneath.

The only downsides of placing decals on the top is that they are susceptible to damage and might be visible as raised features. I can spray clear acrylic paint to form a protective barrier over the top. The frosting paint (white or gray) can still be applied to the underside.

I chose to use one sheet for each panel, although I was able to make four copies of the indicator panel on a single sheet so there is redundancy to handle any errors while building decals. The EL panel does not give me the same redundancy, but I printed several copies instead.

xxxx

Another cosmetic detail to build is the framework around the indicator light sections on the indicator panel. I can laser cut them from thin cardboard stock of the correct color, apply to the top of the panel once I have it constructed, then cover all with clear acrylic spray paint. I did have to check to verify whether I have enough margin around the text to build the framework, but that worked out fine after I shrank the font size slightly.

Framework from a real DSKY indicator panel

After designing the framework, I decided to use the remainder of the 7" x 10" sheet of material, the minimum size from the lasercutting service (Ponoko) to make a mask to fit under the EL panel. My idea here is to narrow the size of the illuminated line coming from the EL wire below, plus to constrain the size of the lighted area behind the PROG, NOUN, VERB and COMP ACTY legends to match a real DSKY panel.

This would sit atop the displays, with cutouts for the lighted legend areas, the three lines, the 21 display digits and the sign area, but block all other light from reaching the panel. I measured the spacing carefully and drew up the cut lines with Inkscape.

The lasercutter requires blue lines that are 0.01mm wide to control the laser for a full cut-through of the cardboard stock. Works fine for the cutter but makes the image impossible to print successfully or view on the screen. I made a copy where I could thicken the lines and make them black, solely for the purposes of printing the design.

I printed the design, cut it out and verified that it exposed and blocked all the portions that I intended of the LEDs, 7-segment displays and EL wires on the PCB. That verified, the order was placed with Ponoko and should be back before the end of the week.