Wednesday, November 28, 2018

PCBs sent to fab house and other work started for continued Apollo Guidance Computer restoration

PREPARATION FOR REPAIR OF ERASABLE MEMORY MODULE FROM AGC

Building realistic test modules to practice techniques

Mike has ordered the same potting materials, magnet wire and fiberglass boards that are used in the erasable memory module. He is casting sections that match the actual module as accurately as we can, including breaks in the wire at the points we suspect.

This will let us practice milling, dissolving and laser welding before we work on the irreplaceable artifact. We want to figure out tools, speeds, pressures and directions with these test modules, plus practice to be sure we do the real work as well as possible.

We have time scheduled on a 3D tomographic X-ray machine in mid December, which will give us micron level resolution of the actual module interior and help us find the place where the wire has broken. Once we have that information, we can do a couple more practice runs on test modules configured the same way.

BUILDING THE DSKY SUBSTITUTE

Verification of circuits before PCB fabrication

I set up a MAX7219 and some 7-segment displays in order to validate the exact circuit on the PCB, just in case I got a pin wrong or made some other error that would cause the DSKY to malfunction. Basically I wired up a few of my new green 7segment displays (the components to sit on the PCB) and four discrete LEDs.

An Arduino UNO drove it to prove that the digits display properly and the signs work. I cycled through all ten digits on each display and cycled through plus, minus and blank for the sign.  This worked well, plus I saw that my high output (90 mcd) LEDs were moderated while in circuit with the MAX by some 5K resistors. I suspect the MAX chip would maintain the current regardless but this is a safety feature.

Next up, I temporarily wired up four of the small surface mount LEDs and placed them in their approximate relationship, to verify that it looks like a plus or minus sign. These LEDs are very tiny, 0603 size, which makes tack soldering very difficult. I did manage to light one up and felt comfortable with the light it gave off and its shape, so I will consider this test completed.

The last circuit to prove out are the chips driving the various LED lights that are under AGC control, mainly caution and warning lights but also the computer activity light. These sit on an I2C serial chain and sink the current through up to eight channels of LEDs.

I had used a module with these installed in my breadboard DSKY, but out of caution I wired up the same circuit today to test it again, since I was wiring this directly on the PCB instead of on a module and thus could have the pins wrong. This worked fine when tested.

Final check before sending out the design is to step through the schematic and PCB view, ensuring that every net is correctly attached to the right pins of all components:
  • Twenty-three input circuits with six components apiece, plus input from the AGC IN connector and output to the Arduino connector. 
  • Eight relay circuits with five components each, plus input from Keypad connector and output to AGC Out connector
  • Standby power sense relay circuit with four components, input from AGC EXTRA connector and output to Arduino connector
  • Two loop circuits on AGC EXTRA connector
  • CompActy light circuit with eight components and output from Arduino connector
  • Legends and lines lighting circuit, with 36 resistors and 36 surface mount LEDs
  • Caution and warning lights on I2C chain, with 72 components including 28 LEDs
  • First MAX7219 chain with five 7-segment digits for R1, two digits for Noun, and four discrete LEDs for the plus//minus sign, with 18 components but many nets/traces
  • Second MAX7219 chain for R2, Verb and plus/minus sign, another 18 components
  • Third MAX7219 chain for R3, Prog and plus/minus sign, another 18 components
Almost 1200 pins/pads, 400 components and quite a few connectors comprise this board, so checkout was a long careful process. By the end of Wednesday I had this completed.

Submission to fab house to create the PCB

I transmitted the GERBER files and drill list to the PCB fabrication site once I was satisfied it was (probably) correct. It will take about 10 days to produce, plus shipping time from China to here.

CORE ROPE SIMULATOR DRIVER

Ken finished his design of the driver to hook to the two Core Rope Simulator modules that came with the AGC. It makes use of a Beaglebone controller and a PCB. The PCB was submitted to the fab house yesterday. He is still developing testing plans for proving this out as well as possible before we cable it up to the real AGC next year.

Monday, November 26, 2018

Readied PCB to build DSKY Substitute for Apollo Guidance Computer

PREPARING DSKY SUBSTITUTE PCB FOR FABRICATION

Adjustments to design and components

While I had initially thought that the five pins I needed to connect out of the 16 on the alphanumeric display components were situated so that they don't conflict, upon a double check I found that pins 1 and 8 are both involved, foiling my plan to stack the three signs close enough in a vertical line.

I found a different alphanumeric display component, this one surface mount, that is not as tall. The three fill fit so that the plus/minus signs are aligned with row 2 digits, slightly low on row 3 and slightly high on row 1, but close enough to proceed. It has some contacts that are underneath the component, like a ball grid array, but I think I can manage these with my reflow oven.

The bonus with this part is that it was common cathode. I can drop the separate driver chip and just drive these with the one unused position on each of the eight digit MAX7219 chips. If I wired up the five wires properly, I can light minus by sending a 1 digit to the chip and by sending a 4 digit I will get the plus sign.

Ultimately, however, I discovered that there was not enough room to stack these vertically and fit in the size of a row on the DSKY. If they were pushed together so close that they didn't look badly out of alignment in rows 1 and 3, they could not be routed. I pulled them out and inserted small surface mount LEDs in the pattern of the plus and minus sign.

I dropped out some more connections, mainly by the decision to have the keyboard circuit (yet to be designed) produce the eight output signals that go to the AGC - a five bit keycode plus reset, keypress and proceed/standby. If I had to decode the keyboard it would have taken thirteen signals linked to the Arduino and then the eight outputs, but with the keyboard doing the heavy lifting, the Arduino need not be involved at all, saving 21 pins

Routing of the PCB

My attempted routing was failing even with a 10 layer board, in large part because of all the vias that are generated attempting to route the connections between all the parts. I burned about a dozen hours on the laptop attempting to get a complete routing solution.

To ease the job, I would look at intermediate routing attempts, find components whose orientation could be changed to avoid vias and to simplify the paths. This is iterative with long routing runs in between.

After yanking the 14 segment alphanumeric displays and directly configuring the plus and minus signs, the routing work got slightly easier but several very dense areas continue to cause problems. For example, there are three rows of five 7-segment displays side by side, making it hard to connect the same pin of all five together in the narrow space between pins. This ends up using space on other layers and adding vias in the narrow ridge between one row and the next.

The solution to this was to hand route the pins of the digit displays, also to organize and route all the legend, line and Comp Acty LEDs to minimize via use. I worked on just a few of the driver transistors and relays, but then let the autorouter try to complete things.

My trace sizes are not aggressive but not too conservative either. Minimum signal of 5 mils and power traces at 20 mils. Could go quite a bit lower with the signal traces and a bit smaller on power, but I think the vias are the limiting factor not trace width. After shrinking the vias commensurately, I was able to get a nearly complete routing.

I had a few dozen errors, like trace to trace flaws, trace to via flaws, three connections that weren't completed and quite a few vias stuck in the middle of a pad. Finally, I had a board ready to send to the fab.
Projected view of front of the PCB (courtesy of DesignSpark)
Verification and testing

I began to breadboard the exact wiring I have built on the PCB, checking that it worked properly. First up were the relays with their driver circuits - using the same components I proved out its operation. Second, I verified that the AGC interface circuits correctly resolve the 28V spacecraft level inputs to inverted TTL outputs that my code in the Arduino expects.

I have three more breadboard checks to make - I2C driver chips hooked to LEDs, the MAX7219 driving two 7-segment displays and the custom plus/minus sign, and the various light emitting devices to determine the appropriate resistors to balance out their brightness. These will be done tomorrow.

Wednesday, November 21, 2018

All parts placed on PCB for DSKY substitute for AGC testing

INTERIM WORK FOR RESTORATION OF APOLLO GUIDANCE COMPUTER

First two videos are up

Marc has posted the first two videos of our restoration work onto YouTube at first video of series and second video of the series

DSKY substitute testing

I want to know that the design is 100% validated before I transfer it to PCB as a misstep is relatively expensive. I used an Arduino Uno plus an I/O expander board on the I2C chain to drive some inputs into the DSKY substitute in lieu of the AGC. I hooked these test signals to the 5V output side of the AGC interface boards.

After some time playing with funky wires and connections, I was able to drive the intended digit to every display position. I then tried to light all the caution/warning and activity lights, but my I2C chain was blown by a mistake back when it was unpacked in Houston. I could see the I2C signals coming out of the Arduino and replies from the boards but none of them would do anything with their outputs.

Final DSKY design

Eventually the bulk of the DSKY will be implemented on a single PCB that fits under the spaces for the caution/warning lights and the Electroluminescent display of digits on a real DSKY. It will mount the Arduino underneath, have all the 7 and 14 segment displays plus the LEDs fit on top, plus host underneath all the surface mount driver chips for these displays.

This PCB will have the 28V inputs from the 23 signal lines coming from the real AGC, implementing the level shifting and inverting function of my existing discrete interface boards. The outputs of these interface circuits run to the Arduino. All outputs from the DSKY substitute to the AGC are accomplished with a separate relay board supporting the eight signals.

A few connectors must come off this master board:
  • One will connect the keyboard section below, allowing the Arduino to scan the key contacts for any depressed buttons, using 10 wires
  • A second will connect to the relay board that drives the AGC input lines, requiring 10 wires. 
  • The AGC output lines coming to my device have a third connector with the 23 lines plus ground and 28V pullup. 
  • A complex of connectors that fit the shape of the Arduino
This will be a dense PCB because of all the signal lines for the display digits (147 for digits, 15 for signs), all the caution/warning and activity LEDs (15), the various driver chips, and the need to connect an Arduino Mega with through hole pins and the four connectors. There is not that much room inside the DSKY case which forces the PCB to be 6 1/2" x 4 1/2". 

The 7-segment digit displays are physically the same height as the EL panel digits, 0.5", although the actual digit displayed is smaller at 0.39". They are .372" wide physically and will fit in the space alloted on the panel of .410". I can use the standard part layout in the PCB design software.

The 14-segment display used for the + and - sign is larger - .820" high and .390" wide. This is a problem because the row to row spacing on the EL panel is just .760". I would need to shave 0.03" off the top and bottom, but also deal with the interference of the pin pattern.

Luckily I only need five pins out of the 16 on the chip and they are clustered up on the top half. That means I can cut off the unused pins and not install the holes on the PCB for those, allowing just enough space with the shaving of the display to wedge them top to bottom for the three rows of register digits. To handle this I have to make a custom parts layout for the PCB design software.

The EL panel has lighted lines separating the three register rows, three lighted text areas over the Verb, Noun and Prog digits, and a large square lighted area for Comp Acty. I will use small surface mount green LEDs to form all of these. The cumulative light emission of these surface mount devices will be similar to that of the 7-segment and 14-segment displays.

The caution and warning lights will be surface mounts LEDs, seven each white and yellow sections. For each section I will use several diffused LEDs to spread the light a bit more evenly. The same area which will hold these lights on the top will also contain the single row and dual row headers to mount the Arduino underneath. My other connectors sit in the space between the EL panel and the caution/warning panels.

The surface mount chips to drive all these segments, surface mount resistors for all the LEDs and drivers, and any miscellaneous parts will be spread over the PCB to minimize traces and avoid any thru hole parts. I began mechanical layout of the parts first to be sure everything fit, before going back to a schematic view to indicate connections.

The toughest part was fitting the Arduino Mega headers to mount the controller below the board. My space was already quite limited just from the displays and LEDs sprinkled across most of the PCB.

I grabbed six different dimensioned drawings, allowing me to cross check them and weed out any inaccuracies. Given that, I could figure out the spacing of the header strips for pins 2 to 53, plus ground and 5V. This minimized the locations where I needed a thru hole header strip, increasing the chances that I could locate the Arduino somewhere underneath my board. Part of the problem is the length of the Arduino, which is about 4 1/2".

After laying out all the connectors and the display driver chips, it was time to place the resistors and capacitors. All the LEDs require resistors to control the current through them, adding many dozen resistors for that purpose.

Routing all this circuitry is going to be quite a challenge, even with a four layer board. I might have to switch to a six layer board, bumping up the cost - if I can't get this to route. I won't buy the current limiting resistors for the LEDs until I test out the various LED lights with various resistor values to achieve the appropriate brightness and some balance between all the lighted elements.

Aspirationally, it would be great to fit every circuit needed on this one board (except for the keyboard of course. That is, have the 23 input circuits with their four resistors, one capacitor and one transistor apiece plus the 8 output circuits with miniature relays. Whether that is even possible is the question.

To get the right amount of illumination I needed larger LEDs and multiple of them. To get the power handling capacity for the LED current, the limiting resistors can't be smaller than 0805. Some parts come in only one surface mount size, and the displays themselves are thru-hole. The rest can be shrunk, as in the resistors and capacitor for the input interface circuits.

The relays for the 8 output lines took a bit of ingenuity to stuff onto the already overcrowded board, but I finally found positions where they could sit and not interfere with other devices. They are surface mount, which makes routing much easier.

The brightness of one segment of the 7-segment display is about 3 mcd (millicandela) at 10ma. The alphanumeric (14-segment) display is roughly 2.2 mcd at the same current, but I will run these at 15ma to compensate. My tiny LEDs used for lines are about 3 mcd at 10ma, thus that is the initial setting for them.

The larger LEDs used for the lit legends, caution, warning and activity lights are 90 mcd at 10ma, so they need to be toned down quite a bit. For the caution and warning area, I will use two apiece and run them at 2 ma or less to generate a 20 mcd warning light. However, for the Comp Acty and legends I think I will shoot for 1 ma and install just 2 LEDs for the legends, but 4 for the wider Comp Acty light. That last area might need to be further reduced, but I can make that determination later.

The last few components were placed. First, a driver for the sign displays because they were common anode, unlike the digit display. Next, a pair of 8 port I/O expander chips that attach to the I2C chain and drive the LEDs for the caution, warning and Comp Acty lights. Since my LEDs are high output units that are barely driven at low current to limit the actual light output, these are easily handled by the two PCF8574 chips and their four associated pull up resistors.

My first cut at placement for all the parts is complete. No routing yet, but I will take that on in the next day or so.  I have to very carefully check clearances for all parts and orientation of connectors before the board is submitted to manufacturing.

All parts placed but before routing

Saturday, November 17, 2018

Some repair work or preparation far from the AGC

RESTORATION WORK FOR APOLLO GUIDANCE COMPUTER

Building driver for Core Rope Simulator modules

Ken joined us today to work on a few remaining issues with the Core Rope Simulator modules before building his driver box that will restore these to operation feeding ROM contents to the AGC. Two issues pertained to the external connector and the other two focused on internal connectivity.

The modules have a round Amphenol 39 pin connector which would be connected by cable to an external box that held a tape drive and circuitry. The purpose was to serve up a ROM image to the AGC in lieu of having the six physical core rope modules in the computer.

Ken had found a mating Amphenol connector on ebay but it wouldn't fit into the module's connector because of a mismatch with the keys. Keys are raised edges around the barrel of the connector that need a corresponding slot milled in the other side. The key guarantees that the proper cable is attached and ensure that the pins are oriented correctly.

We had milled some of the keys off but one mismatched key remained. Marc took it off with his vertical mill and we then tried to put the new connector into the one on the module. It barely made contact. We had discovered that the connector bought on eBay had recessed female pins but the original connector in the module expected the pins to be flush with the face.

We then had to mill off the face of the connector until the recessed holes, formed with hard rubber, were removed and the female pins were flush. The connector was ready to insert but we still had some severe oxidation on the male pins.

One of the modules had a screw on cover on its connector, but the other connector was exposed to the elements for decades. The pins on the second connector were so oxidized that they were electrical insulators.  We used Deoxit with an electrical toothbrush until we had removed enough of the oxide layer that the connectors could be mated with good connectivity.

Moving on to the internal issues, we first examined some wires welded to the top of the cordwood assembly in the module. To the eye they appeared to have solder blobs potentially shorting on adjacent pins. Under the microscope, however we discovered both good and bad news. The blobs were some form of sealant placed on the connection after it was welded, not solder blobs. However, we did see some corrosion of the component leads and rust. We don't think they affected connectivity.

The other problem was with the DipStick modules. These are a prototyping or production method of connecting DIP chips with wire wrap. They are a tray that supports five 14 or 16 pin DIP chips, with the chip sitting on a triangular top part and that part inserted into a triangular valley in the base.

The legs of the chip are sandwiched between the gold contacts of the top and the gold contacts of the valley in the base. A metal bar would snap across the chips on the top, holding them in place. The metal bar deteriorated in all of the DipStick modules and fell into pieces, thus the chips are no longer supported.

We tested connectivity using the +5V and ground pins which are an easy set of pins to check, but found that only about half the chips had a functioning connection. After some investigation, Marc found that the issue was tied to the shape of the DIP chip pins.

The top with its triangular shape has no springiness to its contacts. The chip legs must be making good contact as the contact won't bend to span a gap. DIP chips come with their legs splayed outward slightly compared to the angle needed to insert a chip in a socket, as any hobbyist will remember. The user generally has to bend the legs slightly inward in order to insert them in a socket.

Perhaps the broken metal bar would have forced the chip down against the top and pressed the legs into contact, but with the bar missing the chips could sit on the top without having the legs touch the inside contacts. The top has the contacts on the inside, under the body of the chip.

Our solution was to bend the chips to make the legs slightly closer together, so that they will slide on the top and its inside contacts with no gaps. Once the chips were properly prepared before reinsertion on the top, we had restored connectivity.

We are hunting for a second Amphenol connector to connect to the other module. At the same time, Ken is designing a circuit board to mount suitable connectors and a Beaglebone  to serve as the driver of these core rope modules. He will wire up cables to the two Amphenol connectors and plug the other ends into his circuit board.

Erasable Memory Module repair, advance work

Mike has built a 3D model of the inside of the erasable memory module (RAM) which helps us visualize exactly where we need the high resolution X-ray images and to plan out a repair process.
At this point we are going to mill a hole through the epoxy outer potting, open up the fiberglass board near where the broken wire is routed, and then laser weld a repair.

If the wire broke off right at the inner potting (RTV-11 surrounding the RCA built core memory stack), we can then dissolve a bit of that soft potting with relatively safe solvents to expose enough of the #38 magnet wire to accomplish the weld.

All this is based on our expectation for where the wire is broken - based on all the examined failures during the entire AGC program. When we do the CAT scan in mid December we will have exact information about the location of the break. This will inform our plan of attack for a repair.

Thursday, November 15, 2018

More planning for the Apollo Guidance Computer restoration

NEXT STEPS IN APOLLO GUIDANCE COMPUTER RESTORATION

DSKY substitute planning and construction

Now that I have bought displays for the right hand panel of the DSKY substitute, I am beginning to do the mechanical design followed by PCB layout.

I will use a green 7-segment display for all 21 of the digit positions. The size is reasonable but not exact to the size, shape and tilt of the real EL panel on a DSKY.

Spec for digits on the DSKY EL panel
7-segment display for my DSKY substitute
The conundrum was image size. The real DSKY has digits that are 0.5" high but any 7-segment display that would produce such a digit is too wide and high to fit side by side in the 2.6" width of the panel, since I need five digits plus a sign display. Thus, my digits are only 0.39" high, slanted differently and have a different shape for the digit and segments.

The plus/minus sign display in front of the three registers use a green 14 segment display, the same size as the digit displays at 0.39". This will make the sign smaller than on the real DSKY. All the 7 and 14 segment displays fit across the space properly.

The EL panel also has a square in the upper left for COMP ACTY which will flash based on computer activity signals from the AGC. There are text blocks which are illuminated to describe the PROG, VERB and NOUN display areas.

Finally, there are three horizontal lines, separating the three registers, which light up. I bought a good quantity of surface mount green LEDs  to fit on the PCB to create all these lit areas. I will have to put some 'light dams' in place to keep the LEDs for one area from spilling over to the other sections of the panel.

I need a slightly frosted greenish-gray plexiglass cover over the displays and LEDs, with the text legends applied. I know how to make and apply the legends. The EL panel is rimmed with an aluminum cover, which I will need to figure out how to make for my replica.

Real DSKY whose appearance I will try to approximate
The left upper panel, with the caution and warning lights, is going to be relatively easy to make. Regular thru hole LEDs in white and yellow, respectively, will be fit on a simple board, fitted with 'light dams', and covered with a frosted plexiglass (whiter in color than the right hand panel) having the appropriate legends applied.

The buttons on the keyboard are more of a challenge to create. My aluminum housing is already cut to the size of the button openings on a real DSKY, forcing me to come up with something similar to keep the appearance reasonable.

I am still investigating materials for the 'light dams' as well as the keyboard buttons.

Erasable Memory Module repair plans

We continue to research the failures encountered and the various versions of the erasable memory modules produced. Wire failures in the modules were a continuing problem during the Apollo program, leading to a variety of fixes and versions.

The module type we own is a relatively early design. It is possible that the wire is cracked inside a fiberglass bolster plate where the wire feeds through from the outer epoxy potting material to the inner RTV-11 potted core stack. This would be more challenging to reach and repair than the later versions whose failures were out near the mounting pins.

The actual plan of action will depend on the images we can get of the inside of the module. We will be using a micron resolution tomographic scanner, similar to a medical CAT scan, that gives us a 3D image based on many slices taken throughout the module. This will give us the exact location of the break and a view to how it may be possible to access the wire for repair.

Core Rope Simulator driver design

Ken Shirriff has almost completed his reverse engineering of the two core rope simulator modules and determined what he has to build to connect to the two 39 pin circular connectors on their faces. He will based the driver on a Beaglebone board and has selected cables and connectors for his driver box.

Monday, November 12, 2018

Restoring an Apollo Guidance Computer, part V

RESTORATION OF APOLLO GUIDANCE COMPUTER

Background of this AGC

The owner of the AGC, Jimmie Loocke, was browsing through a scrap metal recycler's warehouse in search of some parts for a project when he spotted a large pile of surplus NASA items that he recognized. Among them the AGC! On the spot, he bought two tons of the material for scrap metal prices and hauled it home.

In an odd twist of fate, Jimmie had worked for a contractor that was performing thermal tests of the Apollo spacecraft at the Manned Spaceflight Center in Houston. His job was to control the thermal profile of the test chamber as they checked out the response of a Lunar Module to the expected thermal conditions in space and on the surface of the moon.

NASA bought some test versions of the LM in addition to the machines that were used on flights. This AGC was installed in LTA8, test article 8. LTA8 is on display in the Johnson Space Center (formerly MSC), but the AGC from inside it was sold off by General Services Administration.

This is a block II AGC, the type used on all the manned flights. It was number 14 of the 15 prototypes built by Raytheon, which is why it has only two modules that are potted. We are very fortunate to have a machine that is identical in wiring and logic to the production machines but have access to the circuitry in all the unpotted modules.

The AGC is hermetically sealed and sat with an internal nitrogen atmosphere for decades before its recent opening by the owner and by Eldon Hall, the lead designer of the computer, Mike Stewart and now the restoration team who are volunteering their efforts to get this historic artifact in full glorious operation once again.

Addition of erasable memory to the tool and further traces

Mike added in the Verilog to provide erasable memory for the AGC, wired up the additional lines to the AGC backplane, and now the computer is running long sequences of instructions with almost every one acting as expected.

The use of breadboards, jumpers with pins and the large mess of wires running between AGC and tool is likely the cause of the sporadic misbehavior. At least we would consider the wiring to be the most likely cause of the occasional misbehavior and not yet suspect a defect in the AGC.

It was great to watch hundreds of instructions execute with the correct results as the AGC ran the Aurora restart code from o04000 until glitches in data returned from the tool caused the stream to diverge.

Design of final DSKY after testing of the prototype

The prototype unit testing last week uncovered some weaknesses in the design that need correction. The AGC and DSKY design refer to the least significant digit of each display as digit 1, but the prototype numbered the digits left to right starting with one. The method of flashing the Verb and Noun displays stressed the MAX7219 chips which drove the seven segment modules.

We have a milled aluminum case that is very close if not exactly the dimensions of a real DSKY, into which I will build the DSKY substitute. It won't be an actual replica of the gate by gate and relay by relay design, but is intended to appear similar and behave the same as far as the user and AGC are concerned.

It would be very hard to achieve a full gate by gate replica, in part because the electroluminescent panel on the right side of the DSKY is unobtainable. That obviates the need for 250V 800 Hz power which is what the relays switched to the various EL panel segments, making the internal construction different from the real DSKY.

In the prototype, I used three MAX7219 chips each on a separate module that I bought online. I did this for speed, since I had just a bit over one week to cobble this together and wanted to skip any assembly and design steps I could to speed things up. In a real design, those three chips would be strung together on one serial chain, but the prototype used three independent chains.

Since a MAX7219 supports eight 7-segment displays, I apportioned the DSKY digits among the three by using the first five digit positions for a register, two more digits for one of the Verb, Noun or Program displays, and ignored one position. For example, Register 1 and Verb were on one of the boards.

My first cut at blinking the Verb and Noun digits, while maintaining a very tight loop in the Arduino, was to use relays to interrupt the common cathode wires to those four 7-segment modules. A 555 timer set at 1.5Hz was gated through an IC to fire the relays on an off the shelf eight relay module.

This approach didn't seem to sit well with the two boards hosting these displays, which began to exhibit odd behavior that suggests I fried the MAX7291 chips. It worked initially but after about 20 minutes of operation, everything went downhill.

I might make use of a newer driver chip, the MAX6955, which can blink individual 7-segment modules without the Arduino doing more than initiating the state and rescinding it. I will also chain these together to make use of a single serial link.

I found that the design could be reading a command word from the AGC surrounding its change, because of the quick and dirty use of the digitalRead() method which is slow and non-atomic. By directly accessing AVR registers in the Arduino I can see all 15 bits at one time. Lastly, when I see a non-zero value coming in from the AGC, I will delay 5ms before sampling again to eliminate the risk of reading a glitched word.

Fitting in the aluminum case will require sourcing and placing 7-segment and 14-segment displays more precisely and fitting everything tightly on a PCB. I found green colored display modules that are close enough to the actual DSKY size that they will appear almost full sized and fit across in the allotted space.

The prototype used a cheap membrane switch array but I need to support the large square button format of the DSKY in the final version. I may need to construct the buttons and related mechanism myself, embedded a microswitch inside to actually read the contacts. More research is needed on this.

Discovery about potting materials used on the erasable memory module

After some archive detective work by Mike and Ken, we are of the opinion that the potting material that we need to open to repair the wire break is not RTV-11, but Stycast 1090, an epoxy resin. Different solvents or methods will be needed to make a small hole for our repair work.


Saturday, November 10, 2018

Restoring an Apollo Guidance Computer, part IV

RESTORATION OF APOLLO GUIDANCE COMPUTER

Looking for suspected design flaw/race hazard

Mike Stewart had modeled the exact schematic of the AGC, gate by gate, which he had used both to understand the system using simulation and to create his own instance of an AGC in an FPGA board. One of the discoveries from his simulation was a glitch or transient signal that impairs operation of one of the hardware alarms. The question - does this occur in the real AGC?

Since the Apollo Guidance Computer acts as the digital autopilot for the astronauts, is in between the hand controllers and the attitude jets, and is responsible for navigation, it must be very reliable. A number of hardware checks are built in to detect software bugs by recognizing undesirable behavior.

One of the clever checks added to the hardware is a timer that watches to see how long it has been since a control transfer instruction (e.g. branches) have been executed. If code is in an infinite loop, this timer would signal an alarm and force a restart of the computer, hopefully restoring function to the spacecraft.

Mike had discovered a transient that would occur relatively frequently that resets the timer, the way it would be reset by a control transfer instruction. Since this glitch is occuring independent of branches, it has a good chance of defeating the check and delaying, perhaps indefinitely, the detection of the infinite loop.

We set up the oscilloscope to watch the condition he had observed in his simulator and indeed this glitch occurs relatively often. He had developed a test for this problem, creating a two instruction loop with a clever program that does not use transfer of control instructions. When he runs it in the simulator, in the software AGC simulator available on the web, and in his FPGA, it proves that the detection of the loop is seriously delayed because of this design flaw.

Testing a longer stream of instructions

Mike adjusted his memory tool and achieved much longer run times, for example having the AGC execute a few thousand instructions of the Aurora core rope software beginning with the restart/start code that the hardware runs at o04000

All of the instructions examined produced the proper results, although the flow of the real AGC diverted from the flow under the simulator at the 100th instruction. The deviation was not due to a defect in the AGC but completely reasonable because of the state of the memory tool right now.

The memory tool is currently wired up to serve up only the fixed memory (core rope memory) contents. This means any access to erasable (RAM) memory returns o00000 as a result. As code executes, at some point a conditional branch will go a different way, because the erasable memory would have had a non-zero content but under the tool it is different.

The actual case we monitored was not an issue of branch conditions, but due to a programming technique that was used. The address of a routine to execute was stored in erasable memory and then the programmer attempted a Transfer Control through that address. Under the tool, we branched to location 0 instead.

PROJECTS TO CONTINUE AFTER LEAVING HOUSTON

Our team reached the end of the trip and we are flying back home this weekend. It has been a very satisfactory first week of work but more needs to be done to achieve our goal of a demonstration of a fully restored AGC running actual Apollo mission software.

Different memory tool implementation

The temporary memory tool is connected to the AGC by a myriad of wires pushed onto the wirewrap pins of the tray A backplane. This is not a rugged or cosmetically appealing approach for the long term. Instead, Mike is building a different version of the tool that will connect to the AGC using the maintenance connector (A52) that is accessible on the outside of the computer with all covers closed.

A52 was used to connect to Ground Support Equipment (GSE) and test out the AGC in a number of locations. Raytheon used it during manufacture, North American Aviation used it to test while installed in the Command Module, Grumman Aerospace Engineering Company used it to test while installed in the Lunar Module, NASA used it to check out the spacecraft in the Manned Spacecraft Operation Building (MSOB) at Kennedy Spaceflight Center, and also used it for testing while the vehicle was sitting on the launchpad.

This connector allows the monitoring of key conditions and control of the computer, including the ability to deliver data that overrides the contents of AGC memory. Initially, A52 allowed override of core rope memory contents but not erasable memory. Later changes added the ability to override the erasable as well.

Our AGC, which was one of the last prototypes before the production versions of the computer were shipped, varies only slightly from the final machine. Our chassis is made of aluminum instead of magnesium, as the weight reduction was important to the production machines that would fly. Most of our modules are not potted, which is great for testing and repair. However, the function to allow override of erasable memory was not installed on our box. We intend to add the option.

Adding MAMU pin to A52 maintenance connector

The signal to override erasable memory (MAMU) is a trivial change to implement and simply adds a standard feature of all production machines. We found a spare NOR gate with an open collector, allowing us to tie that to the existing NOR gate which drives the strobe to read erasable memory. Thus, when our gate activates, it will suppress the strobe.

This requires that we add three wires to the backplane. One will run from the output of the spare gate we chose to the output of the existing eraseable strobe gate. One will run from the unused input of our spare gate to ground. The third wire will run from the open input gate of our spare module over to the MAMU pin on connector A52.

We will wire wrap these new wires, using a different color wire and documenting what we did, allowing the AGC to be restored to its 'as found' state if desired by the owner sometime in the future. Restorations of rare artifacts typically apply a similar process, saving any removed parts, marking any substitutions, and providing a path to restore the artifact to its prior state.

Core Rope Simulator activation using a team built driver system

Our AGC was run with Raytheon built modules installed in the core rope slots. These modules were part of a core rope simulator that connected via two thick cables to some GSE which would respond to any fixed memory request with the appropriate data word. Unfortunately, we did not receive the thick cables nor the GSE, just the two modules which plug into the AGC.

Ken Shirriff spend much of the week reverse engineering the Raytheon modules. We have no schematics or other documentation on these, but think that it would be possible to connect to the modules with some electronics of our own devise that would drive them. In other words, a partial substitute for the GSE associated with the rope simulator.

He is wrapping up the documentation and study of these modules. With a good definition for the 31 data and control lines running over the thick cable, he can control these modules using simple digital signals over differential wire pairs. We have found an old connector that will mate with the Raytheon modules, except that some keys on the connector block insertion.

We will mill off the keys that don't match so that we can insert our connector into the module, substituting extreme care for mechanical keys to ensure that the connector is oriented properly.

Once we have circuits to control the Raytheon modules, they can respond to fixed memory address requests and serve up the data words from within our unit. This is the preferred way to provide fixed memory to this computer, since the modules are standard Raytheon units that were on the machine and historically accurate.

Rope software is available in many versions, corresponding to most Apollo missions. We will have these in our box and selectable to run the AGC as a CM or LM on whatever flight we desire.

Repair of erasable memory module and checkout in the AGC

The preferred way to supply erasable memory to the AGC is by repairing the actual module from this computer and running with it in tray B. As previously described, the module has a pin with open connectivity where we should have a inhibit line for bit 16. We hope to repair this defect.

The module is fully potted - filled in with a solid material covering all the wires, pins and interior components. We believe this material to be RTV-11 but will do some testing to confirm. If RTV-11, we know of solvents that will slowly dissolve the potting material allowing access.

We don't want to remove all the potting. Instead, by analogy to arthroscopic surgery, we want to make the smallest hole feasible, accomplish the repair, and fill the space back in using new RTV-11. To make this happen, we must know exactly where the break is within the module and be sure it is in a location where it can be repaired.

Apollo era documentation informs us that quite a few of the modules failed during testing or use, but these all were breaks where the wire runs from the external pin down to the memory stack itself, not somewhere inside the memory stack. The stack is folded multiple times to fit in the small space, making access to wires inside the stack impossible.

To spot the location, we are going to make use of a 3D tomographic X-Ray machine to give us an exact picture of the location of the fault. Based on the X-Ray, we will build a plan for the repair, make our opening into the potting and hopefully fix the module. The X-Ray is scheduled for late December, based on availability on the high tech imager.

Conversion of DSKY substitute prototype to a finished and rugged state

My DSKYsubstitute was a breadboard hodgepodge used to prove out the design before I commit it to printed circuit boards and more permanent construction. We are not able to drive it from the AGC yet, because the software that communicates with it hasn't run and won''t until we can provide erasable memory, either with a tool or the repaired module.

I was, however, able to test it with a pattern generator and look at the function of much of the unit. We discovered a few places where the design has to change, such as in the method of flashing the Verb and Noun displays. We also discovered the potential to react too quickly to the AGC output and select the wrong digits, signs or lights.

All these learnings will be fed into the design, an improved prototype will be tested and then I will build a PCB based substitute that can fit into an aluminum case milled to match the real DSKY. We were fortunate enough to receive the milled case from the owner of the AGC and can use it to house the substitute.

Producing realistic spacecraft inputs to the AGC

One final aspiration would be to produce realistic inputs over the main spacecraft connector (A51), such as the pulses that come from the inertial navigation unit, telemetry uplink, and so forth. Similarly we would find ways of displaying important outputs of the AGC.

As an example, if we have the inertial gyros and accelerometers providing input, and can display the pulsing of the Reaction Control System (attitude jets), maybe implement the hand controllers and watch RCS activity as we request rotation or translation, then we would have a more compelling demo for the 50th anniversary of the Apollo 11 landing in July 2019.

Friday, November 9, 2018

Restoring Apollo Guidance Computer, part III

RESTORATION OF APOLLO GUIDANCE COMPUTER

Developing tool to substitute for both fixed and erasable memories

Having a computer with no RAM or ROM is a recipe for boredom. Since we have to wait until we can x-ray the erasable module before we start opening up the potting near the break and repairing the wiring, we needed an alternative way to exercise the computer.

Mike had a work in progress to use connector A52, the ground maintenance connector, to do this job but it requires a connector that we don't have. The AGC uses mini-WASP connectors which are unobtainable, thus we would need custom manufacturing of a replacement.

Using the NASA sourcing control documents (SCD) from the NARA archive, we can exactly specify the shape and dimensions of this very non-standard connector. We are hopeful that we can get the main spacecraft/DSKY connector A51 and the maintenance connector A52 produced because a high technology connector company is offering to do this to sponsor our work. It will take some time to receive such a connector.

In the meantime, Mike adapted his FPGA based design to connect via backplane pins. This would watch for addresses and the strobe signal that normally triggers a core read (two strobes, one each for fixed and erasable memory). It would store the entire 36K contents of core rope and host the entire 2K of RAM on the board, feeding the appropriate word to the AGC at the correct time.

The first cut of this would only implement fixed (core rope) memory and serve as a proof of concept before wiring in the lines to update RAM as needed to emulate erasable memory. We wired it up and connected it to the AGC for our first test.

Nothing happened. The board didn't appear to see any addresses or control signals. The interface chips that we were using, chips that work fine converting between 3.3 and 5 volt systems, weren't working for the AGC.

Logic level challenges hooking up the memory substitute tool

The RTL logic used in the AGC, while defined as ranging between 0 and 4V for logic 0 and logic 1, produce much lower voltages for a logic high. We see valid 1 bits with voltages as low as 0.8 volts at the inputs, which is just too low for the MOSFET based level shifter we were using.

Since the actual voltages were lower, not anywhere near 4V, we then tried wiring the AGC directly to the 3.3V level FPGA pins but that didn't work either. LVCMOS logic levels define the lowest valid high as a bit higher than 2 volts, so nothing would register.

Thus, we have to build discrete level shifter circuits to make sure that a 0.8V or higher input will give a crisp high output into the FPGA.

Building lots of interface circuits between memory tool and AGC

There are 26 input lines, each requiring one transistor and three resistors for its interface. After wiring up the interface circuits and connecting them between AGC and FPGA, we did successfully see all the signals from the computer. Using pull up resistors and open drain output mode on the FPGA, we can drive in the data signals to the AGC just fine.

FPGA tool with level shifter circuits, attached to AGC
Testing the DSKY substitute tool using a pattern generator

Marc set up his pattern generator to drive my DSKY substitute while we worked on a few wiring and power issues. We improved its operation quite a bit, although more work is needed to avoid glitches on input from the AGC and to fix suspected damage to some of my control boards.

Testing out DSKY with pattern generator
I think the trauma of moving it on the plane and resulting shorts and dislodged wires caused a few chips to fry. It will come home and be improved, tested and then a final version of the hardware will be built.

There are some places where I need to shift to direct access of the AVR chip registers, bypassing the Arduino functions such as digitalRead(), in order to see all 15 bits from the AGC at one time.

Testing the memory substitute tool

Marc had to leave with his large logic analyzer before we got the memory substitute tool working, leaving us with oscilloscope traces as a diagnostic snapshot into the processor. Still, that gave us enough to see what words we were proving to the AGC. Coupling that with the listings of the Apollo software that are running let us verify both the tool and the AGC operation.

There are a few bugs in the tool right now, so that after a short burst of correctly delivered words, the tool messed up then went into a hung state. Nothing systemic or difficult. Given a bit of time to clean things up and test, this tool should be reliably feeding core rope contents to the computer.

Now, to the more important question. Did the computer work properly? Admittedly we have a small sample, only a few dozen instructions running in any test shot, but it would seem that the AGC is flawlessly executing everything it receives. This includes complex behaviors like inhibiting interrupts, extend mode processing, and use of the ALU within instruction execution.

Without the erasable memory support wired up, eventually the code would go awry even if the tool keep feeding fixed memory data perfectly.  Once the tool is whipped into shape and the erasable memory support is turned on, we should be able to run the Apollo software at least to the point where it requires some input from the spacecraft or the DSKY.

Checking out first few instructions executed by AGC

TO BE CONTINUED

Restoring Apollo Guidance Computer, part II

RESTORATION OF APOLLO GUIDANCE COMPUTER


Testing the analog alarm module from the B tray

The alarm module has a number of circuits that look for error conditions and raise alarms. The alarm will cause the computer to restart, by forcing in a branch instruction to a fixed address in core rope memory (octal 4000). The AGC will continue to restart until the alarm goes away, at which point it completes the restart sequence and is back in operation.

We were able to get all of the circuits verified except for one. This was the circuit that looks at the 28V in from the spacecraft bus, the 14V produced by one power supply and the 4V produced by the other supply in tray A.

It should trigger an alarm if our two generated voltages are above a max level or below a minimum voltage, and it should trigger an alarm if the spacecraft main bus (A or B) falls below a minimum.

When we varied the voltages and watched the alarm signal, it did turn on when we went into the invalid ranges. However, it latched in that condition and wouldn't turn off when the power returned to healthy levels. In a real machine, this would be a problem because the only way to recover would be to pull circuit breakers, a cumbersome and impractical action; thus we knew this should not be happening.

However, when we inserted the module in tray B and did the first bringup (below) we found that the alarms automatically reset when power reached good levels. Not sure why we couldn't recreate that on the bench, but it is significant that this one module is a different level from all the schematics we use.

First bringup of the computer (without memories)

We powered up the computer with some oscilloscope probes attached and saw that the system appeared to be executing, so it was time to wire up the logic analyzer to record enough to know what is going on.

The analyzer showed that we were in restart until the power alarm went off, but then took a different alarm and went back to restart. That alarm was parity error! Since we have no erasable or fixed memory, any read of a location will see the contents as all zero. Since this means the parity bit isn't delivering odd parity, we raise an alarm condition.

Erasable memory module (with open circuit on inhibit line)
This makes us loop, as each time we restart, our second instruction tries to execute at o4000 but gets a parity error, taking us back to restart ad infinitum.

Bypassing the parity error alarm to continue testing

A maintenance connector on the AGC provides a wire that we can raise to logic 1 to disable alarm conditions. We connected it to a pullup resistor and indeed see that the processor does not loop on restart any more.

Traces and verification in the absence of memory contents

There is a lot we can validate even with all memory accesses returning o00000 so we ran traces on the logic analyzer and pored over the entries. Everything we see is correct operation, no sign of a flaw so far.

Marc studying the logic analyzer traces
Once the system comes out of restart, it branches to location 0. This happens because the contents fetched as the first instruction come out o00000 which is interpreted as a transfer control to 0. Normally you would expect that this would loop forever fetching a transfer to zero, but a few other things are going on.

The AGC supports hardware counters that track spacecraft information. A function in the spacecraft may request that a counter be incremented or decremented by sending some pulses to the AGC. The hardware will stop executing instructions and instead force a special instruction like INKL or DINKL to change the counter.

Counters are fixed addresses in erasable memory which are updated by the cycle-stealing function of the INKL, for example. The software doesn't see the increment or decrement, but can read the current counter value at any time by looking at the memory location. With the decrement instruction, if the counter reaches zero an interrupt can be generated

This is what we see happening. After restart, we see INKL and DINKL commands executing because they are top priority and are set up by logic. Of course, the instructions fetch the current value which is always zero, add or subtract, then try to store the updated value but no memory so still zero.

The DINKL sees it result, the zero, and triggers the interrupt. Once the counter increment and decrements are done, the interrupt is processing by branching to the interrupt handlers up in fixed memory. The instruction comes back as o00000 a transfer to location zero.

The other non-intuitive result is that we don't execute the transfer to location zero, because the read request returns with a non-zero value! What? We have no memory. Well, it turns out we do have a limited memory.

The architecture of the AGC uses the first seven or so locations of memory as registers - accumulator and others. The registers need to be much faster than a core memory cycle so they are implemented with flip flops in the machine. Reading memory at 0 will return the current value of the accumulator and treat it as an instruction.

This turns into an instruction referencing memory location o3777 because the flip flop starts out with all bits turned on. We see that happen in the trace. Once the processor steps through the first few register contents thinking they are memory contents, we reach the first non-register location, get the transfer control to 0 instruction and start the loop over.

We need to put some data into the system to try out different instructions. Eventually we need to provide fixed and erasable memory plus the software that was coded into the core rope modules.

TO BE CONTINUED

Restoring Apollo Guidance Computer, part I

RESTORATION OF APOLLO GUIDANCE COMPUTER


System oscillator (2MHz)

We provided 14V and 4V to the clock oscillator module (B7), limiting current to protect against any failing capacitors or other shorts. The output was beautiful. We have a pulse. It is running at 2.04801 MHz, amazingly close to spec for a machine that hasn't run in almost 50 years.

Pulse of the Apollo Guidance Computer

Testing the logic modules (contents of tray A)

The AGC is built of two trays that are bolted together, with mostly digital circuits in tray A and mostly analog circuits in tray B. For example, the master oscillator is module B8 because it is analog, but all the further processing of the clock is done digitally, in modules A1 and A2 in tray A.

The logic modules are constructed exclusively of dual NOR gates, each with three inputs. These are Resistor-Transistor Logic (RTL), which was quickly obsoleted in the industry by DTL, TTL, CMOS and other schemes. After a while working with the schematics, one gets proficient at quickly understanding designs using only NOR gates. No flip flops, no other type of boolean operations at all.

The RTL chips operate with 4V power supply and have a pretty low threshold between logic 0 and logic 1 condition. We found voltages as low as 0.8V that were successfully recognized logic 1 states in the running AGC. Logic 0 inputs were typically under 0.3V in actual operation.

Clock dividers and scaler, other timing signals

The oscillator delivers 2MHz to modules A1 and A2, which first divide the signal to the master oscillator of 1MHz and then produce various timing cycles that will drive the rest of the computer. Most notably, the computer is designed around core memory cycles which take almost 12 us, using twelve of the 1MHz pulses as stages 1 to 12.

We injected appropriate clock signals into the two modules as they sat on the workbench, out of the AGC trays, and verified that all of the timing signals were being produced at the proper time. We soon discovered a failure!

The AGC uses a variety of timers of different durations, from almost master clock frequency down to one that has a pulse once every 36 hours. This is implemented with a chain of dividers in A1. We found that the time pulses stopped somewhere in the middle of the chain.

All the other timing logic in A1 and A2 worked fine. With the failure of our clock scaler, the machine would not properly detect certain errors (watchdog alarms) that depended on slower pulses, it would never go into Standby mode, and of course mission software that needed longer timers wouldn't work right. Most of the machine would still work even with the one gate failing.

Timing signals (some of the 12 steps of a memory cycle
Investigating the failing NOR gate

After some probing around and other testing, we concluded that the output of one NOR gate was up at 4V. This means it was shorted to the VCC power supply. Inside the NOR gate there is a pullup resistor between VCC and the output, but this was shorted. It is an odd failure, one that seemed only possible if some bit of conductive material had fallen across some the two traces, VCC and output; these traces must be close to each other somewhere for this failure to have occured.

We shook the module with the package facing down, retested, and found that the problem had cleared. We will perpetually have the conductive junk, probably a solder blob, inside the sealed gate. Problems could recur but we know that some shaking will clear it up again. We will live with this situation.

SQ (instruction op code) register and decoding module

The next module we placed on the bench, A3, provided the storage of the instruction operation code and performed some of the initial decoding. A module has two boards, one per side. Each board has up to 60 flat pack ICs with two NOR gates apiece.

We worked to find combinations of inputs that would put each and every gate through all its states and give us a way to observe its output. That is, we wanted to see that the output was high when all three inputs are low, but that each input when raised would drive the output low.

This process was very labor intensive and slow, because of the need to find a way to test every input of every gate, isolating others. Since many of the 120 gates on a board in a module are only connected internally on the multilayer circuit board, this is logically challenging.

Some checking of the logic modules
After ten hours of work we had confirmed that every gate worked properly. One possibility was to continue this way, bench testing modules A4 to A24, but at ten hours each that would be a very long process.

Decision to check for destructive failures only in the remaining logic modules

We decided that our sample of 720 gates with only one intermittent failure was a sign that we would expect a low rate of defects. Thus, it would be reasonable to put all the logic together, power up, and debug using logic analyzers. A small number of defects can be chased down this way, but a system with high rates wouldn't do enough to capture anything meaningful.

The only type of failure that would be physically damaging would be short circuits that might produce high current and melt something. We have power supplies that will limit the current to a set level, which means we can power a module and see if the current is too high but not let it go high enough to cause damage. 

We proceeded to power each of the remaining logic modules, board by board, while monitoring current drain. They all passed and ran at levels between 80ma and 110ma per board. Given this, we could stuff it all into tray A and power up the machine. 

Testing the twin power supply modules

Tray A also has two power supply modules. These are identical but look at the connections across certain configuration pins to decide if they will produce 14V or 4V, the two main voltages used across the AGC. We bench tested each module with both configurations, even though in actual operation each module is assigned one of the voltages. 

During power supply testing
The power supplies produce two outputs - switched and unswitched - so that the computer can sit in standby mode with very low power consumption since only a small subset of circuits are powered. We checked that both the switched and unswitched outputs were proper and clean. 

Testing the analog interface circuits between the rest of the spacecraft and the AGC

There was five remaining modules in tray A and they were not digital logic. These provided the interface circuit to couple a spacecraft signal, typically implemented as 28V / 0V logic levels and the AGC gates that run with 4V / 0V levels.

We set up the bench to power each circuit in each module, inject the appropriate input conditions and verify the correct output. This work only took a half day to wrap up modules A25 to A29, and once done, let us fully populate tray A for testing.

Testing the analog modules
Working on tray B - the memory and alarm functions

Tray B has the master oscillator (B8) which is needed, plus an alarms module that checks for various analog style failures. For example, it verifies that the clock isn't running too fast, that the 28V input power and the internally generated 14 and 4 volt power are within limits, and other checks.

We did not install any of the modules involved in memory access - rope or erasable memory drivers, selection logic or sense amps. Nor did we install the erasable memory module itself. The six core rope module slots on the outside of the AGC are also empty.

We then plugged tray A and B together. Power and control signals must move between the two trays over connectors that only mate with the two trays are together. The module faces of the two trays face each other when mated, with the wired wrapped backplanes facing outward.

Trays mated, tray B on bottom with open slots for core rope modules
Examination of the core memory module (erasable memory)

Mike checked continuity of all the pins on the erasable memory module. Since this is one of the few modules that is potted, we couldn't see inside and only had the pins to test. Modules on computers that would fly on missions had every module potted. A material like RTV-11 hardens around the components and holds them from vibration.

Unfortunately, we discovered that one of the pins was an open circuit. Each bit of the memory has a power feed and two inhibit line returns (among other pins used for other purposes), but bit 16 has an open circuit on one of the two inhibit lines.

That will mean that at least half, if not the full 2K of memory will always have bit 16 set to 1. Not only will this limit what op codes and data we can use, even for limited testing, but it will cause the parity check to fail on average half the time (i.e any time when bit 16 should be zero but is read as one).  We will need to correct this in order to have a workable erasable memory.

Examination of Core Rope Simulator modules

This AGC came with a rare Core Rope Simulator rather than six actual rope modules. The two boxes we have plug in and act as the six rope modules, but are cabled to an external box that we don't have to complete the functionality.

Typically, the rope simulator has a magnetic tape drive and some RAM. A new version of software is read from the tape, put in RAM and then served to the AGC as the contents of rope. Test and development centers would find a unit like this convenient, instead of waiting for a real rope module to be wired and installed to make any change.

Since we don't have real rope modules and don't have a complete Core Rope Simulator system, Ken is reverse engineering the simulator boxes we have. We need to do this in order to create hardware to drive the boxes. Alternatively we will need to build a different kind of tool or box to respond with the proper data word anytime the AGC requests it.

Ken reverse engineering the core rope simulator boxes
Verification of level of the AGC

Mike went through the backplane wiring and logic module, looking at documentation of each of the versions of the AGC and comparing what we had to those variants. He discovered that our backplanes were brought up to the latest revisions, although a couple of the modules were not equally updated.

All this meant was that some added functions, while wired on the backplane, aren't instantiated in the modules. Based on the product code and serial number, our AGC was initially built as one of the last of the block II prototype systems, just before Raytheon switched over to production versions.

The prototypes went to the Kennedy and Johnson space centers and to the major contractors such as North American Aviation and Grumman. Ours was used at Johnson space center, although it didn't have that name back during the Apollo program.

The reworking of our unit made it functionally identical to the production versions that would fly on the various missions. This was good news since we want to be able to run any mission's software.

First bringup of the computer (without memories)

In the next part of this blog we will bring up the AGC, without memories, to see in what shape the computer is after all these years.

TO BE CONTINUED

Friday, November 2, 2018

DSKY substitute as complete as I could get it before flying to work on AGC

PREPARING FOR RESTORATION OF APOLLO GUIDANCE COMPUTER


Final construction and testing of the DSKY substitute

I finally finished building all the input interface boards that will safely and compatibly hook the Apollo Guidance Computer outputs based on 28V logic levels with the Arduino Mega 2560 driving the DSKY substitute.

This took half of the morning to wrap up. Twenty four components and ten wires are soldered on the board, which has no traces so the component leads have to bend over to form connections. I then had to inspect all of them for shorts or other errors.

Fifteen of the input circuits hook directly to Arduino input pins. Another eight input circuits are hooked to an I/O expander board on the I2C chain. All eight of the output circuits run through an 8 channel relay board that makes use of my output interface board.

I finished up the wiring of the second 8 channel relay board, which is used to interrupt the cathodes of the 7-segment displays and LEDs to make them flash. A small board gates a 1.5Hz signal through to the relay triggers when the control signal from the AGC is on. The relay clicks on and off to darken the displays.

I am having some difficulties getting the solid wire to hold in the relay board - the design of the connectors are a bit rickety. I tried folding the wire in a U shape, hoping that the two bits of wire would provide a flatter surface for the connector to hold. That did the trick.

The only displays that flash are Operator Error and Key Release caution lights, as well as the four digits of the Verb and Noun displays.That requires six of the eight relays on the board to interrupt their cathodes.

I struggled to find a way to mount the main breadboard panel with the displays and lights  raised off the tabletop, to allow all the support boards and wires to be tucked under or made less visible. Perhaps I could tilt it vertically to the same orientation as a real DSKY, I thought. The keyboard should be at the same level as the display and stationed appropriately.

Cable ties were judiciously applied to neaten up the rats nest of wires between all the boards and modules of this prototype. Transporting this safely will be a major concern. My carry on bag will hold this and a few other fragile items; my laptop and all my other personal items will be wedged into a small 'personal bag'.

All the connections from the myriad of boards and breadboards came to a prototype shield and were soldered in place. Trying to align perhaps 70 pins across different headers to all slide into the female connector was quite challenging with all the wires soldered in place on the shield.

The shield was finally inserted over the Arduino and power was applied. Normally the DSKY comes up with blanks in all the displays and the caution and warning lights are extinguished. For the first tests, I put in some dummy values just to verify that the circuitry is working.

I verified I had fixed the bug that kept the keyboard scan routine from emitting any further keycodes after the first. There is a hardware issue with detection of keypresses in row 3 of the left keypad, something I will chase down and correct.

I did see values displaying and the relays working, although there is a lot of functionality I can't test without the driver since this code is watching 23 input lines and acting on them. These lines are unpowered because I didn't hook the input interface boards up to 28V for this test, meaning the input lines were floating. With power, they would either sit at 0 or 5V depending on the input wire.

I ran out of time - my flight to the location of the AGC is tomorrow morning. I had envisioned a full set of tests with the Arduino Uno test driver setup, but I will have to do that from my hotel room, in my spare time.

I carefully moved the unit and all its attached boards over to my carry-on, placing T-shirts between layers since I expect this may get a close inspection by TSA tomorrow.

Thursday, November 1, 2018

Soldering more secure connections for the DSKY substitute

PREPARING FOR RESTORATION OF APOLLO GUIDANCE COMPUTER


Nearing the end of the construction and testing of the DSKY substitute

The first two finished boards for input and output interface to the Apollo Guidance Computer passed their tests. I began to wire up the five addition input boards needed to connect my DSKY substitute to the AGC next week. This is assuming great progress from a standing start, but better to be prepared.

The output board is used in conjunction with relays. When activated, this connects the output line through a 2K resistor to ground. If not activated, the line floats up to +28V. This particular board can support 9 output circuits, although I am only using 8 for the DSKY.

First output board, used with relays to pull AGC input down towards ground
The input board is connected to a circuit in the AGC that will pull the line down towards ground (through a 2K resistor in the AGC and a 10K resistor in line the the transistor base, so the actual achieved voltage is quite a bit higher than 0V but sufficient to cause the transistor to conduct on my interface circuit.

That transistor delivers 28V through a voltage divider to ground, with the center tap sitting at just under 5V when active or at 0V when inactive. Thus this input circuit is also an inverter - a 28V input produces a 0V output, while when the input is pulled down towards ground, the output becomes 5V.

First input board, floats input at 28V, pulled down when AGC output activates it, gives inverted TTL output
I replaced some more breadboards with small solder boards like the ones used above, making all the connections to the I/O expander boards for the LEDs and for eight of the inputs from the first two input boards like the above that I completed.

I2C chain of I/O expander boards to drive LEDs and record some inputs
I soldered the alphanumeric (14-segment) driver onto another board and wired it to the I2C chain. I still have to do quite a bit:

  • wire up the two 8-channel relay modules
  • move the flasher logic to a circuit board
  • finish the input interface boards
  • wire in the 23 input channels
  • wire up the 8 output channels to the relay board
  • wire up the control lines from the Arduino to the output channel relay board
  • finalize the flashing control relay board wiring
  • ensure I have all the +5 and ground wires routed properly
  • check the Arduino prototype shield wiring for shorts or bad solder joints
One stretch goal is to label the keyboard with the proper text and to put a plastic overlay on the display lights with the correct text legends. I printed the text but the process of converting  them into rub-on transfers will take about an hour that I may not have tomorrow. 


Testing Apollo Guidance Computer interface circuits to my DSKY substitute, other build/test activities

PREPARING FOR RESTORATION OF APOLLO GUIDANCE COMPUTER


Building and testing the DSKY substitute

I spent the morning preparing the text legends that will fit over the face of the displays to show the meaning of all the digits, caution, warning and activity lights. I also prepared legends to fit on the keyboards.

I also wrote some some code for testing with an Arduino Uno hooked to the inputs of the DSKY substitute. This will let me try out the various caution and warning signals, flash the Verb and Noun displays, turn on the Computer Activity and put various digits on signs on the Register 1, Register 2, Register 3 Verb, Noun and Program display sections.

This is more complex than it seems, since the way that the DSKY is commanded is to have a four bit code that specifies a relay bank, consisting of 11 relays per bank, that usually control two digits of the display and in some cases the sign. Thus if I have a command to change one digit on the DSKY, I have to have remembered the state of the other digit and the sign so I can insert my one digit without changing the other items controlled by that register bank.

I got a new supply of breadboard PCBs - cards that reproduce the pin holes and connections of a traditional breadboard - that I can solder connections rather than have wire ends or pins which could come loose. I soldered some of those onto various components.

Another supply of perfboard PCBs arrived, which will hold the input interface circuits that accept the 28V logic levels from the AGC, compatibly, but output a 5V logic level for the Arduino. When the DSKY pulls the line down to 0V, this circuit produces a logic 1 to the Arduino. There are six input circuits per interface card and I need 23 of them to handle all the signals coming to my DSKY.

The output interface boards are much simpler, since they are simply a pair of resistors that are hooked to relays and switch the line to ground or let if float. I built one of these as well as one of the interface boards, giving me 9 usable output circuits and 4 usable input circuits.

I tested the input and output boards using bench power supplies and my voltmeter. Supplying 28V to a relay and to the input boards, driving the relay board with 5V and switching 0V to the relay board input caused the relay to turn on and off, delivering 28V to the output circuit as expected. I then tested the input circuit, pulling the input to 0V and yielding 5V to my controller or leaving the input at 28V to output 0V. to the controller This is exactly what i need to use with the Arduino. They worked perfectly.

To wire up the Arduino Mega 2560 to the DSKY substitute, I made use of a prototyping shield that allows me to solder wires to the shield and simply plug the shield into the Arduino to complete the connections.

Overall I need to build boards sufficient to support the 23 input and 8 output lines that connect to the AGC. One of the output boards handles what I need but it will take 6 input boards with four circuits apiece to cover the 23 incoming lines. The remainder of the components needed to build all six boards are arriving tonight but I built what I could.

I discovered some code flaw in my keyboard scanning routine, such that once it registers one keypress it will no longer register additional ones until the Arduino is rebooted. I can see the PRO and RSET keys are detected, since these also send discrete signals in addition to the RSET keycode, but I only get one keycode emission until the microcontroller is rebooted.