RESTORATION OF APOLLO GUIDANCE COMPUTER
Background of this AGC
The owner of the AGC, Jimmie Loocke, was browsing through a scrap metal recycler's warehouse in search of some parts for a project when he spotted a large pile of surplus NASA items that he recognized. Among them the AGC! On the spot, he bought two tons of the material for scrap metal prices and hauled it home.
In an odd twist of fate, Jimmie had worked for a contractor that was performing thermal tests of the Apollo spacecraft at the Manned Spaceflight Center in Houston. His job was to control the thermal profile of the test chamber as they checked out the response of a Lunar Module to the expected thermal conditions in space and on the surface of the moon.
NASA bought some test versions of the LM in addition to the machines that were used on flights. This AGC was installed in LTA8, test article 8. LTA8 is on display in the Johnson Space Center (formerly MSC), but the AGC from inside it was sold off by General Services Administration.
This is a block II AGC, the type used on all the manned flights. It was number 14 of the 15 prototypes built by Raytheon, which is why it has only two modules that are potted. We are very fortunate to have a machine that is identical in wiring and logic to the production machines but have access to the circuitry in all the unpotted modules.
The AGC is hermetically sealed and sat with an internal nitrogen atmosphere for decades before its recent opening by the owner and by Eldon Hall, the lead designer of the computer, Mike Stewart and now the restoration team who are volunteering their efforts to get this historic artifact in full glorious operation once again.
Addition of erasable memory to the tool and further traces
Mike added in the Verilog to provide erasable memory for the AGC, wired up the additional lines to the AGC backplane, and now the computer is running long sequences of instructions with almost every one acting as expected.
The use of breadboards, jumpers with pins and the large mess of wires running between AGC and tool is likely the cause of the sporadic misbehavior. At least we would consider the wiring to be the most likely cause of the occasional misbehavior and not yet suspect a defect in the AGC.
It was great to watch hundreds of instructions execute with the correct results as the AGC ran the Aurora restart code from o04000 until glitches in data returned from the tool caused the stream to diverge.
Design of final DSKY after testing of the prototype
The prototype unit testing last week uncovered some weaknesses in the design that need correction. The AGC and DSKY design refer to the least significant digit of each display as digit 1, but the prototype numbered the digits left to right starting with one. The method of flashing the Verb and Noun displays stressed the MAX7219 chips which drove the seven segment modules.
We have a milled aluminum case that is very close if not exactly the dimensions of a real DSKY, into which I will build the DSKY substitute. It won't be an actual replica of the gate by gate and relay by relay design, but is intended to appear similar and behave the same as far as the user and AGC are concerned.
It would be very hard to achieve a full gate by gate replica, in part because the electroluminescent panel on the right side of the DSKY is unobtainable. That obviates the need for 250V 800 Hz power which is what the relays switched to the various EL panel segments, making the internal construction different from the real DSKY.
In the prototype, I used three MAX7219 chips each on a separate module that I bought online. I did this for speed, since I had just a bit over one week to cobble this together and wanted to skip any assembly and design steps I could to speed things up. In a real design, those three chips would be strung together on one serial chain, but the prototype used three independent chains.
Since a MAX7219 supports eight 7-segment displays, I apportioned the DSKY digits among the three by using the first five digit positions for a register, two more digits for one of the Verb, Noun or Program displays, and ignored one position. For example, Register 1 and Verb were on one of the boards.
My first cut at blinking the Verb and Noun digits, while maintaining a very tight loop in the Arduino, was to use relays to interrupt the common cathode wires to those four 7-segment modules. A 555 timer set at 1.5Hz was gated through an IC to fire the relays on an off the shelf eight relay module.
This approach didn't seem to sit well with the two boards hosting these displays, which began to exhibit odd behavior that suggests I fried the MAX7291 chips. It worked initially but after about 20 minutes of operation, everything went downhill.
I might make use of a newer driver chip, the MAX6955, which can blink individual 7-segment modules without the Arduino doing more than initiating the state and rescinding it. I will also chain these together to make use of a single serial link.
I found that the design could be reading a command word from the AGC surrounding its change, because of the quick and dirty use of the digitalRead() method which is slow and non-atomic. By directly accessing AVR registers in the Arduino I can see all 15 bits at one time. Lastly, when I see a non-zero value coming in from the AGC, I will delay 5ms before sampling again to eliminate the risk of reading a glitched word.
Fitting in the aluminum case will require sourcing and placing 7-segment and 14-segment displays more precisely and fitting everything tightly on a PCB. I found green colored display modules that are close enough to the actual DSKY size that they will appear almost full sized and fit across in the allotted space.
The prototype used a cheap membrane switch array but I need to support the large square button format of the DSKY in the final version. I may need to construct the buttons and related mechanism myself, embedded a microswitch inside to actually read the contacts. More research is needed on this.
Discovery about potting materials used on the erasable memory module
After some archive detective work by Mike and Ken, we are of the opinion that the potting material that we need to open to repair the wire break is not RTV-11, but Stycast 1090, an epoxy resin. Different solvents or methods will be needed to make a small hole for our repair work.
Monday, November 12, 2018
Saturday, November 10, 2018
Restoring an Apollo Guidance Computer, part IV
RESTORATION OF APOLLO GUIDANCE COMPUTER
Looking for suspected design flaw/race hazard
Mike Stewart had modeled the exact schematic of the AGC, gate by gate, which he had used both to understand the system using simulation and to create his own instance of an AGC in an FPGA board. One of the discoveries from his simulation was a glitch or transient signal that impairs operation of one of the hardware alarms. The question - does this occur in the real AGC?
Since the Apollo Guidance Computer acts as the digital autopilot for the astronauts, is in between the hand controllers and the attitude jets, and is responsible for navigation, it must be very reliable. A number of hardware checks are built in to detect software bugs by recognizing undesirable behavior.
One of the clever checks added to the hardware is a timer that watches to see how long it has been since a control transfer instruction (e.g. branches) have been executed. If code is in an infinite loop, this timer would signal an alarm and force a restart of the computer, hopefully restoring function to the spacecraft.
Mike had discovered a transient that would occur relatively frequently that resets the timer, the way it would be reset by a control transfer instruction. Since this glitch is occuring independent of branches, it has a good chance of defeating the check and delaying, perhaps indefinitely, the detection of the infinite loop.
We set up the oscilloscope to watch the condition he had observed in his simulator and indeed this glitch occurs relatively often. He had developed a test for this problem, creating a two instruction loop with a clever program that does not use transfer of control instructions. When he runs it in the simulator, in the software AGC simulator available on the web, and in his FPGA, it proves that the detection of the loop is seriously delayed because of this design flaw.
Testing a longer stream of instructions
Mike adjusted his memory tool and achieved much longer run times, for example having the AGC execute a few thousand instructions of the Aurora core rope software beginning with the restart/start code that the hardware runs at o04000
All of the instructions examined produced the proper results, although the flow of the real AGC diverted from the flow under the simulator at the 100th instruction. The deviation was not due to a defect in the AGC but completely reasonable because of the state of the memory tool right now.
The memory tool is currently wired up to serve up only the fixed memory (core rope memory) contents. This means any access to erasable (RAM) memory returns o00000 as a result. As code executes, at some point a conditional branch will go a different way, because the erasable memory would have had a non-zero content but under the tool it is different.
The actual case we monitored was not an issue of branch conditions, but due to a programming technique that was used. The address of a routine to execute was stored in erasable memory and then the programmer attempted a Transfer Control through that address. Under the tool, we branched to location 0 instead.
PROJECTS TO CONTINUE AFTER LEAVING HOUSTON
Our team reached the end of the trip and we are flying back home this weekend. It has been a very satisfactory first week of work but more needs to be done to achieve our goal of a demonstration of a fully restored AGC running actual Apollo mission software.
Different memory tool implementation
The temporary memory tool is connected to the AGC by a myriad of wires pushed onto the wirewrap pins of the tray A backplane. This is not a rugged or cosmetically appealing approach for the long term. Instead, Mike is building a different version of the tool that will connect to the AGC using the maintenance connector (A52) that is accessible on the outside of the computer with all covers closed.
A52 was used to connect to Ground Support Equipment (GSE) and test out the AGC in a number of locations. Raytheon used it during manufacture, North American Aviation used it to test while installed in the Command Module, Grumman Aerospace Engineering Company used it to test while installed in the Lunar Module, NASA used it to check out the spacecraft in the Manned Spacecraft Operation Building (MSOB) at Kennedy Spaceflight Center, and also used it for testing while the vehicle was sitting on the launchpad.
This connector allows the monitoring of key conditions and control of the computer, including the ability to deliver data that overrides the contents of AGC memory. Initially, A52 allowed override of core rope memory contents but not erasable memory. Later changes added the ability to override the erasable as well.
Our AGC, which was one of the last prototypes before the production versions of the computer were shipped, varies only slightly from the final machine. Our chassis is made of aluminum instead of magnesium, as the weight reduction was important to the production machines that would fly. Most of our modules are not potted, which is great for testing and repair. However, the function to allow override of erasable memory was not installed on our box. We intend to add the option.
Adding MAMU pin to A52 maintenance connector
The signal to override erasable memory (MAMU) is a trivial change to implement and simply adds a standard feature of all production machines. We found a spare NOR gate with an open collector, allowing us to tie that to the existing NOR gate which drives the strobe to read erasable memory. Thus, when our gate activates, it will suppress the strobe.
This requires that we add three wires to the backplane. One will run from the output of the spare gate we chose to the output of the existing eraseable strobe gate. One will run from the unused input of our spare gate to ground. The third wire will run from the open input gate of our spare module over to the MAMU pin on connector A52.
We will wire wrap these new wires, using a different color wire and documenting what we did, allowing the AGC to be restored to its 'as found' state if desired by the owner sometime in the future. Restorations of rare artifacts typically apply a similar process, saving any removed parts, marking any substitutions, and providing a path to restore the artifact to its prior state.
Core Rope Simulator activation using a team built driver system
Our AGC was run with Raytheon built modules installed in the core rope slots. These modules were part of a core rope simulator that connected via two thick cables to some GSE which would respond to any fixed memory request with the appropriate data word. Unfortunately, we did not receive the thick cables nor the GSE, just the two modules which plug into the AGC.
Ken Shirriff spend much of the week reverse engineering the Raytheon modules. We have no schematics or other documentation on these, but think that it would be possible to connect to the modules with some electronics of our own devise that would drive them. In other words, a partial substitute for the GSE associated with the rope simulator.
He is wrapping up the documentation and study of these modules. With a good definition for the 31 data and control lines running over the thick cable, he can control these modules using simple digital signals over differential wire pairs. We have found an old connector that will mate with the Raytheon modules, except that some keys on the connector block insertion.
We will mill off the keys that don't match so that we can insert our connector into the module, substituting extreme care for mechanical keys to ensure that the connector is oriented properly.
Once we have circuits to control the Raytheon modules, they can respond to fixed memory address requests and serve up the data words from within our unit. This is the preferred way to provide fixed memory to this computer, since the modules are standard Raytheon units that were on the machine and historically accurate.
Rope software is available in many versions, corresponding to most Apollo missions. We will have these in our box and selectable to run the AGC as a CM or LM on whatever flight we desire.
Repair of erasable memory module and checkout in the AGC
The preferred way to supply erasable memory to the AGC is by repairing the actual module from this computer and running with it in tray B. As previously described, the module has a pin with open connectivity where we should have a inhibit line for bit 16. We hope to repair this defect.
The module is fully potted - filled in with a solid material covering all the wires, pins and interior components. We believe this material to be RTV-11 but will do some testing to confirm. If RTV-11, we know of solvents that will slowly dissolve the potting material allowing access.
We don't want to remove all the potting. Instead, by analogy to arthroscopic surgery, we want to make the smallest hole feasible, accomplish the repair, and fill the space back in using new RTV-11. To make this happen, we must know exactly where the break is within the module and be sure it is in a location where it can be repaired.
Apollo era documentation informs us that quite a few of the modules failed during testing or use, but these all were breaks where the wire runs from the external pin down to the memory stack itself, not somewhere inside the memory stack. The stack is folded multiple times to fit in the small space, making access to wires inside the stack impossible.
To spot the location, we are going to make use of a 3D tomographic X-Ray machine to give us an exact picture of the location of the fault. Based on the X-Ray, we will build a plan for the repair, make our opening into the potting and hopefully fix the module. The X-Ray is scheduled for late December, based on availability on the high tech imager.
Conversion of DSKY substitute prototype to a finished and rugged state
My DSKYsubstitute was a breadboard hodgepodge used to prove out the design before I commit it to printed circuit boards and more permanent construction. We are not able to drive it from the AGC yet, because the software that communicates with it hasn't run and won''t until we can provide erasable memory, either with a tool or the repaired module.
I was, however, able to test it with a pattern generator and look at the function of much of the unit. We discovered a few places where the design has to change, such as in the method of flashing the Verb and Noun displays. We also discovered the potential to react too quickly to the AGC output and select the wrong digits, signs or lights.
All these learnings will be fed into the design, an improved prototype will be tested and then I will build a PCB based substitute that can fit into an aluminum case milled to match the real DSKY. We were fortunate enough to receive the milled case from the owner of the AGC and can use it to house the substitute.
Producing realistic spacecraft inputs to the AGC
One final aspiration would be to produce realistic inputs over the main spacecraft connector (A51), such as the pulses that come from the inertial navigation unit, telemetry uplink, and so forth. Similarly we would find ways of displaying important outputs of the AGC.
As an example, if we have the inertial gyros and accelerometers providing input, and can display the pulsing of the Reaction Control System (attitude jets), maybe implement the hand controllers and watch RCS activity as we request rotation or translation, then we would have a more compelling demo for the 50th anniversary of the Apollo 11 landing in July 2019.
Looking for suspected design flaw/race hazard
Mike Stewart had modeled the exact schematic of the AGC, gate by gate, which he had used both to understand the system using simulation and to create his own instance of an AGC in an FPGA board. One of the discoveries from his simulation was a glitch or transient signal that impairs operation of one of the hardware alarms. The question - does this occur in the real AGC?
Since the Apollo Guidance Computer acts as the digital autopilot for the astronauts, is in between the hand controllers and the attitude jets, and is responsible for navigation, it must be very reliable. A number of hardware checks are built in to detect software bugs by recognizing undesirable behavior.
One of the clever checks added to the hardware is a timer that watches to see how long it has been since a control transfer instruction (e.g. branches) have been executed. If code is in an infinite loop, this timer would signal an alarm and force a restart of the computer, hopefully restoring function to the spacecraft.
Mike had discovered a transient that would occur relatively frequently that resets the timer, the way it would be reset by a control transfer instruction. Since this glitch is occuring independent of branches, it has a good chance of defeating the check and delaying, perhaps indefinitely, the detection of the infinite loop.
We set up the oscilloscope to watch the condition he had observed in his simulator and indeed this glitch occurs relatively often. He had developed a test for this problem, creating a two instruction loop with a clever program that does not use transfer of control instructions. When he runs it in the simulator, in the software AGC simulator available on the web, and in his FPGA, it proves that the detection of the loop is seriously delayed because of this design flaw.
Testing a longer stream of instructions
Mike adjusted his memory tool and achieved much longer run times, for example having the AGC execute a few thousand instructions of the Aurora core rope software beginning with the restart/start code that the hardware runs at o04000
All of the instructions examined produced the proper results, although the flow of the real AGC diverted from the flow under the simulator at the 100th instruction. The deviation was not due to a defect in the AGC but completely reasonable because of the state of the memory tool right now.
The memory tool is currently wired up to serve up only the fixed memory (core rope memory) contents. This means any access to erasable (RAM) memory returns o00000 as a result. As code executes, at some point a conditional branch will go a different way, because the erasable memory would have had a non-zero content but under the tool it is different.
The actual case we monitored was not an issue of branch conditions, but due to a programming technique that was used. The address of a routine to execute was stored in erasable memory and then the programmer attempted a Transfer Control through that address. Under the tool, we branched to location 0 instead.
PROJECTS TO CONTINUE AFTER LEAVING HOUSTON
Our team reached the end of the trip and we are flying back home this weekend. It has been a very satisfactory first week of work but more needs to be done to achieve our goal of a demonstration of a fully restored AGC running actual Apollo mission software.
Different memory tool implementation
The temporary memory tool is connected to the AGC by a myriad of wires pushed onto the wirewrap pins of the tray A backplane. This is not a rugged or cosmetically appealing approach for the long term. Instead, Mike is building a different version of the tool that will connect to the AGC using the maintenance connector (A52) that is accessible on the outside of the computer with all covers closed.
A52 was used to connect to Ground Support Equipment (GSE) and test out the AGC in a number of locations. Raytheon used it during manufacture, North American Aviation used it to test while installed in the Command Module, Grumman Aerospace Engineering Company used it to test while installed in the Lunar Module, NASA used it to check out the spacecraft in the Manned Spacecraft Operation Building (MSOB) at Kennedy Spaceflight Center, and also used it for testing while the vehicle was sitting on the launchpad.
This connector allows the monitoring of key conditions and control of the computer, including the ability to deliver data that overrides the contents of AGC memory. Initially, A52 allowed override of core rope memory contents but not erasable memory. Later changes added the ability to override the erasable as well.
Our AGC, which was one of the last prototypes before the production versions of the computer were shipped, varies only slightly from the final machine. Our chassis is made of aluminum instead of magnesium, as the weight reduction was important to the production machines that would fly. Most of our modules are not potted, which is great for testing and repair. However, the function to allow override of erasable memory was not installed on our box. We intend to add the option.
Adding MAMU pin to A52 maintenance connector
The signal to override erasable memory (MAMU) is a trivial change to implement and simply adds a standard feature of all production machines. We found a spare NOR gate with an open collector, allowing us to tie that to the existing NOR gate which drives the strobe to read erasable memory. Thus, when our gate activates, it will suppress the strobe.
This requires that we add three wires to the backplane. One will run from the output of the spare gate we chose to the output of the existing eraseable strobe gate. One will run from the unused input of our spare gate to ground. The third wire will run from the open input gate of our spare module over to the MAMU pin on connector A52.
We will wire wrap these new wires, using a different color wire and documenting what we did, allowing the AGC to be restored to its 'as found' state if desired by the owner sometime in the future. Restorations of rare artifacts typically apply a similar process, saving any removed parts, marking any substitutions, and providing a path to restore the artifact to its prior state.
Our AGC was run with Raytheon built modules installed in the core rope slots. These modules were part of a core rope simulator that connected via two thick cables to some GSE which would respond to any fixed memory request with the appropriate data word. Unfortunately, we did not receive the thick cables nor the GSE, just the two modules which plug into the AGC.
Ken Shirriff spend much of the week reverse engineering the Raytheon modules. We have no schematics or other documentation on these, but think that it would be possible to connect to the modules with some electronics of our own devise that would drive them. In other words, a partial substitute for the GSE associated with the rope simulator.
He is wrapping up the documentation and study of these modules. With a good definition for the 31 data and control lines running over the thick cable, he can control these modules using simple digital signals over differential wire pairs. We have found an old connector that will mate with the Raytheon modules, except that some keys on the connector block insertion.
We will mill off the keys that don't match so that we can insert our connector into the module, substituting extreme care for mechanical keys to ensure that the connector is oriented properly.
Once we have circuits to control the Raytheon modules, they can respond to fixed memory address requests and serve up the data words from within our unit. This is the preferred way to provide fixed memory to this computer, since the modules are standard Raytheon units that were on the machine and historically accurate.
Rope software is available in many versions, corresponding to most Apollo missions. We will have these in our box and selectable to run the AGC as a CM or LM on whatever flight we desire.
Repair of erasable memory module and checkout in the AGC
The preferred way to supply erasable memory to the AGC is by repairing the actual module from this computer and running with it in tray B. As previously described, the module has a pin with open connectivity where we should have a inhibit line for bit 16. We hope to repair this defect.
The module is fully potted - filled in with a solid material covering all the wires, pins and interior components. We believe this material to be RTV-11 but will do some testing to confirm. If RTV-11, we know of solvents that will slowly dissolve the potting material allowing access.
We don't want to remove all the potting. Instead, by analogy to arthroscopic surgery, we want to make the smallest hole feasible, accomplish the repair, and fill the space back in using new RTV-11. To make this happen, we must know exactly where the break is within the module and be sure it is in a location where it can be repaired.
Apollo era documentation informs us that quite a few of the modules failed during testing or use, but these all were breaks where the wire runs from the external pin down to the memory stack itself, not somewhere inside the memory stack. The stack is folded multiple times to fit in the small space, making access to wires inside the stack impossible.
To spot the location, we are going to make use of a 3D tomographic X-Ray machine to give us an exact picture of the location of the fault. Based on the X-Ray, we will build a plan for the repair, make our opening into the potting and hopefully fix the module. The X-Ray is scheduled for late December, based on availability on the high tech imager.
Conversion of DSKY substitute prototype to a finished and rugged state
My DSKYsubstitute was a breadboard hodgepodge used to prove out the design before I commit it to printed circuit boards and more permanent construction. We are not able to drive it from the AGC yet, because the software that communicates with it hasn't run and won''t until we can provide erasable memory, either with a tool or the repaired module.
I was, however, able to test it with a pattern generator and look at the function of much of the unit. We discovered a few places where the design has to change, such as in the method of flashing the Verb and Noun displays. We also discovered the potential to react too quickly to the AGC output and select the wrong digits, signs or lights.
All these learnings will be fed into the design, an improved prototype will be tested and then I will build a PCB based substitute that can fit into an aluminum case milled to match the real DSKY. We were fortunate enough to receive the milled case from the owner of the AGC and can use it to house the substitute.
Producing realistic spacecraft inputs to the AGC
One final aspiration would be to produce realistic inputs over the main spacecraft connector (A51), such as the pulses that come from the inertial navigation unit, telemetry uplink, and so forth. Similarly we would find ways of displaying important outputs of the AGC.
As an example, if we have the inertial gyros and accelerometers providing input, and can display the pulsing of the Reaction Control System (attitude jets), maybe implement the hand controllers and watch RCS activity as we request rotation or translation, then we would have a more compelling demo for the 50th anniversary of the Apollo 11 landing in July 2019.
Friday, November 9, 2018
Restoring Apollo Guidance Computer, part III
RESTORATION OF APOLLO GUIDANCE COMPUTER
Developing tool to substitute for both fixed and erasable memories
Having a computer with no RAM or ROM is a recipe for boredom. Since we have to wait until we can x-ray the erasable module before we start opening up the potting near the break and repairing the wiring, we needed an alternative way to exercise the computer.
Mike had a work in progress to use connector A52, the ground maintenance connector, to do this job but it requires a connector that we don't have. The AGC uses mini-WASP connectors which are unobtainable, thus we would need custom manufacturing of a replacement.
Using the NASA sourcing control documents (SCD) from the NARA archive, we can exactly specify the shape and dimensions of this very non-standard connector. We are hopeful that we can get the main spacecraft/DSKY connector A51 and the maintenance connector A52 produced because a high technology connector company is offering to do this to sponsor our work. It will take some time to receive such a connector.
In the meantime, Mike adapted his FPGA based design to connect via backplane pins. This would watch for addresses and the strobe signal that normally triggers a core read (two strobes, one each for fixed and erasable memory). It would store the entire 36K contents of core rope and host the entire 2K of RAM on the board, feeding the appropriate word to the AGC at the correct time.
The first cut of this would only implement fixed (core rope) memory and serve as a proof of concept before wiring in the lines to update RAM as needed to emulate erasable memory. We wired it up and connected it to the AGC for our first test.
Nothing happened. The board didn't appear to see any addresses or control signals. The interface chips that we were using, chips that work fine converting between 3.3 and 5 volt systems, weren't working for the AGC.
Logic level challenges hooking up the memory substitute tool
The RTL logic used in the AGC, while defined as ranging between 0 and 4V for logic 0 and logic 1, produce much lower voltages for a logic high. We see valid 1 bits with voltages as low as 0.8 volts at the inputs, which is just too low for the MOSFET based level shifter we were using.
Since the actual voltages were lower, not anywhere near 4V, we then tried wiring the AGC directly to the 3.3V level FPGA pins but that didn't work either. LVCMOS logic levels define the lowest valid high as a bit higher than 2 volts, so nothing would register.
Thus, we have to build discrete level shifter circuits to make sure that a 0.8V or higher input will give a crisp high output into the FPGA.
Building lots of interface circuits between memory tool and AGC
There are 26 input lines, each requiring one transistor and three resistors for its interface. After wiring up the interface circuits and connecting them between AGC and FPGA, we did successfully see all the signals from the computer. Using pull up resistors and open drain output mode on the FPGA, we can drive in the data signals to the AGC just fine.
Testing the DSKY substitute tool using a pattern generator
Marc set up his pattern generator to drive my DSKY substitute while we worked on a few wiring and power issues. We improved its operation quite a bit, although more work is needed to avoid glitches on input from the AGC and to fix suspected damage to some of my control boards.
I think the trauma of moving it on the plane and resulting shorts and dislodged wires caused a few chips to fry. It will come home and be improved, tested and then a final version of the hardware will be built.
There are some places where I need to shift to direct access of the AVR chip registers, bypassing the Arduino functions such as digitalRead(), in order to see all 15 bits from the AGC at one time.
Testing the memory substitute tool
Marc had to leave with his large logic analyzer before we got the memory substitute tool working, leaving us with oscilloscope traces as a diagnostic snapshot into the processor. Still, that gave us enough to see what words we were proving to the AGC. Coupling that with the listings of the Apollo software that are running let us verify both the tool and the AGC operation.
There are a few bugs in the tool right now, so that after a short burst of correctly delivered words, the tool messed up then went into a hung state. Nothing systemic or difficult. Given a bit of time to clean things up and test, this tool should be reliably feeding core rope contents to the computer.
Now, to the more important question. Did the computer work properly? Admittedly we have a small sample, only a few dozen instructions running in any test shot, but it would seem that the AGC is flawlessly executing everything it receives. This includes complex behaviors like inhibiting interrupts, extend mode processing, and use of the ALU within instruction execution.
Without the erasable memory support wired up, eventually the code would go awry even if the tool keep feeding fixed memory data perfectly. Once the tool is whipped into shape and the erasable memory support is turned on, we should be able to run the Apollo software at least to the point where it requires some input from the spacecraft or the DSKY.
Checking out first few instructions executed by AGC
TO BE CONTINUED
Developing tool to substitute for both fixed and erasable memories
Having a computer with no RAM or ROM is a recipe for boredom. Since we have to wait until we can x-ray the erasable module before we start opening up the potting near the break and repairing the wiring, we needed an alternative way to exercise the computer.
Mike had a work in progress to use connector A52, the ground maintenance connector, to do this job but it requires a connector that we don't have. The AGC uses mini-WASP connectors which are unobtainable, thus we would need custom manufacturing of a replacement.
Using the NASA sourcing control documents (SCD) from the NARA archive, we can exactly specify the shape and dimensions of this very non-standard connector. We are hopeful that we can get the main spacecraft/DSKY connector A51 and the maintenance connector A52 produced because a high technology connector company is offering to do this to sponsor our work. It will take some time to receive such a connector.
In the meantime, Mike adapted his FPGA based design to connect via backplane pins. This would watch for addresses and the strobe signal that normally triggers a core read (two strobes, one each for fixed and erasable memory). It would store the entire 36K contents of core rope and host the entire 2K of RAM on the board, feeding the appropriate word to the AGC at the correct time.
The first cut of this would only implement fixed (core rope) memory and serve as a proof of concept before wiring in the lines to update RAM as needed to emulate erasable memory. We wired it up and connected it to the AGC for our first test.
Nothing happened. The board didn't appear to see any addresses or control signals. The interface chips that we were using, chips that work fine converting between 3.3 and 5 volt systems, weren't working for the AGC.
Logic level challenges hooking up the memory substitute tool
The RTL logic used in the AGC, while defined as ranging between 0 and 4V for logic 0 and logic 1, produce much lower voltages for a logic high. We see valid 1 bits with voltages as low as 0.8 volts at the inputs, which is just too low for the MOSFET based level shifter we were using.
Since the actual voltages were lower, not anywhere near 4V, we then tried wiring the AGC directly to the 3.3V level FPGA pins but that didn't work either. LVCMOS logic levels define the lowest valid high as a bit higher than 2 volts, so nothing would register.
Thus, we have to build discrete level shifter circuits to make sure that a 0.8V or higher input will give a crisp high output into the FPGA.
Building lots of interface circuits between memory tool and AGC
There are 26 input lines, each requiring one transistor and three resistors for its interface. After wiring up the interface circuits and connecting them between AGC and FPGA, we did successfully see all the signals from the computer. Using pull up resistors and open drain output mode on the FPGA, we can drive in the data signals to the AGC just fine.
 |
| FPGA tool with level shifter circuits, attached to AGC |
Marc set up his pattern generator to drive my DSKY substitute while we worked on a few wiring and power issues. We improved its operation quite a bit, although more work is needed to avoid glitches on input from the AGC and to fix suspected damage to some of my control boards.
 |
| Testing out DSKY with pattern generator |
There are some places where I need to shift to direct access of the AVR chip registers, bypassing the Arduino functions such as digitalRead(), in order to see all 15 bits from the AGC at one time.
Testing the memory substitute tool
Marc had to leave with his large logic analyzer before we got the memory substitute tool working, leaving us with oscilloscope traces as a diagnostic snapshot into the processor. Still, that gave us enough to see what words we were proving to the AGC. Coupling that with the listings of the Apollo software that are running let us verify both the tool and the AGC operation.
There are a few bugs in the tool right now, so that after a short burst of correctly delivered words, the tool messed up then went into a hung state. Nothing systemic or difficult. Given a bit of time to clean things up and test, this tool should be reliably feeding core rope contents to the computer.
Now, to the more important question. Did the computer work properly? Admittedly we have a small sample, only a few dozen instructions running in any test shot, but it would seem that the AGC is flawlessly executing everything it receives. This includes complex behaviors like inhibiting interrupts, extend mode processing, and use of the ALU within instruction execution.
Without the erasable memory support wired up, eventually the code would go awry even if the tool keep feeding fixed memory data perfectly. Once the tool is whipped into shape and the erasable memory support is turned on, we should be able to run the Apollo software at least to the point where it requires some input from the spacecraft or the DSKY.
Checking out first few instructions executed by AGC
TO BE CONTINUED
Restoring Apollo Guidance Computer, part II
RESTORATION OF APOLLO GUIDANCE COMPUTER
Testing the analog alarm module from the B tray
The alarm module has a number of circuits that look for error conditions and raise alarms. The alarm will cause the computer to restart, by forcing in a branch instruction to a fixed address in core rope memory (octal 4000). The AGC will continue to restart until the alarm goes away, at which point it completes the restart sequence and is back in operation.
We were able to get all of the circuits verified except for one. This was the circuit that looks at the 28V in from the spacecraft bus, the 14V produced by one power supply and the 4V produced by the other supply in tray A.
It should trigger an alarm if our two generated voltages are above a max level or below a minimum voltage, and it should trigger an alarm if the spacecraft main bus (A or B) falls below a minimum.
When we varied the voltages and watched the alarm signal, it did turn on when we went into the invalid ranges. However, it latched in that condition and wouldn't turn off when the power returned to healthy levels. In a real machine, this would be a problem because the only way to recover would be to pull circuit breakers, a cumbersome and impractical action; thus we knew this should not be happening.
However, when we inserted the module in tray B and did the first bringup (below) we found that the alarms automatically reset when power reached good levels. Not sure why we couldn't recreate that on the bench, but it is significant that this one module is a different level from all the schematics we use.
First bringup of the computer (without memories)
We powered up the computer with some oscilloscope probes attached and saw that the system appeared to be executing, so it was time to wire up the logic analyzer to record enough to know what is going on.
The analyzer showed that we were in restart until the power alarm went off, but then took a different alarm and went back to restart. That alarm was parity error! Since we have no erasable or fixed memory, any read of a location will see the contents as all zero. Since this means the parity bit isn't delivering odd parity, we raise an alarm condition.
This makes us loop, as each time we restart, our second instruction tries to execute at o4000 but gets a parity error, taking us back to restart ad infinitum.
Bypassing the parity error alarm to continue testing
A maintenance connector on the AGC provides a wire that we can raise to logic 1 to disable alarm conditions. We connected it to a pullup resistor and indeed see that the processor does not loop on restart any more.
Traces and verification in the absence of memory contents
There is a lot we can validate even with all memory accesses returning o00000 so we ran traces on the logic analyzer and pored over the entries. Everything we see is correct operation, no sign of a flaw so far.
Once the system comes out of restart, it branches to location 0. This happens because the contents fetched as the first instruction come out o00000 which is interpreted as a transfer control to 0. Normally you would expect that this would loop forever fetching a transfer to zero, but a few other things are going on.
The AGC supports hardware counters that track spacecraft information. A function in the spacecraft may request that a counter be incremented or decremented by sending some pulses to the AGC. The hardware will stop executing instructions and instead force a special instruction like INKL or DINKL to change the counter.
Counters are fixed addresses in erasable memory which are updated by the cycle-stealing function of the INKL, for example. The software doesn't see the increment or decrement, but can read the current counter value at any time by looking at the memory location. With the decrement instruction, if the counter reaches zero an interrupt can be generated
This is what we see happening. After restart, we see INKL and DINKL commands executing because they are top priority and are set up by logic. Of course, the instructions fetch the current value which is always zero, add or subtract, then try to store the updated value but no memory so still zero.
The DINKL sees it result, the zero, and triggers the interrupt. Once the counter increment and decrements are done, the interrupt is processing by branching to the interrupt handlers up in fixed memory. The instruction comes back as o00000 a transfer to location zero.
The other non-intuitive result is that we don't execute the transfer to location zero, because the read request returns with a non-zero value! What? We have no memory. Well, it turns out we do have a limited memory.
The architecture of the AGC uses the first seven or so locations of memory as registers - accumulator and others. The registers need to be much faster than a core memory cycle so they are implemented with flip flops in the machine. Reading memory at 0 will return the current value of the accumulator and treat it as an instruction.
This turns into an instruction referencing memory location o3777 because the flip flop starts out with all bits turned on. We see that happen in the trace. Once the processor steps through the first few register contents thinking they are memory contents, we reach the first non-register location, get the transfer control to 0 instruction and start the loop over.
We need to put some data into the system to try out different instructions. Eventually we need to provide fixed and erasable memory plus the software that was coded into the core rope modules.
TO BE CONTINUED
The alarm module has a number of circuits that look for error conditions and raise alarms. The alarm will cause the computer to restart, by forcing in a branch instruction to a fixed address in core rope memory (octal 4000). The AGC will continue to restart until the alarm goes away, at which point it completes the restart sequence and is back in operation.
We were able to get all of the circuits verified except for one. This was the circuit that looks at the 28V in from the spacecraft bus, the 14V produced by one power supply and the 4V produced by the other supply in tray A.
It should trigger an alarm if our two generated voltages are above a max level or below a minimum voltage, and it should trigger an alarm if the spacecraft main bus (A or B) falls below a minimum.
When we varied the voltages and watched the alarm signal, it did turn on when we went into the invalid ranges. However, it latched in that condition and wouldn't turn off when the power returned to healthy levels. In a real machine, this would be a problem because the only way to recover would be to pull circuit breakers, a cumbersome and impractical action; thus we knew this should not be happening.
However, when we inserted the module in tray B and did the first bringup (below) we found that the alarms automatically reset when power reached good levels. Not sure why we couldn't recreate that on the bench, but it is significant that this one module is a different level from all the schematics we use.
First bringup of the computer (without memories)
We powered up the computer with some oscilloscope probes attached and saw that the system appeared to be executing, so it was time to wire up the logic analyzer to record enough to know what is going on.
The analyzer showed that we were in restart until the power alarm went off, but then took a different alarm and went back to restart. That alarm was parity error! Since we have no erasable or fixed memory, any read of a location will see the contents as all zero. Since this means the parity bit isn't delivering odd parity, we raise an alarm condition.
 |
| Erasable memory module (with open circuit on inhibit line) |
Bypassing the parity error alarm to continue testing
A maintenance connector on the AGC provides a wire that we can raise to logic 1 to disable alarm conditions. We connected it to a pullup resistor and indeed see that the processor does not loop on restart any more.
Traces and verification in the absence of memory contents
There is a lot we can validate even with all memory accesses returning o00000 so we ran traces on the logic analyzer and pored over the entries. Everything we see is correct operation, no sign of a flaw so far.
 |
| Marc studying the logic analyzer traces |
The AGC supports hardware counters that track spacecraft information. A function in the spacecraft may request that a counter be incremented or decremented by sending some pulses to the AGC. The hardware will stop executing instructions and instead force a special instruction like INKL or DINKL to change the counter.
Counters are fixed addresses in erasable memory which are updated by the cycle-stealing function of the INKL, for example. The software doesn't see the increment or decrement, but can read the current counter value at any time by looking at the memory location. With the decrement instruction, if the counter reaches zero an interrupt can be generated
This is what we see happening. After restart, we see INKL and DINKL commands executing because they are top priority and are set up by logic. Of course, the instructions fetch the current value which is always zero, add or subtract, then try to store the updated value but no memory so still zero.
The DINKL sees it result, the zero, and triggers the interrupt. Once the counter increment and decrements are done, the interrupt is processing by branching to the interrupt handlers up in fixed memory. The instruction comes back as o00000 a transfer to location zero.
The other non-intuitive result is that we don't execute the transfer to location zero, because the read request returns with a non-zero value! What? We have no memory. Well, it turns out we do have a limited memory.
The architecture of the AGC uses the first seven or so locations of memory as registers - accumulator and others. The registers need to be much faster than a core memory cycle so they are implemented with flip flops in the machine. Reading memory at 0 will return the current value of the accumulator and treat it as an instruction.
This turns into an instruction referencing memory location o3777 because the flip flop starts out with all bits turned on. We see that happen in the trace. Once the processor steps through the first few register contents thinking they are memory contents, we reach the first non-register location, get the transfer control to 0 instruction and start the loop over.
We need to put some data into the system to try out different instructions. Eventually we need to provide fixed and erasable memory plus the software that was coded into the core rope modules.
TO BE CONTINUED
Restoring Apollo Guidance Computer, part I
RESTORATION OF APOLLO GUIDANCE COMPUTER
System oscillator (2MHz)
We provided 14V and 4V to the clock oscillator module (B7), limiting current to protect against any failing capacitors or other shorts. The output was beautiful. We have a pulse. It is running at 2.04801 MHz, amazingly close to spec for a machine that hasn't run in almost 50 years.
Testing the logic modules (contents of tray A)
The AGC is built of two trays that are bolted together, with mostly digital circuits in tray A and mostly analog circuits in tray B. For example, the master oscillator is module B8 because it is analog, but all the further processing of the clock is done digitally, in modules A1 and A2 in tray A.
The logic modules are constructed exclusively of dual NOR gates, each with three inputs. These are Resistor-Transistor Logic (RTL), which was quickly obsoleted in the industry by DTL, TTL, CMOS and other schemes. After a while working with the schematics, one gets proficient at quickly understanding designs using only NOR gates. No flip flops, no other type of boolean operations at all.
The RTL chips operate with 4V power supply and have a pretty low threshold between logic 0 and logic 1 condition. We found voltages as low as 0.8V that were successfully recognized logic 1 states in the running AGC. Logic 0 inputs were typically under 0.3V in actual operation.
Clock dividers and scaler, other timing signals
The oscillator delivers 2MHz to modules A1 and A2, which first divide the signal to the master oscillator of 1MHz and then produce various timing cycles that will drive the rest of the computer. Most notably, the computer is designed around core memory cycles which take almost 12 us, using twelve of the 1MHz pulses as stages 1 to 12.
We injected appropriate clock signals into the two modules as they sat on the workbench, out of the AGC trays, and verified that all of the timing signals were being produced at the proper time. We soon discovered a failure!
The AGC uses a variety of timers of different durations, from almost master clock frequency down to one that has a pulse once every 36 hours. This is implemented with a chain of dividers in A1. We found that the time pulses stopped somewhere in the middle of the chain.
All the other timing logic in A1 and A2 worked fine. With the failure of our clock scaler, the machine would not properly detect certain errors (watchdog alarms) that depended on slower pulses, it would never go into Standby mode, and of course mission software that needed longer timers wouldn't work right. Most of the machine would still work even with the one gate failing.
Investigating the failing NOR gate
After some probing around and other testing, we concluded that the output of one NOR gate was up at 4V. This means it was shorted to the VCC power supply. Inside the NOR gate there is a pullup resistor between VCC and the output, but this was shorted. It is an odd failure, one that seemed only possible if some bit of conductive material had fallen across some the two traces, VCC and output; these traces must be close to each other somewhere for this failure to have occured.
We shook the module with the package facing down, retested, and found that the problem had cleared. We will perpetually have the conductive junk, probably a solder blob, inside the sealed gate. Problems could recur but we know that some shaking will clear it up again. We will live with this situation.
SQ (instruction op code) register and decoding module
The next module we placed on the bench, A3, provided the storage of the instruction operation code and performed some of the initial decoding. A module has two boards, one per side. Each board has up to 60 flat pack ICs with two NOR gates apiece.
We worked to find combinations of inputs that would put each and every gate through all its states and give us a way to observe its output. That is, we wanted to see that the output was high when all three inputs are low, but that each input when raised would drive the output low.
This process was very labor intensive and slow, because of the need to find a way to test every input of every gate, isolating others. Since many of the 120 gates on a board in a module are only connected internally on the multilayer circuit board, this is logically challenging.
After ten hours of work we had confirmed that every gate worked properly. One possibility was to continue this way, bench testing modules A4 to A24, but at ten hours each that would be a very long process.
Decision to check for destructive failures only in the remaining logic modules
We decided that our sample of 720 gates with only one intermittent failure was a sign that we would expect a low rate of defects. Thus, it would be reasonable to put all the logic together, power up, and debug using logic analyzers. A small number of defects can be chased down this way, but a system with high rates wouldn't do enough to capture anything meaningful.
Testing the analog interface circuits between the rest of the spacecraft and the AGC
There was five remaining modules in tray A and they were not digital logic. These provided the interface circuit to couple a spacecraft signal, typically implemented as 28V / 0V logic levels and the AGC gates that run with 4V / 0V levels.
We set up the bench to power each circuit in each module, inject the appropriate input conditions and verify the correct output. This work only took a half day to wrap up modules A25 to A29, and once done, let us fully populate tray A for testing.
Working on tray B - the memory and alarm functions
Tray B has the master oscillator (B8) which is needed, plus an alarms module that checks for various analog style failures. For example, it verifies that the clock isn't running too fast, that the 28V input power and the internally generated 14 and 4 volt power are within limits, and other checks.
We did not install any of the modules involved in memory access - rope or erasable memory drivers, selection logic or sense amps. Nor did we install the erasable memory module itself. The six core rope module slots on the outside of the AGC are also empty.
We then plugged tray A and B together. Power and control signals must move between the two trays over connectors that only mate with the two trays are together. The module faces of the two trays face each other when mated, with the wired wrapped backplanes facing outward.
Examination of the core memory module (erasable memory)
Mike checked continuity of all the pins on the erasable memory module. Since this is one of the few modules that is potted, we couldn't see inside and only had the pins to test. Modules on computers that would fly on missions had every module potted. A material like RTV-11 hardens around the components and holds them from vibration.
Unfortunately, we discovered that one of the pins was an open circuit. Each bit of the memory has a power feed and two inhibit line returns (among other pins used for other purposes), but bit 16 has an open circuit on one of the two inhibit lines.
That will mean that at least half, if not the full 2K of memory will always have bit 16 set to 1. Not only will this limit what op codes and data we can use, even for limited testing, but it will cause the parity check to fail on average half the time (i.e any time when bit 16 should be zero but is read as one). We will need to correct this in order to have a workable erasable memory.
Examination of Core Rope Simulator modules
This AGC came with a rare Core Rope Simulator rather than six actual rope modules. The two boxes we have plug in and act as the six rope modules, but are cabled to an external box that we don't have to complete the functionality.
Typically, the rope simulator has a magnetic tape drive and some RAM. A new version of software is read from the tape, put in RAM and then served to the AGC as the contents of rope. Test and development centers would find a unit like this convenient, instead of waiting for a real rope module to be wired and installed to make any change.
Since we don't have real rope modules and don't have a complete Core Rope Simulator system, Ken is reverse engineering the simulator boxes we have. We need to do this in order to create hardware to drive the boxes. Alternatively we will need to build a different kind of tool or box to respond with the proper data word anytime the AGC requests it.
Verification of level of the AGC
Mike went through the backplane wiring and logic module, looking at documentation of each of the versions of the AGC and comparing what we had to those variants. He discovered that our backplanes were brought up to the latest revisions, although a couple of the modules were not equally updated.
All this meant was that some added functions, while wired on the backplane, aren't instantiated in the modules. Based on the product code and serial number, our AGC was initially built as one of the last of the block II prototype systems, just before Raytheon switched over to production versions.
The prototypes went to the Kennedy and Johnson space centers and to the major contractors such as North American Aviation and Grumman. Ours was used at Johnson space center, although it didn't have that name back during the Apollo program.
The reworking of our unit made it functionally identical to the production versions that would fly on the various missions. This was good news since we want to be able to run any mission's software.
First bringup of the computer (without memories)
In the next part of this blog we will bring up the AGC, without memories, to see in what shape the computer is after all these years.
TO BE CONTINUED
We provided 14V and 4V to the clock oscillator module (B7), limiting current to protect against any failing capacitors or other shorts. The output was beautiful. We have a pulse. It is running at 2.04801 MHz, amazingly close to spec for a machine that hasn't run in almost 50 years.
 |
| Pulse of the Apollo Guidance Computer |
Testing the logic modules (contents of tray A)
The AGC is built of two trays that are bolted together, with mostly digital circuits in tray A and mostly analog circuits in tray B. For example, the master oscillator is module B8 because it is analog, but all the further processing of the clock is done digitally, in modules A1 and A2 in tray A.
The logic modules are constructed exclusively of dual NOR gates, each with three inputs. These are Resistor-Transistor Logic (RTL), which was quickly obsoleted in the industry by DTL, TTL, CMOS and other schemes. After a while working with the schematics, one gets proficient at quickly understanding designs using only NOR gates. No flip flops, no other type of boolean operations at all.
The RTL chips operate with 4V power supply and have a pretty low threshold between logic 0 and logic 1 condition. We found voltages as low as 0.8V that were successfully recognized logic 1 states in the running AGC. Logic 0 inputs were typically under 0.3V in actual operation.
Clock dividers and scaler, other timing signals
The oscillator delivers 2MHz to modules A1 and A2, which first divide the signal to the master oscillator of 1MHz and then produce various timing cycles that will drive the rest of the computer. Most notably, the computer is designed around core memory cycles which take almost 12 us, using twelve of the 1MHz pulses as stages 1 to 12.
We injected appropriate clock signals into the two modules as they sat on the workbench, out of the AGC trays, and verified that all of the timing signals were being produced at the proper time. We soon discovered a failure!
The AGC uses a variety of timers of different durations, from almost master clock frequency down to one that has a pulse once every 36 hours. This is implemented with a chain of dividers in A1. We found that the time pulses stopped somewhere in the middle of the chain.
All the other timing logic in A1 and A2 worked fine. With the failure of our clock scaler, the machine would not properly detect certain errors (watchdog alarms) that depended on slower pulses, it would never go into Standby mode, and of course mission software that needed longer timers wouldn't work right. Most of the machine would still work even with the one gate failing.
 |
| Timing signals (some of the 12 steps of a memory cycle |
After some probing around and other testing, we concluded that the output of one NOR gate was up at 4V. This means it was shorted to the VCC power supply. Inside the NOR gate there is a pullup resistor between VCC and the output, but this was shorted. It is an odd failure, one that seemed only possible if some bit of conductive material had fallen across some the two traces, VCC and output; these traces must be close to each other somewhere for this failure to have occured.
We shook the module with the package facing down, retested, and found that the problem had cleared. We will perpetually have the conductive junk, probably a solder blob, inside the sealed gate. Problems could recur but we know that some shaking will clear it up again. We will live with this situation.
SQ (instruction op code) register and decoding module
The next module we placed on the bench, A3, provided the storage of the instruction operation code and performed some of the initial decoding. A module has two boards, one per side. Each board has up to 60 flat pack ICs with two NOR gates apiece.
We worked to find combinations of inputs that would put each and every gate through all its states and give us a way to observe its output. That is, we wanted to see that the output was high when all three inputs are low, but that each input when raised would drive the output low.
This process was very labor intensive and slow, because of the need to find a way to test every input of every gate, isolating others. Since many of the 120 gates on a board in a module are only connected internally on the multilayer circuit board, this is logically challenging.
 |
| Some checking of the logic modules |
Decision to check for destructive failures only in the remaining logic modules
We decided that our sample of 720 gates with only one intermittent failure was a sign that we would expect a low rate of defects. Thus, it would be reasonable to put all the logic together, power up, and debug using logic analyzers. A small number of defects can be chased down this way, but a system with high rates wouldn't do enough to capture anything meaningful.
The only type of failure that would be physically damaging would be short circuits that might produce high current and melt something. We have power supplies that will limit the current to a set level, which means we can power a module and see if the current is too high but not let it go high enough to cause damage.
We proceeded to power each of the remaining logic modules, board by board, while monitoring current drain. They all passed and ran at levels between 80ma and 110ma per board. Given this, we could stuff it all into tray A and power up the machine.
Testing the twin power supply modules
Tray A also has two power supply modules. These are identical but look at the connections across certain configuration pins to decide if they will produce 14V or 4V, the two main voltages used across the AGC. We bench tested each module with both configurations, even though in actual operation each module is assigned one of the voltages.
 |
| During power supply testing |
The power supplies produce two outputs - switched and unswitched - so that the computer can sit in standby mode with very low power consumption since only a small subset of circuits are powered. We checked that both the switched and unswitched outputs were proper and clean.
Testing the analog interface circuits between the rest of the spacecraft and the AGC
There was five remaining modules in tray A and they were not digital logic. These provided the interface circuit to couple a spacecraft signal, typically implemented as 28V / 0V logic levels and the AGC gates that run with 4V / 0V levels.
We set up the bench to power each circuit in each module, inject the appropriate input conditions and verify the correct output. This work only took a half day to wrap up modules A25 to A29, and once done, let us fully populate tray A for testing.
 |
| Testing the analog modules |
Tray B has the master oscillator (B8) which is needed, plus an alarms module that checks for various analog style failures. For example, it verifies that the clock isn't running too fast, that the 28V input power and the internally generated 14 and 4 volt power are within limits, and other checks.
We did not install any of the modules involved in memory access - rope or erasable memory drivers, selection logic or sense amps. Nor did we install the erasable memory module itself. The six core rope module slots on the outside of the AGC are also empty.
We then plugged tray A and B together. Power and control signals must move between the two trays over connectors that only mate with the two trays are together. The module faces of the two trays face each other when mated, with the wired wrapped backplanes facing outward.
 |
| Trays mated, tray B on bottom with open slots for core rope modules |
Mike checked continuity of all the pins on the erasable memory module. Since this is one of the few modules that is potted, we couldn't see inside and only had the pins to test. Modules on computers that would fly on missions had every module potted. A material like RTV-11 hardens around the components and holds them from vibration.
Unfortunately, we discovered that one of the pins was an open circuit. Each bit of the memory has a power feed and two inhibit line returns (among other pins used for other purposes), but bit 16 has an open circuit on one of the two inhibit lines.
That will mean that at least half, if not the full 2K of memory will always have bit 16 set to 1. Not only will this limit what op codes and data we can use, even for limited testing, but it will cause the parity check to fail on average half the time (i.e any time when bit 16 should be zero but is read as one). We will need to correct this in order to have a workable erasable memory.
Examination of Core Rope Simulator modules
This AGC came with a rare Core Rope Simulator rather than six actual rope modules. The two boxes we have plug in and act as the six rope modules, but are cabled to an external box that we don't have to complete the functionality.
Typically, the rope simulator has a magnetic tape drive and some RAM. A new version of software is read from the tape, put in RAM and then served to the AGC as the contents of rope. Test and development centers would find a unit like this convenient, instead of waiting for a real rope module to be wired and installed to make any change.
Since we don't have real rope modules and don't have a complete Core Rope Simulator system, Ken is reverse engineering the simulator boxes we have. We need to do this in order to create hardware to drive the boxes. Alternatively we will need to build a different kind of tool or box to respond with the proper data word anytime the AGC requests it.
 |
| Ken reverse engineering the core rope simulator boxes |
Mike went through the backplane wiring and logic module, looking at documentation of each of the versions of the AGC and comparing what we had to those variants. He discovered that our backplanes were brought up to the latest revisions, although a couple of the modules were not equally updated.
All this meant was that some added functions, while wired on the backplane, aren't instantiated in the modules. Based on the product code and serial number, our AGC was initially built as one of the last of the block II prototype systems, just before Raytheon switched over to production versions.
The prototypes went to the Kennedy and Johnson space centers and to the major contractors such as North American Aviation and Grumman. Ours was used at Johnson space center, although it didn't have that name back during the Apollo program.
The reworking of our unit made it functionally identical to the production versions that would fly on the various missions. This was good news since we want to be able to run any mission's software.
First bringup of the computer (without memories)
In the next part of this blog we will bring up the AGC, without memories, to see in what shape the computer is after all these years.
TO BE CONTINUED
Friday, November 2, 2018
DSKY substitute as complete as I could get it before flying to work on AGC
PREPARING FOR RESTORATION OF APOLLO GUIDANCE COMPUTER
Final construction and testing of the DSKY substitute
I finally finished building all the input interface boards that will safely and compatibly hook the Apollo Guidance Computer outputs based on 28V logic levels with the Arduino Mega 2560 driving the DSKY substitute.
This took half of the morning to wrap up. Twenty four components and ten wires are soldered on the board, which has no traces so the component leads have to bend over to form connections. I then had to inspect all of them for shorts or other errors.
Fifteen of the input circuits hook directly to Arduino input pins. Another eight input circuits are hooked to an I/O expander board on the I2C chain. All eight of the output circuits run through an 8 channel relay board that makes use of my output interface board.
I finished up the wiring of the second 8 channel relay board, which is used to interrupt the cathodes of the 7-segment displays and LEDs to make them flash. A small board gates a 1.5Hz signal through to the relay triggers when the control signal from the AGC is on. The relay clicks on and off to darken the displays.
I am having some difficulties getting the solid wire to hold in the relay board - the design of the connectors are a bit rickety. I tried folding the wire in a U shape, hoping that the two bits of wire would provide a flatter surface for the connector to hold. That did the trick.
The only displays that flash are Operator Error and Key Release caution lights, as well as the four digits of the Verb and Noun displays.That requires six of the eight relays on the board to interrupt their cathodes.
I struggled to find a way to mount the main breadboard panel with the displays and lights raised off the tabletop, to allow all the support boards and wires to be tucked under or made less visible. Perhaps I could tilt it vertically to the same orientation as a real DSKY, I thought. The keyboard should be at the same level as the display and stationed appropriately.
Cable ties were judiciously applied to neaten up the rats nest of wires between all the boards and modules of this prototype. Transporting this safely will be a major concern. My carry on bag will hold this and a few other fragile items; my laptop and all my other personal items will be wedged into a small 'personal bag'.
All the connections from the myriad of boards and breadboards came to a prototype shield and were soldered in place. Trying to align perhaps 70 pins across different headers to all slide into the female connector was quite challenging with all the wires soldered in place on the shield.
The shield was finally inserted over the Arduino and power was applied. Normally the DSKY comes up with blanks in all the displays and the caution and warning lights are extinguished. For the first tests, I put in some dummy values just to verify that the circuitry is working.
I verified I had fixed the bug that kept the keyboard scan routine from emitting any further keycodes after the first. There is a hardware issue with detection of keypresses in row 3 of the left keypad, something I will chase down and correct.
I did see values displaying and the relays working, although there is a lot of functionality I can't test without the driver since this code is watching 23 input lines and acting on them. These lines are unpowered because I didn't hook the input interface boards up to 28V for this test, meaning the input lines were floating. With power, they would either sit at 0 or 5V depending on the input wire.
I ran out of time - my flight to the location of the AGC is tomorrow morning. I had envisioned a full set of tests with the Arduino Uno test driver setup, but I will have to do that from my hotel room, in my spare time.
I carefully moved the unit and all its attached boards over to my carry-on, placing T-shirts between layers since I expect this may get a close inspection by TSA tomorrow.
I finally finished building all the input interface boards that will safely and compatibly hook the Apollo Guidance Computer outputs based on 28V logic levels with the Arduino Mega 2560 driving the DSKY substitute.
This took half of the morning to wrap up. Twenty four components and ten wires are soldered on the board, which has no traces so the component leads have to bend over to form connections. I then had to inspect all of them for shorts or other errors.
Fifteen of the input circuits hook directly to Arduino input pins. Another eight input circuits are hooked to an I/O expander board on the I2C chain. All eight of the output circuits run through an 8 channel relay board that makes use of my output interface board.
I finished up the wiring of the second 8 channel relay board, which is used to interrupt the cathodes of the 7-segment displays and LEDs to make them flash. A small board gates a 1.5Hz signal through to the relay triggers when the control signal from the AGC is on. The relay clicks on and off to darken the displays.
I am having some difficulties getting the solid wire to hold in the relay board - the design of the connectors are a bit rickety. I tried folding the wire in a U shape, hoping that the two bits of wire would provide a flatter surface for the connector to hold. That did the trick.
The only displays that flash are Operator Error and Key Release caution lights, as well as the four digits of the Verb and Noun displays.That requires six of the eight relays on the board to interrupt their cathodes.
I struggled to find a way to mount the main breadboard panel with the displays and lights raised off the tabletop, to allow all the support boards and wires to be tucked under or made less visible. Perhaps I could tilt it vertically to the same orientation as a real DSKY, I thought. The keyboard should be at the same level as the display and stationed appropriately.
Cable ties were judiciously applied to neaten up the rats nest of wires between all the boards and modules of this prototype. Transporting this safely will be a major concern. My carry on bag will hold this and a few other fragile items; my laptop and all my other personal items will be wedged into a small 'personal bag'.
All the connections from the myriad of boards and breadboards came to a prototype shield and were soldered in place. Trying to align perhaps 70 pins across different headers to all slide into the female connector was quite challenging with all the wires soldered in place on the shield.
The shield was finally inserted over the Arduino and power was applied. Normally the DSKY comes up with blanks in all the displays and the caution and warning lights are extinguished. For the first tests, I put in some dummy values just to verify that the circuitry is working.
I verified I had fixed the bug that kept the keyboard scan routine from emitting any further keycodes after the first. There is a hardware issue with detection of keypresses in row 3 of the left keypad, something I will chase down and correct.
I did see values displaying and the relays working, although there is a lot of functionality I can't test without the driver since this code is watching 23 input lines and acting on them. These lines are unpowered because I didn't hook the input interface boards up to 28V for this test, meaning the input lines were floating. With power, they would either sit at 0 or 5V depending on the input wire.
I ran out of time - my flight to the location of the AGC is tomorrow morning. I had envisioned a full set of tests with the Arduino Uno test driver setup, but I will have to do that from my hotel room, in my spare time.
I carefully moved the unit and all its attached boards over to my carry-on, placing T-shirts between layers since I expect this may get a close inspection by TSA tomorrow.
Thursday, November 1, 2018
Soldering more secure connections for the DSKY substitute
PREPARING FOR RESTORATION OF APOLLO GUIDANCE COMPUTER
Nearing the end of the construction and testing of the DSKY substitute
The first two finished boards for input and output interface to the Apollo Guidance Computer passed their tests. I began to wire up the five addition input boards needed to connect my DSKY substitute to the AGC next week. This is assuming great progress from a standing start, but better to be prepared.
The output board is used in conjunction with relays. When activated, this connects the output line through a 2K resistor to ground. If not activated, the line floats up to +28V. This particular board can support 9 output circuits, although I am only using 8 for the DSKY.
The input board is connected to a circuit in the AGC that will pull the line down towards ground (through a 2K resistor in the AGC and a 10K resistor in line the the transistor base, so the actual achieved voltage is quite a bit higher than 0V but sufficient to cause the transistor to conduct on my interface circuit.
That transistor delivers 28V through a voltage divider to ground, with the center tap sitting at just under 5V when active or at 0V when inactive. Thus this input circuit is also an inverter - a 28V input produces a 0V output, while when the input is pulled down towards ground, the output becomes 5V.
I replaced some more breadboards with small solder boards like the ones used above, making all the connections to the I/O expander boards for the LEDs and for eight of the inputs from the first two input boards like the above that I completed.
I soldered the alphanumeric (14-segment) driver onto another board and wired it to the I2C chain. I still have to do quite a bit:
The first two finished boards for input and output interface to the Apollo Guidance Computer passed their tests. I began to wire up the five addition input boards needed to connect my DSKY substitute to the AGC next week. This is assuming great progress from a standing start, but better to be prepared.
The output board is used in conjunction with relays. When activated, this connects the output line through a 2K resistor to ground. If not activated, the line floats up to +28V. This particular board can support 9 output circuits, although I am only using 8 for the DSKY.
 |
| First output board, used with relays to pull AGC input down towards ground |
That transistor delivers 28V through a voltage divider to ground, with the center tap sitting at just under 5V when active or at 0V when inactive. Thus this input circuit is also an inverter - a 28V input produces a 0V output, while when the input is pulled down towards ground, the output becomes 5V.
 |
| First input board, floats input at 28V, pulled down when AGC output activates it, gives inverted TTL output |
 |
| I2C chain of I/O expander boards to drive LEDs and record some inputs |
- wire up the two 8-channel relay modules
- move the flasher logic to a circuit board
- finish the input interface boards
- wire in the 23 input channels
- wire up the 8 output channels to the relay board
- wire up the control lines from the Arduino to the output channel relay board
- finalize the flashing control relay board wiring
- ensure I have all the +5 and ground wires routed properly
- check the Arduino prototype shield wiring for shorts or bad solder joints
One stretch goal is to label the keyboard with the proper text and to put a plastic overlay on the display lights with the correct text legends. I printed the text but the process of converting them into rub-on transfers will take about an hour that I may not have tomorrow.
Subscribe to:
Posts (Atom)