Sunday, July 14, 2019

Final preparations for 50th anniversary demonstration tour with AGC

FINISHING THE LEGO LM MODEL AND STAND

I glued together an acrylic stand that would allow the LM to sit up in the air about a foot. It has a pivot to set the LM with the descent engine facing forward during the initial part of our landing demonstration and then to pivot to place the engine bell downward for the last portion.

It required a bit more bracing than I expected but a quick trip to Tap Plastic got me the extra parts I needed. It can be disassembled and carried in my suitcase for setup at each demonstration event.

I needed a small enclosure to hold the resistor boards, controller and wiring out of site. I rummaged through my parts bins and found something useful, even if not visually appealing.

BUILDING STAND FOR PATCH PANELS AND PLUGBOARDS

The stand to hold the electronics panels up is more challenging due to the combined weight. I stated with two vertical strips of 1/4" acrylic but, just as with the LM stand, I had to add bracing before it was ready for use on the trip.

Starting from the bottom, we have the right hand patch panel placed. This panel will have the 35 banana plug leads from the DSKY replica pulling forward, which is why I chose it for the bottom spot. Next is the main interfaces board which will convert spacecraft signals to modern TTL levels.

Above that is the left hand patch panel. I placed the interfaces board between them to shorten the runs of banana plug cables that splay from the interfaces to the two patch panel boards. Finally, the top position is held by the secondary interface board which provides the last five AGC C level output signals needed for my LM model lighting. These needed aluminum holders to connect them to the acrylic vertical rails.

I picked up some better bolts and nuts for assembling everything and stowed them in my luggage with the stand pieces. I had to reinforce the acrylic with thicker rectangular braces just to keep every thing standing vertically. It was quite solid and dependable once completed.

Our 'rack' of patch panel and interface boards

TESTING THE LEGO LM MODEL LIGHTING

I first set up a temporary version of the controller firmware without the input pullup resistors, thus making each input stay at ground unless I explicitly inject a high level. This let me test the key inputs - the 16 RCS jet signals that cause LEDs to light on the model's RCS quadrants. Along with that, I tried out the engine on and engine off commands, viewing the descent engine bell light at minimum thrust brightness and watching it go out again.

To perform a more comprehensive test, and to help with checkout, I set up a self-test capability at power up of the controller board. It will light each logical group of RCS thrusters in turn; up, down, forward/back and side.

It then will turn on the engine, throttle it up to full thrust, take it back to half thrust, and turn it off. This allows me to check quite a bit without needing a tester to drive the controller.

An external tester (another Arduino) produced pulse patterns to further test the throttle control portion of this model. I had it all wired together ready for testing Sunday morning. By afternoon I want to start packing everything for my departure Monday morning.

MINIMIZING HEAVY POWER SUPPLIES IN MY LUGGAGE

Our lab and my home testbench makes use of triple section power supplies as well as more than one supply box.  That would be painful to have to haul around on the demonstration tour next week. Mike is bringing the big 28V supply that will feed the AGC and supply my interface circuits.

I need to bring a beefy 5V supply to power the DSKY and the controller for the LEGO LM model. There is another voltage necessary if we will be sending pulses into the AGC (the Y type circuits) - 14V - and I don't want to drag around a separate supply. Fortunately, the AGC delivers 14V on another pin of the patch panel.

Thursday, July 11, 2019

Programming Arduino to drive LEDs on LEGO LM model based on signals from AGC flying mission

PROGRAMMING THE RCS AND  DESCENT ENGINE CONTROLLER FOR THE LM MODEL

The Descent Propulsion System (DPS) is the engine for the LM landing, while the Reaction Control System (RCS) is a set of thrusters that control rotation and translation of the LM by emitting short bursts out of one or more of the 16 thrusters on the spacecraft.

The DPS is turned on and off by two signals from the AGC, Command Engine On and Command Engine Off. Its initial thrust level is 10% when it is turned on. This gives time for the AGC to move the engine on gimbals to reduce and pitch or roll movement due to off-axis thrust.

The computer then will throttle the engine up and down through its usable range of 10 to 65% or 100%. For technical reasons, it can't run long in the range between 65 and 100. This is implemented with a counter that is incremented or decremented by pulses from the AGC (Increase Thrust and Decrease Thrust signals).

When the counter is 0, the engine is at its minimum 10% thrust level and when the counter reaches 3200 the engine is at 100% thrust. That means that each increment or decrement changes thrust by .028125% but the LED I will be using does not have the granularity to show 3200 steps of brightness with an 8 bit value.

Instead, I will use 18 steps from 10% to 100% brightness. Taking the counter modulo 178 will give me a step number from 0 to 17 which selects from the table of 18 brightness values for the LED I am using.

The two thrust change signals are pulses that add or subtract one from the counter, but limiting it to values no lower than 0 and no higher than 3200. I set up interrupt routines triggered by the rising edge of those two pulses; these routines simply bump the counter up or down by 1.

The main routine monitors the counter modulo 178 and selects the appropriate brightness value accordingly, but only when the new step number has changed. The main routine also turns the engine completely off or switches it on based on the two engine command signals.

RCS jets will illuminate the LED that represents that thruster. A pulse on the RCS jets is a minimum of 13 milliseconds long, so I will set a start time when I illuminate the LED, and won't switch off the light if the control signal becomes LOW unless at least 13 ms has elapsed.

I coded this up and loaded an Arduino Mega 2560. I now plan to whip up a testing circuit using an Arduino Uno to generate various control signals, different durations, and try stepping the engine up and down.

INVESTIGATING ACCESSING THE DOWNLINK TELEMETRY FROM THE AGC

The Apollo Guidance Computer outputs some words of data on the downlink to Mission Control in Houston, which are joined to all the other sensor and biomedical data on the same link. The telemetry circuits in the Apollo spacecraft control the link and request data from the AGC at the appropriate time in the transmission. This data rides on the same microwave (S-band) signal that carries voice and television signals.

Telemetry runs at 51.2 Kbps, high rate, as long as the unified S band link is available. When the spacecraft uses the backup omnidirectional VHF links the data rate is slashed to 1.6Kbps, called low rate. The AGC operates at either rate, controlled by the telemetry system.

The telemetry system sends a pulse on the Start Downlink signal line, followed by a stream of 40 Downlink Sync pulses at the high or low data rate, terminating the session with a pulse on a separate Stop Downlink signal line. Each session, begun with a Start Downlink pulse, occurs at a rate of 50 times per second for high rate and 10 times per second for low rate.

When each Downlink Sync pulse arrives, the AGC sends out another bit by emitting a pulse on Downlink 0 Bit signal line if the data is zero or skipping a pulse if the data value is one. This was chosen so that the stream from an inactive AGC consists of all one bits.

I am thinking of developing a box that will sent the Start, Sync and Stop pulses to the AGC, capturing the bitstream it sends back. Time is extremely limited before I head out for the demonstration tour, but if it is easy to bang this together then I will try.

Improving the interface board and patch panels for demos, plus LM Model set up to display jet firings

NEW INTERFACES BOARD CHECKOUT

I designed a replacement board to house interface circuits that allow use of modern 5V TTL or 3.3V LVCMOS devices to drive or monitor the AGC signals. The spacecraft itself uses either 28V logic levels or short 10-14V pulses on wire pairs for signalling, neither of which are friendly to an Arduino or FPGA.

Each of these new boards is essentially a rack width (almost 19 inches) and houses one third of the total input-output circuits used by the AGC. I chose that fraction because the PCB fab I used had a minimum order of three boards for their fast turnaround prototype service. Thus one of the PCBs houses 19 type D, 13 type C, 11 type Y and 18 type XT circuits. D and Y circuits are inputs to the AGC of discrete and pulse type respectively. C and XT circuits are outputs from the AGC, discrete and pulse.
Apollo to modern logic interfaces board
I built the board and then subjected the circuits to testing to be sure they worked properly. Using signal generators, oscilloscopes and the like, I produced pulses with my type Y circuit and read them with a type XT on my board. In the same way, I toggled logic levels with my type D circuit which were read correctly on the type C. I used a signal generator first but later an Arduino produced both discrete and pulse test signals and detected the results from the receiving circuits

This board will be mounted just above one of the two rack wide patch panel boards we use to route all the AGC signals from the dense A51 connector to banana jack receptacles that are easily accessible. Banana plug cables hook any chosen circuit from the AGC to the interface board and from there to switches, Arduinos etc to work with the signals.

BUILDING VISUAL DISPLAY FOR DEMONSTRATION TOUR

I began the task of soldering small 30 gauge wire to tiny LEDs that I will place inside the nozzles of the Reaction Control System (RCS) quadrants on the Lego Lunar Lander kit. I had to figure out a way to route the wires that is not distracting to the viewer. The LEDs I will mount inside the Ascent and Descent engine bells are larger.

The Lego kit also arrived today. Only 1087 parts to assemble; somehow it took five hours to build. I skipped the moon surface portion as well as the astronaut figures, concentrating on the portion I will display, but that was a small fraction of the kit.

LEGO model to install LEDs in RCS and engine bells
I needed to work on an anchor point to hold and orient the LM. I can't use the top of the LM because the ascent stage pops off the descent half, the weight would be enough to have it fall aprt if supported only by the top. I needed  something that holds the descent stage and rotates, which will allow me to orient it engine bell forward for part of the descent and then engine bell down for the last portion of the landing.

The solution was some clear acrylic with a stand and pivot. A trip to Tap Plastics nearby will give me the materials I needed. While there I will buy materials for the rack stand to hold the two patch panels and my interface circuit boards. They have to disassemble to travel in my luggage for the demonstration trips, otherwise I would just build permanent stands or use a half height rack enclosure.

Installing the LEDs for the RCS engines was quite challenging. My first plan was to use 0603 surface mount LEDs with wires tacked onto them, super-glued into the model. The problem is the delicateness of the LED and the small size. I had wires break off ruining the LED as I tried to place them into a thruster cone.

Preparing 3mm LEDs

Inserting 30 ga wire through drilled holes in LEGO pieces

I then bought some 3mm thru hole LEDs, bright red when illuminated but transparent white otherwise,  and prepared them for mounting. Using my 30 gauge black wire to tack onto the base of the LED leads, I could cut the rest of the leg off and use heat shrink tubing to insulate. Two wires per LED until downstream where I can combine all the common wires.

I put a 1/8" plastic drill in a small press, placed the lego pieces in a soft vise and drilled out openings inside the four thruster cones. I also drilled out an opening near where the quadrant attaches to an arm on the LM body. This gave barely enough room to fit eight wires through the final hole.

Using my stereo microscope, various tweezers and lots of patience, I threaded the wires through the LEGO parts and pushed in the LEDs. After a bit I had two of the four quadrants done and found that it was not too frustrating as long as I took my time on each.

By Thursday morning I had all four quadrants, 16 thrusters in all, with LEDs wired out to a terminator breadboard which also housed the 16 resistors. The wires ran into the descent stage and out along the bottom, where the black wires blended pretty well against the black LEGO bricks.

Light in RCS quadrants in place, wiring exiting at bottom

It was a simpler job to wire up an RGB serial addressable LED into the Descent Engine Bell. That completed all the lighting I need to use the LM model to show in real-time the thruster and engine firings that occur during the lunar landing we will perform for our demonstrations.

PATCH PANEL COMPLETION

I wired up every signal from the AGC to its circuit pin on the patch panels. This gives us the flexibility to interact however we want, making use of the interface board to do so using regular TTL and LVCMOS devices and controller boards. There are more than 300 wires involved in completing this task, spread over two days to give my back a rest between sections.

Next up was building the stand for the two patch panels and the interface circuits board. The main principle was to use screws and nuts to assemble the stands but allow them to fit compactly in my luggage for transport. It will take a few days for Tap Plastic to cut all the pieces I need but by Saturday I should be assembling the stands.

PROGRAMMING ARDUINO TO LIGHT MODEL LEDS BASED ON ENGINE FIRINGS

I put an Arduino Mega 2560 into duty as a controller for my LEGO Lunar Module display. It hooks to 18 of my type C interfaces and 2 of my type XT circuits. Since my one interface board implements only 13 type C interfaces, I had to partially build a second board to provide the patch positions for the remaining 5 C interfaces.

That build was done quickly, although I was short three red banana plug receptacles which are needed for type C circuits. I had to make do with three yellow jacks which normally would be used for XT circuits.

From the Arduino viewpoint, a type C circuit coming from my interface PCB is simply an INPUT_PULLUP, an input with a weak pull-up resistor inside. The type XT circuits produce pulses that must be detected by the Arduino so I used interrupt service routines for those two pins triggered on rising edges.

Using the MIT documentation of the equations used in their hybrid digital/analog simulator of the LM, I collected the information I needed to properly simulate the throttle of the descent engine. Our AGC will increment or decrement a counter in the engine controller to vary thrust. That engine has a minimum thrust level of 10%, achieved even when the counter is at zero. The counter goes up to a maximum equivalent of 3200 which represents full thrust of the engine.

I also have the lag characteristics they used for the engine. It takes 2.3 seconds from when the Engine On command is received until it is producing the minimum 10% thrust, and when switched off it takes 380 ms. Every thrust change command requires about 300ms to take effect.

I am attempting to model the engine LED in the model based on the MIT defined characteristics. This adds some complexity to the Arduino program but the results should be worth it. More on this soon.

Friday, July 5, 2019

Final preparations for public display and demonstrations for the 50th anniversary of Apollo 11

Current Switch Module failed diodes

We will mill away all the rest of the potting on this module, drill out the 26 diodes that we haven't yet replaced and populate the module with suitable replacements. That will restore the erasable (traditional core RAM) memory to full operation.

Improving demonstration for spectators

While we can accomplish a lunar landing using just the AGC and DSKY, it requires the spectator to visualize quite a bit after they are taught to interpret various numbers displayed by the DSKY. We want to help visitors visualize what is happening so we worked on some improvements.

Mike worked on changes to his test monitor board, the one that plugs into the AGC test connector A52 and also runs with his FPGA replica of the AGC. He made corresponding changes to the NASSP plug-in for the Orbiter simulation, such that the simulation will use the real AGC (or FPGA AGC) rather than a virtual AGC inside the package.

Using this, we can interact with our AGC but see the results on the screen showing the inside of the Lunar Module and the view outside the windows. All the displays such as the altitude tape meter and FDAI flight director work properly. This gives a compelling simulation of the landing, with the engine thrust climbing, the view pivoting as the LM is maneuvered, and appropriate sounds emitting from the screen.

In addition to this, we will bring the LEGO Lunar Lander with LEDs placed in the RCS thruster jet openings and in the bells of the Ascent and Descent engines, driving them based on the pulses emitted by the software running in the real AGC during the mission. An Arduino interprets the pulses and controls the LEDs.

Organizing demonstration tour for 50th anniversary of Apollo 11

Now that the AGC is running well and we are prepared to demonstrate it supporting portions of the Apollo 11 mission, we have to set up public appearances for later this month. Having just completed booking the trip, I can share where we will be appearing.

At the start, we will bring the AGC to the home of the lead designer, Eldon Hall, and let him enjoy the sight of his masterpiece working again after all these years gone by. This will take place in Florida, after which we will fly up to the Long Island, NY area for the first public appearance.

We expect to give short talks and demonstrate landings at the Cradle of Aviation Museum, near LM 13 and CM 2 plus a LM Simulator and other Apollo era artifacts. This will occur on the 18th of July. We are waiting for the museum to define times more exactly.

The following day, July 19th will have us setting up in the MIT Museum in advance of our public appearance all day and evening of July 20th, the fiftieth anniversary of the landing on the moon. We will give talks and demonstrate portions of the flight including landings multiple times on the 20th. Once again, we don't have exact times yet.

That concludes the public demonstration portion of our trip, but we will have a chance to meet with many of the original programmers and designers from MIT and others who supported the success of the Apollo program.

Tuesday, June 25, 2019

Another diode bites the dust in the AGC Current Switch Module

Setback with erasable memory, failure in Current Switch module

Near the end of our time with the AGC, we were running various portions of the Apollo missions such as LM powered descent initiation, LM rendezvous calculation to rejoin CM, or CM burn calculation to hit the earth entry interface.

Suddenly, we were seeing hardware restarts and the software was not running correctly. Using Mike's test monitor we quickly realized that erasable memory was not working properly. We could rapidly trace this down to a failure in the Current Switch module (again).

Testing diodes after powering down the computer led us to a third diode that failed open, just as the first two had been. We didn't have enough time left to mill out new windows in the polyurethane foam encapsulation, drill out the bad diode and insert a replacement.

This module has 28 such diodes - three have already failed and NASA documents show that Raytheon switched from the original diodes to new types due to failures similar to what we experienced. The original diodes had a mesa construction, something the semiconductor industry quickly moved beyond.

We kept the Current Switch Module here so we could remove all the encapsulation on the module, then replace the 26 original diodes with modern 1N914B diodes like we did with the first two we found. This should be completed so that we can rejoin the module with the AGC for live demonstrations we will do in July.

Back to Houston until our next session

The AGC flew back to its home in Houston, until our next session and subsequent demonstrations.

Sunday, June 23, 2019

Final upgrade to our AGC and working on simulating missions such as lunar landing

Adding the PRO key support to this AGC

The original version of the block II computer had a key on the DSKY marked STBY that was used to put the computer into sleep mode (and bring it out again). Soon, it was replaced by the label PRO and was used for two purposes - standby same as before, and signaling PROceed when the astronaut was ready to approve some action proposed by the computer.

Prior to having a PRO key, the astronaut had to push VERB, push 3, push 3, and push ENTR to confirm a proposed action. When there are a sequence of requests from the program, such as during the start of a landing sequence, having to mash four keys to give assent took time. In some cases, the astronaut had just five seconds to approve the start of a burn; if the VERB 33 ENTR had a typo, it might be impossible to redo it fast enough.

The original STBY function is purely done in hardware. Once the software sets a bit that allows standby, the key itself was wired directly to circuitry that dropped the switchable 14V and 4V supplies, leaving only a base level of power for functions such as pulse generation needed by other hardware in the spacecraft.

The new second use of the key for approving action requires that the software be able to see when the key is pressed. This was accomplished by a small hardware change that routed the key signal from the standby circuitry to set a previously unused bit in one of the I/O channels. Our AGC, the last of the prototypes before the flight versions, did not have the PRO key wired to the channel yet.

We recreated the change, using wire wrap, such that when my DSKY has its PRO key pushed, it sets the bit in the channel to approve a program proposed action, but will still switch off the computer if held long enough to activate the standby circuit.

Challenges running the PDI program for the Apollo 11 landing

We have been able to run program 63, the way you accomplish Powered Descent Initiation (PDI) to lower the orbit of the LM and transition through to landing on the surface. However, once the program sets up for the burn of the descent engine and is given permission by the PRO key to fire, the lack of acceleration is detected. The software flashes Verb 97 which means thrust failure and we can't slow down or reach the surface.

Marc set up an accelerometer substitute using an ATMEL processor and three potentiometers, allowing us to indicate (crudely) that acceleration is occurring on any or all of the three axes. However, the question of which direction the acceleration should be injected is not a simple one. In fact, the answer necessary to bring the LM to the ground with zero forward velocity is devilishly hard to calculate.

The inertial measurement unit of the LM (and CM) consists of a platform (called the stable member) which has three gyroscopes and three accelerometers (PIPAs) mounted so that X, Y and Z rotation and acceleration can be measured.

This stable member is suspended on three concentric gimbals so that regardless of the rotation or movement of the spacecraft, the stable member will remain locked in the same position relative to the milky way galaxy. It drifts slightly over time, which is why it is aligned using the star sighting telescope to return it to its ideal position.

Each gimbal is used to read off rotation of the spacecraft around the stable member in one axis. For technical reasons, having only three gimbals means that the inner and outer gimbals can move so they are parallel to each other. This is bad because a rotation can't be assigned to only one axis, it instead is seen as a movement on both axes simultaneously. This foils the ability to recognize movement properly and is called Gimbal Lock.

Certain rotations of the spacecraft can move it so that it approaches or reaches Gimbal Lock. Once in gimbal lock, the stable member must be reinitialized and star sightings are necessary to align it, a time consuming process. To avoid this, the people in Flight Planning (and in Mission Control) have to plan out the orientation of the stable member so that all the planned maneuvers of the spacecraft can be made without approaching gimbal lock.

Consider the way the spacecraft evens out heat and cold as it flies between earth and moon - the Passive Thermal Control mode - where the spacecraft has to spin on an axis like food on a barbecue spit, with the axis at right angles to the sun. Turning one rotation every two minutes means the stable member is rotating on its gimbals at the same rate and that motion can't be allowed to cause gimbal lock.

At many times during the mission, the orientation of the stable member relative to the galactic background is changed to put it in the best position for the maneuvers ahead. The orientation information is sent up as a REFSSMMAT, a reference stable member matrix either vocally or by the ground using 'remote control' to press DSKY keys to enter the new data in the AGC.

The flight director/attitude indicator display (the 8 ball) is tied to the stable member, thus it is showing where the stable member is sitting while the spacecraft rotates around it. That works out find when in a straight line trajectory such as during translunar coasting, but introduces problems when in orbit around the earth or the moon.

Since the stable member remains fixed relative to the background of the universe, the orbital path will cause the member to appear to rotate as the spacecraft circles the globe. If we start with the stable member parallel to the moon's surface at one point and leave the spacecraft in that same position, as we orbit our nose will appear to pitch up then nose over until we seem to be upside down relative to the moon when we are at the opposite point of the orbit, gradually rotating again until we are flat and parallel at the starting point of the orbit.

The flight director/attitude indicator will rotate around with the stable member (and spacecraft). If we cause the spacecraft to pitch at exactly the right rate, then the spacecraft 'floor' will always be pointing down towards the moon. Our stable member, as a consequence, will be rotating as we orbit and our 8-ball won't make sense.

A special device named ORDEAL will permute the 8-ball so that it seems to match the 'floor' of our spacecraft, and cause the correct pitch rate, so that the pilot does not see the stable member rotating. However, regardless of the tweaked display, the reality of an orbit means that the stable member is rotating while we circle the moon.

That means that the accelerometers idea of X, Y and Z is based on the stable member and are rotating compared to our spacecraft. During the landing, therefore, the direction of the thrust vector of the descent engine is rotating and has to be calculated based on the original setting of the REFSSMMAT, the place we are in the lunar orbit and the orientation of the LM compared to the moon's surface.

The landing of Apollo 11 begins with the crew oriented to look down at the moon's surface with the descent engine bell pointed in the direction of the orbit. Later in the descent, the LM pitches around so that the astronaut's feet are pointed down at the surface and they are looking forward in the direction they are moving. All these occur while the stable member stays fixed in inertial space.

As you can now see, the direction of acceleration we have to input to the AGC, which are given strictly to the X, Y and Z direction of the stable member, have to be manipulated based on all the factors above.

One simplification is that we don't care how the LM moves from face down to face forward since the stable member doesn't change at all. What does matter, however, is the effect of that movement on the thrust vector relative to the 1/6 G acceleration of the moon on the spacecraft. That must be factored in, both by the AGC in computing the burn and pitch up timing, but also in our injected acceleration.

We will need to issue AGC commands to look at the orientation of the stable member at the time we begin the engine burn, calculate how this rotates as we move around the orbital path/descent path, and apply acceleration appropriate to that.

So much more complicated that simply delivering input on the axis of the descent engine bell of the LM. We expect we will even see an alarm thrown after we think we have landed, unless we provide acceleration at 1/6 G in the proper direction to match where the moon surface is relative to the stable member.

Wednesday, June 19, 2019

Interacting with the running Apollo Guidance Computer using DSKY

Testing the DSKY substitute

We hooked up the DSKY substitute to the Apollo Guidance Computer and verified that all the lights, displays and keys worked properly. We now have a physical manifestation of the interface used by the astronauts to interact with the computer.

Video of DSKY working with AGC

Testing Ben Krasnow's electroluminescent panel

Ben (Applied Science YouTube channel) came over with the EL panel he built to NASA specs and we worked to interface it to my DSKY substitute. After a bit of work, I got it mirroring the display on my DSKY as the AGC ran various Apollo era software.

Video of Applied Science electroluminescent panel used with AGC

It was great to see a real electroluminescent display providing the output from the computer. Realism of the interaction took a step up.

Using the DSKY with the AGC

We ran several portions of the Apollo 11 LM mission, using the DSKY to interact with the Luminary 99 flight software that flew in the spacecraft. This included calculating the rendezvous burn after ascent from the lunar surface and of course the powered descent down to the surface. No 1202 alarms for us, because we didn't have a simulated Rendezvous Radar feeding pulses to overload the computer.

Monday, June 17, 2019

Erasable (core memory RAM) of Apollo Guidance Computer repaired and working properly

New flaw in Current Switch module

After having replaced the two failed diodes in this module, we began testing it with pulses to ensure that it worked properly at the currents and timing needed to drive core memory. This module consists of a number of large ferrite cores with four sets of windings on each. One winding is tied to all the cores and is used to switch them all back to the 'off' magnetic orientation. A second winding is used to select a core by switching it to the 'on' orientation.

Certain cores are selected by flipping them on. This flip of magnetic state induces a pulse in the third wiring, which causes a transistor to conduct to drive current through an X or Y addressing line in the B12 core memory module. Later, when all the cores are reset by the first winding, only the ones that had been selected will flip back. This induces a pulse in the fourth winding, turning on a transistor to drive current in the opposite direction through the X or Y addressing line.

This scheme is clever because the core retains the selected address from when it is selected. This selection induced the pulse to read out a word of the B12 module but also remembers which address lines were selected. The AGC logic does not have to retain the address in a register until later in the memory cycle when the word of erasable memory is rewritten - the selection cores themselves hold that information.

One one of the cores, we had a 4 ohm short in some module I/O pins that shouldn't be connected at all. After excavating the potting material around the suspicious area, we quickly excluded simple to repair causes such as shorting wires, shorted interconnect, etc. We found that the first and fourth windings were somehow shorting inside the core - these are the reset winding and the winding that produces a pulse when a previously selected core is reset.
Shorted windings in B11 Current Switch Module circuit
Careful examination showed us that it would be impossible to extract the core/windings from inside the cordwood module where they were epoxied in place. At least, impossible to remove it without severe damage to that core and its windings.

We have a suitable replacement core but putting four windings of 50, 32, 32 and 20 turns would be extremely challenging given the small diameter of the core. Fortunately, Marc and Mike figured out a clever hack to give us an equivalent functionality. It began by completely disconnecting the fourth winding, the one that switches on a transistor when a selected core is reset. That effectively cured the short since the shorted winding was no longer connected to anything.

Then, we installed a transistor of opposite polarity tied to the third winding, the one that is normally used only while the core is selected. When the selection pulse flips the core, the third winding sees a positive direction pulse, causing the attached NPN transistor to conduct. When the core is later reset, the pulse is in the negative direction thus the transistor doesn't turn on.

Yet, if we hook that third winding to a PNP transistor as well as to its NPN transistor, the pulses from the windings will cause one transistor to turn on with a positive going pulse, and the other transistor to turn on with the negative going pulse. This produces the same behavior as the original circuit, but requires only three windings instead of four. That is fortunate because we don't have four useful windings.

We inserted a small PNP transistor and did some rewiring, which gave us a module that passed all tests with flying colors. We were then ready to install the erasable memory driver modules, this current switch module, and our erasable memory (core stack) into the AGC.

PNP transistor wired to third winding of circuit
Archiving prior contents of the B12 erasable memory

We still have the flaw in the core memory stack - the inhibit wire for one of the data bits (bit 16) is an open circuit. In the read portion of a memory cycle, the selected X and Y address lines flip all cores of that word to zero. Any of the bits in that word which had a 1 stored in it will flip, causing a pulse to be detected by the sense amplifiers. This is how core memory does a read, by destructively flipping the word to zero.

During the reset pulse from the current switch module core circuits, the cores for the selected X and Y address lines, those whose current switch cores had been set on, will flip all the bits of the word to a 1 state. However, we don't want them to be 1, as some should be 0. That is the purpose of the inhibit wire - a signal on the inhibit wire will block the reset from flipping that particular bit to 1.

Reading data involves erasing it first, then rewriting the original value (or putting a new value in if this is a write operation). With a bad inhibit wire, bit 16 will be rewritten (or written) to 1 regardless of our desired value. Thus the loss of the inhibit wire renders that bit useless throughout all 2K words of erasable memory.

If the inhibit wire damaged happened after the computer was last powered down, then it hadn't yet forced a 1 into every bit 16 in the core stack. We had the opportunity to read the contents of the core memory correctly, since the read portion of a memory access doesn't use the inhibit wire. However, this is a one shot opportunity, as the process of reading any word will jam in the 1 in bit 16 during rewrite.

We powered up with memory access blocked through the test connector, put the machine in single instruction mode, and tested by reading a few memory addresses. We did find that bit 16 sometimes had the value 0, sometimes 1, verifying that the damage to the module was not present when it was being accessed 40-50 years ago.

We wrote down the values we saw in the test locations, then used Mike's test monitor to run through all of memory and retrieve the values. Folding in the few manual reads gave us a complete file of the prior contents of our memory module B12.

Mike studied the contents and could confirm that this was running a version of Aurora software, similar but not identical to the release for which there is an archived image. He could decode the contents of the DSKY display. It showed that the last command entered was a coarse alignment of the IMU (gyroscopes and accelerometers) to angles of 0, 0, and 90. The Gimbal Lock warning light was on, which may have been the event that caused the operator to perform a coarse alignment. Finally, the code stores a latitude when aligning the gyros - it was the location of the Johnson Spacecraft Center in Houston.

Rewiring to swap parity and data bits in B12 core memory module

Now that the prior contents are safely archived, we can implement our modification to make the core erasable memory functional again. Our great fortune is that the erasable memory module contains an error detection method called parity, to capture the cases where some radiation event might have flipped a data bit randomly.

This works by adding a sixteenth bit to each word - data bits are 0 to 14 plus 16, with bit 15 representing parity. The rule for parity is that the number of bits in a word which are 1 has to be an odd quantity. If not, the parity bit is set to 1 thus making the entire word have an odd count. if the data bits themselves have an odd count of 1s, the parity bit is set to 0.

At the end of a read from a word of core memory, the processor counts the 1 bits, determines if parity should have been 1 or 0 and then compares it to the parity bit read from memory. Mismatches raise an erasable memory parity alarm.

The erasable memory is quite reliable, especially since our AGC is not out in space subject to radiation events. We therefore don't really need parity checking. That gives us a working bit 15 which we can substitute for the broken data bit 16. As long as we can disable the parity checking, blocking the alarm, we will have a working memory.

Some backplane wiring was introduced to swap bits 15 and 16, as well as block the signal that performs a parity check. With the wirewrap changes completed, we closed up the AGC and tested. Success! The erasable memory is fully functional and we can run software fully out of the repaired erasable memory and the core rope simulator boxes providing the fixed (core rope) read only memory, just as the machine

Main spacecraft connector and patch panel finished, accelerometer simulator plugged in

Main connector

All connections between the spacecraft and the Apollo Guidance Computer take place through the main connector (A51). This has 360 pins, six rows high and 60 columns wide. The DSKY itself connects through A51, which made this a priority to implement.

As I detailed in an earlier post, this is a labor intensive cleaning process but finally we had all of them ready for a male connector to be inserted. We had designed an aluminum plate to hold the 360 Malco mini-wasp pins and to position a printed circuit board we designed atop it. The pins were soldered to the PCB and the far end held eight HD50 connectors (familiar to those who used these as SCSI cables).
Connector attached to AGC carrying 360 signals out for our access
The connector was inserted with a partial population of pins - those we needed for our demonstrations. We will finish installing the remaining pins later. The eight cables were run from the PCB out to the patch panels that gave us the same six row by 60 column array as banana plug receptacles. We used this to hook up the DSKY and simulated switches and other spacecraft systems.

Accelerometer simulator developed

Marc designed an Arduino based module that would simulate the accelerometers of the LM spacecraft. These Pendulous Integrating Pulse Accelerometers (PIPAs) swing a small pendulum back and forth kicked by three pulses in one direction followed by three in the other. When no external acceleration exists, the PIPA outputs a steady stream of three positive then three negative pulses.

PIPA simulator plugged into patch panel
The AGC software checks for this pattern and throws up a Program alarm (212) if they are not present, since that means the inertial navigation and digital autopilot functions can't be performed. Marc's device would deliver this pulse stream into the appropriate pins of the A51 connector, through our patch panel. He also designed it to allow him to add extra pulses in either direction to reflect acceleration of the LM.

Retrieving previously lost Apollo programs from a museum's core rope modules

Archiving

We visited the Computer History Museum which has a pair of core rope modules containing the Retread 50 software. These are part of their exhibition of an Apollo Guidance Computer, but we arranged to archive the contents of the modules using our working AGC, in support of their software history effort and to deepen understanding of what is in their artifacts.

Before we connected these to our AGC, it was important to verify that there were no open circuits on the strand and module select lines or other inputs as our computer could be damaged if that was the case. Following a long test plan, we determined that the rope modules were safe to insert.

We first started the AGC in single instruction mode, turned on the core rope access and tried a couple of accesses to see if they appeared to be delivering valid contents, based on the known archive of the predecessor Retread 44 software. It matched.

Mike then used his test monitor tool to read through all of the core rope modules and save their contents. We did spot some parity errors which were confine to specific parts of the modules. In fact, all errors were in the first (B1) module and they were in strands 1 and 9. Modules have 12 strands each holding 512 words of data (thus 6K capacity per module).

For strand 1, a particular data bit was always reading as a 1 value. This appears to be from a failed component inside the core rope module. We did not open the modules as they are artifacts that should not be disturbed. When a word legitimately had a 1 value in that bit, no parity error ensued, but every location that intended a 0 value would trigger a parity alarm.

Strand 9 had more erratic results, but again for just a single bit location in every word. That meant it too could be recovered by looking at the parity bit value.

We wrote a quick post-processing routine to use the parity bit to interplate the correct value of our bad bit. In early programs such as Retread, they left any unused words unwired - thus they would legitimately produce a parity error with all zero results. We had to find those unused words since we had improperly corrected it to have a non-zero word value.

With the data recovered, it could be shared and analyzed. Many of the expected changes were there - since Retread 44 was written on the emulator before the first block II hardware was built, it had some code that failed on real hardware due to inaccuracies in the emulator. All of these flaws in code were corrected, many as were expected but some through alternate changes.

There is some added code, parts that are seen in later (Aurora) software but some that are unique to this previously lost software version. Analysis continues

We hope to archive other lost software versions as owners of other rope modules permit us to plug them in to our system.

Sunday, June 9, 2019

Finishing equipment to read core rope programs and drive the AGC with real Apollo software

Core Rope Simulator box debugging

The core rope simulator boxes that came with the Apollo Guidance Computer were built by Raytheon and designed to cable up to a large console that would host the ROM program in rewritable (RAM) storage. We only had the two boxes inserted in place of the AGC core rope modules. not the large console nor the cables.

After Ken had reverse engineered the boxes and developed a Beaglebone-based driver to serve up the ROM programs to the boxes, we set into the process of debugging. First up, we had to find and repair about a dozen faults, many of them broken wires, as well as reshape IC leads to work in the very unreliable DipStik connectors used in the box.

After the Raytheon hardware was working it was time to test the function of Ken's driver box. It mostly worked right out of the chute, but we had to chase down some defects that caused errors reading certain blocks of memory.

The core rope modules that are normally used with the AGC consist of six separate modules which represent six blocks of 6K words. We will call them R1, R2, S1, S2, T1 and T2. The lowest addresses are in R1 and the highest contained in T2. The faults we experienced were when accessing S2, T1 and T2.

The Raytheon box would latch in the request for a particular module, then which of the 512 cores in that module should be read and which of 12 words from that module are passed into the computer sense amplifiers. What we found was that bad addresses were being latched due to transient short pulses that were due to poor logic design, a race hazard.

We could see the glitch, a pulse on the order of 100 ns compared to the real pulses that occur later and are microseconds in duration. The particular way that the addresses were decoded inside the Raytheon boxes allowed these to pass to the trigger that set flipflops, even though some of the address information had not yet made it through the other gates.

The solution was to add a small capacitor to the trigger line, such that it would ignore pulses that were too short but still lock in addresses when the desired signals arrived. The time constant that would block the glitch kept the integrated very short signal from climbing high enough to trigger the flipflop.

When we inserted a capacitor that was about 100 times too large, it began to delay the real signals - as their leading edge was spread out over a long slope. This caused some other addressing issues, but a capacitor that was just a couple times larger than the glitch did the job perfectly.

These boxes were serial number 3, but it is hard to see how they could have worked correctly on an Apollo Guidance Computer. That is, when the ROM was more than half full. They might have sufficed for small programs that remained in sections R1, R2 and S1. In any case, the core rope simulator works just fine now, letting us run real software such as Luminary 99 that was installed in the LM for the Apollo 11 mission.

Core Rope Module jumpers designed and manufactured

We intend to read existing core rope modules from both museums and private collectors, but a few early sets don't use all six rope modules. The code can fit in just two for Retread 50, as an example, thus four slots are empty. To work properly, jumper plugs must be installed in the four unused slots.

We didn't have any jumpers, but we could work out what was needed. Mike designed some connector housings, built them using a 3D printer, and populated them with the Samtec built Malco Mini-Wasp connectors. He installed small circuit board to complete the jumper circuit, and added handles for insertion and removal. These worked fine, setting us up for the opportunities ahead to read and archive the contents of core rope modules.

Rope jumper connector in place
Connector removal using handle

Saturday, June 8, 2019

Continuing the restoration of the Apollo Guidance Computer

RESTORING THE AGC

Checkout to be sure all modules are working properly

Finally we were in front of the Apollo Guidance Computer (AGC) again and could continue the restoration. First step was to check that all the prior parts of the system were still working properly. a task that took a day.

Main connector (360 pin A51)

The AGC is hooked to the rest of the spacecraft through a single 360 pin connector (A51) which routes signals between the computer and Lunar Module components like the Display Keyboard (DSKY), reaction control jets, rocket engines, control panels, gyroscopes and telemetry.

The AGC had been hermetically sealed and filled with nitrogen for most of time, almost five decades, that it sat exposed to the humidity of Houston. That left the bulk of the machine in pristine condition. The second smaller external test connector (A52) with its 144 pins had a sealed cover in place; this would be removed to allow technicians to hook up ground support equipment and test out the AGC.

Unfortunately, A51 had no cover and its 360 tiny female contacts were not protected. We inspected this carefully with a microscope and found that the pins were mostly straight, but had gunk that may be corrosion on them and lots of foreign material wedged into the holes.

Cleaning this requires tools like dentists use, but sharpened to be even smaller, plus cleaning instruments, applied laboriously one pin at a time. Since the pins are plated with 50 mils of gold, they weren't corroded, just coated with contaminants that could be removed. We are achieving about 10 cleaned pins per hour but can't work on the connector while the computer is together and being tested. It will be several more days before it is fully cleaned.

Checking out memory drive electronics

While in Houston last time, we didn't test the analog modules in the AGC that would drive access to the erasable (RAM) and fixed (ROM) memories. These consisted of modules such as Strand Select, Core Rope Drivers, Current Switch, Sense Amplifiers, and Erasable Memory Drivers.

We hadn't tested these last time because we had found that the erasable core memory module B12 had a fault that would cause bit 16 of every word to become a 1. We planned to repair B12 before continuing.

We successfully tested the Strand Select, Core Rope Driver and Sense Amplifier modules using copies of the original checkout procedures. The procedures were partial documents, so we had to study schematics and fill in the blanks.

Current Switch module debugging and repairs

When we got to the Current Switch module, however, we discovered two circuits inside with faults. The path to drive addressing current through a X or Y wire involves a special transformer core with four windings. These flip the core one way to drive current in the read direction, flip it another way to drive current in the write direction, and have the windings that output the current pulses in each direction.

One X wire and one Y wire circuit for addressing set of the transformers showed as open circuits, meaning that the computer couldn't address any words contained in 1/64th of the rows and 1/32nd of the columns, not just the one word that was at the coincidence of the failed X and Y circuits. 

Flight versions of the AGC had every electronic module and the backplane filled with some form of epoxy or other foam material to hold all parts in place during extreme accelerations and vibrations. These seal in the parts, making inspection and replacement very difficult.

Fortunately for us, only two modules were encased (potted is the common term used for this). Sadly, they were two broken modules - the core memory itself (B12) and the Current Switch module. While we had no feasible path to open and fix B12 due to the placement of the fault deep inside a sandwich of core planes, we could proceed on the newly discovered faulty unit.

Armed with mechanical drawings of our module from the NASA archives and pictures of an unpotted copy of the module located at the Computer History Museum, we knew exactly what lay under the potting and where it was to great accuracy.

Excavating the parts to inspect and repair was similar in some ways to excavating a dinosaur fossil from surrounding rock. We milled down through the epoxy until we found the first trace of white wire insulation. From that point, careful work with picks, tiny screwdrives, tiny chisels and a small hammer broke off small bits of the potting.

The encasing material was brittle enough to break without putting any damaging force on the components themselves. The result was a window through the potting, with the face of the cordwood module and its components exposed cleanly. Since these modules have parts that run through the cordwood from one face to the other, we needed to expose four windows in order to reach the two failed circuits inside.
First excavation to find and repair bad diode

Second excavation to find and repair the second bad diode
The serial number of the Current Switch module we worked on was not the same as the item shown in records to be installed in our AGC. Perhaps our machine was used as the donor for a working Current Switch, with the bad part stuck in our machine before it went to be sold off in a GSA auction.

The bad part in both circuits was a signal diode. It was a part that is still common today, 1N914B, so we drove to a local electronics shop and picked up some. The old diodes were detached and drilled out, our replacements slid into the module and soldered into circuit. Repaired!

Ground Support Equipment Monitor tested and put into use

Mike designed an FPGA based board to replicate the function of the three floor standing racks of ground support equipment used at MIT for checkout and debugging of the AGC. Samtec had delivered the new pins they manufactured to allow us to build or repair AGC connectors - this required creating all the tooling to accomplish a limited production run, quite expensive even with the original mechanical drawings of the pins.

This donation was greatly appreciated as it allowed us to build the 144 pin connector A52, put Mike's monitor board atop it and fit it to the AGC. Of course, we also had to make the special jacking screws that tightened the connector to the AGC and pulled it up and out of computer smoothly on removal. Marc had found screws that were close and then milled the remaining features to give us a new supply.

The newly created A52 connector with its attached monitor board was installed on the AGC and we turned on power to both. The design was good, all functions checked out well, so we could control and test the AGC from this point forward with it closed up and in its flight configuration.

A52 test connector with our ground support monitor being installed
The control panel of the ground support equipment was replaced by a graphical user interface that included a virtual DSKY as well as the hundreds of blinking lights, buttons and switches. It communicates over USB with the monitor board in A52.

The monitor can simulate both erasable and fixed memory along with all its other checkout functions. The erasable memory function needed a gate that was only added to AGCs after ours, but we did some wirewrap to enable that function, borrowing a spare gate inside the machine.

Mike could fire up Apollo software, such as Luminary 99 that flew on first moon landing. Initially, we had a Program alarm (0210) and certain virtual DSKY keys came out garbled. This was quickly fixed as we found two signals, one from the DSKY and one from the Intertial Measurement Unit (IMU) that needed to be set high. We wired up a jumper since the pins of A51 had still not been cleaned adequately.

Everything worked great. We ran through the entire self test suite of diagnostics using MIT written software such as Retread 44. The AGC logic was working flawlessly and, using the simulated fixed and erasable memory, we had a running computer.

Ground Support Equipment interface hooked to the A52 connector
In future days we need to complete cleaning of A51, build a replacement male connector and find a way to cable up signals to emulate parts of the spacecraft. My DSKY substitute also connects through 34 of the pins on A51, so it can't be wired into the machine and used until we have that connector completed. 

Sunday, June 2, 2019

The DSKY substitute is completed and ready to attach to the AGC

COMPLETING THE DSKY

Assembly and mechanical improvements

The issues discovered with the keyboard were caused by the shape of the cardboard shims that guide the keys up and down in their channels. Rob Lion worked on those and on the mechanical assembly of the keyboard and the display parts to the case.

All that remained was to drill the holes to attach the acrylic base to the aluminum enclosure, to fasten the faceplate on, and to do some final gluing of loose parts. One faceplate panel was sliding in its cover and keycaps were coming off because the prior glue was not adequate. The DSKY substitute is now sitting ready to be wired to the Apollo Guidance Computer.

Wrapping up the acceptance testing

The remaining series of DSKY tests after the last post were:
  1. set up the register value (M12-M15) for lights and toggle each of the remaining
    1. Vel 
    2. Alt
    3. No DAP
    4. Prog
    5. No Att
    6. Prio Disp
  2. Set up register value and toggle signs
  3. Set digits in Prog display
  4. Set digits in Verb display
  5. Set digits in Noun display
  6. drop 14, verify they turn off, turn 14 on, verify they are blank
  7. Set up Verb, Noun and Prog, then turn on VNflash and observe
  8. Oscillate VNflash
  9. set digits in registers
    1. R1
    2. R2
    3. R3
  10. verify split register changes both R characters
  11. flash key rel light
  12. verify Pro key detected
  13. verify keypress release works (on while key is depressed)
  14. verify reset key discrete
  15. verify all key codes sent
I completed everything down to step 12, trying every character value and all positions. I had to make some changes to my code to properly handle the VNflash discrete. That cycles at 1.5Hz and when it is on, the Verb and Noun are supposed to blank out. It causes them to blink, in other words. My initial spot in the code wasn't always getting executed, but I fixed that quickly.

Testing the keypresses is even easier - no need for 14 or 28 volt supplies, just the 5V line into the DSKY and hook the eight output wires up to eight pins on an Arduino. I discovered some mechanical issues with the keys that needed to be addressed but electrically they were perfect.

Thursday, May 30, 2019

Wrapping up 'acceptance testing' of the DSKY substitute and working on AGC patch panel

COMPLETING ASSEMBLY OF DSKY SUBSTITUTE

Adding in serial link to drive external EL panel

The Arduino Mega has four serial ports, one carried over the USB cable but three more that can be accessed by the software and pins on the board. I chose Serial1, using pins 18 and 19. Now I have to add code to command the EL panel built by Ben Krasnow (Applied Science YouTube channel) to have it mirror the output of my own.

I haven't written or debugged the code to drive his board, something I will defer until next week. My first priority is getting the DSKY complete and fully tested in advance of wiring it up to the Apollo Guidance Computer.

Activating EL wire in my display

My display uses electroluminescent wire for the three horizontal green lines on the display that separate the outputs for registers 1, 2 and 3 from the upper displays. These wires require high voltage AC, which is generated by a driver module inside the acrylic base. The driver is switched on and off by a relay board.

The driver should only be on when the BPLUSSW (14V switched power) from the AGC is present. The AGC can be in standby mode, with no power on BPLUSSW, which causes the panel on the DSKY to blank out.

I have been sensing the BPLUSSW using a voltage divider on my PCB that produces a 5V signal when the computer is not in standby. This seemed like a perfect source for the drive to fire my relay board, however the Arduino has an input pin connected as well. The result is that the Arduino drags the voltage way too low because of how the resistances of my divider and the Arduino input are set.

The Arduino gets the signal because the driver on my PCB that lights up the legends under Verb, Noun and Program requires a ground level to turn on. Thus it is logically inverted compared to the output of my voltage divider - when no BPLUSSW, I light the legends and when the computer comes out of standby, I extinguish them.

I had to put in an inversion in the Arduino code and drive another output pin, so that when BPLUSSW is on, the output pin is pulled to ground and causes my legends to light. I now have to set up another output pin that is not inverted but mirrors the BPLUSSW voltage divider, since my EL wire driver board relay needs +5 to activate, not ground.

I do have spare pins on the Arduino Mega 2560, although they are not connected to headers on my PCB. I can insert a discrete wire (carefully) into the chosen pin and route that to my EL driver relay board. A quick update to the Arduino code and it was ready for testing.

Soldering the external female connector pins to the PCB wiring

The task of soldering my 'inside' connections - short cables that run from the header jacks on my PCB down into the acrylic base where they get connected to the solder lugs of the female 85 pin Bendix connector - was tedious and made slower by the constant double checking to be sure everything is right.

Completing the cable on the external male connector

Once I had all the wires extended the soldered joints needed to be covered with shrink wrap and then the entire cable wrapped in electrical tape to make for a neater wiring harness. As part of this job I tested all the connections through to the inside connectors where they would plug into the PCB, to be sure I had a solid set of circuits.

Each external wire was hooked to a continuity tester and I located the PCB connector where it terminated, then stuck on a label with the signal associated with the wire. Repeat 34 more times and I had all the signals and power lines identified and verified, ready for testing.

SETTING UP TESTING RIG

Testing the DSKY replica is complicated by the odd voltages used. My PCB and some minor boards are driven by the usual 5V DC, but we also have to provide 14V and 28V for testing. The AGC signals coming into the DSKY swing between 28V and ground. The computer as a 14V supply that is turned off when the AGC is in standby, thus we have to take the switched 14V into the DSKY as a sense input.

Incoming signals work by having 28V fed through my PCB interface circuits and the input pin pulled to ground to activate the signal (inverted logic). Outgoing signals pulls their output pin down to ground when activated (also inverted logic). Fortunately, I can feed whatever voltage I want on the output pins, so I chose to use pullup resistors on the tester side.

My 23 inputs to the DSKY are handling 28V, so I made use of banks of relays that will pull the signal to ground through a 2K resistor when the relay is activated. The relay trigger circuit is fired by the Arduino which I use for the test driver. The 8 outputs of the DSKY are hooked to input pins on the Arduino set with internal pullup. I use the serial link over USB to a computer to control the test and read results, in addition to observing the DSKY behavior.

I didn't have enough 2K resistors for the setup today - five 2K and 7 2.2K were all I could scrounge up - but I can test a fair amount of the DSKY with that, stepwise. As long as the codes for the digits don't have more than eight total ones out of the ten bits to be set in a relay command word, I have enough resistors. I can do a more complete test later this week when I get more resistors.

TESTING THE DSKY

My series of tests:

  1. Power up 5V, nothing should show
  2. add 28V, nothing should change
  3. Add 14V, the legends (Verb, Noun, Prog) and the three lines should light
  4. Seven of the fourteen lights have discrete lines to trip them, toggle each
    1. Stby
    2. Restart
    3. Temp
    4. Key Rel
    5. Uplink Acty
    6. Opr Err
    7. Comp Acty
  5. set up the register value (M12-M15) for lights and toggle each of the next eight
    1. Vel
    2. Alt
    3. No DAP
    4. Prog
    5. No Att
    6. Gimbal Lock
    7. Tracker
    8. Prio Disp
  6. With some of the lights lit, turn off 14V, verify they remain on, then turn on
  7. Set up register value and toggle signs
  8. Set digits in Prog display
  9. Set digits in Verb display
  10. Set digits in Noun display
  11. drop 14, verify they turn off, turn 14 on, verify they are blank
  12. Set up Verb, Noun and Prog, then turn on VNflash and observe
  13. Oscillate VNflash
  14. set digits in registers
    1. R1
    2. R2
    3. R3
  15. verify split register changes both R characters
  16. flash key rel light
  17. verify Pro key detected
  18. verify keypress release works (on while key is depressed)
  19. verify reset key discrete
  20. verify all key codes sent

I ran through the tests and at step 3, I wasn't getting the electroluminescent wires lighting to switch on. That was because of the funky way I was inserting the newly added wire that drives the EL driver board relay - it kept popping out.

After I found a way to secure those wires down inside the headers on the Arduino but then discovered that the Arduino and the relay activation were cycling - as the relay flipped on due to presence of 14V signal, the driver pulled so much power that the Arduino reset. I realized I need a humongous filter capacitor to support the surge as the EL driver turns on. That fixed the behavior.

I modified the DSKY firmware to emit diagnostic information over the serial port. I will be able to see whether the input is ever checked and if it is detected. I saw that when I pulled down the Oper Error input, it was properly detected. The Caution/Warning lights illuminated too, thus this part of the test is good.

The COMP ACTY light did not illuminate with the 14V missing, which is correct. I then hooked up the 14V to the sense line to verify that when unblanked all will work well. The other lights behave in a way consistent with the real spacecraft.

Six of them are lit or dropped based on a discrete signal coming from the AGC, so they will turn off as soon as the AGC goes into standby. Eight of them are relays inside the DSKY that are only set or reset by writing to register bank 12 with specific bits on or off. If they were lit, they remain lit while the computer is in standby and don't turn off into the software writes to turn them off as part of coming out of standby.

Next up I tried to test the signs, which are turned on or off by writing a specific pattern to various register banks. I set up the wiring but I didn't see the sign words light. It was time to do some diagnostic output from inside my firmware and figure this out.

DESIGNING INTERFACE PATCH PANEL FOR AGC

The AGC communicates with the rest of the spacecraft (and the DSKY) through a 360 pin connector (A51). There are four main types of interface circuits for sending and receiving discrete digital signals and for sending and receiving pulses.

The spacecraft systems mostly operate with discrete digital signals which are 28V when logic high and near ground when logic low. Pulse circuits use approximately 10V for the on state of the pulse. My panel will convert those voltages down to levels that can be wired to TTL or LVCMOS digital devices.

The panel will be constructed as ground support equipment, fitting in two racks side by side. Each side implements half of the A51 connector signals; the connector is six rows of 30 pins, a break and 30 pins. Thus, one side will have six rows by thirty columns of receptacles for the patch cables.

The cables use 4mm banana plugs on both end. Above the rack panel with the 6 x 30 connectors hooked to A51 will be five sets of interface circuits, each with 26 interface circuits (8 type C, 7 type D, 6 type XT and 5 type Y).

Monday, May 27, 2019

Readying the DSKY substitute and preparing to interface to the Apollo Guidance Computer

PLANNING FOR AGC MAIN CONNECTOR AND SIGNAL INTERFACING

Reviewing and documenting signals

I spent about a dozen hours poring over different documents and schematics to be sure that I had a complete map of the interfaces between the Apollo Guidance Computer and the rest of the spacecraft. There are 360 pins on connector A51 that provide these connections between the interface modules A25, A26, A27, A28 and A29 inside the computer and the external systems.

Some of the pins are used for power connections, some are spare but the majority are important input and output signals that we may need to connect with in order to create a realistic reenactment of some portion of the Apollo 11 mission. In addition to the signals that operate the DSKY, there are 264 pins that implement 176 input/output signals.

One document I built lists all the pins, which interface module and circuit it connects, and the external system purpose of that signal. Another document lists each interface circuit in the five modules and cross references it to the A51 pin and purpose.

I used those to look for holes in the lists - interface circuits that aren't listed in some document or pins that aren't understood. I am down to two pins on the A51 connector, one of which I know has a thermistor wired to it for analog temperature sensing; likely the other has a similar analog monitoring role.

From these documents I moved on to look for discrepancies. As an example, the schematic for module A26 has different interface circuits hooked to the wires in the spacecraft used to turn the LM Ascent or Descent engines on and off, compared to the other documents I have been using.

Fortunately, the interface circuits are associated with known software locations thus I can look to the Luminary code to determine which set of interface circuits is correct. The engine is either controlled by channel 12, bit 7 or the on and off lines are controlled separately by bits 13 and 14 of channel 11. In fact, this was a late change and the proper signals are controlled by bits 13 and 14 of channel 11.

The control signals for the landing and rendezvous radars were misdocumented in some places but the schematics and LM wiring showed the correct assignments, which I rolled into my master spreadsheets.

Listing needed interface circuits for AGC signals

I also built a spreadsheet listing the AGC pins that get routed over my cable through the 85 pin connector to make use of the DSKY substitute when running the real AGC.

CONSTRUCTING BASE OF DSKY SUBSTITUTE AND WIRING CONNECTORS


I installed the connectors for the AGC and modern power cables into the acrylic panels and glued together the base upon which the aluminum enclosure will sit. This gives me plenty of room inside for the Electroluminescent wire boost converter and driver circuits, plus room for the cabling between the PCB up inside the aluminum enclosure and the external connectors down in the acrylic section.

New acrylic base and external connectors
I installed wiring for the signals between the AGC and the DSKY that are relevent to me. Some provide 115V 400Hz and other power feeds which aren't necessary with my replica. Others activate relays fit inside the DSKY whose contacts are routed back out to external spacecraft circuits but never affect the DSKY operation. These I didn't implement; if necessary we can wire them externally to the AGC and house the relays elsewhere. 

Tuesday, May 21, 2019

AGC connectors and chance to use Applied Science EL panel

DSKY SUBSTITUTE FOR THE APOLLO GUIDANCE COMPUTER

Bottom case

The DSKY aluminum case is not deep enough to comfortably fit the power supplies for the EL wire and any connectors I add. Thus, I designed a 3 1/2" box that will fit underneath. On a side note, the resulting combined size is closer to a real DSKY.  I designed this from 9mm thick acrylic and sent the work out to Ponoko.

For connectors to the outside world, I chose to use a very similar connector to the one selected by MIT in their design of the DSKY. It is a Bendix 85 pin circular connector, readily available on eBay and on its way to me. I chose a similar but smaller two pin connector for my main 5V supply to the unit.
Power connector - total overkill for 5V supply

Connector to AGC signals and power
Planning for possible replacement EL panel

Ben Krasnow created a true electroluminescent panel as a replica of the one used on the MIT DSKY, as announced on his (watch video) Applied Science YouTube channel. He has loaned it to us and I now need to find a way to make use of this marvelous creation in the short time left before we are again working on the AGC.

Once I pick it up, I will see if its size is suitable to fit inside the aluminum enclosure I am working with on my DSKY replica. If it is, then I can remove my 7-segment display parts from my PCB, drop the mounting of the PCB lower and place his panel in the enclosure opening. His replica is programmable over USB serial link, which I should be able to accommodate from my Arduino in my replica.

The worst case situation is that I can't fit it in, thus we will need to use my replica as built with his panel sitting beside it and linked over USB cable. More on this soon, once I have completed the measurements and detailed planning.

Friday, May 10, 2019

Restoring and testing the Core Rope Simulator boxes for the Apollo Guidance Computer

Testing the Core Rope Simulator boxes

Ken Shirriff and I have been testing the two Raytheon built boxes that provide the Core Rope functionality for the AGC. We have been using some pulse generators and power supplies to send in the signals that will come from the AGC as it attempts to read a word of core rope memory.

One of the two core rope simulator boxes; Dipsticks on left, cordwood modules on right
There were a few problems we found and fixed early - a couple of broken off wires we resoldered, a screw that failed in one of the IC chip holders (DipStick), materials failure in the DipStick holders that compromised connectivity of chip leads to the holder terminals.

The early testing of the input side, where we introduced the various address bits and control signals coming from the AGC,  went well. All our address, module and strand selection signals, plus the set and reset control signals, were properly detected in the simulator boxes.

We then focused on the outputs from the simulator boxes to the AGC - 16 differential signal pairs representing the 15 data and 1 parity bits of the word that was being read. The last data bit, number 16 as the AGC numbers memory bits, did not produce a good pattern.

The signal from the AGC requesting to read some address is a 1 microsecond pulse. If the differential data coming from the external system over the 85 pin connector is a binary 1, then the pulse from the AGC causes a pulse in a transformer for that bit position. The other side of the transformer produces a 500 millivolt differential signal, one side of the winding going negative while the other side goes positive.

What we saw on the 15 working bit transformers was such a differential signal, but the failing transformer showed us zero volts on one lead and a much larger positive voltage on the other lead. It still added up to 500 mv, but it wasn't swinging the two leads in opposite directions like the other transformers.
Blue and green signals are not moving in opposite directions
After some thinking and some investigation, we realized that one of the output leads of the transformer had to be shorted to ground. Both the transformer and the resistor that is connected to that lead are sitting in a cordwood module in the simulator box, with no visible connection to ground.

Cordwood stacks components between two end plates, with the parts fitting perpendicular to the plates. This allows resistors, capacitors, transistors, transformers and other parts to be fit very close together, like bottles of soda sitting in a carton, with a plate at the bottom and top of the bottles. Wires connect the parts only on the outside, along the two end plates.

While the end plates we could see were made of nylon, with the wires from a transformer routed out of the hole in one end plate and connected to resistors or other parts, for heat dissipation reasons, the transformers are actually set in drilled holes in an aluminum body that is only covered on the ends with nylon.

Thus, we realized that the lead from the failing transformer was stretched taut in its connection from transformer to its associated resistor in the adjacent cordwood module. Somehow, we thought, the insulation has rubbed on the edge of the aluminum body and caused a short.

Ken tested the short while I manipulated the transformer lead with a toothpick. Immediately, the short went away! Our permanent fix will be to glue a small bit of insulating plastic under the lead to keep it from rubbing against the aluminum body.

Now that we had proven that all the inputs and outputs between the AGC and core rope simulator modules worked correctly, it was time to test the interior logic. That is some timing and control circuits, the conversion of module, strand and core to linear addresses, and the interaction over the thick cables with Ken's external system.

We began setting up conditions and testing the parts of the linear address generation, but soon found another problem. The generated signal for an even numbered strand worked properly for 5 of the 6 even values, but strand 10 did not work.

There are some diodes and a transformer involved in this process, so we began checking continuity and diode voltage drops. The diodes and wires were fine, but the transformer output was not operating. Again, we did some testing and a lot of thinking. Finally, we began looking at the wiring from the transformer, wondering if we had another short.

One of these orange transformer wires has a failed weld at the  left resistor post
However, we instead saw that a weld had failed. The wire from a transformer stuck up through the end plate of the cordwood module, where another wire was tack welded to carry that signal off to other circuits. Our toothpick moved the other wire, which was not in contact with the transformer lead. Next time we work on the module, we will solder the contact back, verify that our problem is resolved, then move on to the remaining interior logic of the simulator boxes.

When they are checked out, Ken can test his external system and its interaction with the core rope simulator boxes. The final test will have to wait until Jimmie comes to visit with the AGC and we can test the entire system - AGC, core rope simulator boxes and Ken's external system - and get it all working properly.

Recap of Core Rope Simulator function

The Apollo Guidance Computer (AGC) uses Core Rope Modules to store its read-only programming. Each of the six modules hold 6K words of memory, each word being 15 data and one parity bit. The programs are 'stored' in the core rope during its construction when wires are woven either through or around the cores to permanently set the value of that bit to 1 or 0.

During development of the programming it would be too cumbersome to wait many weeks for the weaving of new core rope modules anytime a change is made to read only memory contents. Several methods were used to avoid the delay, by simulating the read-only core rope using some type of regular computer memory. One such method installed two Core Rope Simulator boxes into the AGC in lieu of the six core rope memory modules it normally uses.

The Core Rope Simulator box is connected by two thick cables to some ground-side racks of machinery. The simulator box will intercept the address and control signals that come from the AGC to the core rope modules, communicating those requests to the external system which in turn feeds the word of data that is associated with that address.
One of the two 85 pin connectors on the core rope simulator boxes
The simulators never fly into space, but are used on the ground either for development or testing when rapid changes of the read only memory contents are needed. The AGC we are restoring was installed in the Lunar Module Test Article 8, which is a Lunar Module built by Grumman for ground testing purposes. The LTAs were too heavy and not fully qualified for operation in space.

LTA-8 was the unit used to certify the follow-on LMs for use in space. LTA-8 was set in a large vacuum and thermal test chamber in Houston, where astronauts worked with the machine to ensure that the flighty-ready LMs were safe for human flight.

A core rope module implements the 6K words of 16 bits each using only 512 large cores in a module. Each core stores the data for 192 bits (12 words). It does this by weaving the wires for all 12 words through the single core if that bit is to be a 1, otherwise the wire is woven outside of the core ring.

Connectors representing three of the core rope modules; these plug into the AGC
The address coming from the AGC is broken up to select one of six core rope modules, one of 12 strands (word number inside a core) and one of the 512 actual core rings. The selected core ring in the chosen module is set to the on magnetic state, while the other 511 are left in the off state.

When the control signal requests the reading of the content, the core ring is reset. Any wires that are woven through the center of the core ring will see a pulse as the core ring is rest, while the wires bypassing the core don't see any signal.

The strand number selects which of the 12 words of that core are to be channeled back to the sense amplifiers inside the AGC. Thus, the address and control signals are pulses that select and set cores, then reset them to cause the core to send pulses back to the sense amplifiers.

The core rope simulator boxes we are restoring do not have actual core rings, nor are they organized into six modules or 12 strands of words or 512 core ring addresses. The box has to interface with the same circuits that normally set, reset and select core rings, which it does by using small transformers that electrically act like the cores.

While the core rope modules are organized into six modules, 512 core rings, 12 strands and 16 bits to the sense amplifiers, the actual memory address inside the AGC is a linear range of 36K continguous word addresses. The external system connected to the core rope simulator boxes also sees a linear range of 36K contiguous word addresses, not modules, strands and cores.

Thus, the core rope simulator box accepts pulses from the AGC that would, in real core rope modules, set or reset a core ring at specific times. It has to save those module, strand and core addresses from the six sets of pins where the six real modules plug in. It will convert the addresses back to the linear range and send that address out to the external system.

The external system looks up the contents of the linear range address it received over the thick cable and sends that word back to the core rope simulator boxes. The box will then take the 16 bit value and use it to generate signal pulses via transformers that will look, to the sense amplifiers in the AGC, just like a pulse that was generated by a real core ring resetting.

We received the core rope simulator boxes with the AGC, but did not have any documentation at all about them. No schematics, no diagrams, no description of how they worked. Worse, we didn't have the ground based external system or the thick cables that run between external system and core rope simulator modules.

Ken had to reverse engineer the two boxes, understand their operation and build schematics. He then had to design and build an external system that would connect via thick cables and work properly with the simulator boxes. It had to recognize the linear address coming from the boxes, interact with the control signals at the right time, and feed the desired word of data back to the simulator boxes in time for them to pass the data into the AGC.

The external system is built around a Beaglebone and a custom printed circuit card, makes use of the same 85 pin aerospace connectors that are on the outside of the core rope simulator boxes, and will allow us to hold onboard all of the known versions of Apollo software, so that we can select which version is active to the AGC. For example, we can pick Luminary 210 which is the software that flew in the LM on Apollo 11.