Tuesday, September 3, 2019

Repairing Telequipment CT-71 Curve Tracer - part III

REPLACEMENT KNOBS

The new knob is a hair small, esthetically, but completely functional. I drilled the center hole, installed the new V/div knob and then added the thru rod and inner knob for the Series Resistance. They work great. To wrap this up, I want to add a nut to hold the V/div control a bit more firmly in place, but I can perform any device testing as this sits.

Replacement V/div knob behind the Series Resistance one
The rotating scale for the Series Resistance would align a set of resistance values such that the inner knob, when pointing at the outer .2 V/div scale position, selects the lowest of the eight resistance values that can be picked for the chosen Peak Wattage.

Originally, the Peak Wattage lever would turn the inner scale, but I can just refer to the scale and set up a spreadsheet that lists the settings for each of the four wattage peaks.
Resistance setting guide for .5 Watt peak
This is derived from this rotary scale and the movement imparted to it by the Peak Wattage lever.

Rotary scale and teeth that engage with the Peak Wattage lever

NUT TO HOLD CONTROL ASSEMBLY IN PLACE

The nut on the control assembly was a cylindrical shaft around which the scale rotated It was much too deep to allow my replacement knob to work. I tried to match the threads to bolts at local hardware stores, to determine the thread type for the nut I would buy. It was closest to Metric 10 or 3/8" Imperial, but the threads are finer than those in the stores. M10-1.0 was too coarse, as was 3/8-24.

I decided that it must be M10-0.75 although I couldn't find any of them at the big box stores or local hardware shops. Based on the likelihood that this is the proper size, I ordered some nuts online that were that size; should be here within a week.

Sunday, September 1, 2019

Repairing Telequipment CT-71 Curve Tracer - part II

FURTHER DIAGNOSING OF THE HORIZONTAL POSITIONING BEHAVIOR

With the fuse FS1 installed, horizontal and vertical positioning worked properly. I ran through some initial checks of a PNP transistor I had but was still not getting the curves I expected. I had not yet put on the new outer knob and thus had no clear idea what V/div and load resistance values I was using.

Once the knobs and interlocks were working properly,  I set up the tracer on the bench and made use of my oscilloscope and VOM to verify that it was behaving properly. First up was a test of the collector voltage levels and the load resistances. The voltage was easy to look at but to I needed to drive it through my VOM to measure the current in order to properly assess the load resistors and other details.

With that working properly, and the current settings for vertical scale working smoothly, all that was left to test was the step generator that drives the base of the transistor. It was immediately obvious that I had the knob pointer set wrong.

I used the wiring diagrams and back of the control to verify when I was on a particular target setting, then moved the knob to match. Now the behavior of the curve tracer is perfect.

INSTALLATION OF NEW OUTER KNOB FOR VOLTS/DIV SELECTOR

I searched for a suitable knob to use - it had to have dual setscrews to tighten to a 1/4" round shaft and should permit drilling out the front where the inner control shaft will emerge. I picked a larger knob with a distinct pointer on it.

It arrived but it is too large in diameter, blocking the scale on the faceplate for V/div. For the time being I will use it, having drilled the hole and mounted the series resistor knob in front of it. I tried to rotate the controls to their extreme ranges but found that the V/div knob, which should allow for ten positions, would only move through eight of them. I suspect that there is a mechanical interlock from the rotating scale that I removed.

I partially disassembled the controls to check out that possibility. I had to play with this several times before I could get proper operation of the two controls, V/div and Series Resistance. I still need to acquire a smaller outer knob and to add a nut to hold the control assembly better but I am happy with the operation. The new knob is on its way, only needing some drilling once it arrives.

SAMPLE CURVE USING THE DEVICE

Here is a PNP transistor (2N2907A) that I hooked up for my initial test.

2N2907A curve tracing

Monday, August 26, 2019

Repairing Telequipment CT-71 Transistor Curve Tracer - part I

Curve Tracer to be repaired

DIAGNOSING THE FAULTS

There were five defects in this unit that I found after inspection and while testing some sample transistors. I will try to repair all of them to bring this into full operability which I can make use of in future repair projects.

One of them is cosmetic - the CRT has a blue filter plate installed under the ruled line cover plate, but that plate slips sideways and doesn't sit covering the entire tube face. I am not certain how this was supposed to be anchored originally, but it definitely needs some adjustment or assistance to sit properly.

Two were clear from a quick visual observation. There is a toggle switch that connects the instrument to the left or right set of transistor sockets, used to make quick A/B comparisons. The handle was broken off. The other issue was a broken rotary selector knob, apparently this is a common problem with Telequipment devices where the plastic decomposes and fragments into powder.

The knob is a complex unit that rotates the clear knob to pick locations on the outer printed scale, with a small concentric plastic knob that independently selects positions on a rotating inner scale that sits inside the broken knob in question. The inner rotating scale will turn based on a nearby lever. Finally, this Rube Goldberg control interlocks positions, so that the outer plastic and inner knobs have constraints on their position based on how the inner scale is positioned.

A set of transistor sockets are built into a plate that plugs into the front of the unit, but that plate is missing entirely from my unit. I have to either secure a replacement or build my own sockets to hook to the unit.

Finally, while testing transistors, I discovered that the horizontal position control is not working properly. For some ranges of motion, the trace jumps wildly or disappears while some ranges seem to work okay. The first diagnosis was a bad control potentiometer, which I treated with contact cleaner. Unfortunately, that made no difference to the faulty behavior.

The potentiometer is hooked across +6V and -6V supply connections, with the wiper feeding into the horizontal defection circuits. This pot can be removed from the circuit and tested by pulling connectors from the main PCB since the pot is fixed to the faceplate and connected by wires.

One complication was that the +6 and -6 wires on this pot were paralleled to the outside lugs of the vertical position potentiometer, thus pulling the wires didn't fully uncouple the pot to be tested. After ensuring that the wipers of both position controls were unhooked from the PCB, as were the +6 and -6 feeds, I could attempt some testing using a VOM.

The two paralleled controls read appropriately, a 22K and a 500K pot yield a combined resistor a bit over 21K. That checked out fine, so I moved on to test the horizontal control wiper individually against the +6 and the -6 ends. I should see a smooth growth in resistance with movement, in reversed directions depending on the end I connected. In fact, I observed exactly that behavior. Observing the vertical pot produced odd results, until I realized that with the 22K resistor across the ends of the 500K device, the wiper would move up and down with a peak at the center, approximately 125K ohms and lower on either end.

The correct operation of the pot told me that I have a more serious flaw in the unit, not caused by the control itself but by the components on the PCB. Drat. This would take more bench time examining the behavior of the horizontal circuit. I quickly discovered that fuse FS1 (250ma fast) was blown. I ordered a new one to replace it as it may have been blown by a test on a bad transistor, but I was be prepared to discover some defect on the board itself that is causing the fuse to blow.

REPAIRING THE SELECTOR SWITCH

I found a suitable replacement DPDT toggle switch and substituted it for the broken component. I verified proper operation and correct wiring.

REPAIRING THE ERRATIC HORIZONTAL POSITION OPERATION

I have to wait for the replacement fuse before I can do more diagnostics on the positioning behavior. More on this in part II.

REPLACING THE SELECTOR KNOB

My first approach was to look for another of the knobs from a junker Telequipment machine - while I could use the acquired knob in the short term, the better approach is to use the knob to make a mold where I can cast replacements of more permanent materials.

This knob was used on two different Telequipment models - the CT-71 curve tracer that I am repairing, and the D75 Oscilloscope. I saw two model D75 on eBay but pricing has been prohibitive. One has a Buy It Now price of about $150 and the other soared to $80 (plus shipping) in a bidding war. I will keep watching but this unit is not worth my spending large amounts of money. All told I would feel remiss spending more than about $50 for this repair.

The small concentric pointer knob is one of two electrical controls operated by this assembly, the other is controlled by the outer clear knob itself. The rotating inner scale is driven by the lever which has its own (third) electrical control. I can operate the small pointer and the inner scale lever just fine, but need a method to rotate the outer knob controls while looking for a good replacement.

The two electrical controls coupled to the outer knob and small pointer are interlocked such that they can only form certain combinations of settings. The inner scale simply displays the value associated with the small pointer setting, as this changes based on the lever settings.

Position of rotating scale with right lever at .1 W setting

Position of rotating scale with right lever set at 10 W
Interlocking controls behind the complex knob assembly
The shaft for the control that is turned by the missing outer knob is set back inside the inner scale. The knob had a metal insert with setscrews that would grip the shaft. This constrains the type of temporary knob I might be able to affix to the shaft.

Since the rotating scale has no actual electrical role, I decided to remove it from the control assembly. This will allow me to mount a set of concentric knobs to rotate the two electrical controls. The outer control sets the volts per division of the horizontal sweep while the inner control chooses the collector load resistor size. The side lever which rotates the scale also picks one of four wattage peaks - 0.1, 0.5, 2 and 10.

As the volts per division goes up, so does the collector voltage. Power is the square of the voltage divided by the load resistor, thus one thing the lever does is set the smallest load resistor value possible to implement the wattage limit.

For example, with a 2W limit, the highest setting of 100V/div applies 400V peak to the transistor under test. The load resistor needed to hit 2W at that peak is 80K ohms. Selecting 0.1V/div or 4V peak and a setting of 2W, a 2 ohm resistor is the low value. The absolute smallest resistor, for 10W and 0.1 V/div is .016 ohm and the very largest resistor needed is for 100V/div and 0.1W would be 1.6 Megohm.

The volts/division and resistor controls have interlocks that limit the maximum spread - higher voltages enforce higher minimum resistances, but the inner control can select even higher values to drop the power below the max selected by the side lever. The lever itself shifts the range of resistors being selected by the inner knob. In a sense the interaction of these three cause the inner knob to select 10 linear steps from least to max power.

Based on this, I determined that the rotating scale is not really necessary to my operation. I can use pictures of the position of the scale for each of the four sliding lever power limits and refer to that if I need to know the load resistance being selected by my inner knob position. In most cases, if the power limit on the sliding lever is safe, I can dial any of the 10 power percentages. Occasionally I will want to limit it partway - a 5W transistor with the power lever at 10W would be safe only with the first five (highest) resistances.

I found a knob that I could place on the outer shaft to indicate the V/div setting. The inner knob was installed to show, by the angular position, which of the power values was selected. My four pictures of the scale, one per power limit, let me correlate that with the resistance value.

REPAIRING THE LOOSE SCREEN FILTER

After examination of the design of all the parts, I could see that this had simply been misinstalled. I rotated it to the proper orientation and put together the stack of filter, gratule and cover on the machine. This is repaired.

REPLACING THE TRANSISTOR SOCKET BLOCK

Initially I hooked up the transistors to be tested using jumpers. I saw an eBay auction of the plate I need, but the opening price including shipping is fairly high already. As well, the sockets provided fit only a few types of transistors making this of limited utility. If I were a collector of Telequipment testers and wanted to restore this, the original block would make sense, but to use this as a bench tool I thought of a better approach.

I whipped up a design for a small project box that would house several useful transistor sockets plus other means of connection. I will build this and hook it to the CT-71 to perform transistor and diode testing. I have a number of sockets and pins on order to make the adapter box. More on this in part II.

Thursday, August 1, 2019

Will perform lunar landings and show some of the AGC support gear at Vintage Computer Festival West

We will be at the Vintage Computer Festival West this weekend at the Computer History Museum in Mountain View, California. August 3 and 4.

Mike will hook up his FPGA gate by gate exact replica of the Apollo Guidance Computer and fly some lunar landings to demonstrate the central role the AGC plays in operating the LM. We will have the DSKY replica and LEGO LM model operating in demo mode, not hooked to an AGC. Also on hand are other devices we created to do the restoration and make use of the computer.

If you are in the area, stop by and say hello. Mike Stewart and I will be there both days, Ken will join us on Sunday and we expect Marc to stop in during the festival.

Tuesday, July 30, 2019

Apollo Guidance Computer ancillary gear - testing, testing and more testing

RAPID UNSCHEDULED DISASSEMBLY OF LEGO LM MODEL (AND REPAIR)

The LEGO model of the LM that I had outfitted with LEDs, to display when the Apollo Guidance Computer was commanding RCS thruster fire or controlling thrust on the descent engine, was in my carryon bag along with the DSKY replica for the recent tour of the east coast demonstrating lunar landings.

As we returned the AGC to its home in Houston, my bag toppled out of the car and smashed into the ground. The LEGO model inside had undergone a rapid unscheduled disassembly into a myriad of pieces and fragments - a pile of LEGO components and not an LM any more. I knew from the sound of hundreds of shifting plastic parts even before I opened the bag to check.

Reassembly was slightly harder because I had used superglue on a few points to keep the model suitably robust for display and rotation on the stand as the LM simulation flew the descent and then pitched forward for the final stages of the landing. Of course, these joints held but the parts they were connected to in turn disassembled, in some cases in a way that blocked straightforward reassembly.

I persevered, gently opening the glued connections and rebuilding until I had the model back together. Next up was a test of the lighting and wire integrity, which was easy because the Arduino controller has a built in self-test at powerup.  The results were mixed and upon inspection I even saw one wire ripped off an LED. With it repaired, I had fifteen of the sixteen thrusters working - the quadrant I up thruster had failed previously and remains offline.

I built in a demo mode in the controller so that I can display the LEGO LM with its thrusters and DPS engine firing. I can easily disable it with a quick firmware update for the next time this will be hooked to an AGC flying a mission.

TESTING DSKY REPLICA AFTER ITS FALL TO EARTH

Since the DSKY replica was in the same bag that fell to the ground in the airport parking lot, I had to verify its correct operation. Visually it appeared intact. I added power and connected +14V to the line that informs the DSKY that the AGC is powered and not in standby mode. The EL lines and legends on the right side panel lit as expected. I then tested a few of the left side warning lights, which properly lit based on the input wire that comes from the AGC.

More complex testing requires that I control 15 signal wires simultaneously as the digits on the display are set by the AGC emitting specific word values on the output channel. To do this and to ease checkout for any future trips, I decided to build a permanent DSKY test fixture. This used 24 relays to switch the 28V signal lines that come from the AGC, plus some resistor voltage dividers to reduce the 8 outputs from the DSKY to 5V from the 28V signalling levels used with the AGC.

An Arduino Mega 2560 was wired up to the relays and those were hooked into the main patch panel that connected the DSKY the the AGC. This involved soldering about 35 wires to a new connector on the patch panel, adding a few new D type circuits on my interface board to use temporarily for the testing and then wiring all the relays and resistors to the Arduino.

After a bit of confusion over the proper direct port access method to flip on a few of the relays, I got the code right and began to test the DSKY. Next up I saw that my relays weren't switching, even though the light was going on that should indicate actuation.

At this point I remembered that the relay board I was using was a 12V part, not a 5V part. Arggh.Enough to light the select light but not enough current to pull in the relay arm. I dropped my BPLUSSW (14V) supply down to 12, which was still high enough to take the DSKY out of standby, but could now do double duty powering my relay coils.

I did some partial tests before all the wiring was in place and was satisfied that the DSKY substitute is still working properly. Now to complete the final wiring. The DSKY will now have an HD50 connector for insertion into a special socket on the right hand patch panel, wired to the proper A51 pins.

The DSKY tester will get the banana plugs, allowing me to wire it into the patch panel to test the DSKY but otherwise leave it clean and empty. My cable had arrived so the wiring party began. Thirty five wires added between patch panel plugs and the new HD50 connector.

Then disassemble the DSKY cable to remove the banana plug wires and add the new HD50 cable. A bit tedious as I have to pick each wire in the cable and beep out up to fifty destinations on the patch panel. Once done, the DSKY is a quick plug into the panel for any future demonstrations of the AGC.

My tester had to be wired to each of the banana plug wires I removed from the DSKY cable. The tester will be plugged into the patch panel, while the DSKY has its HD50 cable connected, to accomplish testing. Many, many hours of sore back and tired eye later, it was done and ready for a test.

I successfully tested the seven discrete warning lights and whether the DSKY switches to and from standby mode based on 14V. Next was testing of the relay commands that light the remaining seven warning lights, Comp Acty light and display digits and signs.  That all worked perfectly as well.

With the tester working properly, I can use it as a demonstration driver to light the DSKY up with simulated output, in addition to its role in validating that our DSKY works. For this purpose, I worked up a demo mode that produces realistic looking displays and behaviors. I will use this at Vintage Computer Festival West this weekend.

The DSKY shows a request to run Program 63 (braking phase of lunar landing), brings up a mission elapsed time monitor, then requests some landing data via noun 68. At this time, Apollo 11 experienced a Program alarm and so do we. I have the PROG light illuminate, show the verb and noun to display alarm codes, and put up the 1202. The AGC goes back to displaying the landing data and then the entire demo loops to run again.

INITIAL TESTING OF AGC DOWNLINK DEVICE

The telemetry downlink from Apollo was driven by a sequencer that sent data from throughout the spacecraft, word by word, including some words that would come from the software running in the Apollo Guidance Computer. It requested the AGC data by a simple serial protocol, with the external sequencer as the master and the AGC a slave device.

I developed an Arduino Mega based machine that would produce, I hoped, the proper sequence of control and timing pulses to cause the AGC to shift out the bits of the downlink words. There is a Start bit, a series of forty Sync bits and a terminal Stop bit for each of the 50 words sent to the downlink each second.

My box should be producing the pattern of Start, Sync and Stop bits, at the correct timing and rates. While I need an AGC running its software to see the data coming out, I could verify the pulses I am sending as the master in the serial protocol, using an oscilloscope.

This is a bit more complicated than it sounds, because the hardware counter/timers in the Arduino are driven by inputs from the AGC of its 1 MHz master clock and won't work without those pulses. I don't have a handy AGC. The solution will be a function generator set up to produce nice 1MHz square waves that will drive the Arduino timers.

First up was watching for the Start pulse followed by a proper train of forty Sync Pulses. The scope triggered on the Start pulse, while I worked the controls to count and time all the pulses. I found and corrected a few flaws in my logic until I was satisfied that I was seeing the trains of Start and Sync pulse.
Start pulse on bottom followed by 40 Data Sync pulses above
These looked great but too short in both duration and spacing. I was expecting to produce pulses of about 4 us duration with an overall spacing of 20 us between pulses, but instead was seeing 2.5us pulses every 12.5us.

I varied all the counter values that should be controlling the AVR processor chip counter/timer, but it had no effect on the pulse widths. This was quite confusing, so I headed down two paths. In addition to reading the detailed ATMEL documentation, I experimented with different counter values.

Once I realized I had to clear existing interrupt requests before turning on the counter with it set to 0, I was able to control both the duration of pulses and the spacing between them. The granularity wasn't perfect, but I was able to tweak the trigger values to give me a 4.5 us pulse width and a cell that was essentially exactly on target at 20us.

Second was a confirmation that the Stop pulse follows the forty Sync pulses correctly. Triggering on the Stop, I backed up to verify that the last of the Sync pulses had been emitted. Third was a check that the next Start pulse follows a Stop pulse by 1/50th of a second, which my scope reported as frequency when I triggered on either Start or Stop. I chose a repeat rate of about 46 Hz, a bit slower than in real life but well within spec for telemetry.

End of forty Sync pulses above, followed by Stop pulse below
I then wired the Sync pulse output to my interface circuits board, into a Y type circuit that would produce a differential pulse output. That was patched over to an XT type circuit that would receive the pulse and produce a TTL level single wire pulse. I used the scope to look at the Sync pulses as received by the AGC, using the output of the XT circuit I just wired.

Output of Y (AGC pulse input) circuit drive by the Data Sync pulse

Back side of interface board while testing XT and Y circuits
Output of XT (AGC pulse out) going into Arduino
The pulses are as I expected, with transits that could go below zero because the diode in place has too slow a recovery time. Soon I will replace those diodes with 8ns recovery time diodes which will make the signals very safe for 5V inputs. For the time being, however, I could test my logic this way.

The AGC sends pulses on the Data bit 0 line to shift out a zero value. Absence of a pulse means that data bit is a one. In the test setup so far, I should be seeing all ones in the captured downlink since I was not injecting any Data bit 0 pulses. If I routed my Sync bit through Y and XT circuits back to the Data Bit 0 input, then I would see a pulse soon after every sync bit which should give me a downlink word of all zeroes.

Data received with no pulses arriving from AGC
Data captured when Sync pulses are fed back as data inputs
Indeed, I do get a zero now that I am hooked to the data sync pulses, but ones when no pulse arrives. This is what I expected. Until I have a more sophisticated test setup that can send a string of mixed one and zero pulses, this is as much as I can validate.

At this point I believe the downlink box to be working properly, although it will have to be hooked to the AGC to fully verify its operation. Before that happens I should learn what words are sent over the downlink, allowing me to format what is emitted into data more like the controllers in the Mission Operations Control Room in Houston would have seen.

I can see that this is already known, based on work by the virtualAGC project, so I only need to use that as a reference source to complete some mission control screens. This will be the next live display hooked to the AGC, for whenever we next have access to the machine.

Thursday, July 25, 2019

Taking the AGC on a tour and demonstrating Apollo 11 landings

TRAVEL CHALLENGES

I flow into Houston on Monday the 15th to meet up with the Apollo Guidance Computer and its owners, Jimmie and June. Early the next morning we would fly to Florida and visit Eldon Hall, the chief designer of the AGC, to demonstrate his 'child' operating once again.

The AGC is large and heavy, but old and fragile. We transported it in a large case with foam padding, placing it into a seat next to me for each flight. This is possible if you buy a seat for the item, it weights less than 165 pounds and can be strapped into the seat securely without blocking other passengers from seeing the overhead signs or exiting the aircraft.

Due to the size, it had a business/first class seat to be sure it could be fit into the seat and fastened in place using a seat belt extender. It was in a window seat and I was traveling next to it. Two big challenges existed - getting it through the army of bridge trolls who would insist it has to be checked as baggage, and lifting a 95 pound unwieldy case up over one seat to place it into the window seat.

On each of the three flights involved, I faced a minimum of five challenges and a peak of seven on the last flight. The ticket was made with a name of Incabin Baggage Claunch and the reservation checked carefully with the airline. Checking in to get boarding passes was no problem, but things went downhill fast.

The guards hired to police the lines into TSA for security checks are focused on making passengers check large suitcases. They enforce size rules and are trained for instant and aggressive action. Usually gentle but persistent discussions, flashing of the boarding pass and unwavering commitment to proceed forward would win out.

In Boston, however, we met a gatekeeper who went ballastic and was dialing for security, refusing to listen to me. She began to berate the porter who was wheeling Jimmie in his wheelchair, until he was able to penetrate her fixation and bring her to listen for a minute. Once she truly comprehended that it had a seat, she let us pass.

TSA was a challenge as well. For two of the three flights, the case with the AGC would fit through the X-Ray scanner throat with about 3/8" of clearance, a tight fit but able to move through. In Boston I had to help move it forward on the exit belt where it was jammed. On the flight from Tampa, the X-Ray machine was smaller and the case could not fit at all. It had to be hand examined there.

The case had wheels that were balky and induced major oscillations as I tried to wheel it, a kind of POGO problem. On the bottom, Jimmie had fastened small casters that would move adequately on stone flooring but drag unmercifully on carpet. They also make an ungodly din, sounding like they were scratching up the floor below; Fortunately it was all bark and no bite.

Airports with an airside tram to the gates have their own gate trolls, fighting viciously to keep the case from traveling on the train. Gate agents needed to be carefully approached and shown the ticket before they understood that the case would be pre-boarded and placed in a seat.

I was able, with some help from Mike or June, depending on the flight, to wrestle the case up into its seat and strap it down. In most cases we were safely past the trolls, but on the last flight one of the flight attendants decided that the bag had to be up front next to a bulkhead, not in the last seat as required by the airline and as it flew on the prior flights.

Yet another troll popping up just when I had it strapped in! I was fine if he wanted to force the people in row 1 to move back to 4 so we could put the case there, but his proposed solution was to strap it into row 7 in economy. I reminded him that I had paid for a first class ticket for the bag. While he was focused on that, his partner went back, saw that the case was securely strapped in, and permitted me to sit down next to it for the rest of the flight.

I don't want to give the impression that all these people were uniformly ill-intentioned or obstructionist. The flight crew on the trip to Tampa were delighted to know what they were transporting. The captain came down to see it and admitted to being a big space fan who happened to live near and know Gene Kranz. We took pictures and they were very helpful.

The other crews offered to help me push it up the ramp when deplaning, or to wheel my carryon bag. Even the ones that initially tried to relocate the bag or get it checked softened and were very friendly and helpful by the end of the flight. Too, TSA agents were willing to work with me, helped to lift the case up and down from the luggage cart to their examination table, and even let me take a luggage trolley into the secure area to wheel down to our gate.

THE AGC PERFORMS FOR ITS FATHER

Eldon Hall, who was the lead designer of the AGC at MIT, is retired and living in Naples Florida. Jimmie had promised to show him the AGC working again, which meant a trip to Naples and setting up in Eldon's condo. The place was packed with family and space fans looking to meet Eldon and see the AGC work.

It took us two hours to assemble everything, cable it up and test it out, but finally we were ready to provide a demonstration. We had to move furniture and place his large TV near the table with the AGC in order for Mike to see the screen for the landing in addition to the audience. There was a minor issue with the LEGO LM model such that it couldn't be hooked up in time to show the thruster and engine fire, but the DSKY and everything else was working fine.

Mike flew the mission from about four minutes before Powered Descent Initiation (PDI) where he began the landing from the 50,000 foot orbit altitude by firing the descent stage engine. Mike took us through the process, from Program 63 which braked and lowered our altitude through the approach phase of Program 64 and then the semi-manual landing under Program 66 where he flew past some boulders and craters to reach a desirable spot to land. The landing was a success, to the delight of all.

Eldon had a souvenir core rope module which he allowed us to read and archive using the AGC. Mike has shared this code with the world and begun working on understanding how it relates to earlier and later versions of Apollo software.

CRADLE OF AVIATION MUSEUM IN LONG ISLAND, NY

We flew up to the NYC area to exhibit at the Cradle of Aviation museum in Garden City, near where Grumman built the Lunar Modules. We had the chance to view a nearly completed LM (LM-13) which would have flown to the moon had not NASA cancelled Apollo 18, 19 and 20 missions. Too, we could see an LM simulator and plenty of Apollo related equipment.

We set up to begin demonstrations at noon, attempting a landing on the moon using the software from the Apollo 11 LM Eagle running on our AGC. It was crowded and hectic, but we had the chance to meet quite a few fans of our restoration, who had read from our blogs and YouTube videos that we would be available today for public viewing.

We didn't have microphones and the space was kind of cramped, set up next to the LM Simulator, but we did our best to run many landings, answer questions and interact with the visitors during the day. Due to a private evening event at the museum, we had to tear down and pack up before five.

We did get the chance to visit the museum archives and see the documents given to them by Grumman - a treasure trove. We hope to be able to digitize some of the more important content in the coming months. One document we spotted would be a great aid in building our own LM simulator. It contained all the data and math equations used to build the simulation. That document was perhaps 5 inches thick.

DRIVING UP TO CAMBRIDGE AND ARCHIVING MORE CORE ROPE MODULES

The following morning, July 19, we drove up to Cambridge Massachusetts to the MIT Museum where we had a chance to examine part of an AGC that they have on display. It contained some core rope modules of the Sundial program, which is the test suite for the Command Module which is analogous to Aurora which tests the LM.

Those modules were read and archived, giving Mike even more disassembly and analysis to distract him from sleep over the coming days. We then set up our equipment that evening for the following days events. Given that setup takes about two hours, it was very helpful to complete it tonight.

It also allowed us to get everything cabled and working, including the LEGO LM model. This model will light LEDs on the RCS quadrants of the model as the AGC commands that the thruster fires. It also lights the descent engine bell and modulates the brightness as the engine is varied from 10% up to nearly 100% thrust by the computer.

DEMONSTRATIONS DURING MIT MUSEUM LUNAR DAY EVENT AND MOON SHOTS!

We had large crowds, with many lined up outside the room after people had filled in aisles and were sitting on the floor in front of the first row. As a result, we ran as many landing demonstrations as we could. While we didn't keep exact count, I believe it was around 11 during the day.

In a couple of them, things went awry but Mike was able to safely achieve a landing even with some challenging conditions. A consequence of how Mike is delivering the landing radar data from the Orbiter/NASSP simulator into the AGC counter memory locations is that the timing between simulator and AGC can misalign.

Since radar data is altitudes and velocities in three dimensions, sequentially entered into the counter, if the timing is off the computer sees a velocity as an altitude, or vice versa, which throws off its computed position and velocity for the LM. In one case, the computer believed we were 1000 feet higher than we really were.

Mike saw based on the shadow, dust clouds and other features that he was much lower than reported, so he took control by moving to Program 66 where he could ask the AGC to control rate of descent while he controlled horizontal velocity. He hovered, nulled out the horizontal velocity over a likely point and gently lowered us to the surface. In that case we were down to 2% propellant left, tight but still safe.

Many more fans of the restoration and the AGC had heard about this visit from our blogs and YouTube videos, as well as the advertising by the MIT Museum. It was a delight to meet with them and talk even though time was tight as we cycled through landing after landing.

Don Eyles sat in the first row and watched one of the landings, quite a distraction to us. He then graciously offered us access to two of his Core Rope modules in order to archive the data on them. After the end of the daytime museum event, there was a break before the adult-only Moon Shots! party began. We used that to read the ropes (and to eat and rest). The evening event allowed us to run through a couple more landings and interact with the visitors before our day drew to a close.

ATTEND 50TH ANNIVERSARY APOLLONAUT BRUNCH AND DEMONSTRATION

On our final day, July 21st, we had the great privilege of being invited to the 50th anniversary brunch for the Apollonauts, the MIT staff who designed, built and programmed the AGC and related gear. There were over 100 people there who had worked at the Instrumentation Labs, later Draper Labs, including children and grandchildren.

We did perform one landing for the crowd, after a hurried assembly of our equipment amid the bustle of a large brunch meeting in a big tent at the Newton Marriott. We didn't bother setting up the LEGO LM as it would be too hard to see across the large tent. Something was wrong and the DSKY wasn't working either, but again only the closer tables would have been able to see it anyhow.

We did ask that any of the group who had Core Rope modules consider loaning them to us for archiving and we did get a number of commitments that will keep us busy studying the evolution of Apollo software for quite some time.

AGC GOES HOME

That evening, I hauled the AGC for its final flight back to Houston. It went home with Jimmie and June while I slept at an airport hotel for my final flight back home to California. It felt odd to board an aircraft without the AGC although the process of check-in, security and boarding went much, much smoother.


Prepared to process telemetry downlink data from the Apollo Guidance Computer running Luminary 99

DOWNLINK OVERVIEW

Apollo spacecraft stream telemetry information down to the ground as part of the Unified S-band radio link. Data is collected from sensors throughout the spacecraft, covering items like propellent quantity, EKG readings, and the state of various switches. A part of the data to be sent down comes from the Apollo Guidance Computer.

The circuitry that is sequencing the different data onto the downlink will poll the AGC for its data when the proper slot in the stream is reached. It does this by triggering a downlink start pulse, sending forty data sync pulses and then emitting a downlink stop pulse. The rate of polling is fifty times per second, with each set of data from the AGC taking a bit over 800 microseconds.

The AGC produces a master clock signal at 1MHz which is used to synchronize other spacecraft systems including the downlink controller which does the polling. The controller sends the pulses roughly once per 20 us with each pulse lasting about 4us.

After each downlink data sync pulse is received, the AGC either sends out a pulse to indicate a zero bit value or does nothing to indicate a one value. This pulse occurs a bit later than the data sync pulse but well within the 20 us alloted to each pulse.

APPROACH USING ARDUINO

This would be trivial to do in hardware, such as an FPGA or even discrete TTL logic, but I am already traveling for the demonstration tour and only have a spare Arduino Mega in my luggage. I therefore have to find a way to produce pulses of 4us, timed around 20us apart, with relationship to the 1MHz master clock timing

The approaches I used exploit parts of the ATMEL processor that are not normally accessed by Arduino programmers using the development environment, but this gives me the control over timing and low overhead that is essential to have the controller keep up with the demands. As a 16MHz system and with an interrupt typically taking about 16 instructions to process, the 1MHZ master clock from the AGC would keep the Arduino pegged at 100% without doing anything useful for me.

The overhead of the usual Arduino method of timing in microsecond ranges, the micros() call, has a resolution of 8 us which is too crude for our purposes. Too, the Arduino sets up a timer to keep track of micros and milliseconds - these take cycles and can shift the accuracy of outputs if you use these for timing.

The chip on the Mega 2560 has six timer/counters built in, with two of them able to be stepped by external pulses rather than the internal 16MHZ clock. I will wire up the AGC master clock to one of them (timer 5) and it will step at 1MHz rate and synchronized to the AGC.

Outputs that I emit based on the interrupts will occur about 1 us behind the actual clock. Since it takes a few instructions to set the output state, I will be lagging the timer change by something on the order of half a microsecond. To compensate for this, I will have the timer step based on the falling edge of the AGC master clock, which puts my delayed output very near the rising edge of the next pulse.

I chose three output pins that are on the same ATMEL port (PORTA) so that I can pass in a bit mask to the interrupt routine which will just OR or AND it with the PORT to flip the pin states. I can choose externally which of the three pulse types we are producing at any time. This keeps the instruction count in the interrupt routine down on the order of half a microsecond duration.

To detect whether the AGC has sent in a pulse, indicating 0 value, or did not, for a 1 value, I hooked that signal up to pin 2 which is wired to an interrupt routine that will fire on a rising edge. Thus, during the time that we are asking for a data bit (one of the forty data sync intervals of 20 us), I will latch in a flag if the pulse arrives from the AGC. The main routine, at the end of the 20 us period, looks at the latched flag to tell if the bit returned was 0 or 1.

A simple integer based state machine will move from idle through the 42 pulses to be sent. Its states are idle, start, sync1, sync2 . . . sync40, and stop. I set up the bit masks to use with PORTA based on whether I am in start, stop, or one of the sync states. At idle, I disable the timer so it doesn't interrupt until we begin the next round of forty bits for the downlink.

It is a simple matter to save the forty bits as they are detected and then write them out on a serial link over USB during the 20 milliseconds we wait before starting the state machine again. This gives us the 50 polls per second rate that the downlink controller would request.

Using external clocks, timer routines, rising edge interrupts and tight coding with direct port access is the key to making this task feasible on an Arduino. I finished the coding over the first couple of days of the trip and hoped to plug it into the AGC to begin testing.

Unfortunately, the pace was more hectic than we had imagined. It took almost two hours to set up for a demonstration, with assembly of panels, cabling and checkout. The demonstrations themselves were public and demanded all our attention. Teardown and packing began immediately after the demonstration time ended.

I never did get to cable the downlink system into the panel while we were running the AGC software. It will have to wait for a future session where we once again are running the computer.

Sunday, July 14, 2019

Final preparations for 50th anniversary demonstration tour with AGC

FINISHING THE LEGO LM MODEL AND STAND

I glued together an acrylic stand that would allow the LM to sit up in the air about a foot. It has a pivot to set the LM with the descent engine facing forward during the initial part of our landing demonstration and then to pivot to place the engine bell downward for the last portion.

It required a bit more bracing than I expected but a quick trip to Tap Plastic got me the extra parts I needed. It can be disassembled and carried in my suitcase for setup at each demonstration event.

I needed a small enclosure to hold the resistor boards, controller and wiring out of site. I rummaged through my parts bins and found something useful, even if not visually appealing.

BUILDING STAND FOR PATCH PANELS AND PLUGBOARDS

The stand to hold the electronics panels up is more challenging due to the combined weight. I stated with two vertical strips of 1/4" acrylic but, just as with the LM stand, I had to add bracing before it was ready for use on the trip.

Starting from the bottom, we have the right hand patch panel placed. This panel will have the 35 banana plug leads from the DSKY replica pulling forward, which is why I chose it for the bottom spot. Next is the main interfaces board which will convert spacecraft signals to modern TTL levels.

Above that is the left hand patch panel. I placed the interfaces board between them to shorten the runs of banana plug cables that splay from the interfaces to the two patch panel boards. Finally, the top position is held by the secondary interface board which provides the last five AGC C level output signals needed for my LM model lighting. These needed aluminum holders to connect them to the acrylic vertical rails.

I picked up some better bolts and nuts for assembling everything and stowed them in my luggage with the stand pieces. I had to reinforce the acrylic with thicker rectangular braces just to keep every thing standing vertically. It was quite solid and dependable once completed.

Our 'rack' of patch panel and interface boards

TESTING THE LEGO LM MODEL LIGHTING

I first set up a temporary version of the controller firmware without the input pullup resistors, thus making each input stay at ground unless I explicitly inject a high level. This let me test the key inputs - the 16 RCS jet signals that cause LEDs to light on the model's RCS quadrants. Along with that, I tried out the engine on and engine off commands, viewing the descent engine bell light at minimum thrust brightness and watching it go out again.

To perform a more comprehensive test, and to help with checkout, I set up a self-test capability at power up of the controller board. It will light each logical group of RCS thrusters in turn; up, down, forward/back and side.

It then will turn on the engine, throttle it up to full thrust, take it back to half thrust, and turn it off. This allows me to check quite a bit without needing a tester to drive the controller.

An external tester (another Arduino) produced pulse patterns to further test the throttle control portion of this model. I had it all wired together ready for testing Sunday morning. By afternoon I want to start packing everything for my departure Monday morning.

MINIMIZING HEAVY POWER SUPPLIES IN MY LUGGAGE

Our lab and my home testbench makes use of triple section power supplies as well as more than one supply box.  That would be painful to have to haul around on the demonstration tour next week. Mike is bringing the big 28V supply that will feed the AGC and supply my interface circuits.

I need to bring a beefy 5V supply to power the DSKY and the controller for the LEGO LM model. There is another voltage necessary if we will be sending pulses into the AGC (the Y type circuits) - 14V - and I don't want to drag around a separate supply. Fortunately, the AGC delivers 14V on another pin of the patch panel.

Thursday, July 11, 2019

Programming Arduino to drive LEDs on LEGO LM model based on signals from AGC flying mission

PROGRAMMING THE RCS AND  DESCENT ENGINE CONTROLLER FOR THE LM MODEL

The Descent Propulsion System (DPS) is the engine for the LM landing, while the Reaction Control System (RCS) is a set of thrusters that control rotation and translation of the LM by emitting short bursts out of one or more of the 16 thrusters on the spacecraft.

The DPS is turned on and off by two signals from the AGC, Command Engine On and Command Engine Off. Its initial thrust level is 10% when it is turned on. This gives time for the AGC to move the engine on gimbals to reduce and pitch or roll movement due to off-axis thrust.

The computer then will throttle the engine up and down through its usable range of 10 to 65% or 100%. For technical reasons, it can't run long in the range between 65 and 100. This is implemented with a counter that is incremented or decremented by pulses from the AGC (Increase Thrust and Decrease Thrust signals).

When the counter is 0, the engine is at its minimum 10% thrust level and when the counter reaches 3200 the engine is at 100% thrust. That means that each increment or decrement changes thrust by .028125% but the LED I will be using does not have the granularity to show 3200 steps of brightness with an 8 bit value.

Instead, I will use 18 steps from 10% to 100% brightness. Taking the counter modulo 178 will give me a step number from 0 to 17 which selects from the table of 18 brightness values for the LED I am using.

The two thrust change signals are pulses that add or subtract one from the counter, but limiting it to values no lower than 0 and no higher than 3200. I set up interrupt routines triggered by the rising edge of those two pulses; these routines simply bump the counter up or down by 1.

The main routine monitors the counter modulo 178 and selects the appropriate brightness value accordingly, but only when the new step number has changed. The main routine also turns the engine completely off or switches it on based on the two engine command signals.

RCS jets will illuminate the LED that represents that thruster. A pulse on the RCS jets is a minimum of 13 milliseconds long, so I will set a start time when I illuminate the LED, and won't switch off the light if the control signal becomes LOW unless at least 13 ms has elapsed.

I coded this up and loaded an Arduino Mega 2560. I now plan to whip up a testing circuit using an Arduino Uno to generate various control signals, different durations, and try stepping the engine up and down.

INVESTIGATING ACCESSING THE DOWNLINK TELEMETRY FROM THE AGC

The Apollo Guidance Computer outputs some words of data on the downlink to Mission Control in Houston, which are joined to all the other sensor and biomedical data on the same link. The telemetry circuits in the Apollo spacecraft control the link and request data from the AGC at the appropriate time in the transmission. This data rides on the same microwave (S-band) signal that carries voice and television signals.

Telemetry runs at 51.2 Kbps, high rate, as long as the unified S band link is available. When the spacecraft uses the backup omnidirectional VHF links the data rate is slashed to 1.6Kbps, called low rate. The AGC operates at either rate, controlled by the telemetry system.

The telemetry system sends a pulse on the Start Downlink signal line, followed by a stream of 40 Downlink Sync pulses at the high or low data rate, terminating the session with a pulse on a separate Stop Downlink signal line. Each session, begun with a Start Downlink pulse, occurs at a rate of 50 times per second for high rate and 10 times per second for low rate.

When each Downlink Sync pulse arrives, the AGC sends out another bit by emitting a pulse on Downlink 0 Bit signal line if the data is zero or skipping a pulse if the data value is one. This was chosen so that the stream from an inactive AGC consists of all one bits.

I am thinking of developing a box that will sent the Start, Sync and Stop pulses to the AGC, capturing the bitstream it sends back. Time is extremely limited before I head out for the demonstration tour, but if it is easy to bang this together then I will try.

Improving the interface board and patch panels for demos, plus LM Model set up to display jet firings

NEW INTERFACES BOARD CHECKOUT

I designed a replacement board to house interface circuits that allow use of modern 5V TTL or 3.3V LVCMOS devices to drive or monitor the AGC signals. The spacecraft itself uses either 28V logic levels or short 10-14V pulses on wire pairs for signalling, neither of which are friendly to an Arduino or FPGA.

Each of these new boards is essentially a rack width (almost 19 inches) and houses one third of the total input-output circuits used by the AGC. I chose that fraction because the PCB fab I used had a minimum order of three boards for their fast turnaround prototype service. Thus one of the PCBs houses 19 type D, 13 type C, 11 type Y and 18 type XT circuits. D and Y circuits are inputs to the AGC of discrete and pulse type respectively. C and XT circuits are outputs from the AGC, discrete and pulse.
Apollo to modern logic interfaces board
I built the board and then subjected the circuits to testing to be sure they worked properly. Using signal generators, oscilloscopes and the like, I produced pulses with my type Y circuit and read them with a type XT on my board. In the same way, I toggled logic levels with my type D circuit which were read correctly on the type C. I used a signal generator first but later an Arduino produced both discrete and pulse test signals and detected the results from the receiving circuits

This board will be mounted just above one of the two rack wide patch panel boards we use to route all the AGC signals from the dense A51 connector to banana jack receptacles that are easily accessible. Banana plug cables hook any chosen circuit from the AGC to the interface board and from there to switches, Arduinos etc to work with the signals.

BUILDING VISUAL DISPLAY FOR DEMONSTRATION TOUR

I began the task of soldering small 30 gauge wire to tiny LEDs that I will place inside the nozzles of the Reaction Control System (RCS) quadrants on the Lego Lunar Lander kit. I had to figure out a way to route the wires that is not distracting to the viewer. The LEDs I will mount inside the Ascent and Descent engine bells are larger.

The Lego kit also arrived today. Only 1087 parts to assemble; somehow it took five hours to build. I skipped the moon surface portion as well as the astronaut figures, concentrating on the portion I will display, but that was a small fraction of the kit.

LEGO model to install LEDs in RCS and engine bells
I needed to work on an anchor point to hold and orient the LM. I can't use the top of the LM because the ascent stage pops off the descent half, the weight would be enough to have it fall aprt if supported only by the top. I needed  something that holds the descent stage and rotates, which will allow me to orient it engine bell forward for part of the descent and then engine bell down for the last portion of the landing.

The solution was some clear acrylic with a stand and pivot. A trip to Tap Plastics nearby will give me the materials I needed. While there I will buy materials for the rack stand to hold the two patch panels and my interface circuit boards. They have to disassemble to travel in my luggage for the demonstration trips, otherwise I would just build permanent stands or use a half height rack enclosure.

Installing the LEDs for the RCS engines was quite challenging. My first plan was to use 0603 surface mount LEDs with wires tacked onto them, super-glued into the model. The problem is the delicateness of the LED and the small size. I had wires break off ruining the LED as I tried to place them into a thruster cone.

Preparing 3mm LEDs

Inserting 30 ga wire through drilled holes in LEGO pieces

I then bought some 3mm thru hole LEDs, bright red when illuminated but transparent white otherwise,  and prepared them for mounting. Using my 30 gauge black wire to tack onto the base of the LED leads, I could cut the rest of the leg off and use heat shrink tubing to insulate. Two wires per LED until downstream where I can combine all the common wires.

I put a 1/8" plastic drill in a small press, placed the lego pieces in a soft vise and drilled out openings inside the four thruster cones. I also drilled out an opening near where the quadrant attaches to an arm on the LM body. This gave barely enough room to fit eight wires through the final hole.

Using my stereo microscope, various tweezers and lots of patience, I threaded the wires through the LEGO parts and pushed in the LEDs. After a bit I had two of the four quadrants done and found that it was not too frustrating as long as I took my time on each.

By Thursday morning I had all four quadrants, 16 thrusters in all, with LEDs wired out to a terminator breadboard which also housed the 16 resistors. The wires ran into the descent stage and out along the bottom, where the black wires blended pretty well against the black LEGO bricks.

Light in RCS quadrants in place, wiring exiting at bottom

It was a simpler job to wire up an RGB serial addressable LED into the Descent Engine Bell. That completed all the lighting I need to use the LM model to show in real-time the thruster and engine firings that occur during the lunar landing we will perform for our demonstrations.

PATCH PANEL COMPLETION

I wired up every signal from the AGC to its circuit pin on the patch panels. This gives us the flexibility to interact however we want, making use of the interface board to do so using regular TTL and LVCMOS devices and controller boards. There are more than 300 wires involved in completing this task, spread over two days to give my back a rest between sections.

Next up was building the stand for the two patch panels and the interface circuits board. The main principle was to use screws and nuts to assemble the stands but allow them to fit compactly in my luggage for transport. It will take a few days for Tap Plastic to cut all the pieces I need but by Saturday I should be assembling the stands.

PROGRAMMING ARDUINO TO LIGHT MODEL LEDS BASED ON ENGINE FIRINGS

I put an Arduino Mega 2560 into duty as a controller for my LEGO Lunar Module display. It hooks to 18 of my type C interfaces and 2 of my type XT circuits. Since my one interface board implements only 13 type C interfaces, I had to partially build a second board to provide the patch positions for the remaining 5 C interfaces.

That build was done quickly, although I was short three red banana plug receptacles which are needed for type C circuits. I had to make do with three yellow jacks which normally would be used for XT circuits.

From the Arduino viewpoint, a type C circuit coming from my interface PCB is simply an INPUT_PULLUP, an input with a weak pull-up resistor inside. The type XT circuits produce pulses that must be detected by the Arduino so I used interrupt service routines for those two pins triggered on rising edges.

Using the MIT documentation of the equations used in their hybrid digital/analog simulator of the LM, I collected the information I needed to properly simulate the throttle of the descent engine. Our AGC will increment or decrement a counter in the engine controller to vary thrust. That engine has a minimum thrust level of 10%, achieved even when the counter is at zero. The counter goes up to a maximum equivalent of 3200 which represents full thrust of the engine.

I also have the lag characteristics they used for the engine. It takes 2.3 seconds from when the Engine On command is received until it is producing the minimum 10% thrust, and when switched off it takes 380 ms. Every thrust change command requires about 300ms to take effect.

I am attempting to model the engine LED in the model based on the MIT defined characteristics. This adds some complexity to the Arduino program but the results should be worth it. More on this soon.

Friday, July 5, 2019

Final preparations for public display and demonstrations for the 50th anniversary of Apollo 11

Current Switch Module failed diodes

We will mill away all the rest of the potting on this module, drill out the 26 diodes that we haven't yet replaced and populate the module with suitable replacements. That will restore the erasable (traditional core RAM) memory to full operation.

Improving demonstration for spectators

While we can accomplish a lunar landing using just the AGC and DSKY, it requires the spectator to visualize quite a bit after they are taught to interpret various numbers displayed by the DSKY. We want to help visitors visualize what is happening so we worked on some improvements.

Mike worked on changes to his test monitor board, the one that plugs into the AGC test connector A52 and also runs with his FPGA replica of the AGC. He made corresponding changes to the NASSP plug-in for the Orbiter simulation, such that the simulation will use the real AGC (or FPGA AGC) rather than a virtual AGC inside the package.

Using this, we can interact with our AGC but see the results on the screen showing the inside of the Lunar Module and the view outside the windows. All the displays such as the altitude tape meter and FDAI flight director work properly. This gives a compelling simulation of the landing, with the engine thrust climbing, the view pivoting as the LM is maneuvered, and appropriate sounds emitting from the screen.

In addition to this, we will bring the LEGO Lunar Lander with LEDs placed in the RCS thruster jet openings and in the bells of the Ascent and Descent engines, driving them based on the pulses emitted by the software running in the real AGC during the mission. An Arduino interprets the pulses and controls the LEDs.

Organizing demonstration tour for 50th anniversary of Apollo 11

Now that the AGC is running well and we are prepared to demonstrate it supporting portions of the Apollo 11 mission, we have to set up public appearances for later this month. Having just completed booking the trip, I can share where we will be appearing.

At the start, we will bring the AGC to the home of the lead designer, Eldon Hall, and let him enjoy the sight of his masterpiece working again after all these years gone by. This will take place in Florida, after which we will fly up to the Long Island, NY area for the first public appearance.

We expect to give short talks and demonstrate landings at the Cradle of Aviation Museum, near LM 13 and CM 2 plus a LM Simulator and other Apollo era artifacts. This will occur on the 18th of July. We are waiting for the museum to define times more exactly.

The following day, July 19th will have us setting up in the MIT Museum in advance of our public appearance all day and evening of July 20th, the fiftieth anniversary of the landing on the moon. We will give talks and demonstrate portions of the flight including landings multiple times on the 20th. Once again, we don't have exact times yet.

That concludes the public demonstration portion of our trip, but we will have a chance to meet with many of the original programmers and designers from MIT and others who supported the success of the Apollo program.

Tuesday, June 25, 2019

Another diode bites the dust in the AGC Current Switch Module

Setback with erasable memory, failure in Current Switch module

Near the end of our time with the AGC, we were running various portions of the Apollo missions such as LM powered descent initiation, LM rendezvous calculation to rejoin CM, or CM burn calculation to hit the earth entry interface.

Suddenly, we were seeing hardware restarts and the software was not running correctly. Using Mike's test monitor we quickly realized that erasable memory was not working properly. We could rapidly trace this down to a failure in the Current Switch module (again).

Testing diodes after powering down the computer led us to a third diode that failed open, just as the first two had been. We didn't have enough time left to mill out new windows in the polyurethane foam encapsulation, drill out the bad diode and insert a replacement.

This module has 28 such diodes - three have already failed and NASA documents show that Raytheon switched from the original diodes to new types due to failures similar to what we experienced. The original diodes had a mesa construction, something the semiconductor industry quickly moved beyond.

We kept the Current Switch Module here so we could remove all the encapsulation on the module, then replace the 26 original diodes with modern 1N914B diodes like we did with the first two we found. This should be completed so that we can rejoin the module with the AGC for live demonstrations we will do in July.

Back to Houston until our next session

The AGC flew back to its home in Houston, until our next session and subsequent demonstrations.

Sunday, June 23, 2019

Final upgrade to our AGC and working on simulating missions such as lunar landing

Adding the PRO key support to this AGC

The original version of the block II computer had a key on the DSKY marked STBY that was used to put the computer into sleep mode (and bring it out again). Soon, it was replaced by the label PRO and was used for two purposes - standby same as before, and signaling PROceed when the astronaut was ready to approve some action proposed by the computer.

Prior to having a PRO key, the astronaut had to push VERB, push 3, push 3, and push ENTR to confirm a proposed action. When there are a sequence of requests from the program, such as during the start of a landing sequence, having to mash four keys to give assent took time. In some cases, the astronaut had just five seconds to approve the start of a burn; if the VERB 33 ENTR had a typo, it might be impossible to redo it fast enough.

The original STBY function is purely done in hardware. Once the software sets a bit that allows standby, the key itself was wired directly to circuitry that dropped the switchable 14V and 4V supplies, leaving only a base level of power for functions such as pulse generation needed by other hardware in the spacecraft.

The new second use of the key for approving action requires that the software be able to see when the key is pressed. This was accomplished by a small hardware change that routed the key signal from the standby circuitry to set a previously unused bit in one of the I/O channels. Our AGC, the last of the prototypes before the flight versions, did not have the PRO key wired to the channel yet.

We recreated the change, using wire wrap, such that when my DSKY has its PRO key pushed, it sets the bit in the channel to approve a program proposed action, but will still switch off the computer if held long enough to activate the standby circuit.

Challenges running the PDI program for the Apollo 11 landing

We have been able to run program 63, the way you accomplish Powered Descent Initiation (PDI) to lower the orbit of the LM and transition through to landing on the surface. However, once the program sets up for the burn of the descent engine and is given permission by the PRO key to fire, the lack of acceleration is detected. The software flashes Verb 97 which means thrust failure and we can't slow down or reach the surface.

Marc set up an accelerometer substitute using an ATMEL processor and three potentiometers, allowing us to indicate (crudely) that acceleration is occurring on any or all of the three axes. However, the question of which direction the acceleration should be injected is not a simple one. In fact, the answer necessary to bring the LM to the ground with zero forward velocity is devilishly hard to calculate.

The inertial measurement unit of the LM (and CM) consists of a platform (called the stable member) which has three gyroscopes and three accelerometers (PIPAs) mounted so that X, Y and Z rotation and acceleration can be measured.

This stable member is suspended on three concentric gimbals so that regardless of the rotation or movement of the spacecraft, the stable member will remain locked in the same position relative to the milky way galaxy. It drifts slightly over time, which is why it is aligned using the star sighting telescope to return it to its ideal position.

Each gimbal is used to read off rotation of the spacecraft around the stable member in one axis. For technical reasons, having only three gimbals means that the inner and outer gimbals can move so they are parallel to each other. This is bad because a rotation can't be assigned to only one axis, it instead is seen as a movement on both axes simultaneously. This foils the ability to recognize movement properly and is called Gimbal Lock.

Certain rotations of the spacecraft can move it so that it approaches or reaches Gimbal Lock. Once in gimbal lock, the stable member must be reinitialized and star sightings are necessary to align it, a time consuming process. To avoid this, the people in Flight Planning (and in Mission Control) have to plan out the orientation of the stable member so that all the planned maneuvers of the spacecraft can be made without approaching gimbal lock.

Consider the way the spacecraft evens out heat and cold as it flies between earth and moon - the Passive Thermal Control mode - where the spacecraft has to spin on an axis like food on a barbecue spit, with the axis at right angles to the sun. Turning one rotation every two minutes means the stable member is rotating on its gimbals at the same rate and that motion can't be allowed to cause gimbal lock.

At many times during the mission, the orientation of the stable member relative to the galactic background is changed to put it in the best position for the maneuvers ahead. The orientation information is sent up as a REFSSMMAT, a reference stable member matrix either vocally or by the ground using 'remote control' to press DSKY keys to enter the new data in the AGC.

The flight director/attitude indicator display (the 8 ball) is tied to the stable member, thus it is showing where the stable member is sitting while the spacecraft rotates around it. That works out find when in a straight line trajectory such as during translunar coasting, but introduces problems when in orbit around the earth or the moon.

Since the stable member remains fixed relative to the background of the universe, the orbital path will cause the member to appear to rotate as the spacecraft circles the globe. If we start with the stable member parallel to the moon's surface at one point and leave the spacecraft in that same position, as we orbit our nose will appear to pitch up then nose over until we seem to be upside down relative to the moon when we are at the opposite point of the orbit, gradually rotating again until we are flat and parallel at the starting point of the orbit.

The flight director/attitude indicator will rotate around with the stable member (and spacecraft). If we cause the spacecraft to pitch at exactly the right rate, then the spacecraft 'floor' will always be pointing down towards the moon. Our stable member, as a consequence, will be rotating as we orbit and our 8-ball won't make sense.

A special device named ORDEAL will permute the 8-ball so that it seems to match the 'floor' of our spacecraft, and cause the correct pitch rate, so that the pilot does not see the stable member rotating. However, regardless of the tweaked display, the reality of an orbit means that the stable member is rotating while we circle the moon.

That means that the accelerometers idea of X, Y and Z is based on the stable member and are rotating compared to our spacecraft. During the landing, therefore, the direction of the thrust vector of the descent engine is rotating and has to be calculated based on the original setting of the REFSSMMAT, the place we are in the lunar orbit and the orientation of the LM compared to the moon's surface.

The landing of Apollo 11 begins with the crew oriented to look down at the moon's surface with the descent engine bell pointed in the direction of the orbit. Later in the descent, the LM pitches around so that the astronaut's feet are pointed down at the surface and they are looking forward in the direction they are moving. All these occur while the stable member stays fixed in inertial space.

As you can now see, the direction of acceleration we have to input to the AGC, which are given strictly to the X, Y and Z direction of the stable member, have to be manipulated based on all the factors above.

One simplification is that we don't care how the LM moves from face down to face forward since the stable member doesn't change at all. What does matter, however, is the effect of that movement on the thrust vector relative to the 1/6 G acceleration of the moon on the spacecraft. That must be factored in, both by the AGC in computing the burn and pitch up timing, but also in our injected acceleration.

We will need to issue AGC commands to look at the orientation of the stable member at the time we begin the engine burn, calculate how this rotates as we move around the orbital path/descent path, and apply acceleration appropriate to that.

So much more complicated that simply delivering input on the axis of the descent engine bell of the LM. We expect we will even see an alarm thrown after we think we have landed, unless we provide acceleration at 1/6 G in the proper direction to match where the moon surface is relative to the stable member.

Wednesday, June 19, 2019

Interacting with the running Apollo Guidance Computer using DSKY

Testing the DSKY substitute

We hooked up the DSKY substitute to the Apollo Guidance Computer and verified that all the lights, displays and keys worked properly. We now have a physical manifestation of the interface used by the astronauts to interact with the computer.

Video of DSKY working with AGC

Testing Ben Krasnow's electroluminescent panel

Ben (Applied Science YouTube channel) came over with the EL panel he built to NASA specs and we worked to interface it to my DSKY substitute. After a bit of work, I got it mirroring the display on my DSKY as the AGC ran various Apollo era software.

Video of Applied Science electroluminescent panel used with AGC

It was great to see a real electroluminescent display providing the output from the computer. Realism of the interaction took a step up.

Using the DSKY with the AGC

We ran several portions of the Apollo 11 LM mission, using the DSKY to interact with the Luminary 99 flight software that flew in the spacecraft. This included calculating the rendezvous burn after ascent from the lunar surface and of course the powered descent down to the surface. No 1202 alarms for us, because we didn't have a simulated Rendezvous Radar feeding pulses to overload the computer.

Monday, June 17, 2019

Erasable (core memory RAM) of Apollo Guidance Computer repaired and working properly

New flaw in Current Switch module

After having replaced the two failed diodes in this module, we began testing it with pulses to ensure that it worked properly at the currents and timing needed to drive core memory. This module consists of a number of large ferrite cores with four sets of windings on each. One winding is tied to all the cores and is used to switch them all back to the 'off' magnetic orientation. A second winding is used to select a core by switching it to the 'on' orientation.

Certain cores are selected by flipping them on. This flip of magnetic state induces a pulse in the third wiring, which causes a transistor to conduct to drive current through an X or Y addressing line in the B12 core memory module. Later, when all the cores are reset by the first winding, only the ones that had been selected will flip back. This induces a pulse in the fourth winding, turning on a transistor to drive current in the opposite direction through the X or Y addressing line.

This scheme is clever because the core retains the selected address from when it is selected. This selection induced the pulse to read out a word of the B12 module but also remembers which address lines were selected. The AGC logic does not have to retain the address in a register until later in the memory cycle when the word of erasable memory is rewritten - the selection cores themselves hold that information.

One one of the cores, we had a 4 ohm short in some module I/O pins that shouldn't be connected at all. After excavating the potting material around the suspicious area, we quickly excluded simple to repair causes such as shorting wires, shorted interconnect, etc. We found that the first and fourth windings were somehow shorting inside the core - these are the reset winding and the winding that produces a pulse when a previously selected core is reset.
Shorted windings in B11 Current Switch Module circuit
Careful examination showed us that it would be impossible to extract the core/windings from inside the cordwood module where they were epoxied in place. At least, impossible to remove it without severe damage to that core and its windings.

We have a suitable replacement core but putting four windings of 50, 32, 32 and 20 turns would be extremely challenging given the small diameter of the core. Fortunately, Marc and Mike figured out a clever hack to give us an equivalent functionality. It began by completely disconnecting the fourth winding, the one that switches on a transistor when a selected core is reset. That effectively cured the short since the shorted winding was no longer connected to anything.

Then, we installed a transistor of opposite polarity tied to the third winding, the one that is normally used only while the core is selected. When the selection pulse flips the core, the third winding sees a positive direction pulse, causing the attached NPN transistor to conduct. When the core is later reset, the pulse is in the negative direction thus the transistor doesn't turn on.

Yet, if we hook that third winding to a PNP transistor as well as to its NPN transistor, the pulses from the windings will cause one transistor to turn on with a positive going pulse, and the other transistor to turn on with the negative going pulse. This produces the same behavior as the original circuit, but requires only three windings instead of four. That is fortunate because we don't have four useful windings.

We inserted a small PNP transistor and did some rewiring, which gave us a module that passed all tests with flying colors. We were then ready to install the erasable memory driver modules, this current switch module, and our erasable memory (core stack) into the AGC.

PNP transistor wired to third winding of circuit
Archiving prior contents of the B12 erasable memory

We still have the flaw in the core memory stack - the inhibit wire for one of the data bits (bit 16) is an open circuit. In the read portion of a memory cycle, the selected X and Y address lines flip all cores of that word to zero. Any of the bits in that word which had a 1 stored in it will flip, causing a pulse to be detected by the sense amplifiers. This is how core memory does a read, by destructively flipping the word to zero.

During the reset pulse from the current switch module core circuits, the cores for the selected X and Y address lines, those whose current switch cores had been set on, will flip all the bits of the word to a 1 state. However, we don't want them to be 1, as some should be 0. That is the purpose of the inhibit wire - a signal on the inhibit wire will block the reset from flipping that particular bit to 1.

Reading data involves erasing it first, then rewriting the original value (or putting a new value in if this is a write operation). With a bad inhibit wire, bit 16 will be rewritten (or written) to 1 regardless of our desired value. Thus the loss of the inhibit wire renders that bit useless throughout all 2K words of erasable memory.

If the inhibit wire damaged happened after the computer was last powered down, then it hadn't yet forced a 1 into every bit 16 in the core stack. We had the opportunity to read the contents of the core memory correctly, since the read portion of a memory access doesn't use the inhibit wire. However, this is a one shot opportunity, as the process of reading any word will jam in the 1 in bit 16 during rewrite.

We powered up with memory access blocked through the test connector, put the machine in single instruction mode, and tested by reading a few memory addresses. We did find that bit 16 sometimes had the value 0, sometimes 1, verifying that the damage to the module was not present when it was being accessed 40-50 years ago.

We wrote down the values we saw in the test locations, then used Mike's test monitor to run through all of memory and retrieve the values. Folding in the few manual reads gave us a complete file of the prior contents of our memory module B12.

Mike studied the contents and could confirm that this was running a version of Aurora software, similar but not identical to the release for which there is an archived image. He could decode the contents of the DSKY display. It showed that the last command entered was a coarse alignment of the IMU (gyroscopes and accelerometers) to angles of 0, 0, and 90. The Gimbal Lock warning light was on, which may have been the event that caused the operator to perform a coarse alignment. Finally, the code stores a latitude when aligning the gyros - it was the location of the Johnson Spacecraft Center in Houston.

Rewiring to swap parity and data bits in B12 core memory module

Now that the prior contents are safely archived, we can implement our modification to make the core erasable memory functional again. Our great fortune is that the erasable memory module contains an error detection method called parity, to capture the cases where some radiation event might have flipped a data bit randomly.

This works by adding a sixteenth bit to each word - data bits are 0 to 14 plus 16, with bit 15 representing parity. The rule for parity is that the number of bits in a word which are 1 has to be an odd quantity. If not, the parity bit is set to 1 thus making the entire word have an odd count. if the data bits themselves have an odd count of 1s, the parity bit is set to 0.

At the end of a read from a word of core memory, the processor counts the 1 bits, determines if parity should have been 1 or 0 and then compares it to the parity bit read from memory. Mismatches raise an erasable memory parity alarm.

The erasable memory is quite reliable, especially since our AGC is not out in space subject to radiation events. We therefore don't really need parity checking. That gives us a working bit 15 which we can substitute for the broken data bit 16. As long as we can disable the parity checking, blocking the alarm, we will have a working memory.

Some backplane wiring was introduced to swap bits 15 and 16, as well as block the signal that performs a parity check. With the wirewrap changes completed, we closed up the AGC and tested. Success! The erasable memory is fully functional and we can run software fully out of the repaired erasable memory and the core rope simulator boxes providing the fixed (core rope) read only memory, just as the machine