Stumbled upon defect in the Raspberry Pi PICO library routines for SPI links

VIRTUAL 2315 CARTRIDGE FACILITY USES SPI LINK BETWEEN PICO AND FPGA

The heart of the Virtual 2315 Cartridge Facility (V2315CF) is a unit that houses a Lattice Field Programmable Gate Array (FPGA), enough RAM memory to hold one disk's worth of data, and a Raspberry Pi PICO. The PICO mainly manages the user interface while the FPGA mainly implements the disk drive emulation and controls the RAM. 

The PICO implements a Serial Peripheral Interface (SPI) link to the FPGA, with the PICO acting as the master. The PICO exchanges two bytes with the FPGA as a transaction, the first byte having the command from the PICO and the second byte recieves one byte from the FPGA based on that command. 

For example, a transaction code of x88 is a request for the FPGA to return one byte of data from the RAM. The address for memory access was previously transmitted by the PICO as a set of three x05 commands that set the 24 bit RAM address. When the PICO is writing back the contents of a virtual 2315 disk cartridge to the microSD card file that stores the data, it sets the RAM address for each of the 1,624 sectors, reads 642 bytes sequentially from the FPGA and then writes the sector's 321 words of data to the file. 

PICO LIBRARY PROVIDES ROUTINES TO CONTROL THE SPI LINK

Exchanging data over an SPI link is done with a few library calls provided for the PICO. The caller first starts the transaction by asserting the SPI chip select line, then calls the  spi_write_read_blocking function giving it the address of the buffer holding the outbound data to send to the FPGA, the address of a buffer to receive the data coming back from the FPGA, and a count of how many bytes to send. The caller then deasserts the chip select signal. 

The transfer proceeds with the SCLK signal being alternated for the number of bits being transmitted - in this case, that is 16. The MOSI signal sends the bits serially from the PICO to the FPGA. The MISO signal comes from the FPGA and provides the inbound data that is returned to the caller. Thus, an SPI transaction is an exchange. 

The routines that I inherited from the RK-05 Emulator had defined the outbound and inbound buffers as arrays of uint8_t, an eight bit unsigned value (byte). The length of each was 2, since the intent is to exchanges two bytes between master and slave in a transaction. There was also a 2 byte array of uint8_t declared in the routine but never used. 

EXPERIENCED ERRATIC RESULTS AND LOOKED CLOSELY AT THE FUNCTION CALLS

When my logic tried to write back the contents of RAM to the microSD card in the virtual 2315 cartridge, I would get some correct results and some corrupt data returned. I looked for every possible cause and investigated, but wasn't getting anywhere.

The routines that I inherited from the RK-05 Emulator had defined the outbound (OUTBUF) and inbound (INBUF) buffers as arrays of uint8_t, an eight bit unsigned value (byte). The length of each was 2, since the intent is to exchanges two bytes between master and slave in a transaction. There was also a 2 byte array BUF of uint8_t declared in the routine but never used. 

After adding code to verify that the last bits had been exchanged before deasserting the chip select, yet seeing no improvement, I was considering slowing the SPI clock just in case I was getting errors due to hardware signal issues. However, this didn't make sense because at startup the same link transferred 1,624 sectors of data down from the microSD card file to the RAM without a single failure. 

I happened to remove the unused variable BUF from the routines but when I next tested, the corruption rate increased to almost 100%. That was unexpected. 

MY ASSUMPTION WAS THAT THE FUNCTION WAS RETURNING TOO MANY BYTES

Perhaps the issue was the function, which might be overrunning the buffers I passed it. The count was a hard coded integer 2, which should be what controls this completely, but perhaps it was instead returning integers, a longer data type than the uint8_t bytes in the arrays. 

I increased the size of OUTBUF and INBUF to four from two and tried again. The code worked flawlessly. Somehow it was stepping on the stack and causing issues until I added extra room in teh buffers. The code is below. 

uint8_t read_write_spi_register(uint8_t reg, uint8_t data)

{

    uint8_t out_buf [BUF_LEN+2], in_buf [BUF_LEN+2];

    uint8_t buf[2];

    out_buf[0] = reg;

    out_buf[1] = data;

    cs_select();

    spi_write_read_blocking (spi_default, out_buf, in_buf, 2);

    cs_deselect();

    return(in_buf[1]);

}

The cs_select and cs_deselect calls are inline routines:

static inline void cs_select()

{

    asm volatile("nop \n nop \n nop");

    gpio_put(PICO_DEFAULT_SPI_CSN_PIN, 0);  // Active low

    asm volatile("nop \n nop \n nop");

}

I still don't understand what is going wrong but increasing the buffer fixed the problem. This must be why the original programmer left the spurious BUF variable in the routines, something they found solved a problem even if the root cause wasn't understood. 

Testing a write after conditioning the Sector Marker signal for the Virtual 2315 Cartridge Facility

FIXES MADE BEFORE DIAGNOSTIC TESTING RESUMED

The new Signal Conditioner board is in place and ensuring that glitches on the -Sector Marker signal receive by the IBM 1130 logic won't lead to incorrect results. Since that signals looked really good now, I expected everything to work properly now. 

LOGIC ANALYZER SHOWS THAT THE SECTOR MARKER ISSUES ARE GONE!

I recorded the same signals that showed me that doubled edges were caused by the ringing of the signal and led to a spurious termination of the write after only half a sector - 5 milliseconds instead of 10. This time, everything was perfect and the write continued for the full 10 ms. 

The flipflop logic that blocks every other 5ms raw sector marker pulse (+Alt pulse and -Alt pulse) are alternating just as they should, thus the -Sector Marker signal at 5ms is blocked and only the second one is passed through as -4 Sector Pulses and that terminates the write. The probe lead had fallen off the raw sector marker signal, which would have been signal 5 (green) in the trace, but we know the pulse comes at the 5ms point in the write operation. 

ALMOST PERFECT TEST OF WRITE IN VIRTUAL MODE ACHIEVED

This time I did achieve a good capture of the data I wrote to the virtual 2315 cartridge, including the entire sector. I set up a known pattern in a memory buffer in the 1130 and issued an XIO instruction to write to a given sector of the disk (cylinder 0, head 0 and sector 1). I issued an XIO to read in the same sector to a different buffer, confirming that the contents matched what I had written. An offline view of the mini 2315 cartridge using the Showsector.exe program I provided confirmed the content of the sector as well. 

It was not flawless, however, as there was a single bit shift in the data for the last 48 words of the 321 in the sector. If you look at the data from the sector as it was extracted with Showsector.exe, you will see starting at word 0x111 the data was recorded as 0x0224 when it should be 0x0112. Every word from that point forward is the same as the data I tried to write out except for the one bit shift, ending with 0x0282 which was intended to be 0x0141. 

The data that I wrote out was almost the same as in my previous tests, however, words 2 and 3 were changed to 0x2600 in the buffer where they had been 0x0002 and 0x0003 in my original test. Thus I wrote what we see above, at least up to word x111 where things went slightly astray.

I had previously identified that the signal -Write Clock Phase B which comes from the V2315CF into the 1130 might cause this kind of issue. The signal will advance the shift register after each clock. A spurious double edge would cause the register to shift the data by one bit. I had finessed the signal termination to really ameliorate signal issues but they were not ideal. It does appear that we had this one error in about 6,500 clock intervals during the write. That is quite good but not good enough.

The solution is the sledgehammer! One board has five channels of signals that it can process. The -Write Clock Phase B signal is definitely worth slamming into shape, so it is going to be processed. On my next visit to the shop I will be wiring the other four signals I identified so that they pass through the Signal Conditioner board. I don't anticipate any errors after I sledgehammer the signal into shape. 

Testing signal sledgehammer for the Virtual 2315 Cartridge Facility

FILTERING OUT RINGING AND UNDERSHOOT OF SIGNALS

The Virtual 2315 Cartridge Facility (V2315CF) sits in between the internal 2310 disk drive and the IBM 1130 computer's logic gates, either relaying signals between the two end points or generating a signal in place of the one that the disk drive might produce. I found that in virtual mode (disk drive not turned on, the V2315CF simulating it entirely), certain signals exhibit an undershoot to under -3V and quite a bit of ringing, when the nominal voltage levels are +5V and 0. 

The ringing caused the 1130 logic circuits to see the falling edge of the signal as multiple falling edges, thus confusing the logic and creating incorrect results. The path from the V2315CF to the logic gate of the 1130 is complex, with transitions between PCB traces, ribbon cables, more PCB traces, twisted wire cables, additional PCB traces, a pin on a backplane and then wiring in the logic card. There are many impedance transitions and places where signals can be distorted - made worse by a very fast falling edge because of the modern FPGA and logic gates in the V2315CF which switch so quickly. 

SIGNAL SLEDGEHAMMER IMPLEMENTS SCHMITT TRIGGER FOR HYSTERESIS

I designed a circuit that will employ hysteresis, so that the output of the circuit changes from logic high (+3V) to logic low (0V) or back at different thresholds depending on the direction of the change. That is, the output won't switch to logic low until the incoming signal from the V2315CF drops below 0.85V and then it won't switch back to logic high until the incoming signal rises above 4V. Any ringing in between these levels is blocked from the output. 

TESTING ONE CHANNEL WITH THE SECTOR MARKER SIGNAL

I initially only located and cut the wire that carries the signal causing me the most problems, -Sector Marker which produces a falling edge for 160 microseconds once every 5 milliseconds. This marks one eighth of a rotation of the disk platter, used by the logic circuit in the 1130 to determine where the four sectors begin on one rotation. 

A peculiarity that leads to the incorrect operation of the 1130 logic is that the disk drive reports every one-eighth of a rotation but the 1130 implements only four sectors per rotation. Therefore, it has to ignore every other sector marker pulse. The ringing in the signal from the V2315CF was causing the logic to see two sector marker pulses rather than one, so it assumed it had already moved a full one-quarter rotation. The signal conditioner should ensure it counts correctly and sees only one falling edge for every -Sector Marker pulse. 

WATCHING THE RESULT OF THE SIGNAL CONDITIONER

I put a scope on the input pin of the 1130 logic that receives the -Sector Marker pulses, to observe how the signal was shaped by my conditioner. I performed a write to a target sector, which previously had been aborting halfway through the sector due to the false edge count. The goal was to see it complete the sector write and the signal to look as close to perfect as possible on the scope.

Rising edge

Falling edge

The yellow trace at the topp is the output from the V2315CF main unit before it runs through all the cabling to get to the 1130 circuits. The purple trace is the signal received at the 1130 after the Signal Conditioner finishes with it. The blue trace is the output of the first inverter inside the 1130, showing that it also produces a single edge. 

By comparison, here are the same three signals without the Signal Conditioner aka sledgehammer - 

WILL ADD ADDITIONAL SIGNALS TO THE SLEDGEHAMMER

I picked four other signals that depend on falling edge detection in their operation in the 1130 logic, since they had the potential to trigger incorrect operation of the logic. Some won't malfunction even if they see multiple falling edges close together, such as the -Index Marker pulse that arrives once per rotation to reset the sector number counter to 0. Two edges on that signal wire still generate the same sector number in the counter. 

The incoming clock generated by the V2315CF, -Write Clock Phase B, is used by the 1130 logic to output the pulses (-Write Clock and Data) that get written on the disk platter to record both clock pulses and data bits that have a value of 1. The design of the disk produces no pulse for a data bit if its value is 0, so the only way to determine whether we have a 0 data bit value is to depend on the timing signals coming from the disk drive (or V2315CF in this case).

When reading a sector from disk, the V2315CF generates the clock and data pulses that would be extracted by the disk drive, however the data actually comes from our mini 2315 cartridges and sits in RAM on the V2315CF. Therefore, the V2315CF generates the -Read Clock and -Read Data pulses that would be seen if the disk drive heads were actually reading data from a full sized 2315 cartridge inserted in the 2310 disk drive. Those two signals are given a channel each on the conditioner board.

A feedback signal from the disk drive mechanism, -Access Ready, is used by the 1130 logic to recognize when a step of the disk arm has been accepted and when it completes. The 1130 converts a request from the program to seek a number of cylinders into a chain of small steps, a single 1 cylinder step for an odd count preceeded by 2 cylinder steps to achieve the full count. False edges might cause the 1130 to believe a step has completed when it has not, throwing off the total movement. 

I protect against false edges in the V2315CF where I watch over the arm movement, but the disk drive and the 1130 are fully in control of arm movement when we are in real mode (disk drive is operating with data coming from mini 2315 cartridge). The V2315CF has to shadow the arm movement so that it can present the data from the correct cylinder when a read or write is done - we change the RAM address based on the current arm location between cylinders 0 and 202. 

The terminator resistors for all the signals passing through the V2315CF must be pulled up to 5V in order for the 4V threshold of the signal conditioner to recognize a logic high, even though the disk drive circuits generally use 3V as logic high. This is one on the terminator board of the V2315CF. 

Another complexity of the V2315CF is my desire to support two modes - real and virtual. The intended use is real mode, where the 2310 disk drive inside the 1130 is spun up. The disk drive is passing along the real -Sector Marker and -Index Marker pulses, with the V2315CF just observing or shadowing them. The disk drive arm moves when the 1130 commands a seek to a different cylinder, with the V2315CF shadowing the end location. 

Virtual mode allows operation with the disk drive left turned off. The -Sector Marker and other signals are all produced by the V2315CF itself. It responds to the seek requests and generates -Access Ready as well. Thus, -Sector Marker, -Index Marker, and -Access Ready can come from either the V2315CF or the 2310 disk drive depending on mode.

The signals like -Sector Marker and -Access Ready are passed along through an electronic analog switch on the 2310 Interface Board which merges the wires from the disk drive, 1130 and V2315CF. In real mode, the switch connects the disk drive output to the 1130 input, while the V2315CF leaves its output line undisturbed. In virtual mode, the switch is off and the V2315CF is pulling the line down to logic low to implement those signals. 

I need the signal coming into the 1130 to work properly in both modes - real and virtual - with different electronics producing the signal in each mode. This made it impossible to select terminator resistors for an impedance that worked correctly in both modes - some choices worked for one mode but not the other. 

When these signals are conditioned by the new board, it will be time to rerun all the function tests for the V2315CF, once in real mode and another time in virtual mode. That plus the correct completion of the IBM 2310 disk diagnostic program constitute an adequate test of the V2315CF. 

Preparing wiring of the signal sledgehammer into the IBM 1130

SCHMITT TRIGGER TO CLEAN UP MESSY SIGNALS ENTERING THE 1130

I have some signals that are generated in the Virtual 2315 Cartridge Facility (V2315CF) that I built and they are fed into the disk drive controller logic in the IBM 1130. I had limited ability to change the 1130 circuitry and the connections into it. The main hardware in the V2315CF generated the signals in a FPGA and drove them out with an open collector driver and a terminator board. 

Typically the terminator board had a pullup and pulldown resistor pair to create the logic high voltage and an appropriate impedance to match to the destination. The details of the Solid Logic Technology (SLT) circuitry in the 1130 is odd by today's standards and often would not work properly when fed with the standard resistor pairs in the terminator - it might not detect a logic low reliably as one example. Thus in some cases I couldn't create a good impedance match and also get good behavior from the 1130 circuitry. I would experiment until the results were good.

When the impedance is not well matched - a difficult condition to achieve since the path from the V2315CF to the SLT input pins involved multiple mechanisms with differing individual impedances - the signal would have poor characteristics where it entered the SLT circuits. Signals may ring with substantial swings, thus swinging up and down between levels that would be a logic high and a logic low to the 1130 circuit. Sometimes the signal had an undershoot to a very negative level. 

Parts of the 1130 logic was designed around edge detection - a falling edge would trigger an action. If that action changed a counter or flipped between states, then the ringing could act like multiple edges when the intent was to have just one edge. What I needed was a way to ignore all the junk such as the ringing that occurs as part of a falling edge sent from the V2315CF, so that the 1130 sees a single clean edge. 

A Schmitt Trigger is a form of hysteresis that controls whether a signal becomes a logic low or logic high, using different thresholds for the transitions in each direction. If we don't consider a signal to be logic low until the voltage drops below 1V, but won't change it back to a logic high until the voltage rises to 4V, then all the oscillations between those voltage thresholds are ignored. 

I designed a signal conditioner that I nicknamed the signal sledgehammer because it forces a very ugly incoming signal to be a clean nice output signal. It uses an operation amplifier to force a very clean dramatic snap between logic low and logic high. Normal logic circuits can enter a grey area when the voltage is above the maximum logic low voltage but below the minimum logic high volage. This forbidden zone may produce an output that is not clearly high or low in normal circuits. Using an op amp meant that when it switched, it swung fully to the extreme. 

I used a voltage reference chip and appropriate resistors with the op amp to set it so that it would flip to logic low when the input falls down past 1V and won't flip back until the input rises up past 4V. Much more of a hysteresis spread than a typical Schmitt Trigger logic gate. I then used other circuitry to produce an SLT type output that would swing fully to its logic high (3V) or logic low (nearly 0V) following the op amp.

INTERCEPTING THE WIRING OF THE SIGNALS TO BE CONDITIONED

IBM connected the 1130 disk driver logic circuits to the internal 2310 disk drive mounted in the computer using a cable with about 20 twisted wire pairs, with SLT style connectors on each side. The SLT connector, used on the circuit cards, is a 2 column by 12 row grid of pins. The backplanes into which SLT cards plug are composed of many SLT connector slots. One of the slots in the backplane where the disk driver logic is implemented is where the cable is connected. The disk drive electronics, also built with SLT, has a smaller backplane and one of its slots is used to connect the other end of the cable. 

The V2315CF sits between the 1130 and the internal disk drive, so that it can make a mini 2315 cartridge appear to be spinning on the internal disk drive on a real 2315 disk cartridge. A 2315 cartridge has a single 14" disk platter inside the plastic enclosure. The cartridge is inserted into the 2310 disk drive and read/write heads of the disk drive will read or write data on the platter and provide it to the 1130 computer memory. 

A printed circuit board named the 2310 Interface Board has the cable from the 1130 plugged into it, rather than to the disk drive. Another cable comes from that PCB and is plugged into the disk drive. A pair of ribbon cables come from the 2310 Interface Board and connect to the main unit of the V2315CF. That unit has an FPGA, a Raspberry Pi Pico processor, RAM and a lot of interface circuitry. 

Some signals come from the disk drive and are passed directly on to the 1130. Some signals come from the 1130 and are passed on to the disk drive. The V2315CF will snoop on those signals as necessary to stay in sync with the disk drive. However, signals that would have come from or gone to the read/write heads of the disk drive are instead processed fully in the V2315CF, which creates signals that appear to come from the disk drive but are actually based on data in the mini cartridge. 

A complication is that the V2315CF can operate in one of two ways. It can completely simulate the internal disk drive, while the disk itself remains powered down. More usually, it works cooperatively with the internal drive. The disk drive spins the platter and produces timing signals (e.g. Sector Marker) that indicate where the platter is in respect to the read/write heads at any moment in time. In the other mode, the Sector Marker is generated within the V2315CF. 

Requests from the 1130 to move the read/write heads to one of the 203 tracks (cylinders) on the platter will have the disk drive move the arm with the heads. The V2315CF has to snoop on this so that it knows which track the heads are at. When the 1130 reads a sector of data, the V2315CF has to grab the data from its RAM at the location corresponding to that cylinder, so the two must stay in sync. In the other mode of operation, the disk drive does not move its arm and only the V2315CF is involved in the movement request. 

The cable from the 2310 Interface Board to the 1130 runs through the inside of the 1130, enters compartment C1 of logic gate A and is plugged into a slot of the backplane in the compartment. Its twisted wire pairs were held together with waxed string lacing then covered by a sheath of electrical tape. I can't easily intercept a signal once it enters a slot in the SLT backplane, so the cable is my point of interception.

MOST IMPORTANT CONDITIONING WAS THE SECTOR MARKER SIGNAL

My biggest challenge is when the V2315CF is in the mode where it fully simulates the disk drive. The signal -Sector Marker is produced once every 5 milliseconds during the 40 milliseconds it takes for a single rotation of the 2315 disk platter. For the 1130, IBM decided that it needed a full 10 milliseconds for a reasonable sized sector of information. Therefore the 1130 logic will ignore every other sector marker pulse. 

When doing a write of a sector of data to the V2315CF, the 1130 logic will stop the writing process when the second sector marker pulse arrives marking the end of the logic sector. The falling edge of the sector marker pulse (160 microseconds long) alternates some logic, to alternate whether the sector marker pulse is passed along or blocked. 

Because of the ugly signal arriving at the 1130 logic, the logic might see a sector marker pulse falling edge as two edges instead of one. That would mean that the write ends after only 5 ms since the blocking logic malfunctioned. 

FINDING THE CORRECT WIRE PAIR FOR THE SECTOR MARKER SIGNAL

I opened up some of the electrical tape sheathing, cut the lacing and had access to the individual twisted wire pairs near the end of the cable near where it will enter the compartment of the 1130. Using a signal tone tracer I attempted to locate the wire pair corresponding to the -Sector Marker signal. It wasn't definitive, but I also used a continuity tester hooked across the two connectors at the ends of the cable. I did locate the wire pair and cut it.

PREPARING TO SOLDER IN THE CONNECTORS TO THE SLEDGEHAMMER

I created two short twisted wire pairs that would connect to the two cut ends of the signal wire, as well as continuing the ground in the other wire of each pair. They were hooked to two connectors which will plug into the signal conditioner PCB that I had mounted on the side of logic gate A near where the cable enters the compartment. One of the short pairs goes to the input connector of the PCB, bringing the signal coming from the V2315CF. The other short pair goes to the output connector, bringing the cleaned up signal into the 1130 logic. 

Next session, I will solder these short pairs to the cut ends and insulate everything, then plug the input and output connectors to the PCB. I can then fire up the 1130 and watch the signal on an oscillocope to confirm that it looks clean. As well, I will watch the write process to verify that the full sector is written, not cut short by the defective sector marker blocking.