M1000 reader magnetic pickup fails again, verifying more of my chips, continuing archiving of decks

 TESTING OF INTERNAL CONTROLLER REVEALS MAG PICKUP FAILS AGAIN

I hooked the scope up to the Index Marker (IM) signal line and saw zip. I then noticed that the behavior of the reader indicates that the original problem I had with the magnetic pickup is back. That is, the pickup is not delivering pulses at all. Those are important in order for the reader logic to recognize that a card is feeding through the photocells and to time the points to read the eighty columns.

I was working on finding the open circuit in the pickup when I found it working again, so I simply installed a new cable and put it back into the reader. It seemed to work properly for a while but just as randomly the connection has failed again.

I am going to attempt more significant surgery on the pickup - cutting into the sealed block and inspecting until I can clearly spot the broken wire. I am assuming that it is broken from where it exits the wound coil up to the point where the external cable is attached; in that case, if I can see the break I can repair it. If it has failed deep in the windings, I am out of luck.

The alternative is to get another pickup that can detect the teeth of the wheel and generate pulses to feed into the logic circuitry. That may involve machining a different holder since I would have to be extraordinarily lucky to find a substitute with an identical size and shape. 

WORKING THROUGH MY MANY BAGS OF IC CHIPS TO DISCARD FAILED ONES

Over the years I have acquired plenty of integrated circuits, with some of them coming from auctions, surplus shops or other sources where I can't be certain that the chips are new and working well. The range of chip types is very wide, including many rare or obsolete versions that are not supported by most chip testers. 

The 8bit-museum.de Retro Chip Tester Pro that I built covers a more comprehensive set of devices - far more than the other testers I have used. As a result, it has tests built in for the majority of the devices I attempt to test. 

I spend time on every visit dumping out bags of chips, sorting them into lots of the same part number and then testing them. I have found a few bad chips. Just as importantly, I have satisfied myself that the bulk of these devices work correctly and can be used in future projects. 

ARCHIVING MORE BOXES OF PUNCHED CARDS

I used the 600cpm reader to archive another one and a half boxes of COMMON programs. These have read fairly reliably, although the heavier work is studying, testing and documenting them afterwards. I also tried to read more of the DMS R2 M12 deck but these cards are very touchy to scan. The rate of misreads, either on the first pass or on the verification run, forces me to take them 20-30 at a time and feed them perhaps five or ten times before I get a good captured file. 

Spent hours organizing DMS files and testing all the COMMON programs I have archived

 DMS FILE ORGANIZATION

I took the batches of cards that I had read, clumped them into larger groups and then split them into the individual phases that are loaded during and initial or reload of DMS onto a disk cartridge. You end up with many, many dozens of phase files plus other files that are assembled into a huge virtual card deck. On the real machine, that deck is loaded in the card reader, a blank disk cartridge is inserted in a drive, and then the deck is loaded by pushing the boot button (Program Load) on the 1130.

A deck consists of the following:

Three loader files in sequence. First, a single 1130 loader card that is the subject of the boot operation. That card's program then reads in a single card that is the 1800 loader; the 1800 is the big brother of the 1130 which was used for real time process control tasks. Finally, the 1800 loader program reads in a multi card deck which is responsible for reading in the system loader. 

The system loader is split into two phases. The first begins to format the disk cartridge and reads the next set of cards which are the data and permanently resident system code areas that are put in the special boot areas of the disk cartridge. After this 'EMN' permanent code and data file comes the system loader phase 2, which is the logic that reads most of the deck of cards and stores each phase on the disk.

A few control cards are inserted by the user that define the peripheral configuration, storage size and defines the ranges of phase numbers for each of the major sections of the monitor system. These are functional areas such as the job control, disk update program, fortran compiler, assembler and core image loader. 

The last phase of the monitor is followed by a 'type 81' card which tells the loader that it through with loading phases onto the disk libraries. The machine then fires up the monitor and runs the DUP program to start loading the system library components such as functions and subroutines. These are small object decks with a *STORE directive in front of each one. After the long string of library components is loaded, the process ends and the disk is now ready to be cold started by hitting Program Load with a cold start card in the reader. 

I have archived one complete deck to load the older V2 M11 version of DMS, a second complete deck to load the newest V2 M12 version, and have started on a third deck which is also a DMS V2 M12 load. Since I have multiples of the deck and have versions of the card decks from other sources, I can do a compare to make sure that I didn't garble any code during the archiving. 

I ran a few comparisons already with no differences detected. As a test I compared the System Loader Phase 1 decks between the two version, M11 and M12, which did flag a reasonable number of differences. I will continue to work with the files until I have compared and verified everything. Preservation of the 1130 system files is the most important aspect of this archiving for me.

TESTING THE PROGRAMS FROM COMMON AND IBM PROGRAM DISTRIBUTION

I have reviewed all the files I have archived, written a short description of what they accomplish, and tested them on the IBM 1130 Simulator as much as feasible. In many cases I include part of the virtual listing from the line printer, particularly for printer art. 

Read almost four boxes more, plus started testing of the internal controller on the M1000 reader

 FINISHED DMS V2 M12 SYSTEM LOAD DECK INCLUDING MANGLED EDGE CARDS

Two of the boxes were the system load plus sample programs in Fortran, RPG, Cobol Assembler plus a few utilities. The sections where the edges near column 1 were torn up gave me the most difficulty, but I was finally able to coax them through and verify the results. 

When a box is particularly troublesome I have to move to small batches, 20 to 35 cards apiece, which means I end up with very many of these small deck files to consolidate. I whipped up a python script to pull them together on my PC.

A box of programs from the COMMON user group was also among the work I accomplished today. The work is just begun when I finish reading, as I have to figure out what each deck accomplishes, document it and test it as much as possible. With as many boxes as I read today, I wasn't able to finish all that back end work by the end of the evening. 

I did find the program that produces music on a nearby radio held near the processor, the complement to the decks of music data I had previously archived. Similarly, I had found a nude woman on stool printer art program which called a mystery program wxyz to print the stool itself. Today I found and archived that program, which completes the printer art work. 

I then began on another reload deck for DMS V2 M12, archiving about 1/3 of a box before I left the workshop. These have intact edges but don't slide smoothly apart so that I have a medium rate of misreads and verification errors. 

TESTING OF THE INTERNAL CONTROLLER ON THE M1000 DOCUMATION READER

I powered up and connected to the controller from my laptop. The connection was successful and it appears to reflect the machine status accurately, although I didn't force very combination of states. The pushbutton toggles the EOF lamp on. Finally, the controller code issues a pick command and drives the reader to move cards through the machine. 

However, no data is transmitted to the PC. Tomorrow I need to monitor the IM line to see if the index marker pulses are arriving which is what triggers the Arduino to capture the hole patterns for each column. This could be wiring, bent pins on the shield, or another defect in the card reader electronics, to be resolved as I test the IM line. 

Fixed retro chip tester and archived more card decks

 RETRO CHIP TESTER DEBUGGING AND RESOLUTION

I flashed the test firmware that would allow me to cycle the voltage rails to the pin.s on the testing socket where they could be routed, as well as cycling signals to all the pins. I quickly discovered that pin 1 of the tester had a constant application of -5V applied to it. 

Tracing the pin back I found the circuitry that could deliver voltages to it - one was a transistor routing +5V but the other was a relay that switches in the -5. That in turn was ARbeing activated improperly because, shame on me, I had inserted the UL2003 backwards! The UL2003 is seven darlington transistor pairs but with pin 8 installed where pin 1 belongs, it fed power to the relay.

Putting the chip into the socket in the proper orientation resolved everything. Fortunately the mistake didn't damage any other components. After I flashed the production firmware I used the tester to validate chips like 7423 that weren't covered by my other chip testers. I validated several dozen ICs, fulfilling my hopes when I took on this project build.

ARCHIVING CARDS CONTINUES

I worked my way through a box (2000 cards) of programs contributed by members of COMMON. These read pretty well - some of them flew through with almost zero errors, other sections were a bit funkier, but I was pleased with the progress. I then turned to a reload deck for DMS V2 M11 which had been in a card tray in a cabinet back in Kansas, where the humidity the basement caused the cards to swell so much they were locked in place. 

The team helping me extract the 1130 could not release the latch to remove the cards in those trays; they were forced to pry sections out to relieve enough pressure to remove the rest. Sadly, this DMS deck had considerable damage on the two sides of the cards, near columns 1 and 80. 

Leading edge of card deck quite damaged

Amazingly the Documation could still get the cards through, albeit with time sucking manipulation that sometimes meant fighting individual cards to read. This will take many many hours to finish reading. 

Continuing the card archiving, finished retro chip tester pro, wiring internal Documation controller

 THREE MORE BOXES OF CARDS READ AND ARCHIVED

The number of reading errors varies dramatically from box to box of card and I don't yet see a pattern to warn me of troublesome decks. A two box set of cards that load DMS V2 M11 (the disk monitor system, compilers and utilities that are the equivalent of the operating system for the IBM 1130) read so smoothly that I only had three cards fail to verify, two of them were errors in the verify pass and one was an error when reading a card on the first pass. A box of contributed programs from COMMON, the IBM users group, on the other hand, was a nightmare where a hundred card deck might require ten or twenty passes to get the data read and verified. 

With the card contents captured, I could move on to phase 2 of the archiving. That is an examination of the decks to understand what they are, which guides me to further work. Sometimes I realize I have two more more logical items in one marked deck and split them up.

I want to end up with a folder for each deck with a descriptive name. In it there will be a text file where I describe what this deck represents and highlights any errors or missing elements. In many cases I run the code on the IBM 1130 simulator to make sure I understand it properly. Where it makes particular sense, I include a text file that is the printer listing output. 

The files I captured include printer art, games, utility programs and the disk monitor. As an example of a very useful utility, I came across programs that write various pause and error conditions to the console printer. In normal practice, a program issuing a PAUSE command will display four hexadecimal digits in the accumulator, visible only on the lights. There is no hard copy evidence of this, so that once you push Start and move past the pause, all record is gone. 

Other conditions produce monitor errors as four hex digits in the accumulator. This utility will type ERROR xxxx on the console, drawing more attention to the issue and making a hard copy. This was handy when I was trying to run a printer art program, because it had a defect that lead to the monitor detecting error F001 and pausing execution. It was helpful to see ERROR F001 printed because you might miss the error otherwise. 

When the card reader runs out of cards, the system waits with a particular address displayed in the console lights - one is accustomed to the monitor waiting and may not carefully examine the lights to distinguish an error from a normal end of job stop. 

I wrote up the order of the decks for someone who wanted to run a fresh install of DMS V2 M11 on a blank disk cartridge - along with the decks themselves. This makes the archived decks more useful than an assemblage of randomly named deck images. 

FLASHED FIRMWARE FOR RETRO CHIP TESTER PRO, FINISHING ASSEMBLY

The last step in building the chip tester designed by Stephan at 8bit-museum.de is to load the code onto the Atmel 2560 chip. The PCB he provides comes with the Atmel soldered on, but not programmed. I picked up a suitable programmer, hooked up and flashed the chip.

I brought it up. All lights were good and the menu system works on the LCD display. I started in testing some chips that weren't covered by the other testers I own, such as the 7475 and CD4050. However, some other chips that are supported came up with errors upon test. More worrisome, either the 12V or the -5V power supply LEDs blinked off during these failed tests. 

I suspect that something is not working properly, applying the wrong rail voltages or hooking more than one supply voltage to the same pin. A construction error undoubtedly. There is a diagnostic firmware load that will selectively apply the rails and signals to each pin, letting me track down the issue.

INTERNAL CONTROLLER FOR M1000 READER GETTING WIRED UP

I removed the old socket from my previous controller and moved the wires over one by one to the Arduino shield that will connect a Mega2560 with my controller sketch. I finished all the signals that were used by Brian Knittel's controller but two additional lines from the reader are used by my controller - Motion Check (MOCK) and Busy. I have to identify the wires for these two and hook them up, along with the LED and pushbutton. 

Card reader interface fully working, beginning the archiving of all my decks

 FINAL FIX MADE TO BYTE ORDER OF WORDS, FLASHED ARDUINO WITH THE CODE

After I adjusted the code to send the two bytes for each column in the order expected by the CARDREAD.EXE application, I flashed that sketch, without a bootloader, to the Arduino Mega2560 inside the external interface box. It was ready to test. That was the last fix, as the unit now reliably read the card images into the file as displayed by the companion DECKVIEW.EXE application.

Cards faithfully captured

STARTED WORKING ON THE ARCHIVING OF WELL OVER 100,000 CARDS 

I can now begin the highest priority task I have, the reading and archiving of all the boxes of punched card programs and data that I hauled with me from California. I begin with around two full boxes worth of cards, consisting of various programs and test data. 

It takes a while for a deck of cards to read correctly - they are a bit sticky from the years of storage, thus from time to time there is a bit of drag as one card slides off the deck. This can cause the data in certain columns to be read as the adjacent column. 

To catch these issues, I read the file and then pass the deck through again in verify mode, where it reads them and compares the data to the stored file. When I get a verify error, it can be an error made during the second pass or it can be an error in the pass that was captured to the file. It takes several tries to figure out which is correct and to get a stored file which verifies properly. Different decks, varying in condition, read with more or less issues and take different numbers of passes to complete. 

The time it takes to make all these passes and yield a verified fail slows down the rate of reading dramatically from the nominal rate of the hardware. The Documation model 600 can read 600 cards per minute at full speed but my net rate of finished reading is only 1200 cards per hour. That is about 30 times slower than the theoretical. This means that it will take about 100 minutes, one and two-thirds hour, per box to archive.

BUILDING INTERNAL INTERFACE FOR DOCUMATION 1000 READER

I now have a reliable Arduino based controller for the reader which I will install inside the other card reader I own, removing the need for an external box and cabling to drive the machine. A USB port on the rear of the reader will let me link it to the card reading application on a PC. 

I have a shield that consists of a set of holes I can attach to wires, linked to each of the Arduino pins. It forms a compact sandwich with the Arduino that will easily fit in the gap inside the Documation card cage between PCBs 3 and 6. The two unimplemented card positions 4 and 5 are a comfortable space. 

I began soldering the wires from the Documation cards that had run down to the external cable connector, but are now available to hook to my shield. When these are done, the only remaining task will be to build a card sized base that slides into slot 5 and supports the Arduino/shield sandwich. 

I had previously installed a pushbutton and LED for the End of File function that is part of the card reading application process. These hook to assigned pins on the shield just as they do in the external box  I finished debugging this morning.

Existing CARDREAD.EXE drives Documation reader properly, formatting mistake in returned data; P390 minor issues

 FLASHED INTERFACE BOX WITH SKETCH SANS BOOTLOADER

I brought my Arduino loader over to the external interface box I made, wired the two Arduinos together and downloaded the sketch without bootloader. As expected, the original CARDREAD.EXE program, written by Brian Knittel, could now faithfully communicate with and control the card reader.

Interface reading cards controlled by this application

I had it read a couple of cards into a file in binary mode, but the resulting file was garbled. After examining it for a bit, I realized that it was my error in how I returned the data from my interface controller up to the PC.  The fix is easy, just a tweak to the sketch and reload it without a bootloader.

Data as passed to the application

Actual cards being read

Brian had his controller return the lower four rows of the card in the first word, padded at the end with four zero bits, then followed it with the word reflecting card rows 12, 11, 0 down to 5. I unfortunately reversed the order of the two words. It was easy to spot when reviewing the pattern of the data in the file. 

Brian's protocol document excerpt

P390 SMALL ISSUES TO WORK ON LATER

The P390 booted up with a post-BIOS error 184, which states that the startup password was corrupted. Since I have not set a password, this suggests I still have some issue with the CMOS battery and storage for configurations. Perhaps it is caused by damage from the twisting that the motherboard sustained. 

OS/2 reports that the gigabit ethernet NIC card is not found, although it is sitting in slot 3. This could be caused by many things. The planar could be damaged causing that slot to not work. The card itself might have been damaged in the twisting. Just as likely, the IRQs and other configuration details were lost when the CMOS lost its data and the default way it was configured is blocking access to the card.

Existing CARDREAD.EXE can communicate to Arduino interface with bootloader removed

 LOADING SKETCH WITHOUT THE BOOTLOADER

The Arduino IDE includes a sketch that will turn an Arduino into a programmer that can load the flash memory of another (target) Arduino with a sketch. The two are wired up with their SPI interfaces connected, such that the 'programming' Arduino will reset the 'target' and transfer an incoming sketch into its flash. It is possible to load the sketch without any bootloader on the 'target' system. In that way, the sketch begins execution almost instantly.

RESULTS AT HOME WITH THE ARDUINO INTERFACE

I loaded my interface controller sketch into my Arduino using the method above, then started up the CARDREAD.EXE application pointing at the COM port to my Arduino. The app connected and was happily exchanging status requests with my controller. 

I used various jumpers on the Arduino to alter various status signals, which were then correctly represented on the GUI of the application. I feel that this will permit me to connect the Documation reader to my controller using this method of installing a sketch without a bootloader. I intend to test the reader tomorrow using the existing Windows application. 

Visitors gone - back in the workshop

 DEVELOPMENT OF ALTERNATIVE CARDREAD APPLICATION IN PYTHON

I am writing an application to drive the Documation card reader and store the data in files using the 'binary' file mode of Brian Knittel's card reader interface programs. I adopted the same protocol that he uses with the Arduino based interface hardware I engineered, thus potentially allowing his program to drive my interface.

However, while I work on resolving the issues running his program on my Windows 10 based laptop, I have been working on my own program to replace CARDREAD.EXE just in case I can't resolve the problems using his code.

I started out with a command line program but have been switching it to a quick and dirty GUI making use of the tkinter library that comes with Python. Eventually, if I have to continue using the program, I will pretty it up including displaying a card with the holes displayed for the card that was just read. 

TESTING WITH DOCUMATION READER

I did some testing with the command line version of my program and worked out some issues. It does report status of the reader hardware properly and will make the reader pick cards. However, I don't yet have it capturing the incoming data words from the cards as they are read. 

BUILDING RETRO CHIP TESTER PRO (DESIGN OF 8-BIT MUSEUM)

I have a huge collection of chips of all types, most of which I couldn't test with the existing chip tester systems I own. I was impressed by the range of devices that are supported by the Retro Chip Pro tester developed by Stephan Slabihoud at his 8-bit computer museum ( http://8bit-museum.de ) and purchased the PCB and related documentation from him. 

I ordered all the myriad parts required to populate the board, so that when the board arrived a couple of days ago, I would be ready to begin assembly when I returned to the shop. That moment was today. I carefully mounted and soldered all the parts to the main PCB, the daughter-card power supply board and set it up for multiple powering options. 

Up next will be careful examination of all the solder joints with my stereo microscope, prior to doing a first power-up and test of the completed project. When this is working, I can revisit the large pile of untested chips I own and validate the condition of many more of them. 

More wrinkles with Arduino and windows serial over USB; making CARDREAD.EXE connect to Arduino

 NEW DISCOVERY ABOUT THE SPURIOUS CHARACTER ISSUE

I was foolishly satisfied that the one millisecond delay I added to my code was going to be a long term workaround for the spurious characters that arrived at startup of the Arduino sketch, in spite of doing this blindly. I had no clear understanding of the cause of the issue and nothing that suggested that a delay was related to the problem. 

I made some unrelated modifications to the sketch in an entirely different area further along in the code, after which the flaw came back. At this point I wanted to know exactly what was being received as that might give me a clue as to the agent which was sending them. 

I set up the sketch to blink out the eight bit character being received, with one clock blink followed by a second blink only when the bit was one, separating the clock-data pair by a longer interval to permit me to see the data unambiguously. 

What I found was two xF0 characters. This reminded me that I was seeing them earlier with the Arduino based 3174 controller board I was experimenting with. Those spurious characters arrived at various times during a sketch, along with an xF8, not just at startup. My workaround at that time was to 'eat' those characters. Only possible if xF0 and xF8 are not valid incoming characters to the Arduino.

NEW WORKAROUND

The protocol between the application and the card reader interface sketch sends only ASCII characters, a small set - C, P, R, S, and X - which means I can safely eat the bad characters even though I still don't know why they arrive on my laptop USB link. With that added, the sketch seems to be solid again.

QUICK RECAP ON ISSUE WITH CARDREAD.EXE

An Arduino puts a bootloader in upper memory along with the compiled user sketch. When the Arduino is reset, it runs the bootloader which listens on the port to see if the IDE is downloading a new sketch. This introduces a delay of almost 2 seconds on startup of the code. 

The reset is triggered by the DTR control line going low, which occurs every time that Windows opens a COM port. This was introduced into the Arduino system to allow sketches to be loaded without the requirement that the user push the reset button on the controller. 

The CARDREAD.EXE program opens the COM port, sends the character R to reset the controller and reads the incoming serial line for a response character. It does this send and receive ten times in quick succession, looking for a valid status character - ASCII between x40 (@) and x7F which includes the capital letters A to Z. 

The problem is that this rapid fire transmission occurs faster than the bootloader will stop, so that the program nevers sees a valid response character and the Arduino never gets through bootloader to run my actual sketch code. 

I don't know why the program doesn't block waiting for the response character. Instead, it seems to look at the incoming port even if there is no data there yet. This is a second sign that the Windows virtual COM port functionality is broken on my laptop. It is also broken on the Windows Surface Pro tablet so it is not just a Windows 10 problem. My guess is this is a compatibility issue mixed with a defect in Windows.

SOLUTION IS TO REMOVE BOOTLOADER FROM ARDUINO

If I load the sketch to the Arduino without the bootloader, it will start up immediately and look at the incoming characters. That should give us the R command and send back the response character @ fast enough so that CARDREAD.EXE is satisfied. At least, that is the plan.

Discovered why CARDREAD.EXE is not communicating with Arduino

 LED ON PIN 13 FLASHES BUT I DON'T USE THAT PORT

I noticed that when I fire up the CARDREAD.EXE program which is unable to communicate, I see the LED light on the Arduino, the one that is controlled by pin 13. I am not making use of that pin and thus nothing should cause it to light, go off, and then flash periodically, but that is what happens.

Using a terminal emulator or the serial monitor does not trigger this behavior, it is unique to the failing connection from the older Windows based application. 

IT APPEARS THAT BOOTLOADER INTERACTION IS BEING TRIGGERED

I did some quick searching online and found that the bootloader on the Arduino manipulates pin 13 in this pattern. On a first connection to upload code, the loader waits for a second (the long LED interval I see), then subsequent fast resets (triggered by DTR sent over the link) are much faster before they decide that code is NOT being uploaded.

This implies that something in the older program and Windows is causing DTR to pulse downward on each open of the port, then the code sends the 10 tries of the R character and gives up apparently before the boot loader has abandoned its control of the link. 

POTENTIAL FIXES

The bootloaders required the reset button to be pressed on an Arduino in the earlier days, but this was 'enhanced' to use DTR as a reset such that the button did not need to be touched anymore. There are ways I can modify or block the bootloader so that the Arduino does not do its reset, allowing me to test this theory. 

There are two separate but related actions taking place, one is desirable but the other causes my problems. The practice of Windows in dropping DTR during the opening of a COM port triggers the auto reset of the Arduino; this is useful and I want to preserve it. When the Arduino resets, the boot loader takes control of the serial link to look for code being uploaded from the IDE, taking a second to wait for code to arrive; this is the behavior that causes me problems.

The Windows code sends a control sequence to the port to set special characters, but they are all zero valued instead of the proper codes for things like XON and end of transmission. It think this is the root cause of my problem, because that sequence seems to cause the Arduino bootloader to wait for code to begin streaming. That wait is long enough that the CARDREAD.EXE program drops the port and starts over.

If I can block the bootloader from taking control, it should all work well. I would still get the reset when the COM port is opened. The only function I would lose is that the normal way of uploaded code to the Arduino would not work. If I have a tenable workaround to upload sketches and a way to block the bootloader otherwise, I can overcome this problem. 

It seems the best method is to upload my sketch without any bootloader. This will cause it to run immediately after it resets and ignore any sequences that would trigger the loader wait. To this end, the IDE allows you to save the compiled sketch, which is stored both with and without the bootloader. 

I then will have to upload it via a programmer. I decided to use another Arduino as the programmer for now, but might look into my programmer devices at the shop and switch over. These all access the ICSP header on the board to update the flash memory with my sketch. 

Dealing with spurious input with first read on Arduino

 RESOLUTION DISCOVERED BY TRIAL AND ERROR

I determined that as short as a one millisecond delay via the delay(1) command will cure the issue I see, where the Serial.available() reports data in the buffer and immediately after the Serial.readBytes pulls spurious content. Not sure if it is just the time spent or it depends on the delay implementation allowing interrupts to occur but this reliably solves the problem. 

Digging into the serial link challenges with my Documation card reader interface

 DEVICE MONITORING STUDIO USED ON WINDOWS TO WATCH THE COM PORT

The Free Device Monitoring Studio installed filters (shims) on the virtual COM ports on my Windows 10 laptop. It can intercept and record all the data and control commands issued to them even when other programs are in control of the port. I made good use of it to watch what was happening on the link.

CARDREAD PROGRAM SENDS BUT DOES NOT RECEIVE DATA

When I fire up Brian Knittel's CARDREAD.EXE program I see it sending the character 'R' repeatedly and I can see the Arduino receive it based on the blinking LED, but I never see any return traffic which means the program never gets its expected response.

I do see his program request both DTR and RTS handshaking, but whether I set up Windows for hardware handshaking or not, the result is the same. No return data. I don't know why yet.

ARDUINO USE OF DTR SIGNAL ADDS TO THE MESS

The Arduino system has overloaded the meaning of DTR, so that when it drops to logical false, the Arduino does a reset. This is useful both for uploading code and to ensure that the code starts fresh when a link is established. Unfortunately, that interacts with the Windows com driver practice of dropping DTR during the open of a COM port, which is how the reset is triggered.

RANDOM DATA SEEMS TO ARRIVE AT ARDUINO RESET

Regardless of the program I use to connect to the Arduino, when it restarts and opens the serial connection, something happens to the input buffer logic. The Serial.available() function sees waiting data, that triggers my code to read a byte and parse it as a command. It is not a valid command thus the code returns the character '?' without the calling computer or application having sent any command characters. 

If the CARDREAD.EXE program was actually receiving the output of my Arduino code, it would have iterated sending the 'R' until it got a good response, overcoming the one time garbled transmission. That is what happens when I use terminal emulation programs such as PUTTY or the Arduino IDE serial monitor. 

TWO PROBLEMS TO OVERCOME

First, I need to get to the bottom of the issue blocking the CARDREAD program from receiving serial traffic. If I get that working properly I can live with the second problem.

Second, I have to figure out why I am getting the garbage results on opening a serial link over USB. This is a problem that has plagued me in prior projects (on this Windows system) so its resolution will have wider benefits that simply cleaning up this reader interface behavior. 

Setting up to investigate the failure to communicate by cardread.exe to interfaces

 SERIAL PROTOCOLS AND CONTROL LINES

Serial communications protocols like RS-232 were developed over time and included a number of signals that are erratically implemented. These include signals with labels such as RTS, DTR, CTS, DSR, RI, RTR and DCD in addition to the TX and RX lines that carry the outbound and inbound data. Even the voltage levels are subject to some variation, with the original standard involving positive and negative voltages of up to 15V but often with computers today TTL levels of 0 and +5V are used. 

The meanings of the signals were originally defined around connections between a modem and a device, the DCE and DTE respectively in the jargon of the standard. Thus, DCD meant that the carrier signal on the phone line was detected and RI meant that a ring was incoming on the line. With fixed wiring between the two sides of a serial link and no modems involved, these have much less meaning.

Flow control can be implemented using the lines, but again the way this occurs varies. One can imagine the use of RTS (Ready to Send) and CTS (Clear to Send) but even that has evolved so that RTS is relabeled RTR (Ready to Receive) and thus each signal indicates that data can be sent on its associated RX or TX line. Other flow control schemes include using characters like XON/XOFF sent inband rather than using the control signals.

Control lines such as DTR and DSR indicate that the two sides of the link are ready. In some implementations, the link won't operate unless it sees these positively asserted. Many other implementations ignore these lines. 

Thus, making a serial link work is more involved than just hooking up TX and RX, then setting the baud rate, bits, parity and stop bit configurations. One has to know whether lines like DTR, DSR, CTS and RTS are required and how they must operate. 

CODE IN THE CARDREAD.EXE APP USES DTR AND RTS

Investigating the source code for the application written by Brian Knittel to control the Documation card reader shows that he opens the serial link at 115200 baud, 8-N-1 settings but also that he is specifying that DTR and RTS are to be used for link enable and flow control. 

The big question to resolve is how this is handled by Windows in its com ports, in the USB driver, and again on the remote end inside the Arduino as it is converted back to RS232 signaling. It is possible that some settings are causing the application to block or to not receive the responses from the interface. 

SERIAL COMMS MONITORING TOOL INSTALLED

In order to see exactly what the application is seeing in its requests to Windows for communication with my interface module, I downloaded a tool that installs a filter in front of each COM port on Windows, allowing it to capture every request and data transfer. 

It will let me see whether the windows to CARDREAD.EXE traffic shows the transmission of the character 'r' to the interface and the receipt of the response '@' which indicates that our interface has successfully reset. That will guide me further in diagnosing the failure to communicate which is blocking my use of the existing application to control my card readers. 

Issue found with my card reader interface; can't get existing program to connect

 CAUSE FOUND FOR LACK OF ERROR STATUS SIGNAL

I used an inspection mirror to look between the shield and the Arduino. Problem found! A pin from the shield was bent out of place and not inserted into the header on the Arduino. It was for pins 35 and 37, which includes the failing ERROR signal. 

A quick bit of bending and the problem was gone. I now get proper status for the machine. 

FAILING TO CONNECT FROM BRIAN KNITTEL'S CARDREAD.EXE

The program shared by Brian Knittel as part of his interface solution to the Documation card readers uses a virtual serial port to speak to the interface. It has been failing to connect to my instances of the board designed by Brian, which was the impetus for me to shift over to the Arduino based design I developed that will provide a more solid base for reading all the punched cards I own.

Sadly, it fails to connect to the Arduino either (or doesn't like something coming back from the Arduino which it interprets as an indication that this is not a valid interface). It fails on my laptop and on my older Surface Pro that I had been using with the readers in past years. 

This could be a problem introduced by Windows maintenance, breaking something for this old code. It could be an issue with my Arduino link also. I chat with my new interface over various terminal programs and everything appears good as far as I can tell.

I will proceed on two fronts simultaneously to overcome this last hurdle. First, I will write a Python program to manage the card reading, bypassing the need for the older program entirely. Second, I will write a program to masquerade as the interface to see what the program attempts, what it sends and how it responds to the intended responses. I will also dig around a bit to see if there are reports of other virtual serial port based programs failing due to windows fixes.

Debugging continues for new Documation card reader controller

 INVESTIGATING THE MISSING ROW 9 DATA

To divide up the set of suspected failures I tacked a wire onto the output of the gates driving the D9 (Row 9 data) signal, just before it exits the Control card and flows over the connectors into my controller box. The scope showed me valid D9 data which eliminated the internal Documation circuitry from the list of suspects.

I then did a continuity test between the shield connector on the Arduino and my tacked wire to see if the signals are making it from the reader into my box. I found an open circuit. I then removed the wire from the shield and verified there was a proper connection. The flaw was a poorly connected wire onto the shield.

With that repaired, I read some cards to prove that I am receiving all the data that is punched on the cards, without dropout or glitches. 

INVESTIGATING THE FALSE ERROR STATUS

At the same time as I had tacked on the wire to the Control board for the data line, I tacked a wire to the Error card to check out the error signal. I found that it was improperly reporting an error condition when none existed. It appeared that the buffer gate was bad - input said no error but output flipped to on. 

This buffer chip is not fully used. I first cut the leads from the package for the failing gate (pins 10 and 11), added jumpers to the unused gate at pins 8 and 9, then retested. Whatever went wrong with the one gate was still wrong with the substitute.

Jumpers substitute unused gate for failed on

I pulled out my Hakko desoldering gun, removed the bad chip and soldered in a replacement 7417 open collector buffer chip. The ERROR signal was now properly generated, but the interface still saw it as active.

Next up was the continuity testing, which proved that the ERROR line got to the shield connector for pin 35 of the Arduino. This pin is read by bit 2 of Port C of the Atmel Mega2560 processor on the Arduino (PC2). All eight bits of Port C are inputs accessed by the same method and the other seven are correctly read. However, I am not seeing PC2 reflecting the signal coming from the reader. 

I may have a mechanical issue with the shield to Arduino pin for 35, or the Arduino processor may have been damaged by whatever was wrong with the original buffer chip, leading to PC2 being unusable. I will have to investigate this further on my next session.