HAVING HAD A POOR EXPERIENCE WITH GOOGLE GEMINI, I TRIED COPILOT
I decided to give Microsoft's Copilot a try and posited the same requirement. Use a PICO to connect to a PC over USB as a disk drive showing the contents of a microSD card, accessed by SPI and formatted in FAT13. I chose the approaches that used Visual Studio and the PICO SDK.
EIGHT HOURS OF HAND TO HAND BATTLE, NO SUCCESS WITH COPILOT
The same kind of tomfoolery happened with Copilot. Plenty of recommendations with 'official links' that got 404 errors or were deprecated, followed by "you are correct, here is the proper site" which sometimes was just another bogus stale recommendation it had screen scraped from Bing searches.
It was never able to do a clean compile. Many hours of changes, reinstallations of VS and the PICO SDK and the toolchain. Lots of changes to Visual Studio, which I suspect has left it a shambles. Previously it had been able to cleanly compile the C code for the Virtual 2315 Cartridge Facility project. Many claims that "now we have the evidence" and "this is a rare issue", every single one leading to the same error. Unable to configure the compiler and assembler for the build. Many many tests that never led to a successful diagnosis.
TODAY I TRIED USING CLAUDE
It was the same mission - microSD card accessed by SPI and presented over USB as a disk drive to a PC. It told me there was a straightforward way to do it using the Adafruit PICO board, avoiding a different PICO board due to compile conflicts - but of course that was stale information and it then admitted that the Adafruit board code was not longer supported, to please use the board file it dissed in the first response.
I put up the code in Arduino IDE and the compile failed. Pasting the messages to Claude gave me recommendations. First about removing the board manager then reinstalling it, testing that a simple Blink script compiles. Then about manually removing the Adafruit SDFat library manually.
It actually compiled clean. No guarantee it will work correctly but having something I could actually load to the PICO was an accomplishment after yesterday's fiasco. I took my laptop to the workshop, installed the code on the PICO and gave it a try.
No joy. I then went through about five hours of debugging suggested by Claude, up until I ran out of free chats for the day. It appears that the PICO is freezing during its USB initialization and it may have nothing to do with the code generated for the function, other than certain libraries initializing during the PICO startup prior to one line of my setup function running.
I will return after 2PM when the Claude timeout ends and see where it takes me. I had also started a Grok session since I am interested in seeing how fundamental the issues are with using AI tools for this sort of project, versus different approaches in the various systems.
GENERAL MUSING ON AI AND ITS USE WITH PROGRAMMING TASKS
A few were more upfront than the others, showing me a running set of web searches they performed before offering the advice. It confirmed my belief that these are just digesting and regurgitating the postings of others, thus only as good as the original postings. There didn't seem to be any understanding of the actual operation of the code being proffered, nor of the failures other than repeating how others had debugged the symptoms.
This is different from how I have experienced the AI tools building code for algorithms I requested. They do seem to understand how to structure code and do a very good job with what they produce. I believe the key difference here is that I am asking to do things that involve very complex nested systems, libraries and functions built into the microcontrollers, which are not fully documented other than the source code.
Since the failures occur deep in those nested systems, the AI falls back on web scraping to find similar symptoms and thus reverts to surface level simplistic analysis. If the AI tool were to ingest and study all of the IDE, microcontroller and various libraries, building up an algorithmic understanding of everything, then it might be able to truly evaluate the issues and chart out successful code at the top level calling or interfacing with these complex systems.
I wonder if the AI is learning from its interactions with you. When it says “try this” and you reply that it didn’t work, is it learning anything? I had the same thing with coding, where Claude (? or openai) hallucinated a language construct and I said “there’s no such thing”, will that improve its response for future queries?
ReplyDelete