User interface test verification part 21

DIGGING THROUGH EVIDENCE TRAIL OF U-BOOT AND ATTEMPTED KERNEL LAUNCH

I captured the output of the preloader (SPL) and then U-boot on two different SD cards. The old, good one is a card that launches Linux successfully but does not properly initialize the bridges I need for my project. The new, bad one is the latest try to build a working U-boot. In this case I copied a prebuilt binary for those files, not the one that should be honed by the build process for my exact design configuation.

The new, bad version will come up to the U-Boot console but will not launch Linux. The last line I see is that it is launching the Linux kernel, then nothing at all happens. There are many hundreds of parameters for U-boot and many dozens of environmental variables you can set for U-boot, which makes an exhaustive search of every combination infeasible. I have tried adjusting anything that seems remotely associate with the problem, but to no avail.

PRELOADER BEHAVIOR COMPARISON

Here are the two preloader (SPL) outputs, first the old good card and then the new bad card. 

GOOD OLD
U-Boot SPL 2013.01.01 (Oct 12 2016 - 10:38:03)
BOARD : Altera SOCFPGA Cyclone V Board
CLOCK: EOSC1 clock 25000 KHz
CLOCK: EOSC2 clock 25000 KHz
CLOCK: F2S_SDR_REF clock 0 KHz
CLOCK: F2S_PER_REF clock 0 KHz
CLOCK: MPU clock 925 MHz
CLOCK: DDR clock 400 MHz
CLOCK: UART clock 100000 KHz
CLOCK: MMC clock 50000 KHz
CLOCK: QSPI clock 3613 KHz
RESET: COLD
INFO : Watchdog enabled
SDRAM: Initializing MMR registers
SDRAM: Calibrating PHY
SEQ.C: Preparing to start memory calibration
SEQ.C: CALIBRATION PASSED
SDRAM: 1024 MiB
ALTERA DWMMC: 0

BAD NEW
U-Boot SPL 2017.11 (Apr 24 2018 - 11:02:05)
drivers/ddr/altera/sequencer.c: Preparing to start memory calibration
drivers/ddr/altera/sequencer.c: CALIBRATION PASSED
drivers/ddr/altera/sequencer.c: Calibration complete
Trying to boot from MMC1

First off I have to state that there are different levels of information printed by the preloader depending on the hundreds of parameters set and even the version of the code, so that we can't expect an exact match. However, there is board information that the old version prints out including configuration details that is missing from the new version. Without that same level of output I can't be certain that the preloader correctly understands our board and has configured it properly.

I do see that it uses some awareness of the memory on the board as it is able to calibrate the memory and make it ready for read/write during U-boot and onwards. It is the only commonality I can see here. 

The preloader works from a stripped down device tree produced by the build process. It is created by stripped and manipulating the binary version of the device tree to remove superfluous entries (things that U-boot won't need during its operation) and to ensure that U-boot is able to manage the board correctly. Because this process of generating the stripped down device tree is hugely opaque and ill documented, there is no easy way to check that it has the proper board definitions in the new version. 

Even worse, I am using the version of the binary from the github repository not a version I built with make, so I am certain that it does NOT have any of my project specific requirements such as mux pin settings. When I try to build a version myself I can't get any response at all from the board, while at least the prebuilt version brings up U-boot. 

U-BOOT BEHAVIOR COMPARISON

Again we start with the old, good U-boot that will launch my Linux system. It does not prepare the bridges thus it is unusable for the project but otherwise seems to work. This is all prebuilt with nothing from my design involved. 

GOOD OLD
U-Boot 2013.01.01 (Oct 12 2016 - 10:40:34)

CPU   : Altera SOCFPGA Platform
BOARD : Altera SOCFPGA Cyclone V Board
I2C:   ready
DRAM:  1 GiB
MMC:   ALTERA DWMMC: 0
In:    serial
Out:   serial
Err:   serial
Skipped ethaddr assignment due to invalid EMAC address in EEPROM
Net:   mii0
Warning: failed to set MAC address

BAD NEW

U-Boot 2017.11 (Apr 24 2018 - 11:02:05 +0900)

CPU:   Altera SoCFPGA Platform
FPGA:  Altera Cyclone V, SE/A6 or SX/C6 or ST/D6, version 0x0
BOOT:  SD/MMC Internal Transceiver (3.0V)
       Watchdog enabled
I2C:   ready
DRAM:  1 GiB
MMC:   dwmmc0@ff704000: 0
In:    serial
Out:   serial
Err:   serial
Model: Terasic DE10-Nano
Net:
Error: ethernet@ff702000 address not set.
No ethernet found.

Here we see some board information that seems to align between the two versions. As I mentioned, the new version won't launch the same Linux kernel that my old version does successfully. 

UNKNOWN CONFIGURATON BY THE LOADER SOFTWARE

Somewhere in the execution of the two software programs - preloader and U-boot - the key settings are applied to the pin multiplexing and other configuration registers to make the board match the design I built with Platform Designer and Quartus. I don't know where this occurs and I can't check what is applied to the board. This makes it hard to modify an existing preloader/U-boot to match my design.

MY BASIC PROBLEM REMAINS

The most fundamental problem I am facing here is that regardless of which version of the U-boot/preloader code I downloaded and regardless of which cross compile tools I use, when I make the preloader + U-boot and try to boot the SD card, I don't see ANYTHING. No flickering of the board, no U-boot SPL header messages at all. It does not appear to be executing anything. Since I install the prebuilt version that came from github in the same way onto the SD card, and those prebuilt versions run, I have to conclude that it is something foul in the code being generated by make. 

I have followed dozens of different procedures and 'recipes'. I have used different code bases and tools. Regardless, the preloader just won't run. I think I have to attack this problem first and solve it, as when that is working then I have control over what I put into U-Boot and I will have all the design specific settings from my project included in the code. 

User interface test verification part 20

FOUND A DIFFERENT UBOOT ARCHIVE FOR DE10-NANO

Somebody had uploaded a U-boot set up for my board. I pulled it from github in the hope that this will accomplish the three things necessary - it must bring up U-boot, it must enable the bridges, and it must support launching my Linux kernel to fit my design. 

As a quick initial test I did an install of their prebuilt U-Loader, one that most likely doesn't support the bridge or pin muxing configuration from my design, just to validate that it will successfully start U-boot. It did.

The next step is to build the customized U-boot from that distributions sources. If the updated U-boot will also load to the command prompt it would be a promising sign that I can move on to Linux boot and resume my testing. 

STUMBLING OVER OLD PROBLEMS AGAIN

The source for U-Boot includes an error with duplicate definitions of a label yylloc to which there are many google hits. However, the resolution in all those hits is to assert that one has to have the correct version of gcc versus Ubuntu versus U-boot all without ever stating ANYWHERE what all the versions are that should work together properly. 

AND ANOTHER BRICK WALL

Fortunately I could just comment out one of the duplicates so that the loader can link all the references to the same location. With that the preloader and U Boot images were created. Now all I had to do is to update the SD card, presumably, and it would boot up. 

It did not! 

The bottom line is that nothing I do to generate U-boot and the preloader can create something that will start up on the DE10-Nano board. I have several versions of premade U-boot but they have various problems. The earliest one does not properly activate the bridges I need. The newer one appears to activate the bridges, although I can't test that and only assume so because there are no error messages during the activation process. However, my Linux image will not boot up on that preloader.

Clearly I have to debug the startup of Linux hoping to get a clue to why this process is failing. My guess is that the U-boot versions are not activating key functions that my design uses and Linux is hanging trying to work with those functions.   

User interface test verification part 19 - still slogging through the mire

CROSS COMPILING CONFUSION

I am using a laptop to build the software for the DE10-Nano board, thus we are cross compiling since my laptop has an x86 processor but the board has ARM Cortex A9 processors. This should be straightforward since Intel/Altera provides the SocEDS tool that is the development environment for cross compiling for boards such as mine. The tool starts a shell terminal window that does the cross compilation automatically. When I build my applications to run under Linux on the board it properly compiles them for the ARM A9 chips. 

Apparently the build process for U-Boot software steps on this somehow and makes different choices for cross compiling which produced code that won't execute on my board. ARM chips come in a vast number of versions and feature sets, thus one must be very explicit about the chip used so that the compiler and assembler use the proper ARM instructions. 

On my laptop, the SocEDS tool runs under Linux in a virtual machine and has the cross compile tools installed using prefixes. Thus, the C compiler GCC has versions such as arm-none-linux-gnueabihf-GCC, arm-none-linux-gnueabi-GCC and many others. The cross compilation process has to set up the right prefix such as arm-none-linux-gnueabihf- in order to work properly.

The Cortex A9 chip has hard floating point, meaning that it implements ARM instructions to do floating point operations. This is indicated by the last two characters hf in the prefix. The prefixes without those letters are 'soft floating point' meaning that the code calls a subroutine to perform the operations without using the hardware instructions. The subroutines must be provided somehow in the module in order for those calls to run successfully. 

Also, the code generated by GCC and set up as executable modules differs depending on whether the code will run under Linux or runs standalone. Under Linux there are system calls which provide services the code can request, but they do not exist or must be explicitly created for standalone use. This is the meaning of the linux in the midst of the cross compilation prefixes. 

Each variant of ARM processor version, hard versus soft floating point and whether Linux or standalone requires its own version of the tools for the cross compile. This can involve installing multiple packages as they are not all provided out of the box by the SocEDS tool. 

Once I recognized the necessary version of the cross compile programs, I made sure that I had those installed on my linux system and set up the cross compile environmental variable to point to them. This was not enough to restore U-boot to operation but important.

DEEP GOOGLING TRIPS OVER A VAGUE HINT

In one of the myriad of conversations I read about those having issues with the preloader (SPL) of U-Boot hanging up without printing anything, there was a mention of two reasons that the SPL might go into a loop - incorrect specification of memory or the pmmu details in the device tree. This highlights that there are in fact more than one device tree for my design, the second one hidden inside the process to make the preloader. 

A device tree is a structure of keywords that the Linux kernel uses to identify the various features, facilities and components it has, understand the addresses to control them and load the appropriate software drivers. It is created as text, the format dts, then it is compiled by the device tree compiler dtc into a binary format file dtb. 

Why not just use the one true device tree for the preloader, you might ask. Well, the preloader runs out of very limited RAM in the processor chip before the SDRAM main memory is initialized and U-boot is copied into that SDRAM. Thus, there is precious little space available for a full device tree. 

A Rube Goldberg (Heath Robinson for my UK oriented readers) mechanism takes the compiled dtb file and runs a binary time version of the grep command (fdtgrep) to strip out all entries not needed for the preloader and U-boot. This essentially opaque process strips down the device tree to its smallest in order to include it with the preloader and U-boot. Apparently there are comment tags in the device tree source file dts that indicate what must be included in the preloader version. These tags are set up by Altera/Intel in the files they provide with their toolkits that should in theory produce a fully usable stripped down dtb. 

The device tree compiler dtc can be run in reverse to convert the binary dtb back into source form dts files. I did this, receiving error messages about faulty formats for the memory and the serial port entries. Both of these are extraordinarily central to the preloader initializing RAM, loading U-Boot and printing the first of the boot time messages on the console. 

I therefore have to figure out what might be wrong with the stripped down file, correct it, and see if I can just build the preloader and U-boot from this point forward. All this assuming that this, perhaps the one hundredth possible cause I had to investigate, is actually the reason I can't run U-boot. 

COMPARING OTHER BOARD CONFIGURATION DTBS TO THE DE10-NANO VERSION

I chose to configure and make the software as if I was running on the Intel Cyclone SDK board rather than the Terasic DE10-Nano. I am hoping that whatever fails for the DE10 will not occur on the most mainstream board they have. 

Comparing the two, however, didn't reveal any obvious differences or issues. so I am still nowhere in the quest to get the darned thing to boot up. All I need is to get U-boot to work, the rest is something I can work out. 

NEXT WILD STAB - PERHAPS THE ENVIRONMENTAL VARIABLES CAUSE CONFLICT

The environmental variables used by U-Boot are saved in flash memory on the board. My next desperate attempt to figure out the problem involves looking into or resetting the values. What I suspect might happen is that some value specifying a load address is clobbering the preloader before it is done or failing to branch to the properly loaded U-Boot. 

To fix this, I will use another SD card I have that boots up, use it to restore the environmental variables to their default state and then retry my failing SD card. Nope. I did write down the loading addresses it would set in the environmental variables.

FOCUSING ON DEVICE TREE PROCESS

The process of creating a device tree that can be used by the U-boot as well as Linux involves many many convoluted steps in the toolchain. Some processing has to occur to take the output of the Quartus toolchain and fold in other entries to create the source tree. It then has to be somehow merged into the U-Boot building process.

Complicating this is the fact that the preloader/U-Boot works with a stripped down version of the device tree. Further complicating debugging is that the creation, modifications and use of the device trees are in its binary form (dtb files) thus at every step I am investigating I have to run the device tree compiler dtc to back out to a representative source (dts file) version. 

The files are built by compiling, applying macros, running scripts, running the fdtgrep tool and generally shuffling things around like a skilled practitioner of Three Card Monty. If indeed my U-boot won't run because the stripped down dtb file created for it is defective, then to uncover the source of the defect I have to inch my way backwards. 

The recipes and how-to guides suffer from the massive problem of procedural evolution. That is, the Rube Goldberg processes involved in the device tree and other data slithering from Quartus to the U-Boot programs have changed many times. Some use a BSP editor. That went away. Some use shell scripts, other versions use python scripts. Some of these take files needed to set pin mux tables but not the device tree. Other things fill out and move the device tree information. 

This renders the system a nightmare to work with, since every reference document, recipe or script says something different. I am going to have to reverse engineer the whole process and perhaps even the entire code base of U-boot in order to produce a loader that will bring up my board to let me in turn boot Linux. 

User interface test verification part 18

DISTRACTIONS WITH VISITORS HOWEVER STILL STUCK

I am finally able to put some time back into the effort to restore a U-boot that brings up my Linux image on the DE10-Nano board. I have been unsuccessful recreating whatever I discovered or did last time that produced a working version. Sadly, after some progress testing the design, I had to make changes that would require an updated U-boot and preloader. 

This is because it is the U-boot software that implements the pin multiplexing settings, starts up the DDR RAM and does other initialization such as the F2SDRAM bridge before Linux can begin booting. With changes to the chip configuration, the U-Boot files had to be updated. 

I did figure out something that regained the booting when I last built and installed U-boot, so somehow I will find it again. Alas I didn't write down what I discovered nor document all the permutations and adjustments I had tried during that journey, so I have only hazy memories to guide me this time. If you can all be patient, I will be relentless until I get this fired up again and can continue perfecting the design of the Virtual 2315 Cartridge Facility. 

User interface test verification part 17 - one step backward

DECISION TO REBUILD PRELOADER AND U-BOOT

The mechanism to create a working System on a Chip (SOC) using the Cyclone V involves a number of tools and software products that are hooked together with a patchwork of scripts and steps, versus an integrated tool that would accomplish all of it in one product. This means that there are important times when you have to repeat steps to reflect important changes else the SOC won't work properly.

Since I had made changes to the address widths of some parts of the HPS to FPGA Lightweight (H2F LW) bridge, it might have changed the four files that are produced by Quartus that have to be manipulated and transferred to the U-boot and Preloader software before those components are built. In particular, there is a file that sets the multiplexing state of connections between the sides of the SOC. 

I therefore had to locate the script that grabs these from the Quartus folders, massages them and saves them in the proper place in the U-Boot folders. I found one and ran it. From that point I issued a make command to hopefully correctly rebuild the files I needed. There is one file, u-boot-with-spl.sfp that is copied to a special partition on the SD card to allow it to boot up.

REDISCOVERING PATH TO A GOOD VERSION SINCE THIS ONE WON'T BOOT

When I tried to boot after updating the Preloader/U-boot on the card, it sat with no sign of life, just as it had when I first attempted to build these components. I had figured out the issues and eventually was able to make a proper version. I simply had to rediscover what I did in order to restore my system to booting up. 

So far I am not getting the Preloader to boot and transfer to U-boot. No sign of life at all. I have only a couple of days before house guests arrive and take all my time through to early next week. 

User interface test verification part 16

BOOTING AND SEEMS TO BE OPERATING PROPERLY

I chose to do some verification that my functions were still working before I dove into more general debugging.

DO THE KEY BRIDGES WORK? 

Indeed I appear to be successfully reading and writing over the H2FLW and F2SDRAM bridges. Response are reasonable and I do see the telltale LED turn off when my loop code in the FPGA advances after having completed a write to RAM of the block sent down. 

I starting walking through the code of a load with pauses to verify I had reasonable responses. If I have a problem with holding off the next block coming from Linux to FPGA while I am busy writing to RAM, I won't see it at this slow pace but that is easily checked by removing the pause statements later.

FIRST ISSUES DETECTED AND BEING HANDLED

I knew that the ARM and x86 systems were little-endian, which means that a 16 bit word that is a single long integer x01F6 will be stored as xF601 in memory. I didn't realize that it also flipped the two 16 bit words returned from a file access. In other words, the first two 1130 words in the virtual cartridge image file are x0000 and x0658 but when I fetch a block of 32 bits from the file I see them in memory as 0x5806 0x0000 meaning that both the bytes in a word and the words themselves are flipped. I had hoped that the two words in a block would be N and N+1 from left to right, not the N+1 and N I encountered.

I can easily fix this in the getdata module that reads and writes blocks from the Linux application over the bridge. I reshuffle the bytes so that the disk contents make sense from an IBM 1130, 16 bit big endian perspective. 

Next I discovered that my understanding of how to address the H2F LW bridge was flawed. Initially I intended that I would only send one 32 bit command word and retrieve one 32 bit status word, thus could ignore addresses. Lately I expanded it to allow me to interrogate up to three 32 bit values from the FPGA in addition to the status word, simply by using the associated address for my fetch or store in the Linux application. 

The addressing can be either symbol or word oriented, plus a given number of bits of address which constrains the maximum range of addressability. For the command/status channel of H2F LW bridge, I wasn't using the correct addresses thus when I tried to fetch my current RAM address from the second word of the channel I didn't get the intended value returned.

Once I got to full speed on this, I chose 4 address bits (1 of 16 addresses) with the address specifying the byte. My bridge channel transfers 32 bits (four bytes) at a time thus the relevant addresses for my intended data are b0000 to b0011 for the status word, b0100 to b0111 for the RAM address word, b1000 to b1011 for the cylinder address of the drive emulation, and b1100 to b1111 reserved for a future fourth word. 

I chose the Cylinder value based on a reader's suggestion that I consider showing the arm movement graphically while the drive is in operation. I am still not convinced this will be worth the effort, considering that the box will be hidden on the side of the machine inside its covers, but if I have spare time and can implement it economically, I will have the relevant data at hand in the Linux application. 

Battle with toolchain software part 15 - Linux now starting with modern loader and U-boot

LINUX BOOTS AT LAST!

As I suspected, relocating the loading points of the kernel and the device tree blob resulted in a normal bootup of the Linux image. At this point I expect that I will have workable bridges, but that is the next thing to check. There was a lot of minor tweaking of U-boot environmental variables as well, to get the card to the point where it autoboots successfully. 

I will have to add a command to have Linux mount the FAT partition automatically as that is where we store the virtual cartridge images and bitmaps for the project, but I mounted it manually and did a quick test. The user interface worked correctly on the LCD screen although I still need to debug whether my bridges and my logic is working properly.