What is really in the header of binary files?

[dilwyn]

Of course. But I think his point was that (a) he only has a 128k BBQL to work with so QL-end software might have been a no-no and (b) he's new to it.

Hence my comment about learning to walk before running, i.e. learn to use the QL first before tackling more difficult things.

[tofro]

To be honest, I'm not particularily happy with any of the work-arounds implemented by emulators and compilers for retaining QDOS file headers:

1. The Q-Emulator (and derivates like uqlx) header sits in front of the binary and doesn't include a "branch-around" its inserted header. People tend to copy (or zip) files from the native file system outside the emulator, and that renders the executable invalid. If you unpack such a file on a QDOS system, it will still have no header, some people try to fix it, but then the inserted header gets in the way, which makes it even harder to fix than a "simple" lost header. In hindsight, it would probably have been better to include a branch to the original job start into the header, and thus make such a file in its plain form executable on a native QDOS system as well - even if that would rather hide the problem than solve it. That would, in fact, destroy the QDOS executable header information and mangle the job name display, but that is maybe a minor problem.
2. QPC's former(?) way of encoding the needed data space in the filename extension ("quill_ex8") tend(s/ed?) to work somehow, but is way too easy to break. I also think that support was dropped in favor of Q-Emulator headers in v4 (Me not knowing that probably shows how frequently I used that feature...)
3. xtcc's way of doing things (with a trailer) is fine because it tends to sit at the end of the file and doesn't get in the way when executing such a header-induced file. It has, however, never really been adopted widely, and there really is not a lot of support outside of the xtcc universe (and outside programs compiled with xtcc). It's also hard to explain to a newbie why an xtcc-compiled binary will happily run when it was previously mangled through zip and unzip and not when it was not, and non-xtcc compiled software will simply not run whatever you do.

I know I might upset quite some developers, because they were simply trying to be helpful to newbie end-users, but I think all of these different work-arounds have made the problem even more tricky than it used to be and the various ways of doing things now are more a source of utter confusion than helpful. I am aware you will run into problems whan trying to unpack stuff on a 128k system as a beginner, but, apart from that, I still stand to my statement "unzip on a QL". 128k users can use the ALCHP/SEXEC fix. Maybe I should develop the good habit to somehow include some textual hint at the beginning of new binaries like "SEXEC with 4k data space" after the job start header that could be simply viewed (I tend to hide the version information already there after I found EasyPtr and some QJump programs seem to do that).

[Peter]

[*]xtcc's way of doing things (with a trailer) is fine because it tends to sit at the end of the file and doesn't get in the way when executing such a header-induced file. It has, however, never really been adopted widely, [...]

I think this is the way to go! It is used by UQLX, XTC68, QDOS-GCC, FreePascal and increasingly supported by newer software, e.g. by SMSQemulator and TFTP for the Q68. It is widely known.

I wonder if transparent support for native SD card drivers could be established. In that case, we'd have most machines covered quite well.

[ql_freak]

 
I think Dilwyn's answer is one of the best:

fl = (length of executable program in bytes)
base=respr(fl)
LBYTES mdv1_no_header_prog_exe,base
dataspace_value = (your best guess if you don't know it)
SEXEC mdv1_fixed_exe,base,fl,dataspace_value

If you make dataspace_value too small, the program won't have enough space for its data and will probably complain with an error message. If you make it too big, the program might work, but the too-large value is wasteful of memory, especially on a 128K machine.

There's one problem, the first line: fl = (length of... bytes).

On an unexpanded (no Toolkits) QL there is (if I remember correctly) no way to determine the length of a file :‑( One solution could be:

Format a blank microdrive, and get the number of free sectors on it (with DIR command). Afterwards copy the executable file to it and check the free sectors. The difference*512 is the size (for safety add 512 to it).

Because of the dataspace: The SuperBASIC Compilers had a program to adjust it, because not even the programmer does know, how much dataspace is necessary. Assume the following SuperBASIC program:

100 REMark Define an array with size specified by INPUT (from user):
110:
120 INPUT "Enter number of elements (floating point numbers):", size
120 DIM elems(size)
130 ...

If you enter 10 for size, you won't have any problems if you compile this program and use the default dataspace the compiler is using. If you enter 1000000 for size, the compiled program won't run/will crash. In this case the user can use the supplied program (for SuperCharge/Turbo it was IMHO DATASPACE_EXE, which was allowed to give away with your [commercial] program[!]) to increase the dataspace. I.e. not even the author of a program does know the dataspace a user requires.

The first (real K&R) C Compiler (and even the first versions of C68!) for QDOS (Metacomco aka Lattice C) is using a similar method. When starting the program, you can give a command line argument, which adjusts the available memory for data. Newer versions of C68 expand the memory dynamically (using the QDOS heap, which can lead to heap fragmentation [very problematic on a multitasking operating system without support {and hardware} for MMU {memory management unit aka virtual memory}]).

[dilwyn]

A simple (but slow) way to work out the file length on a QL without toolkit extensions is to open the file and keep reading bytes from it until you run into EOF for that channel, incrementing a counter for every byte read. If used to help with restoring file headers, it will only be run once for each file so the slowness shouldn't be too much of a problem.

Code: Select all

1000 DEFine FuNction FileLength (filename$)
1010   LOCal fl,loop
1020   fl = 0 : REMark file length counter
1030   OPEN_IN #3,filename$
1040   REPeat loop
1050     IF EOF(#3) THEN EXIT loop
1060     loop = CODE(INKEY$(#3))
1070     fl = fl+1
1080   END REPeat loop
1090   CLOSE #3
1100   RETurn fl
1110 END DEFine FileLength

Not really tested it (I typed it from memory, so apologies if I made any typing mistakes). If I got it right, PRINT FileLength ('filename') should slowly return the length of the given filename.

(Should be possible to test it by comparing with PRINT FLEN(\filename$) for anyone with Toolkit 2 extensions)

Uses channel #3 - change that if already used in your program.

Should work with all types of files for which EOF(#channel) works.

[Peter]

On an unexpanded (no Toolkits) QL there is (if I remember correctly) no way to determine the length of a file :‑(

Isn't this a pretty academic problem?
The reason for the lost dataspace info was that the file came from a PC, where you easily see the length.
And in the worst case of no PC involved and the program being too long to also load a toolkit, the length could be determined with a toolkit prior to actually loading it to memory, then reboot.

[ql_freak]

Hi Peter,

think I should join t.lienhard forum.

toolkit prior to actually loading it to memory, then reboot.

NEVER EVER!

The QL is a (was the first affordable) Multitasking computer. You normally shouldn't need to reboot a multitasking system. I have had running my QL (in my days) often nearly a week without rebooting AND thereby also writing programs in (Lattice-) C or (and) Assembler.

[tofro]

Hi Peter,

think I should join t.lienhard forum.

toolkit prior to actually loading it to memory, then reboot.

NEVER EVER!

The QL is a (was the first affordable) Multitasking computer. You normally shouldn't need to reboot a multitasking system. I have had running my QL (in my days) often nearly a week without rebooting AND thereby also writing programs in (Lattice-) C or (and) Assembler.

Actually, nothing on the Multi-Tasking box says "don't reboot" Especially on the non-expanded QL we're talking about, multi-tasking will not be very much "multi" due to memory restrictions.

[Derek_Stewart]

HI,

This QL Emulator file header attachment is a real mess, I do not store QL executable files in a non-QL environment and extract all compressed files into a QWA or QUB container file. I rarely have a problem with files headers being missing.

I know how the QDOS File Header system works and can add a new header to file that has been extracted on non-QL environment, but I do not know how to acuarely calculate the data space for the SEXEC command. Too large a data space causes problems with a QL with Low memory, which is what Peter meant by rebooting the QL, to regain the memory, if Toolkit 2 is not loaded.

On a QL Emulator which stores it files with an attached QDOS file header, seems not to be standard, Q-emulator, adds it to the start of the file, the XTcc can be added to the start or the end. Also if a QL executable witha QDOS file header has been compressed into a ZIP file and unzipped in a QL environment, the QDOS file header still exists and the executable will not run.

All very confusing, so I make life simple and use QL environments for all files, also backing a single QWA file is easier than backing up hundreds of files.

[stephen_usher]

Derek, if you are cross compiling and importing into a QDOS environment then you have no choice.

My own workflow involves using SMSQmulator which has a virtual drive watching a native system directory. I copy the executable there and then create the resource fork (as it would be known on a Mac) to give the memory size. I then mount either a Microdrive image or a floppy image and copy the file onto there. The image can then be used in a vDrive or a Gotek on a real machine.