Converting software from Q-emuLator to QLAY

Discussion and advice about emulating the QL on other machines.
User avatar
dilwyn
Mr QL
Posts: 2761
Joined: Wed Dec 01, 2010 10:39 pm

Re: Converting software from Q-emuLator to QLAY

Post by dilwyn »

Dave wrote:The flip side, is those other OS do a much better job of making the file format differences transparent. When you consider that the QL came first, and that it did not get traction in the marketplace so didn't set any standards, it's easy to see how we ended here.

All it would take is a concerted effort to have provision in SMSQ/E and Minerva to check the position of data space in executables, then repositioning it in one standard place, and then we wouldn't have a problem.

Especially because it's such a trivial problem, but it causes such severe problems for returning users when they are least tolerant to challenges.
Err, well, the problem with "check the position of data space in executables" is that the dataspace isn't physically in the executable, it's stored in a completely separate place on the disk - the header entry in the directory. This is why QXL.WIN came about - DOS/Windows drives can't replicate this for example. Some emulators get around this by actually extending the file by 64 or 128 bytes to be able to get the header into it, this is what QemuLator does and is the reason why you have to demodify a Qemulator executable for it to work on other systems.

"Concerted effort to..." - excuse me for being the sceptic, we've been here so many times already haven't we?


User avatar
tofro
Font of All Knowledge
Posts: 2702
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: Converting software from Q-emuLator to QLAY

Post by tofro »

Second that.

Even a "concerted effort" wouldn't help much with putting some information somewhere where there's no room for it or picking it from somewhere where it isn't ;)

zip is in my opinion still the best vehicle to exchange "complete" QDOS files between different worlds. And the best a "concerted effort" could possibly hope to achieve would be something remarkably similar.

Tobias

PS: The Mac has very similar problems with its resource fork. It does work arond this on vfat file systems by creating hidden files (that tend to annoy every non-Mac user) containing the information that would live in there on native Apple file systems. That approach does create very much the same problems when a volume is created on a Mac, copied by a Windows PC to some other volume and then re-opened on the Mac.

The QL could probably use a similar approach with hidden files containing the headers using an OS extension - But that wouldn't be of any help for "new" and unexperienced QLers with a non-modified QDOS.


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
User avatar
dilwyn
Mr QL
Posts: 2761
Joined: Wed Dec 01, 2010 10:39 pm

Re: Converting software from Q-emuLator to QLAY

Post by dilwyn »

tofro wrote:Second that.

Even a "concerted effort" wouldn't help much with putting some information somewhere where there's no room for it or picking it from somewhere where it isn't ;)

zip is in my opinion still the best vehicle to exchange "complete" QDOS files between different worlds. And the best a "concerted effort" could possibly hope to achieve would be something remarkably similar.

Tobias
Indeed.

I suppose a universal copier type of program could do a QemuLator and add a file header to the start or end of the file and makes it a type 0 datafile for transfer purposes (e.g. using a command called ENCOPY) if it knows the file type and if it spots it's a file type already with that on, it strips it when it knows where the file is to go to.
In this case, Quill, say, becomes standard length+64 (or plus a few more bytes of 'flag' if you want) and gets copied in modified form. Try executing that, it's a type 0 data file but has its datapace appended. A corresponding command called DECOPY simply strips the extra bytes and saves it as an executable.

As an aside, it might be possible, when (if?) we get a QL email program, to include something like this as an X- facility in the mail headers for direct emailing of QL executables (and viruses :oops: ?)

I might call it by a posh title like QL Intermediate File Format - QLIFF ;)

Downside 1, any computers using this will need the extensions installed (easy enough, make it freeware or an SMSQ/E module or something).
Downside 2, the extensions need to know every file format
Upside 1, universal
Upside 2, probably easy enough to code even in BASIC! I'll have a go tonight if I can get hold of enough file format information to come up with something to trial the principle and provide proof of concept.

If anyone knows why it wouldn't work and can save me the time tinkering...


User avatar
tofro
Font of All Knowledge
Posts: 2702
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: Converting software from Q-emuLator to QLAY

Post by tofro »

Dilwyn,

see no reason why that wouldn't work.

But would it indeed be simpler to explain to a QL-newbie than zip? I guess not, at least not much.
Would it solve the problem that unpacking a QDOS-zip on a PC drops the header information? I guess not.
Would it solve the problem that copying (instead of ENCOPYing) a QDOS file to a non-QDOS media would drop the header? No.
Would it allow files of that type to be extractable on a PC (maybe to read READMEs like you can today with zip) ? No. You had to zip the package up anyways.

As long as you can copy a QDOS file to a non-QDOS media and vice versa, the problem is going to remain.

Don't want to discourage you, but my personal opinion is that adding a new method to exchange files is only prone to add even more confusion.

What you could probably do is add a piece of code in front of every executable that jumps over a specific signature like the config blocks have plus a 64-byte header area that contains the header as it would be on a QL media. Then modify the OS to read the header from there instead from the filesystem in case it doesn't look right. You could even build executables that detect they have been LRESPRed and re-create their header information that way - But that wouldn't help new QL users with unmodified machines in any way. Would add even more confusion and have files floating around in different formats - Not exactly a desirable situation.

My opinion stands: The header fact needs to be explained to every new QL user anyways.
Zip/Unzip needs to be explained to new users as well.
I am pretty sure adding another mechanism (and piece of software needed) that needs explanation wouldn't help much....

Regards,
Tobias


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
RWAP
RWAP Master
Posts: 2839
Joined: Sun Nov 28, 2010 4:51 pm
Location: Stone, United Kingdom
Contact:

Re: Converting software from Q-emuLator to QLAY

Post by RWAP »

Actually, it was not until the weekend that I realised that the qxlwin explorer (on the emulators CD) can extract files suitable for use with either QLAY or q-emulator - creating the necessary windows directories.

It is just unfortunate that there are limitations with this program:
a) You can only select one file at a time
b) As I discovered, it will not extract the files correctly if there is a space at the end of the filename.

The qxl.win format seems to be one of the most widely supported on windows based machines - particularly with the qxlwin explorer. It is unfortunate that q-emulator does not currently properly support level-2 directories.

We also have to think about support for mdv images - bearing in mind the forthcoming SD microdrive emulator. Daniele Terdina wrote a short program to make an image of a microdrive which can then be attached to Q-emuLator, and this will presumably be the same format as that used by the SD microdrive emulator (which connects to the QL's microdrive port). QLAY and QL2K both also support .mdv files, but I don't know if these are the same as these images....

Overall I guess it is always going to be horses for courses.

The main issue remains how people can move data between different emulators and different hardware - at the moment, floppy disk seems the best alternative (or the HxC floppy disk drive emulator). The chances of getting new file formats / support added to the older emulators and hardware is minimal.


RWAP
RWAP Master
Posts: 2839
Joined: Sun Nov 28, 2010 4:51 pm
Location: Stone, United Kingdom
Contact:

Re: Converting software from Q-emuLator to QLAY

Post by RWAP »

As for QL Intermediate File Format - I could see a purpose for this but there is the danger of re-inventing the wheel, or ending up with at least 3 different standards.

Ideally files with anything other than filetype 0 (data) would begin with:
A Jump to the code past the QL-iFF Header - then the files can be lrespr'd on any system without a problem (except it may cause issues with boot files which assume the length of the code).
An identification string (QL-IFF-1)
The 64 bytes file header

Q-emulator already does this in part - so that the device access routines can spot that it needs to create the file header to extract dataspace etc... I think it lacks the jump to the code past the header...

QL Unzip would also need to be amended to recognise this header - I am sure QL Unzip already does something similar, but is its method of encoding files the same as q-emulator (I doubt it)....


User avatar
tofro
Font of All Knowledge
Posts: 2702
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: Converting software from Q-emuLator to QLAY

Post by tofro »

I was maybe a bit too simplistic with the header put in front of the binary - There's also some registers that need tweaking to take account of the extended length, otherwise programs that modify their own code won't be able to do that - C68-compiled code, for example, would need to be tweaked that way in order to relocate properly. Pure position-independant code would work without having the initial register values modified, I guess.

Tobias


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
User avatar
dilwyn
Mr QL
Posts: 2761
Joined: Wed Dec 01, 2010 10:39 pm

Re: Converting software from Q-emuLator to QLAY

Post by dilwyn »

tofro wrote:Dilwyn,

see no reason why that wouldn't work.
(snip)
Don't want to discourage you, but my personal opinion is that adding a new method to exchange files is only prone to add even more confusion.

My opinion stands: The header fact needs to be explained to every new QL user anyways.
Zip/Unzip needs to be explained to new users as well.
I am pretty sure adding another mechanism (and piece of software needed) that needs explanation wouldn't help much....

Regards,
Tobias
Agree 100%.

But when every query involving file transfers starts with "I unzipped it to my windows hard disk, copied the file to the ql, then it doesn't work, just says bad parameter" and my reply is "you should have used ql unzip like I said" and the answer is "I didn't know there was a difference" and you've said that for the tenth time that week, the message isn't getting through and I end up feeling I'm wasting my time and energy.

Anyway, while in 100% agreement with you, I did said tinkering and came up with this simply encopier program in BASIC. To see it working, just put a few files in ram1_ (some executables and some not) and run this, tell it to copy from ram1_ to ram2_.

All the files in ram2_ will now be non-executables but with dataspace preserved in the extra 12 bytes. The12 bytes contain identifier "QLIF" , file type long word and dataspace long word. COuld extend this to 64 byte format if dates etc are to be preserved, this is only a proof of concept.

It would obviously need a more refined copier to allow directories etc to be copied. The only "clever" bit about it is that I realised most file handling commands like COPY on emulators do the conversion of modified executables automatically if you copy files from a native (i.e. Windows etc) to a non-native medium like ram, qxl.win or flp.

Anyway, the point about complexity and explaining yet another complication to a newbie is accepted absolutely. Something like this might be useful to Rich because he could say encopy files from a native qemulator directory to another directory, copy the files in windows to another emulator and just run decopy to do the doings on that computer and finally copy them to wherever they're needed. The non-executables like BASIC and text files aren't modified, the executables are, but carry the extra info with them which is easily undone at the other end.

Anyway, it was fun tinkering with it - here's the code for anyone wanting to build on it.

Code: Select all

100 REMark QLIFF - QL Intermediate File Format for copying and transferring files between systems
110 REMark by Dilwyn Jones 2014
120 REMark needs Toolkit 2
130 :
140 REMark ENCOPY a$ TO b$ to make intermediate file
150 REMark DECOPY b$ TO a$ to convert intermediate to normal EXEC
160 REMark only works for type 1 (EXEC) files, not S-ROFF etc
170 :
180 REMark example QLIFF copier
190 CLS : CLS #0
200 INPUT #0,'Copy from ';from$ : IF from$ = '' THEN STOP
210 INPUT #0,'Copy to   ';dest$ : IF dest$ = '' THEN STOP
220 DIR\ram1_tmp,from$
230 OPEN_IN #3,ram1_tmp
240 INPUT #3,t$ : INPUT #3,t$ : REMark medium name and capacity
250 :
260 REPeat copying
270   IF EOF(#3) THEN EXIT copying
280   INPUT #3,t$
290   Encopy from$&t$ TO dest$&t$
300 END REPeat copying
310 CLOSE #3
320 DELETE ram1_tmp : REMark hide the evidence
330 PRINT #0,'Copying complete'
340 STOP
350 :
10000 DEFine PROCedure Encopy (from$,dest$)
10010   REMark copy executable file to QLIFF encoded format
10020   LOCal fl,ft,fd,base
10030   PRINT 'Encopying ';from$;' to ';dest$ : REMark remove if not required
10040   ft = FTYP(\from$)
10050   IF ft = 0 THEN
10060     COPY from$ TO dest$ : REMark no need for QLIFF, no executable header
10070     RETurn
10080   END IF
10090   :
10100   REMark assume type 1 for now
10110   fd = FDAT(\from$) : REMark dataspace of job
10120   fl = FLEN(\from$) : REMark length of file
10130   :
10140   REMark allocate space in memory
10150   base = ALCHP(fl+12)
10160   IF base < 0 THEN PRINT #0,'Out of memory.' : RETurn : REMark oops
10170   LBYTES from$,base+12
10180   REMark assembler intermediate identifier header
10190   POKE base,CODE('Q')
10200   POKE base+1,CODE('L')
10210   POKE base+2,CODE('I')
10220   POKE base+3,CODE('F')
10230   POKE_L base+4,ft : REMark file type
10240   POKE_L base+8,fd : REMark dataspace
10250   :
10260   REMark save modified file to destination
10270   SBYTES dest$,base,fl+12
10280   :
10290   REMark deallocate memory used
10300   RECHP base
10310 END DEFine Encopy
10320 :
10330 DEFine PROCedure Decopy (from$,dest$)
10340   REMark copy file and remove QLIFF formatting
10350   LOCal fl,ft,fd,base,ch
10360   PRINT 'Decopying ';from$;' to ';dest$ : REMark remove if not required
10370   ft = FTYP(\from$)
10380   IF ft <> 0 THEN
10390     COPY from$ TO dest$ : REMark no decoding needed
10400     RETurn
10410   END IF
10420   :
10430   REMark type 0 but is it a QLIFF file
10440   qliff = 0 : REMark becomes 1 if a QLIFF file
10450   fl = FLEN(\from$)
10460   IF fl > 12 THEN
10470     ch = FOP_IN(from$)
10480     IF ch >= 0 THEN
10490       IF INKEY$(#ch) = 'Q' THEN
10500         IF INKEY$(#ch) = 'L' THEN
10510           IF INKEY$(#ch) = 'I' THEN
10520             IF INKEY$(#ch) = 'F' THEN qliff = 1 : REMark identified as a QLIFF file
10530           END IF
10540         END IF
10550       END IF
10560       CLOSE #ch
10570     END IF
10580   END IF
10590   :
10600   IF qliff = 0 THEN
10610     REMark not a QLIFF file
10620     COPY from$ TO dest$ : REMark no decoding needed
10630     RETurn
10640   END IF
10650   :
10660   REMark allocate space in memory
10670   base = ALCHP(fl)
10680   IF base < 0 THEN PRINT #0,'Out of memory.' : RETurn : REMark oops
10690   LBYTES from$,base
10700   ft = PEEK_L(base+4) : REMark not strictly needed unless you want to double check
10710   fd = PEEK_L(base+8)
10720   SEXEC dest$,base+12,fl-12,fd
10730   :
10740   REMark release memory used
10750   RECHP base
10760 END DEFine Decopy
10770 :
10780 DEFine PROCedure XTV
10790   OUTLN #0,FLIM_W(#0),FLIM_H(#0),0,0
10800   WINDOW FLIM_W(#0),FLIM_H(#0)-52,0,0 : WINDOW #2,FLIM_W(#0),FLIM_H(#0)-52,0,0 : WINDOW #0,FLIM_W(#0),52,0,FLIM_H(#0)-52
10810   BORDER #1,1,255 : BORDER #2,1,255 : BORDER #0,1,255
10820   CLS : CLS #2 : CLS #0
10830 END DEFine XTV
10840 :
10850 DEFine PROCedure Sa
10860   SAVE win1_basic_qliff_bas
10870   QSAVE win1_basic_qliff
10880 END DEFine Sa


User avatar
tofro
Font of All Knowledge
Posts: 2702
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: Converting software from Q-emuLator to QLAY

Post by tofro »

Dilwyn, you're scaring me! From concept to implementation in 30 minutes!

But anyhow: Apple hasn't found a proper solution to that problem in the last 25 years, so I don't think the QL community has much reason to blush here.

Tobias


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
User avatar
Mr_Navigator
QL Fanatic
Posts: 782
Joined: Mon Dec 13, 2010 11:17 pm
Location: UK, Essex
Contact:

Re: Converting software from Q-emuLator to QLAY

Post by Mr_Navigator »

Hey Forum Admin

A sticky somewhere of the QL ZIP and UnZIP post Dilwyn did sometime ago???


-----------------------------------------------------------------------------------
QLick here for the Back 2 the QL Blog http://backtotheql.blogspot.co.uk/
Post Reply