In defence of the QDOS filesystem...

Anything QL Software or Programming Related.
Post Reply
martyn_hill
Aurora
Posts: 909
Joined: Sat Oct 25, 2014 9:53 am

In defence of the QDOS filesystem...

Post by martyn_hill »

Hi everyone!

I chose a provocative title, judging by the general consensus shared over time here on the forum... Slightly playing "devil's advocate", I thought I'd share some thoughts (my own and entirely open to critique!) about our legacy QDOS filesystem - such as it is.

To be precise, I'll be focusing on the so-called "Level-2" directory-device driver enhancement that has become standard since the early days of FLP expansions on the QL.

Some background info and context to set the scene - many of you know this stuff far better than I, but judging by some recent posts, it's not necessarily common knowledge.
  • Prior to the Level-2 (L2) standard, we had the simple QDOS directory driver for our humble MDV - capable of recognising one 'root' directory, with each logical file represented by an entry in this directory - itself a simple file with a specific internal record-structure, but assigned a unique file-number (0). We call each such record a 'file-header'.
  • Probably for reasons of support for file-repair, this same* file-header is then (transparently) duplicated inside the very first block of the file itself. *The file's copy of the header is not always maintained in sync with the directory copy, it seems to depend upon the version of the respective device driver/OS.
  • Fundamentally, this basic directory record structure remains intact today, albeit supplemented with some compatible refinements (file-date stamping, file-versions, etc.)
  • With the advent of TK2, we got 'soft-directories' - an overlay on top of either legacy QDOS or the emerging L2 directory-device driver.
  • As well as a couple of enhancements to the MDV driver, TK2's soft-directories made traversing the directory 'tree' (real or virtual) more straightforward and, more significantly, allowed 'default directories' to reduce the typing needed to specify a given file (or more generally, a group of files sharing the same prefix to their respective filenames.) We might refer to these as default 'paths' - a term more commonly used elsewhere.
  • The L2 device driver finally introduced 'hard-subdirectories' as well as some other extended capabilities to support the growing size of available media.
  • To be quite clear, TK2's default directories and the L2 driver standard are independent entities - and cause for confusion, where it's entirely possible for a system to have TK2 but no L2 device drivers loaded - whereas TK2 is typically already present wherever an L2 device driver is available.
  • Furthermore, the L2 capabilities offered by any given device driver are dependent upon what the driver developer chose to include and not an OS wide-capability - thankfully, all the L2 devices I've used seem to adhere to the same spec. The point being that, the presence of an L2 device in a system does not mean that any other directory device magically becomes L2 capable - e.g. the L2 FLP device on an SGC-equipped QL does not make the MDV device L2-capable (perfectly possible BTW, but of limited practical facility..)
  • In practical terms, the difference can be determined by the presence or lack of the (F)MAKE_DIR command and how the target device driver responds to the associated L2 TRAP code.
  • Coming back to the common directory-file structure, the 64-byte per-file record allows for the usual file properties (length, type, datestamp, etc) plus the all important 36-character 'filename'. On the surface, fairly generous for it's time - after-all, DOS and its brethren still had their hideous "10.3" filename limit and it was some time before 'Long Filenames' (LFNs) came envogue. But...
  • It just so happens that the original QDOS designer(s) chose an unusual paradigm for the QDOS filename that differs from other filesystems - namely, that the entire file-path plus filename is always stored complete in that precious 36-character directory field.
  • From a system or application designer's point of view, I would argue that this approach of storing the 'fully-qualified filename' actually makes coding much easier: What size array do I need to DIM to hold a filename and it's path? How much common-heap do I need to store an arbitrary DATA_USE path? - OK, that one still seems to have been f'bared along the way :-)
  • From a user's perspective however, the result is that, after two or three 'levels' of sub-directory, you rapidly run-out of storage space to record the actual filename (which I here use to refer to just the 'last-bit' of the fully-qaulified filename plus any 'filename-extension'. Here lieth, to my mind, the only limitation that really impacts my day-to-day usage...
  • Another attribute of the QDOS directory design is that EXECutable files have an extra datum stored just for them - namely, the Jobs's 'Dataspace' - the cause of the other principal painpoint for all of us - not just newcomers.
Still with me?

OK, so, to my mind (as representative of perhaps 90% of QL users?), we only have left two 'real' issues with our L2-enabled filesystems:
1. Nested sub-directories gobble-up our once generous 36-character filename limit. We end up either limiting our use of nested directories (a pity), using forgettably short names for the 'sub-directory' parts where we do use them, and still having to limit our creativity in naming our files.
[Aside: As a sys-admin for a secondary-school at the time, I cursed loudly when M$ decided that their new version of Word should take the first line of your new document as a default filename to save under. That was taking LFNs too far and regularly caused my Unix box running SAMBA to complain...]
- Technically, this would appear to pose the most trouble, and any future solution that opts to stay with our QDOS/L2 filesystem would need some serious overhaul. However, I've been thinking about this for some time and it does seem doable with a little 'support' from the OS - I'll update this thread if I feel moved to do so, after a bit more head-scratching. Oh, and nothing quite as sophisticated as guru Nasta's whitepaper on 'meta-drivers', mind-you - well worth a read, if you can still find it online...

2. Those darn 'missing fileheaders' that have rarely any impact and almost no utility until I want to EX(EC) a file that I unzipped in Windows or copied from a DOS device. How silly of me...
- Now, I would argue that this most painful obstacle to our QL file-management is not in fact a limitation of the filesystem at all - or at least, it would not require anything as troublesome as a driver rewrite to address quite readily. There have been some creative approaches to working around this issue over time - but I think it could be achieved far more simply, when we look more closely. I'll outline what I mean here for someone more cleverer than I to pull to pieces (shouldn't take too long, knowing the caliber of intellect that frequent these parts...)

As far as I'm aware, the only part of QDOS that actually cares about the 'Filetype' flag being '1' and the associated 'Dataspace' field when attempting to load and start a new Job is the S*BASIC interpreter as well as TK2's grown-up versions of EX/EW et-al. (OK, HOTKEY's multifarious alternatives, too). My point being that this is all well within the realm of the our extensible Toolkit capability. Hold that thought...

Furthermore, only our Linker applications care about File-type 2 (and how many Linkers are there out there? I only count two of any substance...)

Oh, OK. Then there's those file-types used in some graphics apps. But these are still apps, and not QDOS...

Returning to EX/EW etc. I propose a very simple rewrite of this family of tools that simply accommodates fileimages that do not have Filetype 1 in the fileheader and, in such cases, default the Dataspace allocated - perhaps through a configurable system-wide variable. And/or take an optional parameter to allow the user to specify the Dataspace, if it isn't captured already in the fileheader. We had something similar on the Spectrum - you could over-ride the start-address for a loaded CODE file versus what was saved in it's equivalent 9-byte header...

So, in summary, whilst there is much going for our modern, journal-based, snapshot capable, resilliant and distributed FS's (ZFS is my current best-friend), I believe there is life left yet in our humble QDOS/L2 filesystem, with just a bit if careful (read 'backwards-compatible') tweaking and creativity at the user-interface level.

Now, who's first to shoot-down the above arguments? :-)


User avatar
NormanDunbar
Forum Moderator
Posts: 2251
Joined: Tue Dec 14, 2010 9:04 am
Location: Leeds, West Yorkshire, UK
Contact:

Re: In defence of the QDOS filesystem...

Post by NormanDunbar »

Hi Martin,

Back in 2009 I had a mini rant about the file system over at http://qdosmsq.dunbar-it.co.uk/blog/200 ... le-system/. That still applies.

Re your question of "how much space do I allocate ..." On other OSsses, there is a Defined limit named MAX_PATH which is how much space is required for a path. Mind you, it is still possible in Windoes Explorer, to copy a directory structure to the end of another and still be able to see the files in a directory listing, but try copying one out and you get an error if the file's path exceedes the limit. Weird.

I'm pretty sure that _bas files can be EXEC'd on SMSQ. If I remember correctly, they start an Sbasic daughter job. They are of course type 0.


Cheers,
Norm.


Why do they put lightning conductors on churches?
Author of Arduino Software Internals
Author of Arduino Interrupts

No longer on Twitter, find me on https://mastodon.scot/@NormanDunbar.
User avatar
pjw
QL Wafer Drive
Posts: 1286
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway
Contact:

Re: In defence of the QDOS filesystem...

Post by pjw »

Hi Martin,
Just a wee hint at what we're up against here. The piece below is taken from QL Toady (slightly edited), December 2002 (V7-I4-P21):

One of our great QL luminaries was once out
walking in the woods, when he espied a small
brass lamp lying in the undergrowth. He picks it up
and rubs the dirt off with his sleeve. Somewhat to
his surprise a genie pops out, bowing obsequiously,
"Oh master, in gratitude for liberating me, it is in
my power to grant you a wish. Speak! Thy whim is my
command!" "Hm", quoth said luminary, "I thought it
was customary in such cases to be granted three
wishes, not one." "Alas! lt is the times, lord. What
with 9/11, the current interest rates, etc; even us
spirits have to tighten our belts, One wish it is.
Now what will it be, lord and master?" Our luminary
ponders deeply a moment then, bringing out his pocket
diary, points to a map and says "See, this is the
Middle East. I want you to stop these guys from
fighting and killing each other. I want you to make
peace in the Middle East!" "Good gracious me, my
lord", says genie aghast. "These people have been at
war for decades - centuries! Nay, come to think of it,
millennia! Impossible! Think of something else, I
beg you. How about a shiny new Mercedes 500 SLK? Or a
super-duper new IBM PC?" Our luminary gives him a
withering eye, but then a happy thought strikes him,
"I wish", he says, "that the QL-world would smooth over
their differences, put aside their idiosyncrasies and
extreme individualism - for a little while, at least -
and get together to produce an intelligent, functional
and backwardly compatible file system, allowing for
unlimited directory depth, long file names, and a
sensible solution to the underscore/backslash problem
that we can all agree on. Yes! That is my wish!" The
spirit is silent a moment and then, with a deep sigh
of resignation, says "Lets have a look at that friggin'
map again, eh.


Per
dont be happy. worry
- ?
User avatar
Dave
SandySuperQDave
Posts: 2765
Joined: Sat Jan 22, 2011 6:52 am
Location: Austin, TX
Contact:

Re: In defence of the QDOS filesystem...

Post by Dave »

Now that storage devices are so much faster than they were back then, it would be a huge step forward if the memory consumed by large filing systems could be shunted off to files on the filing system. That by itself would be huge.


User avatar
mk79
QL Wafer Drive
Posts: 1349
Joined: Sun Feb 02, 2014 10:54 am
Location: Esslingen/Germany
Contact:

Re: In defence of the QDOS filesystem...

Post by mk79 »

martyn_hill wrote:- Technically, this would appear to pose the most trouble, and any future solution that opts to stay with our QDOS/L2 filesystem would need some serious overhaul. However, I've been thinking about this for some time and it does seem doable with a little 'support' from the OS - I'll update this thread if I feel moved to do so, after a bit more head-scratching. Oh, and nothing quite as sophisticated as guru Nasta's whitepaper on 'meta-drivers', mind-you - well worth a read, if you can still find it online...
The problem is not the filesystem but the application-facing APIs. If there was an API that could enumerate directories and return bigger filenames AND applications did use these APIs then the rest would have been comparatively easy. But I said the same thing 15 years ago and even then there wasn't much support for it. For some people, no matter what you do, if it's not compatible to AH ROM or whatever it's not worth doing it ;)
As far as I'm aware, the only part of QDOS that actually cares about the 'Filetype' flag being '1' and the associated 'Dataspace' field when attempting to load and start a new Job is the S*BASIC interpreter as well as TK2's grown-up versions of EX/EW et-al. (OK, HOTKEY's multifarious alternatives, too). My point being that this is all well within the realm of the our extensible Toolkit capability. Hold that thought...
Yes, it's just TK2, HK2... and QPAC2, QRAM, CueShell, FileInfo2, Qascade, Make, QDT... ;-) Basically everything that executes something and is not written in Basic.
Returning to EX/EW etc. I propose a very simple rewrite of this family of tools that simply accommodates fileimages that do not have Filetype 1 in the fileheader and, in such cases, default the Dataspace allocated - perhaps through a configurable system-wide variable.
That is downright dangerous. Guessing too big is okay, it's just wasteful, but guessing too small crashes the system! If not immediately then at a later point, which is even worse as it hides the reason for the crash.

The EX call et al could look for the XTCC file epilog, which is already defined and doesn't rely on the filesystem. But as with the directory stuff we do not have an API that abstracts this away, so every single application like QPAC2 would have to be altered along.

Cheers, Marcel


martyn_hill
Aurora
Posts: 909
Joined: Sat Oct 25, 2014 9:53 am

Re: In defense of the QDOS filesystem...

Post by martyn_hill »

Gentlemen, thanks for the feedback - I wasn't sure what response I'd get to opening-up this old can-o-worms...

Thanks Norman for describing so much more succinctly than I (Mr Verbose...) the 36-character issue in your rants blog. Bee-hives, eh? :-)

Thanks Per for adding humour back to what might have been a very dry thread!

Interesting idea, Dave - as well as your thoughts shared elsewhere around a direct online repository accessible through a potential Wifi connection - effectively by-passing the EXecutable header issue altogether.

Marcel - as ever, I'm grateful for the level of consideration and detail you give these half-baked ideas!

I'll re-start by saying that this is just another angle on a well-worn topic. My primary aim was to highlight that (IMHO) the existing QDOS/L2 filesystem is not so very bad. but of course, has these limitations that we have learnt to live with and build-around. And there-in, as Marcel points out, lies much of the challenge.

The 'holes' left behind in the original QL design have, to my mind, been one of the major catalysts behind the sheer creativity and technical genius that we see today across a range of QL software and hardware; were the original design so air-tight and fully implemented in the first instance, the opportunity for novices like myself to identify gaps and explore ways to address the 'problems' (such a First-World conundrum, no?) would have been much reduced. I remain fascinated by the QL/QDOS et-al and admire the characters in our small community involved, principally for this reason. But of course, this means that existing applications have necessarily gone 'DIY' in order to fill the design gaps and would thus be largely incompatible with a new approach.

1. Nested sub-directories gobble-up our once generous 36-character filename limit:
Clearly, the way to address the 'fully-qualified filename' issue is to revert to how other FS designs appear to tackle directory entries - namely to record only the final filename in each 36-character record entry and find an alternative means to be able to remember 'where' in the path the file resides - or more precisely, the path to its parent DIRectory. As we have necessarily found our way 'here' in order to open the file, we could have the OPEN call store the textual path-to-here in the CDB for the opened file.

Meanwhile, outside the CDB, maintaining a 'path-map' of file-numbers representing each interim sub-directory en-route for recently accessed files could help to address the performance hit (and ambiguity of the '_' separator issue) of navigating around the tree (anyone tried 'DUP' when the current default-directory ends in a directory name containing the underscore?)

For sure, the work involved in re-engineering age-old, proven and reliable applications to accommodate an enhanced usage of those precious 36-characters is unlikely to be a practical proposition. My point was rather to highlight that, for most practical applications, the deficiencies of the incumbent design are not so manifold as popular view seems to uphold, but pretty-much down to this single design decision. And therefore, possibly less out of reach than believed - were we (the Royal 'we', of course) of a mind to change things. I'll leave that point hanging for now as I continue to unscramble in my head just what would be involved...

2. Those darn 'missing fileheaders':
Marcel's arguments are of course entirely sound in that, allowing the user to invoke a fileimage with Filetype <anything> as a Job is both inherently 'unsafe' and also limited in scope to being invoked from within the interpreter (command-line, in a boot-file or else via compiled Jobs, using EX/EW et-al.). I put forward the following counter-argument - knowing full well that it has holes of it's own :-)

One of the design-reasons for the Filetype handling in QDOS is surely to avoid random files from blowing-up your QL - or rather your as-yet unsaved Quill document, etc - if incorrectly invoked as a Job. Checking for the XTcc 'tag' post-amble and the standard Job-header pre-amble would probably catch most nasties.

As we are exploring here ways to minimise the 'pain' of the average user who today relies on transferring QDOS EXECutable files via their PC/Mac and an Internet connection in order to play the odd game or try-out some recently downloaded app from Dilwyn's site - or else wishing to enjoy the 'freedom' of working cross-platform from a DOS or other NFA device, we might weigh-up the the 'risk' of mistakingly invoking a non EXECutable file as a Job compared to the utility it affords. Open to debate...

Existing (and currently proposed) approaches all share the requirements that the Job's file-header is already intact, viz:

QEm has one approach (available only to herself), we have the tremendously clever approach of one of our respectable friends to effectively convert an entire QXL.Win filesystem to run from a DOS device as well as QLAY's old-school approach and probably others. Oh, and Marcel's proposal to do clever things when transferring files between QDOS and DOS filesystems in QPC.

But these are either unwieldy for the 'average' user, not transferable and in any case all assume an intact fileheader from the outset, which is the problem we're trying to work-around.

The pragmatist/tinkerer in me would argue that, adding a capability in to the EX/EW toolset to default the missing Dataspace in a non-type-1 file, along perhaps with a 'switch' to enable or disable the feature altogether would be a reasonable balance to address the majority of frustration encountered by the user just wanting to run some games or other apps downloaded via PC/Mac. It of course completely fails to address the file/job/Thing manager approach of invoking a Job, but then is that how our average user is going about it? Not 80% of the time, I would suggest.

Finally, should such a 'dangerous' feature be incorporated in to official versions of SMSQ/E or Minerva? Probably not, but nor would there be any need to - were a (foolish?) soul be prepared to tweak the current EX/EW tools and make available as a separate mini-toolkit, that I believe would suffice. And yes, as I propose it, I guess I should be the first to offer to do the work (after I finish the QLUB adapter project, that is...)

Postscript...
Having just devised a series of methods in SBasic to efficiently manipulate directory-structures via linked lists within simple 2D string Arrays (mostly as a proof of concept for Andrei's terrific little QLC filemanager app), my own view of the QDOS/L2 filesystem was reinvigorated and provoked this thread in an attempt to offer an alternative to the commonly held view that our QDOS filesystem is inherently 'broken' - I think instead that it has a couple of flaws that impact the average user, but otherwise has mileage.

Thanks for reading :-)


User avatar
tofro
Font of All Knowledge
Posts: 2685
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: In defence of the QDOS filesystem...

Post by tofro »

Martyn,

Do you know that QPC2 has (or used to have? I never used it) a feature that allows it to execute headerless executables that have the amount of data space mangled into their filename extension?

This is a very pragmatic solution for people who think they need to have such files...

Tobias


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
martyn_hill
Aurora
Posts: 909
Joined: Sat Oct 25, 2014 9:53 am

Re: In defence of the QDOS filesystem...

Post by martyn_hill »

Hi Tobi - oh yes, that was the other method I had forgotten - thank you!

On reflection, I still 'feel' that modding EX/EW to transparently handle the situation holds some water.

:-)

And, apologies to anyone I've offended with my poor spelling in the topic title - I can't seem to adjust it after the event! Heaven help me if my old English teacher reads this...


User avatar
mk79
QL Wafer Drive
Posts: 1349
Joined: Sun Feb 02, 2014 10:54 am
Location: Esslingen/Germany
Contact:

Re: In defence of the QDOS filesystem...

Post by mk79 »

tofro wrote:Do you know that QPC2 has (or used to have? I never used it) a feature that allows it to execute headerless executables that have the amount of data space mangled into their filename extension?
That's actually a feature of Tony's FAT12 FLP driver and available to all platforms, but only on floppy discs. The extension was then "EX2", "EX3" etc with the number denoting the data space in some increments.

As I mentioned elsewhere QPC2v5 will feature QemuLator style headers. This will still not solve the problem of people unzipping files on the PC side, but everything else is much easier. With the new DOS driver I could actually boot and work without any WIN device.

Cheers, Marcel


Post Reply