I chose a provocative title, judging by the general consensus shared over time here on the forum... Slightly playing "devil's advocate", I thought I'd share some thoughts (my own and entirely open to critique!) about our legacy QDOS filesystem - such as it is.
To be precise, I'll be focusing on the so-called "Level-2" directory-device driver enhancement that has become standard since the early days of FLP expansions on the QL.
Some background info and context to set the scene - many of you know this stuff far better than I, but judging by some recent posts, it's not necessarily common knowledge.
- Prior to the Level-2 (L2) standard, we had the simple QDOS directory driver for our humble MDV - capable of recognising one 'root' directory, with each logical file represented by an entry in this directory - itself a simple file with a specific internal record-structure, but assigned a unique file-number (0). We call each such record a 'file-header'.
- Probably for reasons of support for file-repair, this same* file-header is then (transparently) duplicated inside the very first block of the file itself. *The file's copy of the header is not always maintained in sync with the directory copy, it seems to depend upon the version of the respective device driver/OS.
- Fundamentally, this basic directory record structure remains intact today, albeit supplemented with some compatible refinements (file-date stamping, file-versions, etc.)
- With the advent of TK2, we got 'soft-directories' - an overlay on top of either legacy QDOS or the emerging L2 directory-device driver.
- As well as a couple of enhancements to the MDV driver, TK2's soft-directories made traversing the directory 'tree' (real or virtual) more straightforward and, more significantly, allowed 'default directories' to reduce the typing needed to specify a given file (or more generally, a group of files sharing the same prefix to their respective filenames.) We might refer to these as default 'paths' - a term more commonly used elsewhere.
- The L2 device driver finally introduced 'hard-subdirectories' as well as some other extended capabilities to support the growing size of available media.
- To be quite clear, TK2's default directories and the L2 driver standard are independent entities - and cause for confusion, where it's entirely possible for a system to have TK2 but no L2 device drivers loaded - whereas TK2 is typically already present wherever an L2 device driver is available.
- Furthermore, the L2 capabilities offered by any given device driver are dependent upon what the driver developer chose to include and not an OS wide-capability - thankfully, all the L2 devices I've used seem to adhere to the same spec. The point being that, the presence of an L2 device in a system does not mean that any other directory device magically becomes L2 capable - e.g. the L2 FLP device on an SGC-equipped QL does not make the MDV device L2-capable (perfectly possible BTW, but of limited practical facility..)
- In practical terms, the difference can be determined by the presence or lack of the (F)MAKE_DIR command and how the target device driver responds to the associated L2 TRAP code.
- Coming back to the common directory-file structure, the 64-byte per-file record allows for the usual file properties (length, type, datestamp, etc) plus the all important 36-character 'filename'. On the surface, fairly generous for it's time - after-all, DOS and its brethren still had their hideous "10.3" filename limit and it was some time before 'Long Filenames' (LFNs) came envogue. But...
- It just so happens that the original QDOS designer(s) chose an unusual paradigm for the QDOS filename that differs from other filesystems - namely, that the entire file-path plus filename is always stored complete in that precious 36-character directory field.
- From a system or application designer's point of view, I would argue that this approach of storing the 'fully-qualified filename' actually makes coding much easier: What size array do I need to DIM to hold a filename and it's path? How much common-heap do I need to store an arbitrary DATA_USE path? - OK, that one still seems to have been f'bared along the way
- From a user's perspective however, the result is that, after two or three 'levels' of sub-directory, you rapidly run-out of storage space to record the actual filename (which I here use to refer to just the 'last-bit' of the fully-qaulified filename plus any 'filename-extension'. Here lieth, to my mind, the only limitation that really impacts my day-to-day usage...
- Another attribute of the QDOS directory design is that EXECutable files have an extra datum stored just for them - namely, the Jobs's 'Dataspace' - the cause of the other principal painpoint for all of us - not just newcomers.
OK, so, to my mind (as representative of perhaps 90% of QL users?), we only have left two 'real' issues with our L2-enabled filesystems:
1. Nested sub-directories gobble-up our once generous 36-character filename limit. We end up either limiting our use of nested directories (a pity), using forgettably short names for the 'sub-directory' parts where we do use them, and still having to limit our creativity in naming our files.
[Aside: As a sys-admin for a secondary-school at the time, I cursed loudly when M$ decided that their new version of Word should take the first line of your new document as a default filename to save under. That was taking LFNs too far and regularly caused my Unix box running SAMBA to complain...]
- Technically, this would appear to pose the most trouble, and any future solution that opts to stay with our QDOS/L2 filesystem would need some serious overhaul. However, I've been thinking about this for some time and it does seem doable with a little 'support' from the OS - I'll update this thread if I feel moved to do so, after a bit more head-scratching. Oh, and nothing quite as sophisticated as guru Nasta's whitepaper on 'meta-drivers', mind-you - well worth a read, if you can still find it online...
2. Those darn 'missing fileheaders' that have rarely any impact and almost no utility until I want to EX(EC) a file that I unzipped in Windows or copied from a DOS device. How silly of me...
- Now, I would argue that this most painful obstacle to our QL file-management is not in fact a limitation of the filesystem at all - or at least, it would not require anything as troublesome as a driver rewrite to address quite readily. There have been some creative approaches to working around this issue over time - but I think it could be achieved far more simply, when we look more closely. I'll outline what I mean here for someone more cleverer than I to pull to pieces (shouldn't take too long, knowing the caliber of intellect that frequent these parts...)
As far as I'm aware, the only part of QDOS that actually cares about the 'Filetype' flag being '1' and the associated 'Dataspace' field when attempting to load and start a new Job is the S*BASIC interpreter as well as TK2's grown-up versions of EX/EW et-al. (OK, HOTKEY's multifarious alternatives, too). My point being that this is all well within the realm of the our extensible Toolkit capability. Hold that thought...
Furthermore, only our Linker applications care about File-type 2 (and how many Linkers are there out there? I only count two of any substance...)
Oh, OK. Then there's those file-types used in some graphics apps. But these are still apps, and not QDOS...
Returning to EX/EW etc. I propose a very simple rewrite of this family of tools that simply accommodates fileimages that do not have Filetype 1 in the fileheader and, in such cases, default the Dataspace allocated - perhaps through a configurable system-wide variable. And/or take an optional parameter to allow the user to specify the Dataspace, if it isn't captured already in the fileheader. We had something similar on the Spectrum - you could over-ride the start-address for a loaded CODE file versus what was saved in it's equivalent 9-byte header...
So, in summary, whilst there is much going for our modern, journal-based, snapshot capable, resilliant and distributed FS's (ZFS is my current best-friend), I believe there is life left yet in our humble QDOS/L2 filesystem, with just a bit if careful (read 'backwards-compatible') tweaking and creativity at the user-interface level.
Now, who's first to shoot-down the above arguments?