Let's chat about: Filing Systems!

[Dave]

This is an enquiry about the current state of play with the filing system on the QL, and the realistic or unrealistic prospects for an improved FS.

One of the issues with Qdos is it uses a block filing system. As the system grows beyond anything Sinclair imagined in 1983, the blocks consume more and more memory. SMSQ/E limits this somewhat.

So, in a realistic fantasy world, if there were a replacement what might it be? Something standard like FAT, which would improve portability but not be a natural fit? Or something new and different, yet still compatible with Qdos?

What are the issues involved?

[Dave]

("Let's chat about?!" is a series of posts brought to you by people who don't know any better, in an attempt to get a few knowledgeable people to comment and talk about something in one place.)

[janbredenbeek]

This is an enquiry about the current state of play with the filing system on the QL, and the realistic or unrealistic prospects for an improved FS.
One of the issues with Qdos is it uses a block filing system. As the system grows beyond anything Sinclair imagined in 1983, the blocks consume more and more memory. SMSQ/E limits this somewhat.

I assume you mean the Slave Block system? These do not consume memory but simply make use of memory not otherwise used by the system for buffering data. If more memory is needed by the system, any slave blocks affected are simply reclaimed and written back to storage if necessary.
The problem arises when RAM gets so big that the table used to keep track of slave blocks grows to unmanageable sizes - 16MB RAM means 32512 slave blocks, which have to be searched every time a new block has to be loaded.

So, in a realistic fantasy world, if there were a replacement what might it be? Something standard like FAT, which would improve portability but not be a natural fit? Or something new and different, yet still compatible with Qdos?

What are the issues involved?

One problem are 16-bit block counters - with standard 512 byte blocks this limits file sizes to 16MB. Also, linear searching through FATs and slave block tables is not very efficient, we need more sophisticated algorithms. And last but not least, the 36-character limit of filenames (which includes ALL subdirectories!) is a bottleneck - but extending this will probably break many applications

Jan.

[Dave]

I assume you mean the Slave Block system? These do not consume memory but simply make use of memory not otherwise used by the system for buffering data. If more memory is needed by the system, any slave blocks affected are simply reclaimed and written back to storage if necessary.

I mean the plurality of issues. The filing system was developed in 1983, and it has not aged well. It is generally restrictive, and limits the size, utility and flexibility of how we use storage devices. Especially when they're massive, by 80s standards.

It strikes me, with the character limit, that retaining compatibility with unmaintained software can be easily done simply by having both filesystems side by side - old school on one device and new on another.

The problem arises when RAM gets so big that the table used to keep track of slave blocks grows to unmanageable sizes - 16MB RAM means 32512 slave blocks, which have to be searched every time a new block has to be loaded.

What are alternatives to that? I know SMSQ limits the number of blocks, or something like that, but to me that's a half measure that may make things... not worse but swaps one problem for another. I wonder if the blocks could be in a partition on the drive that acts like swap, memory mapped block space? This thread is to document people's thoughts on the whole area, what might work, what might not, what amount of work might be involved, etc.

One problem are 16-bit block counters - with standard 512 byte blocks this limits file sizes to 16MB. Also, linear searching through FATs and slave block tables is not very efficient, we need more sophisticated algorithms. And last but not least, the 36-character limit of filenames (which includes ALL subdirectories!) is a bottleneck - but extending this will probably break many applications

We are definitely reaching the point, with Q68 and Issue 8 and X, where large memory systems will become the norm. Luckily, all these systems will run SMSQ/E, so there's a living OS that can adapt and grow.

Thank you for your thoughtful and informative reply, Jan.