SMSEQ and file system

[Derek_Stewart]

Hi,

I have never heard of this 36 character file name length in subdirectories, which was the point of QVFS. As SMSQ/E, QDOS, Minerva uses a maximum of 32 characters plus characters for the device, which also takes into account the subdirectories definition.

For example, a directory of my C68 folder, USING:

DIR WIN1_C68_

gives:

C68_SOURCE ->
C68_LIB ->
C68_INCLUDE ->
C68_DOC ->

For the SOURCE sub directory, this is broken down to:

WIN1_
WIN1_C68_ File Type 255 Hard Directory, File Length: 9
WIN1_C68_SOURCE_ File Type 255 Hard Directory, File Length: 16
WIN1_C68_SOURCE_SQLITE_makefile File Type 0 data file, File Length: 31

Please tell me how to get 36 characters per subdirectory.

I posed the question in the QL-USERS mailing list, see answers there...

[Derek_Stewart]

QL-USERS List Reply from Marcel Kilgus:

Derek Stewart wrote:
> Is it possible to create a new directory device in SMSQEmulator to use
> filenames with a length of greater than 36 characters.

It is probably possible, but what's the point? Next to no QL program
will be able to use it. Sure, you could open a file on it in SBASIC if
you happen to know the exakt name. But a simple "dir" command will not
work and all file managers known to me (e.g. QPAC2) will not like it
either.

Marcel

_______________________________________________
QL-Users Mailing List
http://www.q-v-d.demon.co.uk/smsqe.htm

[Derek_Stewart]

QL-USERS List Reply from Wolfgang Lenerz:

Hi,

I agree with Marcel on this point. If there were new software being
developed, it **might** make sense, but since this isn't the case...

I've already looked into that possibility some time ago.

Basically, and I'm speaking for SMSQ/E here, the OS already checks on
the file name length when opening a file. So the OS would have to be
modified. Then you would need new device drivers. In current filing
system device drivers, each file is reserved an entry in the directory
for that drive / (sub)directory. The file entry is 64 bytes long, 36 of
which are taken for the file name (+2 bytes for the length word).

So the device driver would have to have directory entries longer than
that, probably 256 bytes, 220 of which (roughly) could be used for the
file name. Whilst we're at it, replacing the "_" with something else as
directory marker in a name would probably be a good idea.

I personally would recommend introducing a new trap 2, opening a file
with a long file name so that old programs trying to open a file
"normally" could be told that there is no such file/device etc...

And that's where Marcel is, unfortunately, so right.

Just what would you use this nes device with? Any files manager I know
expects a directory entry to be 64 bytes long, and the name part 36
bytes max. Any software I know would keel over when being fed file names
longer than the 36 bytes it expects (ok, so that should expect filenames
a bit log,er i.e. n2-remote_drive_whatever, but this is just a few bytes
more).


Perhaps one could introduce some kind of compatibility mode where the
drive allocates, aso, a "short" name whenever a long filename file is
opened on it.

Again, what would be the use.

(Supposing a programme being able to do that)You'd create a file with a
nice comprehensible filecname like
"dev1_accounts_bookings_december2015_clientx_order_dated_20151214_ext".

Good, but later when you try to find this file again, your file manager
gives you a name like "dev1_gmblkfhatagveye_ext". Is this any easier?


So, changing the filename length would only make sense if
- the OS is modified
- new device drivers are written
- there is software that can actually make use of it.

Wolfgang

>> Is it possible to create a new directory device in SMSQEmulator to use
>> filenames with a length of greater than 36 characters.
>
> It is probably possible, but what's the point? Next to no QL program
> will be able to use it. Sure, you could open a file on it in SBASIC if
> you happen to know the exakt name. But a simple "dir" command will not
> work and all file managers known to me (e.g. QPAC2) will not like it
> either.
>
> Marcel
>
> _______________________________________________
> QL-Users Mailing List
> http://www.q-v-d.demon.co.uk/smsqe.htm

[XorA]

Please tell me how to get 36 characters per subdirectory.

The actual on disk format for a file

typedef struct PACKED {
uint32_t file_len;
uint8_t file_mode;
uint8_t file_type;
uint32_t data_space;
uint32_t spare;
uint16_t fn_len;
char filename[36];
uint32_t date_update;
uint16_t version;
uint16_t fileno;
uint32_t date_backup;
} DIR_ENTRY;

So every file/directory can have upto 36 characters allocated to its name, its a higher layer in the OS that limits the total file name including directories to 36 characters. I would expect this is because the OS support for directories is a massive hack in QDOS.

G

[dilwyn]

As someone who hasn't got the smallest understanding of the technicalities of the QL file system, I have long accepted the explanations that

(1) existing software would be unlikely to cope with any enhanced filing system
(2) doubtful that any new versions of such software or brand new software to use it would be written in any quantity
(3) we should just accept and live within the existing limitations as a necessary evil.

My personal thoughts on how to work around this (I emphasise I don't have the knowledge to know if this ever stands a chance and was told a while back this could not possibly work and forgotten why) was to use the approach of a device name which was not just a device but added a part of a longer path, in a similar way to how DEV etc translate between names, to hide the new filing system from the old software, e.g. DEV_USE 1,win1_quill_ : DEV_USE 'flp' so that flp1_ is now win1_quill_ almost invisibly to the user.

My idea at the time was that a new device like this could be defined to allow existing software to somehow access the new filing system without running into limits like >36 character directory/filename. Suppose it's called NDV (new device). We want existing software to access a long path and file name such as win1_mydocuments_xchangeversion309m_mypersonaldocuments_letter_doc. This is over 60 characters long in total. So to allow existing software to access this, we could hide part of the path name using an NDV driver.

NDV_USE win1_mydocuments_xchangeversion309m_mypersonaldocuments_

Programs would then load the file simply as NDV1_letter_doc. The operating system quietly expands this to long filename without the user program really knowing about it.

It would be cumbersome to use - you'd have to set it up individually for all the directories you wanted to use, although a little app could be written to take care of that like some of the DEV_xxx setters out there.

Like I said, it has been explained to me before why this wouldn't work in practice but I still like the idea if anyone ever thought of a way of working around it.

The notion of an improved filing system is attractive to all of us I'm sure, but perhaps impractical at this stage in the life of the QL. It's one of those conversation that keeps popping up from time to time and goes round and round in circles. I'll still follow the discussion with interest though

[Derek_Stewart]

Please tell me how to get 36 characters per subdirectory.

The actual on disk format for a file

typedef struct PACKED {
uint32_t file_len;
uint8_t file_mode;
uint8_t file_type;
uint32_t data_space;
uint32_t spare;
uint16_t fn_len;
char filename[36];
uint32_t date_update;
uint16_t version;
uint16_t fileno;
uint32_t date_backup;
} DIR_ENTRY;

So every file/directory can have upto 36 characters allocated to its name, its a higher layer in the OS that limits the total file name including directories to 36 characters. I would expect this is because the OS support for directories is a massive hack in QDOS.

G

Hi,

Looks great, do you have an implementation of the Structure, how can this be used?

I can not see it being QPC2 or SMSQmulator,as this would be a chnage to the operating system...

I like Dilwyn solution by using a NDV_USE idea, which does not change the SMSQ/E, QDOS filing system and keeps backward compatiability.

[NormanDunbar]

Derek,

That is the current directory structure, at least it looks like it. The problem is, and I had a rant on my blog about his a while back, the 36 characters are used up by duplicated info. Assume win1_source_c_test_c where test_c is the filename and everything else is a device or directory.

The top level directory for win1 has "source_c_test_c" in it.
The directory for the source dir also has it.
So does the directory entry for the c directory.
It's a cluster f*ck!

Why all the duplication? All that is needed in the top level is "source" we know its file type is a directory, ao we can open it. In there, all we need is a "c" directory entry, where we can finally find "test_c".

The reason for being limited to 36 characters in a path plus file name is all the duplication.


Cheers,
Norm.

[Derek_Stewart]

Hi,

If 36 characters per subdirectory could be implemented, that would be great.

But what would be the limit on the total file length including the directory tree.

Does Linux EXT4 and MS NTFS not have a limit of 255 bytes/characters, which I assume includes the directories.

According to Wolfgang Lenerz in the QL-USERS list, SMSQ/E (assume also QDOS) expects a 64 bytes filename which includes 32 characters for the file name.

It is looking like a new device driver and Trap 2 handler for new modern QL systems and new software, which will be incompatiabile with older QL systems.

Is this a route that should be taken, because it is then a question of altering the operating system, which in the past has other implications.

[tofro]

All,

Note that it is not only the drivers that assume one single directory file holds not only the name of the affected files but also the complete path.

It's also basically everything in the system that actually looks in there - Mainly S*BASIC commands like DIR and OPEN, but also all system extensions like WSTAT.... (and, obviously, the same thing in other languages...) rely on this redundant information. so all of those would need to be re-written (while retaining their functionality for the "legacy" file system - You dont want a special command set for every file system type....). It all boils down to the fact that QDOS never adopted the concept of a "current" directory like DOS or UNIX have. (PROG_USE and DATA_USE somehow claim to do that, but that's just facade and not really built in).

Even if you did all that, you still have the problem that legacy software would not be able to use the new long filenames. I have, long ago accepted the limits and can, with a bit of caution, very well live with them - Even on GB-sized win images, and much better than with any complicated kludge that would be a constant nuisance anyhow....

I think: It's not worth it

Tobias

[Giorgio Garabello]

I read with interest all your comments .
Surely it is a long and difficult change , but do not agree on some conclusions .
it is true that today there are no file manager capable of reading more than 32 characters long paths ... it could not be otherwise , how do you develop a program based on something that does not exist ?
I know very few file manager , I think that most uses qpac2..e most software accesses files via the menu extension..i sw but change are not that many ...

With regard to command DIR STAT etc etc ... there really someone use it yet ???

Obviously I'm good at talking but are not able to make these modifiche..per where my words are worth very little.

Giorgio