No fun funzipping

Anything QL Software or Programming Related.
Post Reply
User avatar
pjw
QL Wafer Drive
Posts: 1286
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway
Contact:

No fun funzipping

Post by pjw »

In some projects I use zip/funzip to archive certain material on the fly via pipes. But what takes zip about a second to do, it takes funzip 10 seconds to undo! The same material can be unzipped with the normal unzip in about a second or two. Im using the latest known versions of Jonathan Hudson's zip suit V2.3 for QDOS/SMSQ/E.

So my questions are: Why? Can I fix it? Can anyone else fix it? Is there something else I could use that does more or less the same? (I dont need individual files, search, encryption etc, just a pure stream of data going in, being compressed, and coming out - and the reverse, of course). Is there a way of fooling unzip to do the same - without the use of intermediary files?

TIA


Per
dont be happy. worry
- ?
User avatar
tofro
Font of All Knowledge
Posts: 2685
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: No fun funzipping

Post by tofro »

Per,

did you try "unzip -p" instead of "funzip" (which should basically do the same thing?

Tobias


ʎɐqǝ ɯoɹɟ ǝq oʇ ƃuᴉoƃ ʇou sᴉ pɹɐoqʎǝʞ ʇxǝu ʎɯ 'ɹɐǝp ɥO
User avatar
pjw
QL Wafer Drive
Posts: 1286
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway
Contact:

Re: No fun funzipping

Post by pjw »

tofro wrote:did you try "unzip -p" instead of "funzip" (which should basically do the same thing?
I was discouraged, as the manual stated "Archives read from standard input are not yet supported,
except with funzip (and then only the first member of the archive can be extracted)."

I did try anyway, but no arrangement of parameters or (pre-opened) channels produced the desired
effect:

Code: Select all

500 id = FEX_M('win2_arc_zip_funzip', fnm$, 'pipe_in_2048')
510 cpi = FOP_IN("pipe_in")
..
BTW: My unzip version was probably 5.2


Per
dont be happy. worry
- ?
User avatar
pjw
QL Wafer Drive
Posts: 1286
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway
Contact:

Re: No fun funzipping

Post by pjw »

Ok, over to plan B: Does anyone have a current email address for Jonathan Hudson? The last contact I had with him was in 1998, and that address no longer works. Please PM it to me if you can.


Per
dont be happy. worry
- ?
User avatar
XorA
Site Admin
Posts: 1358
Joined: Thu Jun 02, 2011 11:31 am
Location: Shotts, North Lanarkshire, Scotland, UK

Re: No fun funzipping

Post by XorA »

Is this a streaming speed issue, because zip algorithm needs to read the end of archive first!


User avatar
pjw
QL Wafer Drive
Posts: 1286
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway
Contact:

Re: No fun funzipping

Post by pjw »

XorA wrote:Is this a streaming speed issue, because zip algorithm needs to read the end of archive first!
You cracked it! :) Thanks a lot!
Increasing the pipe size to 32768 reduced the time to a fraction, even for archives many times this size. (I tried pipe sizes beyond that but it didnt appear to make any difference..)


Per
dont be happy. worry
- ?
User avatar
mk79
QL Wafer Drive
Posts: 1349
Joined: Sun Feb 02, 2014 10:54 am
Location: Esslingen/Germany
Contact:

Re: No fun funzipping

Post by mk79 »

Have a look at bzcat/bzip2, it's specifically for compressing/decompressing streams of data. I've used it to stream huge amounts using SBasic before.


User avatar
pjw
QL Wafer Drive
Posts: 1286
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway
Contact:

Re: No fun funzipping

Post by pjw »

mk79 wrote:Have a look at bzcat/bzip2, it's specifically for compressing/decompressing streams of data. I've used it to stream huge amounts using SBasic before.
Thanks for the tip :)
I checked out the version currently on Dilwyn's site. This is what I found:
Sample: 64 files @ 32792b == 2Mb to be squashed into a single archive, tmp$
:
bzip2:

Code: Select all

..
cpo = FOPEN("pipe_out_2048")
id = FEX_M('win2_arc_bzp_bzip2', 'pipe_out', 'nul', tmp$)
<stuff data into pipe>
..
and
..
id = FEX_M('win2_arc_bzp_bzcat', tmp$, 'nul', 'pipe_in_32768')
<get data out of pipe>
..
pipe_out 32k, zip t= 17s, unzip pipe_in 32k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 32k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 10k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 5k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 2k, t >5min or choked

Compressed size: 183440

zip:

Code: Select all

..
cpo = FOPEN("pipe_out_2048")
id = FEX_M('win2_arc_zip_exe', 'pipe_out', 'nul', 'ram1_tst_zip')
..
and
..
id = FEX_M('win2_arc_zip_funzip', tmp$, 'nul', 'pipe_in_32768')
..
pipe_out 32k, zip t= 2s, unzip pipe_in 32k, t 2s
pipe_out 2k, zip t= 2s, unzip pipe_in 32k, t 2s
pipe_out 2k, zip t= 2s, unzip pipe_in 10k, t 4s
pipe_out 2k, zip t= 2s, unzip pipe_in 5k, t 10s

Compressed size: 176478

183440 - 176478 = 6962 in zip's favour

Conclusion: For the kind of data I tested, zip compresses faster and meaner,
but is slow at decompression. bzip is super fast at decompressing. Sadly bzcat
doesnt decompress zipped files!

Different settings may give different results! If you or anyone knows how any of
this may be improved, please post it here! For now Ill stick with zip/funzip


Per
dont be happy. worry
- ?
Post Reply