IDCshell (tm) - A DOS/SHELL AND ZIPFILE UTILITY


NETWORK COMPATIBLE


(for revision history, see IDCSHELL.REV)


Documentation for IDCshell.EXE


Written in assembler by Gary Conway and Shawn Merrick


Infinity Design Concepts, Inc.


Louisville, Kentucky


Copyright 1987-90 Infinity Design Concepts, Inc. all rights reserved


Version 4.0











IDCshell.EXE is a user supported shareware product. As such,
you have the opportunity to test drive the program before
deciding whether or not to buy. One of the beauties of the
shareware concept, is this "try before you buy" idea, but
note that this does not relieve you of your obligation to buy
if you do find the program to be of value to you and continue
using it. You are also encouraged to give copies to your
friends for their evaluation, as long as the documentation
and program(s) remain intact as they were released by the
author and as long as you DO NOT distribute copies of
ARCCFG.EXE. IDCshell.EXE is and will remain the property of
Infinity Design Concepts, Inc. This program may not be used
in any connection with commercial ventures, nor as a sales
aid, without the expressed written consent of the author. All
rights are reserved.


Shareware distributors may request permission to include our
shareware software in their catalogs by submitting a letter
of intent to the following address.

Infinity Design Concepts, Inc.
1052 Parkway Drive
Louisville, Kentucky 40217
Voice: (502) 636-1234
Modem: (502) 635-5471

Member IEEE AOPA
PCCL WINGS
NSPE KSPE











REGISTRATION


If you find yourself using IDCshell, please take the time to
do the right thing and that is register your copy. You have
been provided the opportunity to freely test the program
before even thinking about registering. This is only fair,
so, in fairness, you should reciprocate and register your
copy, if you continue using the program.

Why register if you use the program ?

1) You get the ARCCFG.EXE program for customizing IDCshell.
2) You get notification of updates to IDC software.
3) You get FREE net-mail services for contacting IDC.
4) It is the right thing to do.


You will find the registration form in the ARChive with this
document under the name REGISTER.FRM. Please use this form
for registration.


MASTERCARD AND VISA are accepted, but you MUST follow the
instructions in the REGISTER.FRM, otherwise, the order
will not be processed.

THIS IS NOT A FREE PROGRAM





........................ ............................
........................ TABLE OF CONTENTS ............................
.............................................................................


Page
FOREWORD .......................................... 1
Running IDCshell .................................. 2
Environment Variables.............................. 2
GENERAL ........................................... 4
HELP .............................................. 5
Copy File(s) ...................................... 6
Directory Tree .................................... 6
Quick Tree Traversal .............................. 6
Erase file(s) ..................................... 7
Move file(s) ..................................... 7
Rename file(s) .................................... 7
Print file(s) ..................................... 8
View .............................................. 9
ANSI view feature ................................. 9
Wildcard tag ...................................... 10
Mouse ............................................. 10
Editor ............................................ 10
Call NARC ......................................... 10
DOS Shell ......................................... 11
File Attributes ................................... 11
ERRORS ............................................ 12
Finding a File .................................... 12
What is a ZIPFILE ................................. 13
Compression Techniques ............................ 14
Storing ................................. 14
Shrinking ............................... 14
Imploding ............................... 18
Tagging files ..................................... 18
Why TAG files anyway ? .................. 18
Tagged files and zipfiles ............... 19
ZIPFILE preview function .......................... 19
Function Keys ..................................... 22



........................................................................
IDCshell (c) 1987-90 Infinity Design Concepts, Inc. all rights reserved
........................................................................

FOREWORD


IDCshell.EXE is the denouement of many hours of rewarding
and painful labor. The long belated release of IDCshell was
the consequence of many factors, not the least of which, was a
dedication to writing a sleek, powerful and fast utility.
Many of the routines have been completely rewritten several
times in the ultimate quest for speed and compactness. These
are the very reasons that ALL IDC software is written in
assembler. There were bugs that took months to fix, once
located, others that subtly emerged and still others that lay
waiting for that right set of circumstances to pop up. We
have made a more than substantial effort to squash bugs in
IDCshell and hope that we have hit the mark, one hesitates to
be too certain in the software industry. There were a number
of people who proved vital in the debugging phases and have
made monumental contributions to the project. I would like to
thank Curt Edwards for being so damned picky in offering
suggestions for features and how they should work. Curt was
an endless source of ideas (and more work for me). I would
also like to thank Jerry Taylor for his uncanny gift for
finding bugs that the entire universe would overlook.

I would like to thank my mom and dad for not killing me at an
early age and my wife for not killing me at a later age,
although, I may not be out of the woods yet.

IDCshell is the grandchild of B29.COM (first released Sept.
1985), a CP/M utility that was very similar to IDCshell and
handled LBR files. Many of the features of IDCshell were
taken from the original DOS version of B29, which I never
released. The directory tree routine is a complete rewrite of
IDCTREE.COM, which was released in late '86.
























Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 1

IDCshell is written entirely in assembler using the Microsoft
MASM 5.1. The program was written on a Kaypro PC and an ALR
386/20. The editor of choice was the Norton Editor from
Norton Computing. Approximately 30 support programs were
developed in QuickBASIC, QuickC or assembler to aid in coding,
debugging and testing algorithms and subroutines used in
IDCshell. Including these files the source code is well over
1MB and some 40,000 lines of code.


Acknowledgments:

We at IDC would like to thank the following people for their
kind direct or indirect assistance in the development of
IDCshell.

Curt Edwards Bob Freed
Jerry Taylor (Creative Applications) Terry Welch
Chuck Crumpton Carlisle Crutcher
Murphy Frank Roemer

Programmer's thought: You are your own worst enemy.


Gary Conway, Pres.



RUNNING IDCshell


When IDCshell is first invoked, it reads the current
directory, sorts and formats the entries and displays them on
the screen. This is called the FILES SCREEN. The box on the
right hand side of the screen shows IDCshell's commands. Note
that the F1 key will bring up a full screen of help. Some of
IDCshell's commands can be accessed through the command bar
at the bottom of the screen. The cursor keys or the mouse may
be used to move the highlight bar left and right. Once the
highlight bar is over the desired selection, the ENTER key
will execute that command. Alternatively, you can use the
first letter of each of the menu bar commands with the same
effect.

Throughout the program, the left mouse button simulates the
ENTER key and the right mouse button simulates the ESC key.


ENVIRONMENT VARIABLES


If you don't know what an environment variable is, then
continue reading. If you DO know, then skip to "The Goods"
section below.




Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 2

What are they ?

An environment variable is a convenient way for programs to
communicate with DOS or with each other through a commonly
shared spot(s) in memory. If you type "SET" at the
DOS prompt, DOS will respond by showing you the current spot
of shared memory which we call the environment. These text
strings can be created,read, modified and deleted at any time
by any program. Each variable has a certain format that must
be adhered to. The format is as follows,

variable_name = string

An example would be IDCED=C:\UTILITY\WS.EXE

Spaces are allowed before and after the equal sign. This
variable will mean absolutely nothing to DOS, but, to
IDCshell, it will show the way to your editor. The variable
name here is IDCED and the string is C:\UTILITY\WS which is
the pathname for finding WordStar on the disk. You can change
this string to anything you want by simply typing --- SET
IDCED= (and type your pathname for your editor). IDCshell
uses several of these environment variables to communicate
with you and you should set them up in your AUTOEXEC.BAT
file.

What do I do if DOS says, "Out of Environment Space" ?

DOS reserves only so much space for the environment strings,
and if you have a long search path or a lot of strings, then
you may run out of space. I can offer two suggestions, one
for DOS 3.1 and one for DOS 3.2. There is a "switch" that you
can add to your CONFIG.SYS file (If you don't have one, then
you should !). One of the lines in your config.sys file
should read Shell=C:\COMMAND.COM or something similar. You
can add the following to the end of this line to increase
your environment space.

DOS 3.1 - /E:nn where nn = 10 up to 62. This
number times 16 is the length
of your environment.

example: Shell=c:\command.com c:\ /P/E:10

This would reserve 160 bytes.

DOS 3.2 - add /E:nnnn where nnnn is the actual
number of bytes to reserve.

example: Shell=c:\command.com c:\ /P/E:160

This would reserve 160 bytes.


Note that the minimum environment size is 160 bytes.



Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 3

The Goods ..

IDCED=c:\editors\ws.exe

Use this variable to tell IDCshell where your
editor is located, so that when the F9 key is
pressed to invoke the editor, IDCshell will know
where to locate it. The drive and path and editors
name must be given (in upper or lower case). The
file extension (.COM or .EXE) MUST be given.

NARCPATH=c:\utility\narc.exe

Use this variable to tell IDCshell where to find
NARC when the "N" key is pressed. As with IDCED,
the drive and path and filename and extension must
be given.

IDCHELP=c:\utility\idchelp.hlp

Use this variable to tell IDCshell where to find
the IDCshell help file when the F1 key is pressed.


IDCTEMP=e:\ramdisk

Use this variable to tell IDCshell where it can
place temporary work files. If you have a ramdisk
set up, then this would be a good use for it since
all file operations will be MUCH faster from
the ramdisk.

All of the above variables can be set up in your Autoexec.bat
file. The format is as follows,

SET variablename = pathname

GENERAL


IDCshell is laid out with many menus, windows, dialog boxes
and screens, all emanating from the MAIN SCREEN. This screen
displays the program name and release number in the upper
left hand corner, followed by the current drive/path, this is
the status line. Just below the status line are headers for
information describing each file. The fields are, filename,
filesize, file date, file time and file attributes. The tag
status indicators are located at the right end of this line
and show how many files in the current directory are tagged
and how many bytes are represented by the tagged files.

At the bottom of the screen is the command line. Here you may
use the cursor keys or the mouse to SELECT any of the
commands and then use the ENTER key or the left mouse button
to EXECUTE the highlighted command. You may also use the
letter in each command that is capitalized. The last line of
the screen displays the name of the current zipfile (if one

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 4

has been selected).

The remainder of the screen is broken vertically into two
halves. The left half displays the filenames and directory
names that are located in the current directory. Directory
names are denoted with in the file size field and
volume labels are denoted similarly with . The cursor
keys, Pg Up, Pg Dn, Home and End keys allow easy navigation
through the list. The mouse may also be used to simulate the
up and down arrows. Note that you may find any file easily
with the ALT-letter combination (see finding files).

The right side of the screen is the command menu window
showing most of the commands that IDCshell provides and a
very brief description of each. Note that much more detailed
help is always on line with the F1 (HELP) key. At the bottom
of the menu window, the total number of files in the current
directory are displayed. Note that this count DOES NOT
include directory names or volume labels, nor does it include
the "." and ".." entries (although IDCshell will not display
them).

At the top of the command menu window, the volume label of
the current disk is displayed.

The command menu window is an interesting story. The first
version of IDCshell (then B29 and later IDC-ARC), looked very
much different than the version you have now. There was a
window for each of the commands on the command line and as
you moved the cursor bar right and left, the menu window
would change giving much more detailed information than is
currently displayed. As commands were added, this windowing
system proved inadequate and the IDC menu compiler was
written so that a complete on-line help facility was almost
always available. This new system gave even more details
about the commands. It was also during this time, that the
help facility was incorporated into NARC.


THE HELP SCREEN


From the files screen, it is always possible to call the
context sensitive help system. Hitting the F1 key will cause
a full screen of help to pop up that has information specific
to the command that the cursor bar was pointing to on the
command line.

Once the help window is opened up, it is then possible to get
help on ANY of the commands. The Home key will take you to
the help index (as will the "I" key). You can put the window
away at any time with the ESC key.

Note that the environment variable IDCHELP must be set for
the help feature to work correctly. See "Environment
variables".


Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 5

COPY FILES


The copy command will allow copying files from one directory
or disk to another. When this command is invoked, IDCshell
first checks to see if there are any tagged files. If there
are none, then it is assumes that you wish to copy the
highlighted file. If there are tagged files, then IDCshell
makes no assumptions and asks you whether you wish to copy
the highlighted file or the tagged files.

After resolving the above, IDCshell will display a directory
tree from which you can select the destination drive or
directory. If you wish to copy to another drive, simply use
the "L" command to log a new disk, select the destination
directory there and the copy will proceed.



DIRECTORY TREE


The directory tree reads the disk directory from the current
disk and locates all sub-directories from this information.
The sub-dirs are then sorted and sub-sorted and formatted for
a visual tree presentation.

The commands available when the tree is displayed are the
following;

ALT-letter this combination of keys will position the
highlight bar on the first subdirectory that
begins with "letter"
Pg UP - goes to the previous page of the tree
Pg DOWN - goes to the next page of the tree
Home - goes to the top of the tree (root)
End - goes to the end of the tree
^Pg UP - goes to the top of the current page
^Pg DOWN - goes to the bottom of the current page
"\" key - finds the root (same as Home key)
R - remove the highlighted directory from disk
(only if directory is empty)
M - make new directory, you are prompted for a name
L - log new disk drive
P - send copy of tree to printer on LPT1:
ENTER - go to highlighted directory
ESC - return to where you were before without changing
directories
^PGUP - moves highlight bar to top of screen
^PGDN - moves highlight bar to bottom of screen


Quick Tree Traversal.

BIG NOTE: There are four other keys that may be used when
moving around your disk tree. When you are NOT on the
directory screen mentioned above, that is, when you are at

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 6

the FILES SCREEN, you may use the "-" or the "G" keys to move
one directory toward the root. Conversely, if you highlight a
directory entry (denoted with in the size field),
you can use the "+" or the "H" keys to select or move down
the tree away from the root. In either case, the current
directory is changed to the newly selected one and the screen
is updated with the new files.


ERASE FILES

The "E" command will first check to see if there are any
tagged files, if there are none, then IDCshell assumes that
you wish to copy the highlighted file, if there are tagged
files, then IDCshell will ask whether you wish to erase the
highlighted or tagged files.

Files that are found to have the Read Only attribute set,
cause IDCshell to ask before deleting them.


MOVE FILES


Files can be moved from one subdirectory to another ON THE
SAME DISK. Moving a file is similar to copying a file, except
that moving is faster and the file is actually moved from one
subdirectory to another, not just duplicated as in copying.
The file will also, no longer exist in the source directory
after the move. The one limitation to moving files, is that
you cannot move a file from one disk to another, here, you
must use copy.

If IDCshell finds no tagged files, then the currently
highlighted file will be moved. If there are tagged files,
then IDCshell will ask whether you wish to move the
highlighted or tagged files. Next IDCshell will take you to
the directory tree, where you can select the destination
directory for the move.


RENAME FILE(S)

The RENAME "R" command allows you to rename any file in the
current directory . To rename a single file highlight the
chosen file and press "R" IDCshell will prompt you with -


"ENTER NEW FILE NAME (OR "*" FOR WILDCARD RENAME)"

At this time you can enter your new file name and press
. However, if you wanted to change several files that
match a wildcard mask (BIG???.*), at the above mentioned
prompt you would enter "*" , IDCshell would then prompt you
with -



Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 7

"ENTER OLD NAME (WILDCARD OK)"

Now you can ENTER something like *.PRG and IDCshell would
then prompt you with -

"ENTER NEW NAME (WILDCARD OK)"

Now you would ENTER something like *.CMD this action
would result in all .PRG files being renamed to .CMD
files.


Note that the rename function can be used to rename sub-
directories as well as normal files. Note also that the
volume label cannot be chaned in this way.



PRINT FILE(S)
The print routine will print the highlighted or tagged
file(s). You have several options here. The first option that
comes to the screen is the printer string option. This allows
you to select a character font for printing the file. There
are four optional fonts. they are ,

0 Send no printer string (use printer default)
1 Set printer to 10 cpi (pica)
2 Set printer to 12 cpi (elite)
3 Set printer to 17 cpi (compressed)

These printer strings come with IDCshell configured for an
EPSON printer. They can be changed to anything you like with
ARCCFG.EXE. Each string has a maximum length of 9 bytes. The
ESC key aborts the print operation completely.

The second option tells IDCshell how to format the document.
There are three options. Each option is toggled between YES
and NO with the SPACE bar. the ESC key aborts the print
operation completely and the enter key accepts the currently
displayed options.

Format -

YES - This option tells IDCshell that you want the document
formatted during printing. The formatting includes page
numbers,page breaks and expanded tabs.

NO - This option tells IDCshell not to perform any formatting
at all during print. This is for use on files that have
already been formatted with formfeeds etc.

Strip High bit -

YES - This option tells IDCshell to strip the high bit off of
each character that is printed. This is to allow WordStar (c)
type documents to be printed correctly since WordStar and
other word processors use the high bit of some characters to

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 8

denote paragraph alignment etc.

NO - Do not strip the high bit from each character.

Strip Control Characters -

YES - This option tells IDCshell to remove all control
characters from the document before printing. The exceptions
to this are carriage returns,linefeeds and tabs.

NO - Do not remove control characters.
VIEW

The VIEW "V" command allows you to view any file in the
current directory by simply moving the highlight bar to the
chosen file and pressing "V". This command is very useful for
viewing ANY type of file, including binary. By using the MODE
"M" function you may toggle between ASCII or HEX views. The
FILTER "F" function toggles the filter so that the out put of
some word processors can be displayed correctly.

Last but not least is the SEARCH "S" function. This function
can be used to find a key word or heading within a file. When
the SEARCH function is chosen IDCshell will prompt you with-

" ENTER TEXT TO FIND "

Simply ENTER the text to be found IDCshell will locate the
first listing of this text and highlight it onscreen. The
search next function will locate the next occurrence of a
matching string. If no matches are located, a message will
appear. Note that CONTROL-R will repeat the last text when
entering the string to find.


ANSI VIEW FEATURE


This feature is invoked by the F3 key and is UNIQUE to
IDCshell. The highlighted file will be displayed on the
screen as though it were a file containing ANSI escape
sequences. You will note that there are no page up/down
commands in this mode. The reason for this is simple, MOST
ansi files are single page files and most of the editors that
create ansi files, create single page files. You may PAUSE
the screen output with the SPACE bar or abort with ESC or
ENTER.

This feature is provided primarily for SYSOPS of bulletin
boards since most BBS software these days supports ANSI files
when transceiving in color. This is an easy method of looking
at your compliment of ANSI files.


NOTE: You MUST have ANSI.SYS loaded in your CONFIG.SYS file
for this feature to function correctly.


Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 9




WILDCARD TAG

If you wish to tag several files that match a wildcard
filemask (BIGTOOL?.*, for example), the current directory use
the WILDCARD TAG "W" function. To use this function press "W"
and IDCshell will prompt you with-

ENTER WILDCARD MASK TO TAG :

Entering something like *.PRG, would result in all files
with the extension of .PRG being tagged. Note that tagged
files are used when arcing, copying, deleting, renaming or
printing files.


MOUSE FUNCTION

The mouse function is always enabled (providing of course
that you have a mouse) and controls the highlight bar and
command line. The left mouse button simulates the ENTER key
and the right mouse button simulates the ESC key. Mouse
sensitivity may be changed in ARCCFG by decreasing or
increasing the values for vertical or horizontal movement
using the appropriate function key. You may also disable the
mouse function in ARCCFG if you so desire.



INVOKING YOUR EDITOR


IDCshell can call your editor with the F9 key from the main
menu. IDCshell must first be told what your editors' name is
and where to locate it. This is done via the environment
variable IDCED. For complete information on how to set this
environment variable, see the ENVIRONMENT VARIABLE section.

Once the environment variable is set up, the F9 key will open
the currently highlighted file with your editor. Note that
the ALT-F9 key combination will call the editor and will not
pass a filename to edit.



CALL NARC (tm)

To execute NARC you must make a change in your AUTOEXEC.BAT
file so that IDCshell will be able to find NARC. The
following line (with your drive and path filled in) should be
put into your AUTOEXEC.BAT file.

SET NARCPATH = DRIVE:\PATH

This is called an environment variable if you don't fully

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 10

understand this it is suggested that you read the section ;


ENVIRONMENT VARIABLES

Once you have set up your autoexec.bat file it is very simple
to call NARC by simply press "N" and NARC will be executed.
If the highlighted file is an zipfile, then this filename
will be passed to NARC and the zipfile opened. If the
filename is NOT an zipfile, then no filename is passed, NARC
comes up in its default condition.



DOS SHELL

If for some reason you wish to exit IDCshell temporarily this
is easily accomplished by using the F5 (dos shell) command.
THIS will allow you to run DOS commands outside of IDCshell
and return by typing EXIT when you are finished. It is also
possible to invoke a second copy of the DOS command processor
by typing "COMMAND" at the first DOS prompt (if you have
enough RAM left).


FILE ATTRIBUTES

Each file on your disk has one or more file attributes.
Here is a brief description of these attributes.

READ ONLY (R) The file cannot be deleted NOR can it be
modified under normal circumstances.

SYSTEM (S) Your MSDOS files are system files

HIDDEN (H) These files can be executed but do not show
up when you request DOS to show the directory
via the DOS DIR command.

ARCHIVE (A) Not to be confused with .ARC files. This
attribute is used primarily for backing up
harddisks.

IDCshell shows these attributes in the "Attr" column next to
each file in the directory using the abbreviations, R H S or
A. If you have not tagged any files IDCshell will assume that
you wish to change the highlighted file. In order to change a
file's attributes press the "Y" key and IDCshell will display
a dialog box showing each of the possible file attributes.
Beneath each of the attributes is displayed either "YES" or
"NO",indicating that the attribute will be either turned on
"YES" or turned off "NO". The left and right arrow keys will
move the highlight bar to the selections in the box. The
SPACE bar toggles the attributes between the "YES" and "NO"
states.

The ENTER key will cause the file(s) attribute(s) to be set.

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 11


The ESC key will abort the operation.

Note that the attributes of the subdirectories may also be
changed with this command, but only one directory at at time
because you cannot tag subdirectories.


ERRORS

Requires DOS version 2.0 or above.

When IDCshell is first invoked, it checks the DOS version, if
it is found that the system is operating on a DOS version
lower than 2.0, IDCshell exits with the above message.


Insufficient Free Memory to Run IDCshell.

When IDCshell is first invoked, it checks to see if there is
enough RAM available to run the program, if not, IDCshell
exits to DOS with the above message.


Error: Can't Move a file to ITSELF, Hit ENTER.


When trying to move files, if you do not select a destination
directory different from the source, then the move is aborted
with the above message.


Error: Unable to Move File, Hit ENTER.

DOS has reported an error in trying to move the file, disk
removal etc.


Error: Writing Destination File, Hit ENTER.
Error: Reading Source File, Hit ENTER.

Either of the above messages can be caused when trying to
copy file(s) if the disk is removed.


Unable to Open Source File, Hit ENTER.


Destination Directory is Full, Hit ENTER.


FINDING FILES


The "F" command will allow you to quickly find any file in
any subdirectory of a disk. This is the global find feature.
Invoked by the"F" key, this function will ask for a wildcard

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 12

filename and will proceed to locate a matching file on the
disk. As always, ESCape or ENTER will abort the operation
while entering the filename. Once the filename is given, a
directory tree appears, along with the search mask. The
cursor bar moves down the tree as it hunts for your file.
When one is found, a menu bar appears at the bottom of the
screen asking for further instructions. At this point you may
do one of three things;

(J)ump to the directory where your file was found
(C)ontinue the search, looking for the next match
(Q)uit the search operation and return to the files
screen

Note that if the directory tree has not been previously
scanned (and remembered) with the "D" command from the files
screen, then it is scanned for the find operation. If the
tree has already been read, then it is not read again,
speeding up the find operation considerably.


You may also find files in the current directory with the
ALT-letter key combination. Here, to find a file that
started with the letter "Q", for example, you would enter
ALT-Q and the first file starting with the letter "Q" would
be highlighted. Note also, that hitting ALT-Q again, will
locate the next filename that starts with the letter "Q".



WHAT IS A ZIPFILE ?


A zipfile is a collection of files all rolled into one
file. You could, for example, store all of last month's
invoices in one ARC file for organizational purposes. Suppose
that all of these invoices totaled 400K when adding their
file sizes. If you only had 360K floppy drives, you could not
put all of this information on a single floppy. With a
zipfile, it is very possible that you can do just that. The
arcing program will compress the files by as much as 99% and
store them into one file, an ARC file, that will very likely
be much smaller than 360K and thus fit on one floppy.

Each file in the zipfile has a header that gives statistical
information about the file such as the filename, actual size
and compressed size, what method of compression was used (if
any), a CRC of the file, and the date and time of the file.


History:

In the CP/M world, we had what were called library files.
These files were collections of ones rolled into one. There
was no compression offered, unless you compressed the files
BEFORE inserting them into the library (.LBR). Also, the
number of entries in the library had to be specified at the

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 13

time the library was built. We had several programs to deal
with libraries, LU, NULU and IDC's B29. Library files were
soon ported to DOS. This was great until Thom Henderson of
System Enhancement Associates (SEA) brought forward his
version of library files, called archives (.ARC). What
ARC.EXE offered was a new file format and compression of the
files as they were inserted. Truly the library file had come
of age, except we now called them .ARC files. The newer file
compression schemes are used in zipfiles, which were jointly
developed by IDC and PKWARE and dedicated to the public
domain in Feb 1989.


STORING

There is really no story to tell here, the input file is
merely copied into the zipfile. This method is really a last
resort. When none of the other compression methods will yield
a smaller output file, then storing is used. You will notice
that not many files in zipfiles are simply stored and when
they are they are usually VERY SMALL files.


SHRINKING


The current crunching and squashing algorithms take their
roots from an article by Terry Welch in IEEE COMPUTER, June
1984, entitled "A Technique for High Performance Data
Compression". This article built on another article by J. Ziv
and A. Lempel, entitled, "A universal algorithm for
sequential data compression. The term "LZW" was coined in the
article by Welch from the three authors' initials.


Shrinking is a variation of the LZW algorithm, since the
original made no provision for variable length output codes,
nor did it call for an adaptive reset. The LZW algorithm
begins by constructing a table in memory, where each entry in
the table contains one of the 256 "atomic" characters along
with a pointer, thus the format for each entry is,





Where POINTER is just that, a number that "points" to one of
the other entries in the table. The reason for the pointer
should become apparent as we proceed. The following IS the
complete LZW algorithm. It appears, at first glance, to be
very simple, but the ramifications get downright nasty very
quickly.

The LZW algorithm,


Initialize the string table as described above.

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 14

Get the first character from the input file (omega).
Rept: Get the next character from input (K).
IF no more characters then output code(omega): DONE
IF omega-K exists in table then
omega = omega-K: GO REPT
ELSE
omega-K not in table, output code(omega)
insert omega-K into table
omega = K
GOTO REPT
ENDIF


This is the algorithm as originally described by Welch. Note
that it does not make provision for the string table filling.
It should be apparent that when the string table fills, no
more strings (omega-K) could be added to the table. More on
this later.

What the algorithm does, in effect, is to look at the input
file, byte by byte, looking for the longest sequence of
characters that it "recognizes". A sequence of characters (a
string) is "recognized" if it is located in the string table.
The omega in the algorithm above, represents this string of
characters and the K represents the last character read from
the input file and appended to this string. As an example,
take a look at the start of this paragraph and imagine that
you are looking at the input file. The algorithm would get
the first character "W" and call it omega. It would then get
the second character "h" and call it K. We would then search
the string table for an entry where points to the
location in the table where the "W" entry was made at
initialization AND = K. If we find this, then we are
ready to get another character and add it to the omega
string. If we don't find a match in the table, then a new
entry is made in the table and omega is sent to the output
file as a "CODE". This sequence continues until the input
file is exhausted.

It is in examining the output "CODES" mentioned above, that
we find one of the major differences between SQUASHING and
CRUNCHING. Both of the algorithms vary the output code
length, they both begin with 9 bits, but crunching ends with
12 bits and squashing ends with 13 bits. When we described
the string table at the start of this discussion, we said
that the table was initialized with 256 entries, one for each
of the 256 atomic characters. Well, the algorithm has shown
us that we are going to be adding strings to the table, so it
seems that we will have more than just the 256 atomic
entries. But how many entries ? In the case of crunching,
remember, we said that our maximum code length was 12 bits,
in binary, 12 bits can represent numbers up to 4096, thus we
have our maximum number of entries in the table. Squashing
has a maximum code length of 13 bits, which allows for 8192
entries. From this, is should be somewhat intuitive, that
squashing allows learning more strings that does crunching,
and indeed it does.

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 15


But what happens when the string table fills ? The first LZW
implementation as described in the algorithm, did not provide
for this case and simply stopped "learning" strings, because
there was no more room to add them to the string table. Later
versions (and squashing) perform what is called an "adaptive
reset" of the string table when it fills. This is done by
sending a special code to the output file that will tell the
uncompressor to reset its table and continue.

The variable length output codes are explained by looking at
the codes that are output. The first "learned" code is output
as the number 257 and the second as 258 and so on up to the
table size, 4096 (crunch) or 8192 (squash). As the numbers
get larger, it takes more bits to represent them, obviously,
you cannot represent 4096 with 9 bits.

In actuality, one never lets the table fill at all, in fact,
the best compression speeds are achieved by not letting the
table get over 75-80% full. This is precipitated by the
method for looking entries up in the table. Remember, that
the algorithm says that we must see if omega-K exists in the
table. This implies that we must search the table once for
each byte of the input file. The EASIEST way to search the
table in by looking at the first entry for a match, and then
the next and so on, until we have examined the whole table.
This is called a linear probe. To see the disastrous
ramifications of a linear probe for LZW, we must turn to a
little mathematics.


MATHEMATICAL FACT 1.
The smallest codeable loop (in assembler) for the linear
probe takes 6.3 micro-seconds to execute on a 20mhz 386
computer.

MATHEMATICAL FACT 2.
Our sample file is 15000 bytes long.

MATHEMATICAL FACT 3.
The table size is 4096 entries, so we CHOOSE an average
figure of 2000 locations searched for each input byte.

Multiplying,

loop iteration time X avg. iterations/search X bytes in file

we end up with 189 seconds or just over 3 minutes to compress
our 15K file. Here we have definitely delved into the
unlivable. Note also, that we have not taken into account the
rest of the compression code overhead, we have only looked at
how much time the compressor would spend/waste searching the
table.

It seems there must be a better way, and indeed there is, the
hash function. Our table has a finite number of entries and
all we wish to do is search the table for a match, so if we

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 16

had a mathematical black box where we could input a number
and have it output a unique number, our searches would be
made a lot faster. We input a number (omega-K) and the black
box outputs another number, which is an entry number in the
table. Well, we run-amuck here again. is a word (2
bytes) and is 1 byte. The number of possible
combinations of the two is 65536 times 256, which is a BIG
number and left as an exercise for the reader. If our black
box is to spit out UNIQUE numbers, then there will be as many
output numbers as there are input numbers and our string
table would have to be HUGE, in fact, it couldn't be run on
any computer that exists in the world today! Then what we
need is a black box (hash function) that will take a huge
number of input combinations and output numbers that are
within our table. The implication here is that we will have
several input numbers that result in the same output number.
This is better known as a "hash collision". We can help
minimize the number of collisions by designing our hash as a
double hash, where the second hash function is a function of
and table size. Technically, what we are doing here,
is minimizing what is called "clustering", that is, the
tendency of certain sets of input numbers to "cluster" around
a given output number. Collisions from the first hash are
called "primary clustering" and those from the second hash
are called "secondary clustering". Our hash function must be
designed very carefully so that it will also "probe" every
location in the table, if need be. Clustering is also avoided
by making the table size a prime number instead of the 4096
or 8192 called for by the bit length. The "fill limit", as
discussed above, is also selected as a prime number. Prime
numbers are divisible by only two numbers, 1 and the number
itself. It may be intuitively observable, that this should
reduce clustering.

The real beauty of hashing, is that when the string table is
25% full, the hash function will have to search an average of
1.15 to 1.33 table locations to know whether or not the omega-K
exists in the table. With the linear probe that we spoke of
before, would be searching an average of 500 locations for
each input byte, quite a difference. As the table fills,
however, the performance of the hash function decreases
logarithmically, thus it is favorable, from the standpoint of
speed, that we limit the load factor of the string table,
that is we limit the number of entries that we will actually
put into the table. When the table hits this limit, we
arbitrarily call the table full and issue an adaptive reset
code to the output file. At this point, the table is cleared
of the least frequently occurring strings. The hash function
in IDCshell was derived from a discussion in "Data Structures
in Pascal", by Reingold/Hansen, pp. 397.

One final note on the codes that are developed by the
compressor and sent to the output file. These codes, you
remember, vary in length from 9 to 12 or 13 bits and are
strung together at bit level in the output file. The most
significant bits from one code are shifted left into the next
code. It does get a bit messy playing with the codes this

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 17

way, but real compression has as its primary cost, coding
headaches. The algorithms are actually very short when fully
coded, a little over 1K each.

And there you have it. It seems like a bit much and is not
casually understood, but if you really want to know how they
work, you should get the original article and study it along
with the above and hopefully, you will gain more insight into
the replete world of data compression. There are many other
forms of data compression that have been in use for years,
bit mapping, null suppression, half-byte packing, diatomic
encoding, relative encoding, statistical encoding and pattern

substitution to name a few and that is not to mention all the
forms of redundancy and apriori structure compression. All
things considered, LZW is definitely one of the best we have.


IMPLODING


The imploding algorithm was developed by Phil Katz of PKWARE
and is a combination of two algorithms. First it uses a
sliding dictionary, whereby the compressor looks backward in
the file (toward the beginning) for matches of strings of
characters. The sliding dictionary size (how far we go back)
is either 4K or 8K. If a match is located, then two numbers
will describe the match itself, namely the LENGTH of the
match and the DISTANCE back in the file where the match was
located. These LENGTH and DISTANCE pairs are further encoded
using the Shannon-Fano technique, which is very similar to
Huffman coding. For further reading about the SF codes, see
HELD, "DATA COMPRESSION" ppg. 107. Any characters that are
not matchable are encoded as literal characters either using
another Shannon-Fano tree or by simply sending the literal
character to the compressed file as is.

The imploder stores the Shannon-Fano trees in the zipfile in
a compressed manner also, further reducing compressed file
size. These trees are then used by the decompressor when
reconstructing the original file.


TAGGING FILES



Why tag files anyway ?

Tagging files started back in the CP/M world with the sweep
programs. The idea is to allow performing the same operation
to more than one file at a time. You would tag the files that
you wanted to copy, for example, and then invoke the copy
command. This is a lot faster than copying each of the files
individually. This same reasoning applies to deleting, moving
and printing.



Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 18


There are several ways to tag files in IDCshell.

SPACE bar or "T" key -

These two keys perform the same operation, they tag the
currently highlighted file. Actually, they toggle the tag
status of the currently highlighted file, that is if it was
tagged, after the toggle, it will not and vice-versa. A
diamond will appear next to the file on the screen denoting
it's tagged status. The number of tagged files and tagged
bytes will also be displayed on the screen, just above the
command window.


F6 - key

This key will tag all files in the current directory.

F7 - key

This key will invert the tag status of all files.


NOTE: Also see the WILDCARD TAG section.


Tagged files and zipfiles


Tagged files are one method of adding/updating files in a
zipfile. When you change directories or drives, IDCshell will
remember the files that you had tagged and where they were so
that when the "Z" (zip) command is invoked, you will be
shown a list of these tagged files to preview/edit before
performing the zip function. Also, if you tag files and
then proceed directly to the preview screen via the "A"
command, the tagged files will be displayed for review. See
the ARC preview section.

ZIPFILE PREVIEW FUNCTION


The major changes in version 4.0 and above as far as the
operation of the preview screen, is that now you have the
option of compressing tagged files -OR- compressing
recusively, files that match a filemask from a user provided
directory name. The recursive function will compress all
files that match the file mask in the directory specified and
all files in all directories off of the specified directory.

The preview screen will display each path where files were
tagged and the files that were tagged in that directory. The
arrow, home, end, page up and page down keys will move you
through the list. The "D" key will delete the currently
highlighted filename from the list -OR- if the highlight bar
is pointing to a path and the "D" key is pressed, all files

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 19

in that path will be deleted from the list.

IDCshell allows moving into and out of the preview screen at
any time, without having to choose the ARC file first. This
allows for a certain degree of playing around before you get
your thoughts together. You can edit the list, return to
select more files for inclusion, or proceed with the arc
function.

Each time that you leave the files screen, all of the tagged
files are saved in memory for inclusion into an ARC. This
method would normally display several copies of the tagged
files in a directory if you had been into and out of that
same directory more than once. IDCshell will remove any
duplicated directories from the saved list and keep ONLY the
last one.

Once you have tweaked the preview list into just what you
want, you can then hit ENTER and proceed with the zipping
operation. The next screen asks how you want to perform the
arc function. This screen is vastly different from the usual
-f, -u, -a etc. commands from the other arcers. The
reasoning for this is simple, How many times have you used
the -u command when you really meant to -f ? IDCshell uses a
much more intuitive approach. The screen is as follows.



Select Archive Options

Add new or update existing files YES
Update existing files only NO

Replace with newer files only YES
Replace files regardless of date NO

Overwrite existing files YES
Do not overwrite existing files NO

Save paths with filenames NO
Do not save paths with filenames YES



The arrow keys are used to move from field to field up and
down. The SPACE bar is used to toggle between YES and NO.
Notice that if you select NO for "add new or update existing
files", that the last field disappears. This is because you
are telling IDCshell to update existing files only, therefore
you MUST overwrite them, but you still have the option of
selecting date checking with the second field.

Add new or update existing files -

This selection is used if you are adding new files to an
existing zipfile and/or updating files that already exist in
the zipfile. Existing files will be replaced only if you have

Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 20

chosen to do so with the second selection in the box.


Update existing files only -

This selection allows you to update a zipfile without adding
any new files. Suppose that you wanted to update a zipfile
and you did not know all the filenames that were in the
zipfile, but you also knew that whatever their names were,
they were in a certain directory. You could tag ALL files in
that directory and then choose "update existing files only"
and only the files that previously existed in the zipfile
would be updated, no new files would be added. The "extra"
tagged filenames would be ignored.

This selection is also subject to the setting of the
"replace" option.

Replace with newer files only -

This option, when YES, will cause any filename in the zipfile
to be replaced with a matching tagged filename, ONLY if the
tagged filename has a more current date/time stamp.


Replace files regardless of date -

This option, when YES, will cause filenames in the zipfile to
be replaced with matching tagged filenames REGARDLESS of the
date/time stamp. The net result here is that NEWER files may
be replaced with OLDER files.


Overwrite existing files -

NOTE: This option only applies if the "add new or update
existing files" option is set to YES.

If this option is set to YES, then new files will be added to
the zipfile and existing files will be updated. Note also,
that this option is subject to the setting of the "replace
with newer files only" option.


Do not overwrite existing files -

If the option is set to YES, then IDCshell will ONLY ADD NEW
FILES to the zipfile, existing files will not be disturbed.


Once the options are chosen, you may hit the ENTER key and
the archiving process will begin. Each file is displayed at
the top of the list as it is analyzed and compressed. The ESC
key can be used to abort the operation. Note that the abort
does not take place until the current file is finished.



Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 21


Save paths with filenames -

This option causes IDCshell to store the pathnames along with
the filenames in the zipfile.

Do not save paths with filenames -

This option causes IDCshell to ignore the paths associated
with filenames and they are subsequently NOT stored in the
zipfile.



FUNCTION KEYS


F1 - invokes the IDC on-line help system
F2 - selects the currently highlighted zipfile for archive
operations.
F3 - ANSI view function
F4 - send a copy of the file list in the main screen to the
printer.
F5 - shell to DOS
F6 - tag all files in the current directory
F7 - invert tagged status of all files
F8 - specify directory sort order
F9 - edit the highlighted file with your editor
F10- clear all files tagged for archiving
ALT-F9 - call your editor
ALT-F1 - change file display mask
ALT-F2 - run highlighted program
ALT-F3 - display machine status screen
ALT-F4 - toggle EGA/VGA 43 or 50 line modes


End of file.





















Copr. 1987-90 Infinity Design Concepts, Inc. All rights reserved

Page 22