TDBS 1.x - Database Packing Under TDBS

Contact:   eSoft, Inc. (Makers of TBBS)
           15200 E. Girard Ave., Suite 3000
           Aurora, CO  80014
           (303) 699-6565      Voice
           (303) 699-6872      Fax
           (303) 699-8222      BBS
           support@esoft.com   E-Mail

DATABASE PACKING UNDER TDBS
---------------------------

WHAT IS PACKING?

"Packing" refers to an ability to physically remove records marked for deletion 
from the database.  Much like the way TBBS handles its message base, in a 
dBASE environment, records are only MARKED for deletion.  They are not 
physically removed from the database until a pack operation is performed. 

In dBASE itself, this is done using the PACK command.  This command is NOT 
supported by TDBS, however. 

WHY DOESN'T TDBS SUPPORT THE PACK COMMAND?

For the same reason you should use the information contained herein sparingly - 
packing is not an operation that is conducive to multi-user environments.  In 
order to pack, the database must be used in exclusive mode (so no other users 
can access the database while the pack occurs).  Further, the operation itself 
is extremely disk intensive - LOTS of disk activity occurs, which loads the 
system more than TDBS normally would.  Depending on the system load at the 
time, the packing operation can also take a considerable amount of time. 

HOW LONG DOES IT TAKE?

In comparison to dBASE, a LONG, long time.  On a sample database of 885 records 
with a single index on one simple field, the entire operation as described 
below took 2 minutes, 28 seconds under TDBS 1.1.  dBASE III Plus performed the 
same operation in just 21 seconds total.  This was on a system with one single 
user signed on.  You can expect it to take even longer when multiple users are 
on the system, especially if any of them are in a TDBS application. 

THE GENERAL APPROACH TO PACKING

Simply stated, packing involves copying out all records that are not marked for 
deletion to a temporary database, zapping all records in the current database, 
then copying them all back again. You could copy ALL records, then copy back 
only those that are not deleted, thereby maintaining a backup of the original 
database, but my feeling is that this takes longer, especially if there are 
large numbers of deleted records involved. 

Other approaches may seem more logical, but do not work in practice.  For 
example, you could copy off the records to the temporary file, then delete the 
original and rename it.  Or you could copy off the records to the temporary 
and copy the temporary over the original.  Neither of these approaches can 
really work, because they require the original database file to be closed. 
This leaves open a small window of opportunity for other users to attempt a 
database access on another line during the operation, risking a file open 
error, or possibly a database usage that has invalid indexes associated with 
it.  The approach outlined here allows the original file to stay open 
throughout the operation (although in exclusive mode) thereby assuring file 
integrity. 

The packing operation MUST occur in exclusive mode for several reasons.  Even 
if it could occur in a shared environment, there would be no guarantee that 
the records would be current throughout the operation.  Further, the zapping 
operation (ZAP command) cannot be used in shared mode (it could dump records 
someone else is using).  Finally, reindexing cannot be accomplished in shared 
mode (records wouldn't necessarily stay current throughout the process). 

THE SPECIFICS

First, an error handler must be put in place in case the database file cannot 
be opened exclusive (another user is already using the database).  A USE 
command (do not reference associated indexes) is called, exclusive mode.  The 
error handler can now be cancelled or the reference changed as required.  The 
records are COPYed TO a temporary database file.  The records in the current 
database are ZAPped.  Records from the temporary are APPENDed FROM into the 
current database.  The temporary is deleted or moved as required. All indexes 
are rebuilt using INDEX ON as needed.  The packing operation is complete. 

The code below shows the constructs needed with heavy commenting. You will need 
to change the code as required for your application. 

WHY NOT USE WITH INDEXES INSTEAD OF DOING AN INDEX ON AFTERWARD?

You can do a USE <database> INDEX <index(es)> EXCLUSIVE and eliminate the need 
to do an INDEX ON after the operation, but in testing, that method actually 
took longer, at 3 minutes 10 seconds.  For that reason, it is suggested that 
you use the approach outlined herein. 

OTHER CONSIDERATIONS

In any applications that access the same database as the one being packed, you 
must remember to implement some error handling.  While the database is being 
packed, all other access attempts (through USE commands) will NOT be 
permitted.  ON ERROR error handlers must be implemented to alert users of 
other applications that database access is temporarily unavailable. 

Because of the serious demands on the system itself to do a packing operation, 
it is recommended that it be done as infrequently as possible. 

EXAMPLE PROGRAM FOLLOWS

--------------------------------------------------------------------------

*
*       PACK.PRG
*
*       An example of database packing for TDBS
*


*
*       Create an error handler in case exclusive use is impossible
*
ON ERROR DO FILEPROB


*
*       USE the primary database file EXCLUSIVE, but do not reference indexes
*
USE TEST EXCLUSIVE


*
*       Cancel the ON ERROR handler
*
ON ERROR


*
*       COPY TO the database file to a temporary database FOR .NOT. DELETED()
*
COPY TO TEMP FOR .NOT. DELETED()


*
*       ZAP the database of all its records
*
ZAP


*
*       APPEND FROM the temporary database back to the current one
*
APPEND FROM TEMP


*
*       ERASE the temporary file
*
ERASE TEMP.DBF


*
*       If the database was indexed, do an INDEX ON at this time to rebuild
*
INDEX ON MASTER TO TEST


*
*       QUIT or return to program flow
*
QUIT


*
*       Error handler in case USE EXCLUSIVE fails
*
PROCEDURE FILEPROB

? "Sorry, but the database cannot be packed at this time."

QUIT






- END -
TDBS0001
Rev. 10/93

Copyright (C) 1994 eSoft, Inc., All Rights Reserved.  Permission granted
to distribute this file in its entirety, without modification, to any
interested party.  Any other use requires the written permission of
eSoft, Inc.

IMPORTANT:  The information herein is subject to change without notice.
Please call or write to confirm factual information of importance to you
or your organization.

