Possibilities - Efficient TDBS Programming

Contact:   eSoft, Inc. (Makers of TBBS)
           15200 E. Girard Ave., Suite 3000
           Aurora, CO  80014
           (303) 699-6565      Voice
           (303) 699-6872      Fax
           (303) 699-8222      BBS
           support@esoft.com   E-Mail

EFFICIENT TDBS PROGRAMMING
--------------------------

*** From October 1992 Possibilities Newsletter ***
*** Copyright 1992 by eSoft, Inc.  All Rights Reserved ***

Efficient TDBS Programming
Edited from the TDBS Developers Roundtable

TDBS developer Ben Cunningham organized a TDBS developer's roundtable at the 
ONE BBSCON.  It was a lively session where a tremendous amount of 
information on programming TDBS applications was exchanged.  One of the more 
interesting topics discussed at this successful forum was the issue of 
making TDBS programs run faster and more efficiently.  Many good programming
tips emerged from this discussion, and we felt that a compilation of them 
should be reprinted here for the benefit of those who didn't attend.  Return 
with us now to the ONE BBSCON TDBS Developer's roundtable...

Programming in any environment is more of an artform than a defined 
mechanical task.  Virtually anyone can learn the mechanics of TDBS and be 
able to be a functional programmer, but being truly expressive and 
productive as a programmer takes experience, drive and an appreciation for 
performance issues.

TDBS program development is no different.  TDBS may be "dBASE compatible", 
but it operates in an environment extremely different from the typical 
environment of dBASE -- the online world of TBBS.

eSoft has worked very hard to ensure that no TDBS program can be overly 
harmful to the total TBBS/TDBS system.  Certainly no TDBS program should be 
able to crash a system, but it most certainly CAN affect its performance.  
TDBS gives a program developer the tools to be very productive, but it 
doesn't guard against everything.  Like any powerful tool it can create many 
things if it is properly and carefully used, but a little knowledge can help 
you avoid problems.

The tips presented herein are guidelines -- NOT universal, all-emcompassing 
no-no's or must do's.  The tips here focus on obtaining maximum performance 
with minimum system impact.  Virtually all of the things mentioned here as 
"should avoid" issues are perfectly fine when used on smaller databases.  At 
other times the performance penalty is a fair tradeoff to accomplish the 
functions you need.

Complex Indexes...

It is best to avoid complex indexes unless there is no alternative.  Complex 
indexes are those which involve an expression.  For example, the following 
code generates a complex index: 

index on substr(NAME, 1, 5);
        +substr(PHONE, 2, 3) to foo 

The following code generates a simple index:

index on NAME to foo

The difference is that a complex index involves an expression -- not just a 
field name.  For a complex index TDBS must evaluate the expression in real-
time for every record when searching the index.  This consumes additional 
resources and takes additional time.  A simple index, on the other hand, is 
pre-evaluated and has no additional overhead.

Complex indexes can typically consume from 15% to 20% more CPU than a simple 
index -- use them wisely.

Relationships...

The dBASE language SET RELATION TO command makes it easy to automatically 
cause multiple files to move together.  But it is best to avoid the frequent 
use of built-in relationships because setting relationships causes TDBS to 
go to a lot of extra effort to keep two or more databases in synch with the 
related field.  In many cases, this results in a lot of gratuitous re-
seeking of related databases. 

A better approach is to manually relate databases and only re-seek in 
secondary (related) databases when necessary.  Reducing the number of seeks 
will increase the efficiency of your TDBS application.

The technique you should use here is to manually select the secondary 
database, issue a SEEK command, and switch back -- when a match is found on 
the first database.  This approach gives your program full control of when 
secondary seeks occur rather than depending on internal TDBS functionality 
which will do a secondary seek on every primary file movement.  Sample code 
for this technique might look like this: 

select a
use data1 index name1
select b
use data2 index name2
select a
seek "ALAN BRYANT"
if found()
   select b
   seek "ALAN BRYANT"
   select a
endif

NOTE:  If you really will be re-seeking the secondary database EVERY TIME 
you seek in the primary database, then this method may INCREASE overhead as 
contrasted with using SET RELATION TO to do the work for you automatically.  
In those cases you have a true application for SET RELATION TO.

Filters...

SET FILTER TO is another command which can adversely affect the performance 
of your TDBS system.  There is almost always an alternative to a SET FILTER, 
making it a "lazy way out" approach to data searching.  As with all easy 
programming methods, there can be a big price to pay for saving the 
programmer time.

In fact a SET FILTER TO which screens out most of the records in a large 
database can actually cause your TBBS system to appear to freeze when file 
accesses occur.  Even on smaller databases, however, filters are best 
avoided under virtually all circumstances. 

The best alternative to setting a filter is to index on the first element of 
the filtering criteria.  Perform a seek, then inside of a DO WHILE loop, 
check the other criteria.  Like this: 

use foo index st
seek "CO"
if found()
   do while st = "CO" .and. .not. eof()
   if areacode = "303" .and. pcode = "A"
      * do record handling here
   endif
   skip
   enddo
endif

In this example, we seek on a state abbreviation.  If found, we then sit in 
a loop while the seek criteria is still in scope, and check the other 
criteria we're looking for, which would ordinarily have all been part of the 
SET FILTER  command.  In almost every situation, this approach will yield 
better performance results than using a SET FILTER. 

Macro Expansion...

When you use macros, use parentheses instead of an ampersand.  TDBS will 
evalute: 

USE (dbf_name)

about 20% faster than the more traditional:

USE &dbf_name

Using a text expression rather than a macro will work in many places.  It 
turns out it will work in most places where you will want to use macros.  
The performance increase it very high if the macro is part of a loop.

Complex Macros...

Use complex macros to their fullest advantage.  Many TDBS programmers code 
only with simple macros, such as: 

macro = "my_dbf"
use (macro)

However, macros may also be complex expressions:

macro = "fld1 $ fld2 .or. .not. (upriv() > 100)"

if (macro)
   * code
endif

This technique can add power if used well.

Checking Performance...

Performance of your TDBS code is important -- especially if you'll be 
selling your finished application to others.  TBBS provides a powerful tool 
for checking performance, and it's built right into the system:  the 
performance monitors.

As you're developing and testing your TDBS application, use a terminal hard- 
wired to the TBBS computer so that you can see the local console performance 
monitors as you run your program.  This is best done with NO other users on 
the system so the performance monitors are showing only your impact.

Watch the six meters carefully.  The three on the left (Idle, CPU and Disk) 
don't show a picture of your program's impact on the system, but rather are 
an indication of when your program is idle and when it is not.  If your 
program is pegging the CPU meter when it is waiting for input, it is a sign 
that you probably have an INKEY() loop that needs a time specified in it to 
truly unload the system while it waits.

The three right-hand meters (Resp, Turn and Thpt) give you an accurate 
picture of the impact of your program on other users.  The precise 
thresholds are in the TBBS manual, but if your TDBS program is affecting 
these meters severely, you need to look closely at the program code in 
operation at the time.  Again, in most cases, tight loops, heavy movement 
within many related files, or SET FILTERS cause this type of impact and 
should be (and can be) avoided. 

If possible, test your application on low-end hardware (such as a 286 
computer with a slow disk that is not cached).  If you application runs well 
on a such a machine, it will likely run well on better systems. 

Tuning Performance...

You can use the SECONDS() function to time and fine tune the execution of 
important sections of code.  For example: 



start = seconds()
count = 1
do while count  100
   * code to be tested
   count = count + 1
enddo
? seconds() - start
wait

For example, if you want to determine if string str1 occurs in string str2, 
you will find that: 

str1 $ str2

executes about 20% faster than:

at(str1, str2) > 0

A more advanced technique which you can use to measure the impact of 
alternate programming techniques is to calculate the total time your program 
takes to do a particular function using the TBBS performance monitor.  To do 
this you need to add WAIT commands just before and just after the function 
you want to measure.  Then use the <F2><F9> key to select the "breakout" 
monitor on the line your program is running on.

Log on to the hard wired terminal and execute your program up to the first 
WAIT.  Then press the F8 key on the console which will reset the monitor 
accumulators.  The press enter on your program and execute the code you are 
testing.  You can then read exactly how much CPU and DISK time that code 
took to execute from the performance monitor accumulator.  If you average 
the results of three or four runs of your code, you can compare different 
methods of programming a function.  The one that uses the least CPU and DISK 
time is the most efficient.  This is a powerful technique for discovering 
TDBS optimization techniques.

Opening and Closing Databases...

Many times a TDBS application will appear sluggish to the user because of 
the delay in opening databases and indexes when the application is started.  
This delay will be less apparent if some sort of prompting is provided 
before the USE command is executed.  For example: 

? "  [A] Press A to do this"
? "  [B] Press B to do this"
use myfile index myindex
key = inkey(0)

In this case, the menu is displayed and the files are opened before the 
keypress is even accepted.  In most cases, it will take the user a moment to 
read the menu and make a choice, and in that period of time TDBS will open 
the files referenced by the USE command and remove any apparent delay. 

Another hint about opening databases is to open files ONCE at the beginning 
of the program using multiple work areas, and reference them throughout the 
code where necessary by issuing a select command.  For example, the 
following will have significant overhead: 

use namedata
* name related code here
use proddata
* product related code here
use namedata
* more name related code here

In this example, using a single work area, the NAMEDATA database is opened, 
code executed, closed; PRODDATA is opened, code executed, closed; then 
NAMEDATA is opened again, code is executed.  This results in three file open 
commands and two file close commands (by inference).  Every file open or 
close commands adds overhead to your TDBS application and to the TBBS system 
itself. The following would improve performance:

select a
use namedata
select b 
use proddata
select a
* name related code here
select b
* product related code here
select a
* more name related code here

In this case, there are just two file open commands issued and no file 
closes, reducing potential overhead by about half.  Then, selecting the 
desired work area with a SELECT command, you can operate on the database of 
your choice.

The only exception to this would be a database that is only needed 
occasionally.  In such a case, it might be best to open and close the 
database when needed instead of leaving it open.  All databases and all 
indexes that are open at a given moment each require a file handle -- a 
limited resource in a TBBS/TDBS environment on large systems.

Waiting For A Key...

With the INKEY() function, remember that unlike dBASE, TDBS accepts an 
argument indicating how many seconds to wait for a keystroke before moving 
on.  Most people who learn dBASE wait on a keystroke like this: 

key = 0
do while key = 0
   key = inkey()
enddo

This is an endless loop and although TDBS handles it well, it will affect 
the feel of the system.  Always use an argument, like this: 

key = inkey(30)

On a side note, the TDBS INKEY() function will not timeout if your user does 
not press a key so you should not use an argument of 0 (wait indefinitely) 
unless the circumstances are unusual.  If your inactivity timeout setting in 
CEDIT is set for 5 minutes, you could code the following: 

key = inkey(240)
if key = 0
   ?? chr(7)+"Press any key to avoid auto-logoff"
   key = inkey(60)
   if key = 0
      dotbbs type 10 optdata ""
   endif
endif

- END -
PS1092-5
Rev. 10/92

Copyright (C) 1994 eSoft, Inc., All Rights Reserved.  Permission granted
to distribute this file in its entirety, without modification, to any
interested party.  Any other use requires the written permission of
eSoft, Inc.

IMPORTANT:  The information herein is subject to change without notice.
Please call or write to confirm factual information of importance to you
or your organization.

