Advanced Heap Management for Turbo Pascal 5.5
Version 5.5
January 6, 1990

Overview
--------------------------------------------------------------------------
Turbo Pascal's heap is one of the most useful and powerful features of the
language. By using the heap, programs can access all 640K of DOS memory in a
completely dynamic fashion. With power comes responsibility, however. Managing
pointers to the heap is one of the trickiest subjects in Pascal. And as time
goes by, the 640K of DOS memory doesn't seem all that big -- accessing even
more memory is desirable.

This collection of Turbo Pascal units and utilities offers methods for
managing and extending the heap. It includes the following:

o a patch to TPC.EXE so that the compiler will generate an interrupt to a
user-supplied routine after dereferencing each pointer.
o a unit that checks for dereferencing an invalid pointer.
o a unit that transfers control of the New, Getmem, Dispose, and FreeMem
procedures to user-supplied routines.
o a unit that logs various information about the heap to disk at
convenient points in a program.

This collection of utilities is an updated version of those previously
written for Turbo Pascal 4.0 and 5.0.

Several people have developed virtual heap managers using earlier versions of
the compiler patch. This version of the patch should allow those utilities to
work with Turbo Pascal 5.5. Be careful to note the version 5.5 specific
comments in the documentation below, however. You can find these comments by
searching for the string Note!

Patching TPC.EXE
--------------------------------------------------------------------------
The patching program HPAT55 makes a small but important change to TPC.EXE
(version 5.5 only). Once the change is in place, the compiler is capable of
generating an interrupt each time a pointer is dereferenced. When the
interrupt occurs, a user-supplied routine takes control, thus allowing a
program to validate or modify the pointer value.

To apply the patch, compile the supplied program HPAT55.PAS, and run it while
a copy of TPC.EXE is in the current directory. HPAT55 modifies about 70
bytes of the command line compiler (most of which are required to relocate a
particular table so that it can hold two more bytes). Note that the patcher
works only on TPC, not on TURBO.EXE. While it would be possible to make the
same patch to TURBO.EXE, we rarely use the integrated compiler ourselves, and
it is linked differently enough to make finding the patch points a
time-consuming chore.

HPAT55 assures that the compiler version you have is Turbo Pascal 5.50. If
the file size is wrong, or if any of the locations to be patched contain
unexpected data, HPAT55 will halt with an error message. If this happens, be
sure to restore a clean copy of TPC.EXE from a backup, since a partial patch
may have occurred.

After the patch is successfully completed, the compiler's behavior is changed
in the following manner:

0. The compiler version will be reported as "5.5p" whenever TPC writes its
copyright message.

1. A new compiler switch directive will exist: {$P+} or {$P-}. The default
state of this switch is {$P-}. When the switch is turned ON, the compiler
will generate an interrupt $66 after each pointer dereference. The switch
may be turned on and off as desired in the source code only (not at the DOS
command line nor in a TPC.CFG file). Like other Turbo compiler directives,
the switch returns to its default state at the beginning of each unit.

It is your program's responsibility to install an interrupt handler prior to
the first occurrence of int $66 in the program. The supplied unit BADPTR
automatically installs such a handler for you.

Interrupt $66 is one of the user-definable interrupts described by IBM. If
this interrupt conflicts with your application, you may change HPAT55.PAS to
use a different interrupt. To do so, simply change the constant
DerefInterrupt in HPAT55.PAS, recompile the program and patch a fresh copy
of the compiler. User-definable interrupts range from $60 to $66. ($67 is
used for EMS, so it should be avoided here.)

If you change the interrupt, be sure to modify any related interrupt handler
(such as the one in BADPTR) as well.

2. Upon entry to your interrupt handler, ES:DI will hold the current value
of the pointer being dereferenced. The interrupt handler must preserve the
values of AX, BX, CX, DX, SI, DS, SS, BP, and SP, and it must return either
the same value of ES:DI or a value appropriately mapped by a virtual memory
manager.

While a Turbo Pascal Interrupt procedure will work correctly for the pointer
handler, the performance penalty may be unacceptable. For best performance,
the int $66 handler should be written primarily in assembly language. See
BADPTR.ASM for a simple example of such a handler.

The following fragment shows the code generated by the patched compiler when a

pointer P is a global variable, in this case a pointer to an integer.

P^ := 1;
C43E0000 LES DI,[P] ;load pointer into ES:DI
CD66 INT 66 ;call user routine
26C7050100 MOV WORD PTR ES:[DI],0001 ;assign value

An easy rule to remember is this: for each appearance of a dereferencing caret
(^) in your source code, an interrupt will be generated. Of course, this
occurs only when the $P+ directive is in effect for that statement.

Note!
In version 5.5 of Turbo Pascal, a dereferencing interrupt occurs in one
additional situation that is related to object-oriented programming. The
following code generates an interrupt:

dispose(anobjectptr, done);

This makes sense, since the statement is a kind of shorthand for the
following:

anobjectptr^.done;
dispose(anobjectptr);

The code that actually calls dispose in this case is located within the done
destructor, but the effect is the same.


Using BADPTR
--------------------------------------------------------------------------
BADPTR is a unit that works in conjunction with the HPAT55-patched compiler.
If you compile with the patched compiler, you must be sure that you USE the
unit BADPTR (or another unit with an int $66 handler), or your program may
crash unexpectedly. (Interrupt $66 often points to an IRET by default, but
that is not always the case.) You should USE BADPTR early in the USES
statement of the main program. If you USE BADPTR and then don't compile with
the patched compiler, no harm will result.

BADPTR automatically installs an interrupt handler that is invoked whenever a
pointer is dereferenced and the $P+ directive is active. This interrupt
handler checks that the pointer refers to the normal Turbo Pascal heap,
between HeapOrg and the top of the free list. If the pointer falls into this
range, the interrupt handler returns and the program proceeds. If the pointer
is outside of the normal heap, BADPTR calls an error routine, which writes the
value of the pointer as well as the relative code address where the error
occurred, and then halts. Note that NIL pointers will always fail BADPTR's
test. You can use Turbo's find runtime error facility to correlate the error
address to a source position.

If you are using pointers not allocated by New or GetMem, there are two ways
to keep BADPTR from reporting a false error. A common example of such a
pointer would be one pointing to the DOS command line, initialized with P :=
Ptr(PrefixSeg, $80). First, you could assure that the $P+ directive is not
active wherever you dereference such a pointer. Second, BADPTR interfaces two
WORD variables that allow you to determine the acceptable range for pointers.
HeapBot is the lowest acceptable pointer segment, and HeapTop is the highest.
BADPTR initializes these to the range of the normal Turbo heap. Suppose you
wanted to accept any pointer but the NIL pointer. In that case you could
assign:

HeapBot := $0001;
HeapTop := $FFFF;

BADPTR uses simple WriteLn statements to report an error. If this technique is
not appropriate for a particular application, modify the procedure BadPointer
in BADPTR.PAS. Upon entry to BadPointer, the system variable ErrorAddr
contains the PSP-relative address of the code causing the error, and BadP
contains the faulty pointer. Generally, BadPointer should halt without
returning. If it does not halt, execution will continue after the interrupt,
using the pointer originally supplied in ES:DI. The BadPointer procedure
cannot change the actual pointer value.

If you change HPAT55 to use a different interrupt, be sure to make the
corresponding change in BADPTR as well.


Using GRABHEAP
--------------------------------------------------------------------------
GRABHEAP is a unit that allows a program to take control of memory allocation
and deallocation functions normally handled by the system unit: NEW, GETMEM,
DISPOSE, and FREEMEM. To offer complete control, GRABHEAP interfaces two
procedures:

procedure CustomHeapControl(GetPtr : GetMemFunc; FreePtr : FreeMemProc);
{-Give control of GetMem, New, FreeMem, Dispose to specified procedures}

procedure SystemHeapControl;
{-Restore control to the system heap routines}

A program can call CustomHeapControl to transfer control of the system
routines to specified procedures. To do so, pass the addresses of two FAR,
global procedures that match the following declarations:

{$F+}
function CustomGetMem(Size : Word) : pointer;
begin
...
end;
procedure CustomFreeMem(P : Pointer; Size : Word);
begin
...
end;
{$F-}

To set up for this example, an appropriate call would be

CustomHeapControl(CustomGetMem, CustomFreeMem);

Note!
Due to changes in Turbo Pascal 5.5's heap manager, the declarations for the
custom routines are different than for previous versions of these utilities.

Thereafter, any calls that would normally go to New or GetMem are transferred
to CustomGetMem, and calls for Dispose or FreeMem are sent to CustomFreeMem.

The custom heap management routines can perform any needed actions, including
calls to the original system GetMem and FreeMem routines. To call the original
routines, the program must temporarily restore control to the system runtime
library by calling GRABHEAP's SystemHeapControl routine.

Here is an example of CustomGetMem and CustomFreeMem routines that do nothing
but keep a balance sheet of the memory allocated and deallocated in a program:

var
TotalAlloc : LongInt;
HeapMax : Pointer;

{$F+}
procedure MyFree(var P : Pointer; Size : Word); forward;

function MyGet(Size : Word) : pointer;
var
P : pointer;
begin
Inc(TotalAlloc, Size); {Update balance sheet}
SystemHeapControl; {Give back heap control temporarily}
GetMem(P, Size); {Use the system routine to allocate}
MyGet := P; {Assign it to function result}
if LongInt(HeapPtr) > LongInt(HeapMax) then
HeapMax := HeapPtr; {Keep track of heap high water mark}
CustomHeapControl(MyGet, MyFree); {Take over heap control again}
end;

procedure MyFree(P : Pointer; Size : Word);
begin
Dec(TotalAlloc, Size);
SystemHeapControl;
FreeMem(P, Size);
CustomHeapControl(MyGet, MyFree);
end;
{$F-}

begin
TotalAlloc := 0; {No memory allocated to start}
HeapMax := HeapOrg; {High water mark at base of heap}
CustomHeapControl(MyGet, MyFree); {Take over heap control}

.... {Normal program actions}

WriteLn('Maximum heap usage: ',
16*(LongInt(seg(HeapMax^))-seg(HeapOrg^)), ' bytes');
if TotalAlloc <> 0 then
WriteLn('Allocated memory not freed: ', TotalAlloc, ' bytes');
end.

GRABHEAP can be put to more powerful uses, of course. For example, based on an
installation flag, it could select among data storage in normal, EMS, or disk
memory. The pointer returned by the custom GetMem routine need not be a normal
pointer, but can instead be an EMS page and offset, or a disk page and offset,
combined into a four byte record. Then, based on the same installation flag, a
deref-interrupt handler can access the data on the appropriate media and
return a pointer to the actual data buffered in memory.

Note!
In Turbo Pascal 5.5, there is one form of call to New() that won't be
redirected to the custom routine you install. That's a call to allocate an
object and call its constructor at the same time:

new(AnObjP, Init);

or

AnObjP := new(AnObjPtrType, Init);

In both of these cases the allocation is performed by code within the
constructor itself. GRABHEAP doesn't redirect this code to the custom routines
since the code must perform special checks related to inheritance.


Using HEAPLOG
-------------------------------------------------------------------------
HEAPLOG is a unit that builds on top of the GRABHEAP facility to provide
diagnostic information for programs that make extensive use of the heap. It
keeps a log of all heap allocation and deallocation, and allows the program to
dump this log to disk at any time. By studying the log, you can detect
excessive heap fragmentation and memory that hasn't been deallocated.

You'll get a basic log just by using HEAPLOG in your program. HEAPLOG creates
a file named HEAP.LOG. When a program first starts, a report labeled "Initial"
is written to HEAP.LOG. When the program ends, a report labeled "Final" is
written to HEAP.LOG. The reports themselves will be described momentarily.

HEAPLOG interfaces two procedures that control the logging process:

procedure DumpHeapLog(Msg : string);
{-Write the current heap log to a file}

procedure ClearLog;
{-Clear all entries from the log}

DumpHeapLog adds another report to HEAP.LOG, giving it the label passed in
Msg. ClearLog clears HEAPLOG's internal data structures so that succeeding
reports will be relative to that point instead of relative to the beginning of
the program.

The following is an example of a HEAPLOG report.

Initial

MemAvail: 385568
MaxAvail: 385568
HeapPtr : 41DE:0000
HeapCnt : 0
FreeCnt : 0
Filled : FALSE

Intermediate

MemAvail: 382017
MaxAvail: 380742
HeapPtr : 4309:0002
HeapCnt : 15
FreeCnt : 5
Filled : FALSE

Pointer Size Allocated at
42CF:0008 115 0000:0AE3
42E9:000F 499 0000:0AE3
41F8:000F 387 0000:07F1
4229:0004 396 0000:07F1
4259:0007 125 0000:07F1
4261:0004 415 0000:07F1
427B:0003 116 0000:07F1
4282:0007 269 0000:07F1
4293:0004 223 0000:07F1
42A4:0009 158 0000:07F1
42AE:0007 42 0000:07F1
42B1:0001 112 0000:07F1
42B8:0001 74 0000:07F1
42BC:000B 301 0000:07F1
42D8:0008 279 0000:07F1

Free start Size
42D6:000B 29
42A1:0003 54
4242:0000 375
4211:0002 386
41DE:0000 431

Final

MemAvail: 385568
MaxAvail: 385568
HeapPtr : 41DE:0000
HeapCnt : 0
FreeCnt : 0
Filled : FALSE

HEAPLOG always generates the Initial and Final reports. For programs that
deallocate all their dynamic memory, the Final report should be the same as
the Initial. In the example, the Intermediate report is one generated by
calling DumpHeapLog directly.

Each report shows the values of MemAvail and MaxAvail in bytes, and the
current heap high water mark, HeapPtr. Each also shows the number of separate
blocks allocated on the heap (HeapCnt) and the number of blocks deallocated
and currently on the free list (FreeCnt). If HeapCnt is non-zero, HEAPLOG
shows the value of each allocated pointer, the size of the region it points
to, and the code address where the pointer was allocated. If FreeCnt is
non-zero, HEAPLOG shows the starting address of each free block and its size.

By default, HEAPLOG can track the allocation of up to 1000 pointers in one
run. If the program allocates more than this at one time, the Filled field
will report True. HEAPLOG's capacity can be adjusted by modifying the constant
MaxLog in HEAPLOG.PAS. Note that HEAPLOG itself uses 10 bytes of heap space
for each increment in MaxLog. (HEAPLOG doesn't report its own heap usage,
however.) The performance of calls to GetMem, FreeMem, New, and Dispose will
degrade for large numbers of pointers.

Note!
Because of the GRABHEAP limitation already described, HEAPLOG won't report
dynamic instances of objects that are allocated and initialized with a single
call to New(Obj, Init).

Legalities
--------------------------------------------------------------------------
Chris Franzen of O.K.Soft, West Germany, used earlier versions of HEAP.ARC to
help track down the compiler patch locations for Turbo Pascal 5.5. Thanks to
Chris for spending the time to do this.

Other programs and documentation in this package are copyright (C) TurboPower
Software, 1988, 1989, 1990. All rights reserved. TurboPower Software hereby
grants permission for free distribution of this software, and for use of these
units and techniques within commercial and non-commercial applications. The
units and utilities themselves may not be distributed commercially without
obtaining written permission from TurboPower Software.

We would appreciate hearing about enhancements made to these routines. Contact
Kim Kokkonen at Compuserve ID 76004,2611 or write to:

TurboPower Software
P.O. Box 66747
Scotts Valley, CA 95066