Data Integrity Protection provided by DataFlex for Btrieve on Novell NetWare

by Brad Waddell

Brad is the Designer and co-developer of DataFlex for Btrieve.

Data Integrity issues are prominent in the features of Novell's Btrieve Record Manager, which was the primary reason the DataFlex for Btrieve product was created.

Since Btrieve is a data management module only, and has no programming front end, it's design has been focused on being among the most robust and reliable file systems available for LAN's today. It is primarily used as a Novell NetWare NLM (NetWare Loadable Module), where it can do file management for a variety of front end products (such as DataFlex for Btrieve) in true Client/Server design.

First among these benefits is the fact that the record manager is adding and deleting records and reindexing inside the server, protecting the file from corruption of data and indexes by stations on the network. Stations (Clients) may only request that the specified record be added to a file, but it never actually opens or writes to the file. This puts an end to reindexing due to index corruption because of a station lockup or disconnect.

Btrieve uses a system called Shadow Paging to protect the integrity of updates to a database file. This technique creates a copy of all sections of a file that are going to be changed (data and index pages) before making the change. The changes are then made to the copies, and upon completion, the copies are marked to become the current areas and the originals are marked empty. If a system failure occurs before the changes are completed, the next time the file is opened, the shadow copies would be eliminated and the originals retained, avoiding problems with incomplete processes.

Transaction logging is an important feature of Btrieve which allows all record updates to be saved to a seperate log file (typically on a different drive or device). This log file should be deleted each time a backup is run. If a hard disk failure should occur, the previous backup files can be restored, and the transaction logs can be rolled-forward to re-save all data changes into the file, bringing it back up to the state it was in before the failure occured.

Transaction Tracking (TTS) allows multiple files to be updated as a group, and if anything were to cause the file update to fail, all changes made to that point would be rolled-back as if they had never happened. Unlike NetWare TTS which is file based, Btrieve TTS is record based, allowing multiple TTS transactions to occur on the same file concurrently. The programmer can also code into his program the decision to force an abort to a transaction if he desires.

The continuous operation feature allows a set of Btrieve files to be backed up to tape while they are still open and being written to by users, allowing 24 hour operation. To accomplish this, Btrieve has a special type of transaction to start continuous mode.

When this transaction is started, the original Btrieve file becomes read-only and shareable to allow reads by users and the backup device. During the transaction, file updates are placed in a special delta image file, which mirrors the design of the original file. Records found during this transaction period will first check the delta files for updated records. Once the backup is complete, the delta file changes are saved back into the main file. This mode is completely transparent to the clients requesting data from the Btrieve server.

Btrieve also supports the use of Referential Integrity constraints against a set of database records. This allows you to place data rules with the record manager that must be followed by each client to keep data integrity, and will be enforced by the server. These would include not being able to delete a parent record with active child records, or deleting customers with a balance. This feature can only be activated by using the tools provided with the NetWare SQL product.

DDF files are special Btrieve files which include full definitions of how fields are arranged in a Btrieve file, which allows various tools to use and manipulate data in Btrieve properly without data errors, and to assure that when field locations change that this information is updated the next time the tool opens the file.

Owner names are how Btrieve refers to file passwords, which can be used by your application to insure data integrity and privacy from other applications. The file can be protected completely with encryption, or it can be made read-only to stations which do not have the password. Your application can also change the file passwords at any interval to increase protection levels.

Additional data protection is available through the NetWare file-level security, which will insure that only users with the proper file/directory access rights will be able to use the data in a file, even though it is opened by the file server.

Record locking through Btrieve protects data against multi-user conflicts, in addition to the reread/field locking which is emulated by DataFlex for Btrieve. In fact, Btrieve includes passive concurrency abilities, so if an application were to have no multi-user lock commands, the server would still protect the integrity of the system by creating implicit write locks.

Btrieve rarely creates a corrupted file, but should a server defect of some kind cause one to occur, the record manager would detect it and prevent your application from writing any further records (which would increase the problem) until the file is recovered. The BUTIL application runs as a server NLM to recover data from damaged Btrieve files.

Novell Btrieve also benefits from standard NetWare data protection features such as read after write verify and hot-fix of bad disk blocks.

DataFlex for Btrieve was designed to run DataFlex applications with the Btrieve file system in the same way as standard DataFlex, but the benefits of Btrieve have provided the following data protection features that run differently:

• Automatic index allocation by Btrieve means that there is no need as in standard DataFlex to increase the maximum number of records periodically and reindex to re-size the index tree. This prevents the index file full error from occuring.
• DataFlex for Btrieve prevents the "edited record not saved" message, since it does not do any part of the save until the save command is executed.
• Some versions of DataFlex can skip a record during periods of heavy updates where the index is being split. This problem does not occur under Btrieve.
• You can designate a file as being open in exclusive mode, and when in use it cannot be opened by other users.
• Writing to a data file is prevented while indexes are being created, but the data can still be read.
• The ZEROFILE command is prevented while the file is in use by other stations.

Although not as fast as the Btrieve NLM, the Client versions does allow for the creation of standard Btrieve files from the station, and the use of pre-imaging (similar to shadow-paging), transaction tracking (TTS) and transaction logging to protect data.

Brad Waddell
FLEXquarters
1125 W Baseline #2-F
Mesa AZ 85210
(602) 827-9368
fax: (602) 345-2893
Compuserve: 76477,3147