Xbase64 DBMS Chapter 2

System Overview and Internals

Chapter Updated 11/01/25

The information in this chapter is mainly for the curious and as reference material. It is not really required information for effectively using the library.

The XBase64 library is a cross platform object oriented C++ set of classes for accessing and manipulating the following Xbase file types:

Extension	Description	Supported by Xbase64 library?
*.DBF	Main DBF file or table name	Y
*.DBT	Verson 3 or Version 4 memo file	Y
*.NDX	Single tag index file	Y
*.MDX	Multi tag index file	Y
*.INF	ODBC file used for auto open of NDX files	Y
*.NTX	Clipper indexfile	Future
*.CDX	Fox Pro indexfile	Future
*.IDX	Fox Pro index file	Future

DBF files are comprised of a variable length header record which stores information about the file and describes he fixed length record format, followed by a series of fixed length data records.

Each fixed length data record is preceded by a one byte indicator which identifies if the record has been deleted. If the record is not deleted, the indicator is a space (0x20). If deleted, the indicator contains an asterisk (0x2A). Data fields are stored in records without field separators or record terminators.

In earlier releases of dBASE, there is an ASCII NULL character between the $0D end of header indicator and the start of the data. This NULL was removed starting with dBASE III Plus, making a Plus header one byte shorter than an identically structured III file. The methods documented in the Xbase software and documentation follow the more recent version where the NULL character is not included.

Each database file is comprised of zero, one or many records. A record is comprised of fields. Only one record is accessed at a time.

Zero, one or many database files can be open simultaneously.

The Record Buffer

When using the Xbase routines, each open data file has a record buffer which is manipulated by calling the database, index and field routines.

If AutoCommit is turned on (Default), updates are committed from the record buffer to the database when a write, or append is performed. The library automatically writes updates to the database if the buffer has been updated and the record is repositioned or the database is closed.

If AutoCommit is turned off, updates will need to be explicity committed to the database file with one of dbf->Put(), dbf->Append() or dbf->Commit() command depending on context.. Updates can be cancelled with the Abort() command.

The record buffer is not used for handling the actual data portion of memo fields. When working with memo fields, the application program must allocate enough buffer space for reading and writing memo fields or use the xbString class for handling memo data.

Internal to the library, there is an additional record buffer which stores the original value of the data record before any changes are made. This is used by the index routines for finding and deleting original key values from any open indices before adding the new keys. If the key values are not changed, no index updates occur. Additionally, calling the Abort() method will back out any updates to the record buffer.

Xbase Database File Header

There are two file header version.
Level 5 supports all dBase versions up through version 6.
Level 7 supports dBASE versions 7 and greater.

The Xbase file header, located at the beginning of the database, describes the .DBF database. Knowledge of this structure is not necessary to effectively utilize the Xbase64 libraries.

Level 5 Header Layout

Position	Length	Description
0	1 byte	file version number (03H without a .DBT file) (83H with a .DBT file)
1-3	3 bytes	date of last update (YY MM DD) in binary format
4-7	32 bit number	number of records in data file
8-9	16 bit number	length of header structure
10-11	16 bit number	length of the record
12-13	20 bytes
14	1 byte	Transaction flag
15	1 byte	Encryption flag
16-27	20 bytes
28	1 byte	Index flag
29	1 byte	Language Driver ID
30-31	2 bytes
32-n	32 bytes each	field descriptor record (see below)
n+1	1 byte	0DH as the field terminator

Click here for Level 7 Header Layout Details

Level 7 Constraints

Level 7 DBF files provide support for various constraint types and custom properties.

Standard Constraints

Minimum value

Maximum value

Default value

Required value

Database constraint - For example: field1 >= field2

Referential Integrity Constraints

Referential integrity is used for validating data. It guarantees a data value in one (child) table matches a value in another (parent) table or is null.

Custom Properties

There is a section in the table header for custom properties. Custom properties are user defined and can be estabished at the table or field level.

Version 4.2.6 of the xbase64 library supports:

The ability to read and reference the standard constraints in the table.

The ability to read and reference the referential integrity constraints in the table.

The ability to read and reference custom properties in the table.

The ability to enforce thestandard constraints in a given dbf table, if they are defined.

Future xbase64 libary versions will include support for:

Enforcement of referencial integrity constaints

Improved ability to utilize custom properties

The ability to add or modify constraints and custom properties for a give table.

Xbase Field Descriptor Record

The Xbase field descriptor record stores information about each field in the database.
Knowledge of this structure is not necessary to effectively utilize the Xbase libraries.

Level 5 Field Descriptor Record

Position	Length	Description
0-10	11 bytes	field name in ASCII zero-filled
11	1 byte	field type in ASCII (C N L D or M)
12-15	32 bit number	field data address
16	1 byte	field length in binary
17	1 byte	field decimal count in binary
18-31	14 bytes	reserved bytes (version 1.00)

Field Data Format

Data are stored in ASCII format in the database as follows:

DATA TYPE	DATA RECORD STORAGE
Character	ASCII characters, left justified, right blank filled
Date	(8 digits in YYYYMMDD format, such as 19601007 for October 7, 1960)
Logical	? Y y N n T t F f (? when not initialized)
Memo	10 digits representing a .DBT block number
Numeric	. 0 1 2 3 4 5 6 7 8 9 + -, right justified, left blank filled
Float (Version IV only)	. 0 1 2 3 4 5 6 7 8 9 + -, right justified, left blank filled

Memo Fields

Memo fields store variable length data elements in a seperate .DBT file. The main .DBF file maintains a ten byte field which is used by the Xbase routines for determining the location of the data in the .DBT file.

Xbase DBMS supports both dBASE III+ and dBASE IV version memo files. The version IV files are somewhat more efficient in that they reuse unused memo space when data are deleted or freed from use. With version III files, all new updates are appended to the end of the file and the unused space is not reclaimed until the datafiles are packed.

Memo fields can be used for storing a variety of date type. However, type 3 files are limited to storing textual data because most internal memo field processing in a type 3 file relies on two contiguous 0x1a charaters.

Type 4 memo files are also used for storing binary and OLE data, both of which became available in version 5 of dBASE.

Technical memo file information

The following info on memo fields is for the curious. It is not required reading if you don't need to know the internals.

Memo files are made up of one or more blocks

For version III files, the block size is 512

For version IV files, the block size is a multiple of 512

The minimum amout of space necessary to store one memo field is one block or 512 bytes.

The default block size can be adjusted by manipulating the XB_DBT_BLOCK_SIZE macro in the options.h file.

The main .DBF file maintains a ten byte numeric field which is blank if no memo data exists for a given field. Otherwise it contains a number, which when multiplied by the block size, points to the offset in the file of the head block in the file.

For version 3 memo field files, there are two fields in the head block of the file, NextBlockNo and Version. Depending on the Xbase software, some vendors products update these two fields, some do not. The Xbase library keeps the fields updated, but does not rely on them to be valued with correct data. This helps to support maximum compatibility amoungst all Xbase tools available.

For version 4 memo field files, the first block in the .DBT file is a header block which is comprised of 8 bytes of data which maintain the file's block size and the next free block available in the file. Blocks two through n contain the actual memo data. A chain of empty blocks is maintained within the file for potential future use. When an add or update routine executes, it first attempts to find a spot in a set of blocks which were earlier allocated, but not currently in use for the data. If no free spot is found, data are appended to the end of the file.

The version 3 memo file was used in dBASE version III Plus and probably earlier. The version 4 memo file is used in dBASE versions IV and greater.

The free block chain is sorted in block number order. When blocks of data are freed and added to the free block chain, the routines will attempt to concatonate free block chains togethor where possible. When a delete occurs, or an update which requires less space occurs, the new free space is added to the free block chain.

Memo File Block Types

Memo Block Types
Head Block
Only data block for memo field
First of several contiguous data block set
2-n of contiguous data block set
Only data block in free chain (version IV only)
First of several contiguous free block set (version IV only)
2-n of contiguous free block set (type IV only)

Head Block Structure

Start Pos	End Pos	Len	Type	Description
1	4	4	int	Next Block ID
5	8	4	int	Not used all 0x00's
9	16	8	char	Version IV only. Filename, left justified, right 0x00 filled. Files names longer than 8 bytes are trunctated at 8 bytes.
17	17	1	char	0x03 for Version III 0x00 for Version IV
18	18	1	char	0x00
19	20	2	char	Version 3 is 0x00, 0x00 Version 4 is 0x02, 0x01. Not sure what this represents.
21	22	2	int	Version IV only. Block Size
23	Blocksize	Blocksize - 22		filler

Version IV Head Data Block Structure

Start Pos	End Pos	Len	Type	Description
0	1	2	short int	0xff, 0xff (-1)
2	3	2	short int	Starting position of data (always 8?)
4	7	4	int	Length of data includes first 8 bytes
8	Calc	Calc	char	Data

Version IV Head Free Block Structure

Start Pos	End Pos	Len	Type	Description
0	3	4	int	Next free block in the free block chain
4	7	4	int	Number of free blocks in this contiguous free block set

Version 3 and 4 memo fields are terminated with two contiguous 0x1A bytes of data.

Note: The above information was found from internet research, reviewing data files and general reverse engineering. If you find it to be inaccurate or have additional information, please forward your notes to the group.

Block Reads

As of release 4.1.1, the Xbase library includes functionality for reading a DBF file in blocks, rather than one record at a time.

This functionality can be used to improve application performance in situations where a data file is being read sequentially. For situations where records are retrieved randomly from the file, enabling this probably won't help much. In short, this can be turned on when accessing a file sequentially and should be left off when not processing sequentially.

The logic is handled internally within the library, all that is needed is to enable it and the library handles the rest. Additionally, the block read functionality was designed with for sequential file access and is designed with reporting in mind. It doesn't currently have any auto locking associated with it.

To enable and disable Block Reading for a DBF file, use xbDbf::EnableBlockReadProcessing() and xbDbf::DisableBlockReadProcessing().

Locking Overview

Xbase64 supports multi-user processing through file and record locks. Record locking restricts multiple cooperating programs from simultaneously accessing the same data and corrupting it. Without record and file locking in a multi-user environment, simultaneous access to the data and index files can cause the files to become inaccurate and unusable.

Automatic record locking is on by default in the Xbase64 library. To turn it on or off use method xbXBase::SetMultiUser().

Locking can also be enabled / disabled at the table level with with xbDbf::SetMultiUSer().

If autolocking is disabled and the code base is being used in a multi user environment, it is up to the application program to verify the needed locks are set as there is no checking or setting any locks if autolocking is turned off. It is only safe to turn off the autolocking functionality if the library is being used in a single user environment.

The current Xbase64 record locking logic is modeled after dBASE V7 locking.

The locking methods return either XB_LOCK_FAILED or XB_NO_ERROR. If they return XB_LOCK_FAILED the actual reason can be found in the global variable errno or function perror() can be executed to view the results.

The errno field may contain one of the following values if the lock was not successful.

Error Code	Description
EBADF	Invalid file descriptor
EINVAL	Invalid lock information or file does not support locks
EACCESS EAGAIN	Lock can not be set because it is blocked by an existing lock on the file.
ENOLCK	The system is out of lock resources, too many file locks in place.
EDEADLK	Deadlock condition
EINTR	Process was interrupted by a signal while it was waiting

Linux/Windows File Locking Compatibility Issue

There is a compatibility locking issue to be aware of. Windows environments allow for the exclusive opening of file handles and Linux/Unix platforms do not. If you are writing an application that will be using a tool like dBASE on a Windows machine, accessing a file on a Linux/Samba configure machine, be aware that the file could be opened in exclusive mode by dBASE on the Windows system, and the same file could be simultaneously opened with a program on the Unix box. That could cause some issues.

In Unix, a program can not lock a file so another process can not access it.
In Windows, a program can lock a file so another process can not access it.
dBASE supports routines to open files exclusively, preventing other users from opening a file.
Locking on the Mac/Apple platform only works on NFS shares. It does not work with SMB shares.

Samba settings

If you will be using Samba on Linux/Unix and sharing files between Linux and Windows machines, you will need to disable oplocks. In the smb.conf file, set:

[sharename]
oplocks = False
level2 oplocks = False

There is a known issue with Samba and Windows 7. Xbase64 runs very slow on Windows 7 connecting to a Samba Server. If you know how to resolve this, please send an email to support. Thanks.

iLockFlavor

The library was constructed in a manner so that it could be updated to support alternate lock "flavors". The 4.x.x library is built to mirror the dBASE locking, but the structure is in place to expand to other locking types if needed.

Table Manager

Like most of the information in this chapter, the table manager information is included for reference purposes. Application programs don't need to be concerned with the table manager as the list is maintained automatically.

Xbase64 maintains an internal list (xbTblList) of open tables that is automatically updated by the Open and Close routines.

See class xbTblMgr for more info.

Each entry in the list includes:

Fully qualified file name of the table (dbf file).

Table name (without file path or extension).

Optional unique alias.

Pointer address of the associated xbDbf instance.

Other Notes:

The alias is optional but if used it must be unique.

The same file can be opened multiple times. Each instance must have a unique alias identifier.