DBF/xBase format notes

Copy Markdown

DBF is a family of related formats, not one fully consistent specification. Select a format profile from the version byte, flags, layout, and producer evidence; never infer complete support from the version byte alone.

Agent checklist

  • Treat all offsets and lengths as byte counts, not character counts.
  • Standard dBASE/FoxPro headers store record count, header length, and record length little-endian. Do not generalize that byte order to memo files or every binary field.
  • For common 32-byte-descriptor layouts, the declared header length points to the first record and includes the 0x0D descriptor terminator. Validate the terminator rather than deriving field count from division alone.
  • Declared record length includes the one-byte deletion marker. It should equal 1 + sum(field widths) for fixed-width layouts. The currently accepted fixed-width profiles reject zero-width field descriptors as invalid structure.
  • Records normally start with 0x20 (active) or 0x2A (deleted). Some writers append 0x1A after the final record; do not require it unless the selected profile does.
  • Fixed-width names and values are bytes until decoded. NUL padding, space padding, and text encoding are separate concerns.
  • FoxBase 16-byte field descriptors contain a width but no numeric scale byte. Exact numeric decoding must inspect the fixed-width value: integer text becomes an integer, while a decimal point preserves a Decimal value.
  • Microsoft documents language-driver 0xC9 as Windows-1251 and 0x03 as Windows-1252. 0x57 is commonly treated as Windows-1252 by DBF readers, while some GIS tooling historically labels it ISO-8859-1; retain caller override and raw policies for producer disagreements.
  • Header date bytes usually mean year since 1900, month, and day, but validate dates without bang functions. FoxBASE 0x02 has a different header layout.
  • Preserve raw version, table flags, language-driver ID, field flags, and reserved bytes needed to diagnose producer-specific variants.
  • Validate structure before reading records: file size, header and record lengths, record count, descriptor terminator, field widths, offsets, and memo pointers.
  • Blank, null, invalid, deleted, and unsupported are different states. Visual FoxPro nullable fields require per-record null metadata; blank bytes are not sufficient.
  • Duplicate field names are legal in real files, often after name truncation. Never silently overwrite one value when constructing a map.

Organizing variant-specific code

Treat a format as a composition of header layout, descriptor layout, memo family, record metadata, and field/value capabilities. Do not copy the complete reader into one module per version, and do not scatter raw version-byte checks through parsers.

  • Keep DBF.FormatProfile as the single version-selection point.
  • Dispatch parsers on profile-selected layouts or families.
  • Put schema parsing, duplicate-name validation, and compiled record offsets in DBF.Schema; keep DBF.Field as one field's metadata.
  • Keep genuinely different memo algorithms in DBF.Memo.DBT3, DBF.Memo.DBT4, and later DBF.Memo.FPT, behind a small facade.
  • Keep shared header logic together until another layout makes separate modules materially clearer.
  • Keep record decoding profile-aware through compiled schema metadata, not raw version checks.
  • Defer value-decoder module boundaries until value, blank/null, encoding, and binary/text contracts are settled.
  • Prefer plain internal modules with explicit function contracts. Do not add one large whole-format behavior whose callbacks combine independently varying concerns.

See ADR 0003.

Memo and binary traps

  • .DBT and .FPT are different families. Extension discovery does not select a safe parser; use the selected DBF profile and verify the memo header.
  • dBASE III DBT, dBASE IV DBT, and FoxPro/VFP FPT differ in block headers, termination, block-size rules, and byte order.
  • Visual FoxPro FPT header and block integers are big-endian. Common VFP binary record fields such as Integer and Currency are little-endian.
  • Memo pointers may be space-padded decimal text or a binary integer depending on the format and field width.
  • The verified dBASE IV fixture stores its little-endian block size at DBT header offset 20, uses FF FF 08 00 block signatures, includes the 8-byte block header in each declared memo length, and uses 0x1F as a text terminator.
  • A memo may span several blocks. Bounds-check block * block_size, headers, and declared payload lengths before allocating or reading.
  • Memo, General, Picture, Blob, and binary-flagged Character/Memo values are not interchangeable. Decode only values known to be textual.
  • Missing memo files, empty memo pointers, invalid pointers, and truncated memo data need distinct outcomes.

Evidence rules

  1. Prefer a primary producer specification.
  2. Record disagreements between sources instead of silently choosing one.
  3. Treat a fixture as evidence for its producer/version, not the whole family.
  4. Cross-check representative files with at least one mature independent reader, but store expected values locally so tests have no runtime dependency.
  5. Keep synthetic malformed binaries in test helpers; do not hand-edit fixture files.

Primary and high-value references

Secondary references

Reference implementations

Use these to compare behavior and discover edge cases, not as normative sources:

  • dbase-rs (Rust) — active typed reader/writer with seekable sources, memo support, encoding policies, and structured errors.
  • dbfread (Python) — unusually readable field and memo parsing; inspect field_parser.py and memo.py. Its comments also document unresolved ambiguities, so cross-check them.
  • python-dbf (Python) — broad dBASE III, FoxPro, VFP, Clipper, memo, null, and value-semantics coverage.
  • OSGeo Shapelib dbfopen.c (C) — mature defensive I/O, offset checks, deletion handling, and code-page metadata for the shapefile-oriented DBF subset; it is not a general memo/VFP reference.

When a source and fixture disagree, preserve the raw bytes, identify the producing application if possible, and capture the decision in a regression test or ADR.