SDBM(3)

NAME
sdbm, sdbm_open, sdbm_prep, sdbm_close, sdbm_fetch, sdbm_store, sdbm_replace, sdbm_delete, sdbm_exists, sdbm_firstkey, sdbm_nextkey, sdbm_hash, sdbm_rdonly, sdbm_error, sdbm_clearerr, sdbm_dirfno, sdbm_pagfno, sdbm_datfno - data base subroutines

SYNOPSIS
#include <sdbm.h>

typedef struct {
	char *dptr;
	size_t dsize;
} datum;

datum nullitem = { NULL, 0 };

DBM *sdbm_open(char *file, int flags, int mode)
DBM *sdbm_prep(char *dirname, char *pagname, char *datname,
        int flags, int mode)
void sdbm_close(DBM *db)

datum sdbm_fetch(DBM *db, key)
int sdbm_store(DBM *db, datum key, datum val, int flags)
int sdbm_replace(DBM *db, datum key, datum val, gboolean *existed)
int sdbm_delete(DBM *db, datum key)
int sdbm_exists(DBM *db, datum key)

datum sdbm_firstkey(DBM *db)
datum sdbm_firstkey_safe(DBM *db)
datum sdbm_nextkey(DBM *db)
datum sdbm_value(DBM *db)
int sdbm_deletekey(DBM *db)

long sdbm_hash(char *string, size_t len)

ssize_t sdbm_sync(DBM *db)

int sdbm_set_cache(DBM *db, long pages)
int sdbm_set_wdelay(DBM *db, gboolean on)
int sdbm_set_volatile(DBM *db, gboolean yes)

void sdbm_set_name(DBM *db, const char *string)
const char *sdbm_name(DBM *db)

gboolean sdbm_rdonly(DBM *db)
gboolean sdbm_error(DBM *db)
gboolean sdbm_shrink(DBM *db)
gboolean sdbm_clear(DBM *db)
sdbm_clearerr(DBM *db)
int sdbm_dirfno(DBM *db)
int sdbm_pagfno(DBM *db)
int sdbm_datfno(DBM *db)

gboolean sdbm_is_storable(size_t key, size_t value)

DESCRIPTION

This package allows an application to maintain a mapping of <key,value> pairs in disk files. This is not to be considered a real database system, but is still useful in many simple applications built around fast retrieval of a data value from a key. This implementation uses an external hashing scheme, called Dynamic Hashing, as described by Per-Aake Larson in BIT 18 (1978) pp. 184-201. Retrieval of any item usually requires a single disk access, unless some keys or values are large enough to be stored via indirections, in which case more disk accesses are required. The application interface is compatible with the ndbm(3) library.

An sdbm database is kept in two files usually given the extensions .dir and .pag. The .dir file contains a bitmap representing a forest of binary hash trees, the leaves of which indicate data pages in the .pag file. When facing large keys or values that would not otherwise fit in one page, a third file (carrying the .dat extension) is created to store the large data, with indirection pointers left in the .pag file to be able to access the underlying data.

The application interface uses the datum structure to describe both keys and values. A datum specifies a byte sequence of dsize size pointed to by dptr. If you use ASCII strings as keys or values, then you must decide whether or not to include the terminating NUL byte which sometimes defines strings. Including it will require larger database files, but it will be possible to get sensible output from a strings(1) command applied to the data file.

In order to allow a process using this package to manipulate multiple databases, the applications interface always requires a handle, a DBM *, to identify the database to be manipulated. Such a handle can be obtained from the only routines that do not require it, namely sdbm_open() or sdbm_prep(). Either of these will open or create the two or three necessary files. The difference is that the latter allows explicitly naming the bitmap and data files whereas sdbm_open() will take a base file name and call sdbm_prep() with the default extensions. Furthermore, giving a NULL pointer for datname in sdbm_prep() will disable support for large keys and values. The flags and mode parameters are the same as for open(2).

To free the resources occupied while a database handle is active, call sdbm_close().

Given a handle, one can retrieve data associated with a key by using the sdbm_fetch() routine, and associate data with a key by using the sdbm_store() routine. sdbm_exists() will say whether a given key exists in the database.

The values of the flags parameter for sdbm_store() can be either DBM_INSERT, which will not change an existing entry with the same key, or DBM_REPLACE, which will replace an existing entry with the same key. Keys are unique within the database.

To replace a possibly existing key and know whether the key existed or was actually created, use sdbm_replace() and the parameter existed will be set upon return to indicate whether the key existed already, provided the call is not returning -1.

To delete a key and its associated value use the sdbm_delete() routine.

To retrieve every key/value pairss in the database, use a loop like:

for (key = sdbm_firstkey(db); key.dptr != NULL; key = sdbm_nextkey(db)) {
    datum value = sdbm_value(db);
}

The order of retrieval is unspecified. During traversal, no insertion or deletion of any other key should occur. However, the current key may be deleted through the sdbm_deletekey() routine. Usage of sdbm_value() outside of the outlined iterator loop is forbidden.

The iteration may begin with sdbm_firstkey_safe() instead, which will activate extended checks of each page: keys not belonging to the page are deleted on the fly. This is useful when iterating over a database which could have been corrupted due to I/O errors, and the .dir and .pag files are not consistent with eachother.

If you determine that the performance of the database is inadequate or you notice clustering or other effects that may be due to the hashing algorithm used by this package, you can override it by supplying your own sdbm_hash() routine. Doing so will make the database unintelligable to any other applications that do not use your specialized hash function.

The following functions are also defined in the header file:

    sdbm_rdonly() returns true if the database has been opened read-only.

    sdbm_error() returns true if an I/O error has occurred. After a write I/O error, the database is put in read-only mode until sdbm_clearerr() has been called, to prevent cascading errors.

    sdbm_clearerr() allows you to clear the error flag if you think you know what the error was and insist on ignoring it.

    sdbm_dirfno() returns the file descriptor associated with the bitmap file.

    sdbm_pagfno() returns the file descriptor associated with the page file.

    sdbm_datfno() returns the file descriptor associated with the data file (large keys and values), or -1 if the file was not opened yet because it was not needed so far.

    sdbm_name() returns the database name previous set using sdbm_set_name() or the empty string if none was set.

    sdbm_is_storable() indicates whether a combination of key and value sizes can fit within the size constraints we have.

    sdbm_shrink() attempts to shrink the allocated files on disk to reclaim unused trailing space, after many value deletions for instance. Returns FALSE on errors.

    sdbm_clear() erases all the data held in the database but keeps it opened. Returns -1 on errors.

PAGE CACHING
This sdbm library is equipped with an LRU (Least Recently Used) page cache whose aim is to reduce the amount of read() system calls performed when accessing data. By default, it is set to cache 64 pages, but this may be changed at any time by calling sdbm_set_cache() with the amount of pages desired for caching. Use 1 to disable LRU caching altogether and only keep the last loaded page in memory.

It is also possible to enhance the performance of sdbm by turning write delay on via sdbm_set_wdelay() at any time. This will also greatly reduce the amount of write() system calls performed when changing data, but it could make the database inconsistent in the advent of a crash before sdbm_close() can be called. To mitigate that risk, the application can call sdbm_sync() on a regular basis (say every 5 seconds). That call returns the amount of pages flushed if everything was OK, and -1 if an I/O error occurred during flushing.

Even with deferred writes, there are important operations that are nonetheless flushed immediately to disk, when splitting a page for instance. Otherwise, in the advent of a crash, the disk data could contain twice the same key / value pair. However, if the database is recreated from scratch each time it is opened, there is no need for this precaution, since disk consistency does not matter then. The database is said to be volatile and this behaviour can be turned on by calling sdbm_set_volatile() with a TRUE argument. In which case it also automatically enables delayed writes.

SEE ALSO
open(2).

DIAGNOSTICS
Functions that return a DBM * handle will use NULL to indicate an error. Functions that return an int will use -1 to indicate an error. The normal return value in that case is 0. Functions that return a datum will return nullitem to indicate an error.

As a special case of sdbm_store(), if it is called with the DBM_INSERT flag and the key already exists in the database, the return value will be 1.

In general, if a function parameter is invalid, errno will be set to EINVAL. If a write operation is requested on a read-only database, errno will be set to ENOPERM. If a memory allocation (using malloc(3)) failed, errno will be set to ENOMEM. For I/O operation failures errno will contain the value set by the relevant failed system call, either read(2), write(2), or lseek(2).

AUTHORS
"Ozan S. Yigit" <oz@nexus.yorku.ca>

Raphael Manfredi <Raphael_Manfredi@pobox.com> (for gtk-gnutella extensions)

BUGS
The sum of key and value data sizes must not exceed PAIRMAX (1008 bytes) if large key/value support was disabled by calling sdbm_prep() with a NULL value for datname. Otherwise, with large key/value support enabled, the sum of key and value data sizes cannot exceed 250 KiB currently, but the real limit can be lower depending on the respective sizes of keys and values. Use sdbm_is_storable() to be sure.

The sum of the key and value data sizes where several keys hash to the same value must fit within one bitmap page.

The .pag file will contain holes, so its apparent size is larger than its contents. When copied through the filesystem the holes will be filled.

The contents of datum values returned are in volatile storage. If you want to retain the values pointed to, you must copy them immediately before another call to this package.

The only safe way for multiple processes to (read and) update a database at the same time, is to implement a private locking scheme outside this package and open and close the database between lock acquisitions. It is safe for multiple processes to concurrently access a database read-only.

APPLICATIONS PORTABILITY
For complete source code compatibility with the Berkeley Unix ndbm(3) library, the sdbm.h header file should be installed in /usr/include/ndbm.h.

The nullitem data item, and the sdbm_prep(), sdbm_hash(), sdbm_rdonly(), sdbm_dirfno(), and sdbm_pagfno() functions are unique to this package.

The sdbm_datfno() sdbm_replace(), sdbm_firstkey_safe(), sdbm_deletekey(), sdbm_value(), sdbm_is_storable(), sdbm_sync(), sdbm_srink(), sdbm_clear(), sdbm_set_cache(), sdbm_set_wdelay(), sdbm_set_volatile(), sdbm_set_name(), and sdbm_name() functions are additions made by Raphael Manfredi for gtk-gnutella but anyone is welcome reusing these extensions.

Support for large keys and values was added by Raphael Manfredi in a way that is backward compatible (i.e. databases not requiring this feature should see no difference). When activated (as needed), a third file is created to store the larger keys and values, ending with the extension .dat.