			     Corrupt Database Files
			     ======================

							  Author: Herbert Straub
						  e-mail: herbert@linuxhacker.at
								Date: 2004-09-26

Situation:
----------

Try to fixing database corruption errors, such as this Debian bug:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=52583

or this one (reported via mailinglist):
Aug 24 08:51:06 mail NewsCache[10564]: NewsCache Server Start Aug 24
08:51:11 mail NewsCache[14658]: nnrpd: access entry name matched:
0.0.0.0/0.0.0.0 Aug 24 08:51:11 mail NewsCache[14658]:
mail.thriftinternet.com [127.0.0.1] connect Aug 24 08:51:11 mail
NewsCache[14658]: NServer::NServer() hostname set to:
mail.thriftinternet.com Aug 24 08:51:11 mail NewsCache[14658]: nnrpd caught
Error Exception! File: NVcontainer.cc Function: virtual void
NVcontainer::make current() Line: 135
Desc: NVConatiner(19603): info-record shrunk
Aug 24 08:51:11 mail NewsCache[10564]: receiving signal SIGCHLD: 17



Behaviour since NewsCache 1.3.22
--------------------------------

(This section replaces the 2004 notes on error messages, automatic
recovery, newscacheclean and the list of constraints, which described
NewsCache 1.2rc6.)

The spool's database files are a group's .db (its index, overview
records and small articles) and the active database .active64. NewsCache
maps them into memory, so every offset and size in them comes from the
file. All of them are checked before they are used: the header (size,
version, length of the file), the free list, the index or hash table,
each record and each chain of records, and the text of stored articles.

A file that fails a check is moved to name.bad (e.g. .db.bad,
.active64.bad), and a new, empty database is started under the old name.
It fills itself from the upstream server again: a group's overview and
articles when they are asked for, the active database at the next
access. The log gets one line at error level, e.g.

  corrupt database /var/cache/newscache/at/test/.db: bad index entry;
  moved to /var/cache/newscache/at/test/.db.bad, starting a new one

In a session, the command that ran into the damage is run once more on
the new database if nothing was sent yet; if part of a reply was sent,
it is ended properly (a list with its final "."), so the client stays in
step. Any other error a command doesn't handle gets a 503 reply instead
of ending the session.

newscacheclean logs a damaged database and goes on with the next group;
it removes the .bad files (they are only kept until its next run, for a
look). updatenews prefetches a group whose database was moved aside into
the new one.

Everything in the spool is a cache of an upstream server, so nothing is
lost but the time to fetch it again. (The "local newsgroups" of the
manual are groups from a second upstream server, not groups stored only
in NewsCache's spool.)

The tests are in tests/e2e/corrupt_test.go: damaged headers, cut-off
files, damaged indexes and random damage anywhere in a .db or
.active64.
