Each Syndie instance operates with its own archive of messages, storing
the data extracted from the signed messages
within its local database (by default stored under
$dataRoot/db/
), an archive of those signed messages in the
$dataRoot/archive/
file hierarchy, and an archive of locally
created but not yet distributed messages in the
$dataRoot/outbound/
file hierarchy. The contents of
$dataRoot/archive/
can be wiped out without any loss of
functionality, though doing so prevents the Syndie instance from sharing
the authenticated messages with other people.
Within the $dataRoot/archive/
directory is a periodically
rebuilt file shared-index.dat
, containing a listing of files
in the archive that should be reachable, some metadata surrounding each of
those channel and message files, and some general metadata for the archive
itself. This file is only accessed when serving up the archive on the web -
never for those not publishing an archive. In general, it contain a subset
of the available files underneath the archive directory, so as to hide some
anonymity-related facts (such as delaying the listing of locally created
messages until they have been published elsewhere, etc).
The shared-index.dat
file uses the following format:
$archiveFlags // 2 bytes - see below for meaning $archiveAdminChannel// 4 bytes - index into the channels where the archive admin posts $numAltURIs // 1 byte for (i = 0; i < $numAltURIs; i++) $url // UTF-8 encoded SyndieURI of some other archives $numChannels // 4 bytes for (i = 0; i < $numChannels; i++) $channelHash // 32 byte SHA256 value $channelEdition // 8 byte unsigned integer $channelFlags // 1 byte - see below $numMessages // 4 bytes for (i = 0; i < $numMessages; i++) $messageId // 8 byte unsigned integer $scopeChannel // 4 byte unsigned integer - index into $numChannels above $targetChannel // 4 byte unsigned integer - index into $numChannels above $msgFlags // 1 byte - see below $archiveFlags: first byte: high bit: if true, the archive wants to receive only recent messages bit 6 : if true, the archive wants to receive only messages in known channels bit 5 : if true, the archive will accept passphrase encrypted messages bit 4 : if true, the archive will accept private reply messages bit 3 : if true, the archive requires a passphrase to post to it bit 2-0 : along with bits 7 and 6 of the second byte, configures the hashcash requirements to post second byte: high bit: see above bit 6 : see above bits 5-4: index rebuild frequency 0 = hourly, 1 = every 6 hours, 2 = every 12 hours, 3 = every 24 hours bits 3-0: maximum message size the archive will receive, in kilobytes, bitshifted 0 = 1KB, 1 = 2KB, 2 = 4KB, ..., 14 = 8MB, 15 = 16MB $channelFlags: high bit: if true, the channel info is passphrase encrypted bit 6: if true, the channel info publishes its read key bit 5: if true, the channel info has been updated "recently" bit 4: if true, the archive wants to receive updates to this channel's metadata bit 3: if true, the archive wants to receive new messages in this channel bits 2-0: undefined $messageFlags: high bit: if true, the message is passphrase encrypted bit 6: if true, the message is encrypted to the forum owner bit 5: if true, the message publishes its read key bit 4: if true, the archive considers this message "new" bits 3-0: message size shift, in kilobytes, starting at 4KB 0 = 4KB, 1 = 8KB, 2 = 16KB, ..., 14 = 32MB, 15 = 64MB
Individual posts are found under
$dataRoot/archive/$scope/$messageId.syndie
, and metadata under
$dataRoot/archive/$scope/meta.syndie
. The externally referenced
posts are found under their original scope path, not the targetted channel
path - $dataRoot/archive/$scopeHash/$messageId.syndie
and not
$dataRoot/archive/$channelHash/$messageId.syndie
Given the simple file-based archive hierarchy and index, running a public
Syndie archive is trivial - simply publish your $dataRoot/archive/
in a webserver and tell people the URL. They will then be able to load up
their Syndie instance and use the getindex and fetch
commands to pull posts from the archive into their local Syndie instance.
Syndie also includes a built-in HTTP server (run with the
httpserv command), which both serves up the
content and accepts HTTP posts from others (if allowed), limiting the data accessible
to those messages and channel metadata files published in the
shared-index.dat
file.
The server accepts posts (uploaded through schedule and put), writing them to a temporary directory under the syndie root, and running bulkimport on them after they've all been read.
The HTTP post received is not a mime-encoded post, but a fairly simple concatenation of all of the data to be uploaded (with metadata messages first):
POST /import.cgi\r\n Content-length: $total\r\n \r\n $headerSize // 2 bytes $header // $headerSize bytes, for authorization/etc foreach (msg) $msgFlags // 1 byte. 0x0 for normal posts, 0x1 for metadata posts $msgSize // 4 bytes $msgData // $msgSize bytes