Generic Multilog Toolkit.

Read Grid4All to get some context.

Download the first (alpha) version of the toolkit.

The generic multilog toolkit implements a filesystem structure that includes a number of log files. Each log file belongs to an editor on the network. Editors log their updates to a shared document in their own log files and read all other logs for the others' updates. The toolkit, besides log manipulation, enables each editor to serve his own log from his site so that each other editor can maintain a replica in their own site. The replication process is designed to handle casual network disconnection of any editor.

It uses TCP/IP sockets for communication and Linux inotify to monitor local files. It has four main tools; a log server, a log client, a log reader/writer (logtool) and a multilog daemon.

LogServer

The LogServer monitors a log directory for changes and serves read requests from the network. Read requests are made at an offset relative to the entire log. The LogServer handles the different chunkfiles efficiently. Clients may request to be subscribed for notifications when the log changes. The LogServer will dispatch a short message with the new log size to subscribers. The local participant directly appends to the log. The LogServer uses the system's facilities to be notified of changes (linux inotify).

LogClient

The LogClient is bound to a specific LogServer. The LogClient creates an empty log directory and then updates according to the LogServer. The LogClient will only append to its log. Throttling of synchronization granularity can be controlled by the local user, via special access patterns to files. The local participant directly reads the log. If the LogClient is disconnected from the network, this appears as idleness. The LogClient will attempt to learn about the LogServer's state according to a heartbeat. When there is demand, the LogClient will subscribe for notifications so that latency is minimized.

LogTool

The LogTool makes access to the structure of a log. It provides reading and writing logs, handling log chunks efficiently, and mechanisms for constructing and maintaing an in-memory state of the log. The LogTool is used by the LogServer and LogClient, but can also be used from command line, for reading, writing, or waiting on a log.

MultilogD

MultilogD is a daemonized script that parses a list of participant and populates a multilog directory with one log for each. LogServer is started for the local participant's log and LogClient is started for all the others. MultilogD periodically polls the list of participants and adds any new ones and re-launches daemons for logs, if they have vanished.