Commit b57cb701 authored by David Byers's avatar David Byers
Browse files

Minor updates. Added dbck.texi

parent ef07d126
* Better Text Garbing (nisse@lysator.liu.se)
Keep texts that have comments newer than N days, where N is a
parameter of the conference.
The parameter N has been implemented (keep_commented). The garb
has to be changed (this is also in the TODO file.) It might be
desirable to look even deeper into the comment chain so we keep
*chains* that have additions newer than N days.
* Chop up texts before sending (byers@lysator.liu.se)
We aren't too good at handling large texts. The problem is that we
start off by reading the entire thing into memory from disk, then
shove it all onto an ISC queue. For big texts that means we sit and
wait for I/O of the text, then we sit and wait while we queue it all
up, and while we're doing that, we actually wait for the server to
write the garbage to the pipe!
It would be nice to be able to spread the work around.
I think the trick would be to not read everything from disk at once.
We want to read a bit, send it, read some more, and send that. We
add a function to ISC that lets us push an ISC packet onto the ISC
queue that contains no data, just an instruction to call a callback.
That callback reads some more data from the from the disk and queues
hat in ISC too.
When we want to output a text, we start by enqueueing a callback to
read the second bit of the text. Then we read the first bit and
enqueue that. When we call isc_flush, isc will start by calling the
callback. It will enqueue a callback to read the third bit from
disk, then read the second bit from disk and enqueue that. Then we
return to ISC and let ISC write the first bit to the socket. The
result is that the ISC queue will contain block N of the text, then
a callback to get N+2, then block N+1.
The reason for doing it this way is that if we can get the callback
to do the reads asynchronously, we can have ISC writing a bit of
data to the socket while we are reading the next bit from disk. If
we can do async reads, when ISC gets to the callback, the queue will
contain a callback for block N+1, then block N of the text. The
callback will enqueue a callback for block N+2, then initiate an
async read for block N+1 and return immediately. The server will
write block N to the socket concurrently with the read of block N+1.
When the async read returns, its handler (callback, signal handler,
whatever) enqueues the data in the ISC queue. This should result in
the server spending much less time in IO wait.
If we can't do async reads, then we might as well just put the
callback for block N+1 after the data for block N. Async reads are
available under Solaris, but not in Linux 2.0. Regrettably the
interface is probably vendor-dependent.
A potential problem is that we need to make *sure* that the text
file is not cleaned when a text is deleted. That is not a problem in
the current server, but could be in the future. If we have a pending
request for text N, and someone deletes it before we can send the
whole thing, we have to be able to continue sending it.
Changes need to be made to ISC (typed queue elements, call the
callback); prot_a_output and mux.c (need to output texts in a
special way); plus we have to write the callbacks. This should be
totally doable.
NEWS: The aio_ interface is not supported by Linux, at least not
yet. It is, however, a Posix 4 extension, not just a Solaris thing.
On Linux we'd have to use poll or select to do asynchronous IO. Is
it possible to use ISC for this as well?
* Sync the database without blocking requests (byers@lysator.liu.se)
Currently the database is synced in three phases. Phase one creates
a snapshot of all changed objects by traversing the internal arrays
and moving the data pointer to the snapshot pointer in the cache
nodes. The second phase writes objects to disk. The third phase
frees the snapshot data from memory. The first and last phase block
the server completely.
Rewrite the sync algorithm like this. Assume that all objects in the
database can be ordered, regardless of type. The server can be in
one of two states: syncing or not syncing.
When the server starts syncing, open the target data file and write
the database header to the file. Write kom_info to the database. Set
next_to_sync to zero.
Periodically, sync a few objects. Examine object next_to_sync. If it
has been saved to disk, do nothing. If it has a snapshot, write the
snapshot to disk. If it is in memory, write it to disk from memory.
If it is on disk, copy it from the old data file. Clear the
snapshot. Clear the dirty flag. Clear the saved-to-disk flag.
Increase next_to_sync and repeat this step until a fixed number of
objects have been saved.
When all objects are done, set the state of the server to not
syncing, set the current data file to the one just synced.
While the server is syncing, access to objects needs to be done
carefully. If an object is requested that has not been synced,
create a snapshot of it before returning it, or write it to disk
immediately. If an object that has been synced but is not in memory
is requested, then it must be read from the new data file.
Tricky stuff: if something is added while we are syncing, it should
not be written to disk. If something is deleted while we are syncing
is should be written to disk anyway.
The basic idea here is to make sure that the database file contains
a copy of the database as it was when the sync was started, no
matter what happens with it afterwards. We have like an index;
everything before the index has been synced and can be treated as if
we are not syncing at all. Everything beyond the frontier may be
unsynced, and we need to make sure that it is retained unchanged
until it gets written to disk.
* Hierarchical Conferences (byers@lysator.liu.se)
LysKOM conferences should be organized hierarchically.
Give each conference a parent conference pointer and a children
conference list. Internally, prepend the parent ID to each
conference name, so that we can have two conferences with the same
name contained below different parent conferences. The prepending
should not be visible to the client.
Permissions in a conference are based on the permissions in the
conference closest in child-parent order to the requested conference
in which the person is a member (whew!) Some examples: conference A
is rd_prot and conference B is not. Persons who are members of A can
see anything in A and B. Persons who are members in B can see
anything in B, but not in A. To persons who are members of neither,
B behaves as if it were rd_prot, since its parent conference is.
It has to be possible to do name lookup on a subtree of the
conference tree.
The small_conf_stat structure should probably mirror the tree
structure and contain permission information about conferences.
Don't forget that we want to be able to do sequential traversal of
all elements in a subtree.
This should require surprisingly small changes in the server. Mainly
small_conf_stat, name lookup and storage, and permissions checks.
* Garbage Collecting Conferences (byers@lysator.liu.se)
Conferences have an expire field. Conferences that have not
contained texts for that many days should be removed automatically
by the garbage collector or by dbck.
* Mark Types (byers@lysator.liu.se)
Marks should be fixed. Marks should have flags and stuff. Texts
should know who has marked them. There's a text about this somewhere
in LysKOM.
* Multibyte character compatibility (byers@lysator.liu.se)
The server should be capable of operating in multi-byte
environments. We'll have to be able to specify a coding system and
charset for conference and person names. We'll have to make sure the
regexp engine can deal with multibyte characters. We'll need a new
method of separating the subject line from the text body. The name
matching will have to be more flexible. For eight-bit characters
we'll need a way of specifying a collate table. For multibyte
characters, maybe we'll have to be able to specify a procedure to
determine equivalence of two characters.
Local variables:
mode: outline
paragraph-separate: "[ \t ]*$"
outline-regexp: "[* ]+"
end:
......@@ -2,7 +2,7 @@
@c
@c FIXME: Explain how the garb works with nice and keep-commented
@c
@c $Id: Protocol-A.texi,v 1.48 1999/01/07 15:10:33 byers Exp $
@c $Id: Protocol-A.texi,v 1.49 1999/01/13 12:00:13 byers Exp $
@c %**start of header
@setfilename protocol-a.info
@settitle LysKOM Protocol A
......@@ -2197,7 +2197,8 @@ to see the contents of the structure.
)
Membership ::=
( last-time-read : Time;
( position : INT32;
last-time-read : Time;
conference : Conf-No;
priority : INT8;
last-text-read : Local-Text-No;
......@@ -2213,6 +2214,8 @@ membership in a conference. It is returned by the @ref{query-read-texts}
and @ref{get-membership} calls.
@table @code
@item position
The position of this membership in the membership list.
@item last-time-read
The time when the person last read anything from the conference.
@item conference
......@@ -2762,9 +2765,9 @@ in the example.
* sync-kom:: Save the database (43)
* shutdown-kom:: Shut LysKOM down (44)
* broadcast:: O Broadcast a message. Replaced by call 53 (45)
* get-membership-old:: O Get membership for a person (46)
* get-membership-old:: O Get membership for a person. Use call 99 (46)
* get-created-texts:: O Get texts created by a user. Use call 104 (47)
* get-members-old:: O Get members of a conference (48)
* get-members-old:: O Get members of a conference. Use call 101 (48)
* get-person-stat:: Get status information for a person (49)
* get-conf-stat-old:: O Get status information for a conference (50)
* who-is-on:: O Get current sessions. Replaced by call 63 (51)
......@@ -2784,7 +2787,7 @@ in the example.
* re-lookup-person:: O Look up a person based on name (65)
* re-lookup-conf:: O Look up a conference based on name (66)
* lookup-person:: O Find persons matching abbreviated name (67)
* lookup-conf:: Find conference matching abbreviated name (68)
* lookup-conf:: O Find conference matching abbreviated name (68)
* set-client-version:: Set the name and version the client (69)
* get-client-name:: Get the name of the client (70)
* get-client-version:: Get the version of the client (71)
......@@ -6508,6 +6511,12 @@ The session @code{session-no} does not exist.
-> ( HOLLERITH );
@end example
This call returns the collate table being used by the server to match
names. If index A and index B in the string are the same character,
characters A and B are considered equivalent. Clients should be prepared
for collate tables of any length. An empty collate table indicates that
the server considers all characters different.
@unnumberedsubsubsec Error codes
This call always succeeds.
......@@ -6969,8 +6978,7 @@ complete the call anyway.
@example
query-read-texts [98] (( person : Pers-No;
conference : Conf-No; ))
-> (( position : INT32;
mship : Membership ));
-> ( mship : Membership );
@end example
This call is used to find the number of unread texts in a conference.
......
\input texinfo
@c $Id: dbck.texi,v 1.1 1999/01/13 12:00:17 byers Exp $
@c %**start of header
@setfilename dbck.info
@settitle "dbck Reference Manual"
@setchapternewpage odd
@c %**end of header
@ifinfo
This is the reference manual for the lyskomd LysKOM server version 2.0
Copyright @copyright{} 1999 Lysator ACS.
Permission is granted to make and distribute verbatim copies of this
specification provided the copyright notice and this permission notice
are preserved on all copies.
@end ifinfo
@dircategory LysKOM
@direntry
* dbck: (dbck). dbck reference manual.
@end direntry
@titlepage
@sp 10
@title "dbck Reference Manual"
@sp 2
@subtitle "Server version 2.0"
@sp 2
@author by the lyskomd developers
@page
@vskip 0pt plus 1filll
Copyright @copyright{} 1995-1999 Lysator ACS
Permission is granted to make and distribute verbatim copies of this document
provided the copyright notice and this permission notice are preserved on all
copies.
Modified versions of this document may be redistributed with the added
condition that all modifications not cleared with the LysKOM development group
are clearly marked and that the entire modified work be redistributed under the
same conditions as the original.
Permission is granted to copy and distribute translations of this manual into
another language under the same conditions as for modified versions.
@end titlepage
@ifinfo
@node Top, Copying, (dir), (dir)
@comment node-name, next, previous, up
@top dbck
dbck is a program that can is used for minor database maintenance tasks
and for repairing a corrupt lyskomd database.
@menu
* Copying:: LysKOM is free software.
* Overview:: Overview of dbck.
* Invoking dbck:: How to run dbck.
* Notes:: Notes about running dbck.
* Files:: Files used by dbck.
* Bugs:: Known bugs.
@end menu
@end ifinfo
@node Copying, Overview, Top, Top
@comment node-name, next, previous, up
@chapter Copying
dbck is part of the lyskomd distribution.
lyskomd is free software. It is distributed under the Gnu General Public
License version 2. The file COPYING in the top level of the distribution
contains the text of the license.
@node Overview, Invoking dbck, Copying, Top
@comment node-name, next, previous, up
@chapter Overview
The dbck program is used for minor maintenance of the LysKOM database
and for repairing corrupt databases. In brief it performs the following
functions:
@itemize @bullet
@item Compact the text file to remove deleted texts.
@item Repair inconsistent membership information.
@item Repair invalid recipients
@item Repair inconsistent comment links
@item Correct invalid local text numbers
@item Correct invalid text maps
@item Set special conferences
@item Convert between database formats
@end itemize
@node Invoking dbck, Notes, Overview, Top
@comment node-name, next, previous, up
@chapter Invoking dbck
The functionality of dbck is controlled through command-line switches.
These are documented below.
@menu
* General Options:: Controlling the overall behavior of dbck.
* Database Repair Options:: Repairing errors in the LysKOM database.
* Format Conversion Options:: Converting the database file to a new format.
* Database Maintenance Options:: Options for database maintenance.
* Reporting Options:: Options controlling status reports.
@end menu
@node General Options, Database Repair Options, Invoking dbck, Invoking dbck
@comment node-name, next, previous, up
@section General Options
These options control the general behavior of lyskomd.
@table @asis
@item @code{-v} or @code{--verbose}
Verbose mode. Report not only errors but the status of the database.
@item @code{-F} or @code{--force-output}
This option forces dbck to write the database file. Normally @code{dbck}
will only write a new database file if changes have been made for some
other reason. If you want to simply convert a database from one version
to another, you will probably have to give this option.
@end table
@node Database Repair Options, Format Conversion Options, General Options, Invoking dbck
@comment node-name, next, previous, up
@section Database Repair Options
The following options control database repair.
@table @asis
@item @code{-i} or @code{--interactive}
Run interactively. If any inconsistency is found, a remedial cure will
be suggested and the user must confirm the correction.
@item @code{-r} or @code{--auto-repair}
Repair simple errors without asking.
@item @code{-c} or @code{--set-change-name}
Consider it an error if the @code{change-name} capability of a person is
not set. Due to a bug that capability was never set for newly created
persons in release 1.6.1 of lyskomd. This option can be used to repair
the damage.
@end table
@node Format Conversion Options, Database Maintenance Options, Database Repair Options, Invoking dbck
@comment node-name, next, previous, up
@section Format Conversion Options
dbck can be used to conver the LysKOM database from one storage format
to another. This is necessary only when moving the database to a new
server version.
@table @asis
@item @code{-F} or @code{--force-output}
This option forces dbck to write the database file. Normally @code{dbck}
will only write a new database file if changes have been made for some
other reason. If you want to simply convert a database from one version
to another, you will probably have to give this option.
@item @code{-o} or @code{--output-version}
This option is used to set the output version of the database. This
option will normally be used in conjunction with the @code{-F} option.
@end table
Version 1.9 of @code{lyskomd} requires database version 1; version 2.0
requires database version 2. Versions of @code{lyskomd} prior to 1.9
requires database version 0. Note that information is irrevocably lost
when converting from a higher to a lower database version. This options
requires an argument: the output format version.
@node Database Maintenance Options, Reporting Options, Format Conversion Options, Invoking dbck
@comment node-name, next, previous, up
@section Database Update Options
dbck can be used to update certain aspects of the database that either
were impossible to update in early versions of protocol A or that are
inconvenient in all protocol versions.
@table @asis
@item @code{-g} or @code{--compact-text-mass}
Do garbage collection on the texts part of the database. This removes
all unreferenced texts from the database.
@item @code{-P} or @code{--clear-password}
Clear the password of a specified user. This option is silently ignored
if the user does not exist. This option requires an argument: the ID of
the person whose password is to be cleared.
@item @code{-G} or @code{--grant-all}
Grant all privileges to the specified user. This option is silently
ignored if the user does not exist. This option requires an argument:
the ID of the person who is to be granted all privileges.
@item @code{--pers-pres-conf}
Set the person presentation conference of the server to the specified
conference. Since version 1.9 of lyskomd the @code{set-info} call can be
used to do this.
@item @code{--conf-pres-conf}
Set the conference presentation conference of the server to the specified
conference. Since version 1.9 of lyskomd the @code{set-info} call can be
used to do this.
@item @code{--motd-conf}
Set the message-of-the-day conference of the server to the specified
conference. Since version 1.9 of lyskomd the @code{set-info} call can be
used to do this.
@item @code{--kom-news-conf}
Set the news-about-lyskom conference of the server to the specified
conference. Since version 1.9 of lyskomd the @code{set-info} call can be
used to do this.
@item @code{--motd-of-kom}
Set the message of the day of the server to the specified text. Since
version 1.9 of lyskomd the @code{set-info} call can be used to do this.
@end table
@node Reporting Options, , Database Maintenance Options, Invoking dbck
@comment node-name, next, previous, up
@section Reporting Options
These options control reporting of information about the database.
@table @asis
@item @code{-s} or @code{--print-statistics}
Gather statistics about the lengths of texts. A table containing the
frequence of all lengths that are currently present is printed.
@item @code{-t} or @code{--list-text-no}
Print ``Checking @i{text-no}'' for every text that examined.
@b{Warning:} This produces lots of output.
@end table
@node Notes, Files, Invoking dbck, Top
@comment node-name, next, previous, up
@chapter Notes
The messages ``Conference @i{conf-no} has a bad Text-list. Starts with
0'' and ``Person @i{pers-no} has created @i{num} conferences, no @i{num}
(as said in person-stat).'' are normal. If tou get them when you specify
the @code{-g} option, let @code{dbck} repair them and run @code{dbck -g}
again.
@node Files, Bugs, Notes, Top
@comment node-name, next, previous, up
@chapter Files Used by dbck
dbck uses the same files as @code{lyskomd} (@xref{(lyskomd)}.)
All file names can be changed in the server configuration file.
@xref{(lyskomd)Parameters}.
@table @code
@item /usr/lyskom
Default value of @i{prefix}. The default of this value is set at compile
time, but it can be changed in the server configuration file.
@xref{(lyskomd)Parameters}.
@item @i{prefix}/db/lyskomd-data
Half of the database: all status information. This
@item @i{prefix}/db/lyskomd-texts
The other half of the database: the actual texts.
@item @i{prefix}/db/lyskomd-backup
A backup copy of @code{lyskomd-data}. Never, ever delete this file
unless you know what you are doing, or you may lose the entire data
base. Most of the time this is the only complete database file!
@end table
@node Bugs, , Files, Top
@comment node-name, next, previous, up
@chapter Known Bugs
@itemize @bullet
@item Does not lock the database.
@item Never checks if the database is locked.
@item Should have an unlock database option.
@item Does not check that the data file and text file are consistent.
@end itemize
@contents
@bye
\input texinfo
@c $Id: hacking.texi,v 1.1 1999/01/07 15:10:38 byers Exp $
@c $Id: hacking.texi,v 1.2 1999/01/13 12:00:17 byers Exp $
@c %**start of header
@setfilename hacking.info
@settitle "Hacking lyskomd"
......@@ -58,7 +58,8 @@ another language under the same conditions as for modified versions.
* Adding Configuration Parameters to the Server:: How to add config params.
* Adding a New Protocol Request:: How to extend protocol A.
* Adding Aux-Item Types::
* Modifying an Existing Type::
* Modifying an Existing Stored Type::
* Notes::
@end menu
@end ifinfo
......@@ -263,6 +264,7 @@ not logged on) and @code{03.exp} (normal behavior.)
@menu
* Adding New Input Types:: Procedure for adding input types.
* Adding New Result Types:: Procedure for adding output types.
* Modifying Output Types:: Procedure for modifying output types.
@end menu
......@@ -300,7 +302,7 @@ Initialize the contents of the field in @code{init_connection()}.
@node Adding New Result Types, , Adding New Input Types, Adding a New Protocol Request
@node Adding New Result Types, Modifying Output Types, Adding New Input Types, Adding a New Protocol Request
@comment node-name, next, previous, up
@section Adding New Result Types
......@@ -324,7 +326,62 @@ to a connection. Use the existing functions as templates.
@end table
@node Adding Aux-Item Types, Modifying an Existing Type, Adding a New Protocol Request, Top
@node Modifying Output Types, , Adding New Result Types, Adding a New Protocol Request
@comment node-name, next, previous, up
@section Modifying Output Types
When you modify an existing type you have to rename the old version of
the type since it will still be used in existing calls. The convention
hs previously been to rename @code{Something} to @code{Something_old},
but the preferred method is to append an underscore and the protocol
version in which the current version of the type was introduced. For
example, if the type @code{Gazonk} was introduced in protocol version
11, and a new version is to be introduced in protocol version 15, the
current @code{Gazonk} structure is renamed to @code{Gazonk_11}.
This is to avoid names like @code{Something_older},
@code{Something_oldest} and @code{Something_Truly_Ancient}.
Changes need to be made to the following files:
@table @code
@item Protocol-A.texi
Document the new type in the appropriate section. Rename the existing
type in the type documentation and in all calls that return it. Be
thorough!
@item fncdef.txt
Rewrite all calls that use the modified type so they use the old version
of the type.
@item prot-a.c
Modify the current line in @code{prot_a_reply} for the existing version
of the type, and add a new line for the new version of the type.
@item connections.h
Modify the existing entry in @code{Res_type} and @code{Result_holder},
if necessary. This should only be necessary if the server uses both a
new and old type internally, which is not recommended. Add new entries
for the new version of the type.
@item prot-a-output.h
@item prot-a-output.c
Rename the existing output routing according to the new name of the
type. Write a new output routine for the new version of the type.
@item memory.c
If there are functions for the type in @code{memory.c}, make sure that
your new type is initialized, cleared and copied in an appropriate
manner.
@end table
If the type you modify is stored in the database, make sure it gets
saved properly. @xref{Modifying an Existing Stored Type}.
@node Adding Aux-Item Types, Modifying an Existing Stored Type, Adding a New Protocol Request, Top
@comment node-name, next, previous, up
@chapter Adding Aux-Item Types
......@@ -350,9 +407,9 @@ step.
@end enumerate
@node Modifying an Existing Type, , Adding Aux-Item Types, Top
@node Modifying an Existing Stored Type, Notes, Adding Aux-Item Types, Top
@comment node-name, next, previous, up
@chapter Modifying an Existing Type
@chapter Modifying an Existing Stored Type
If you want to modify an existing type that is stored in the database,
think again. Can the job be done with aux-items instead? Is it really
......@@ -390,7 +447,7 @@ dependent so that they can deal with the new database format.