IDEAS 8.18 KB
Newer Older
David Byers's avatar
David Byers committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
* Better Text Garbing (nisse@lysator.liu.se)

  Keep texts that have comments newer than N days, where N is a
  parameter of the conference.

  The parameter N has been implemented (keep_commented). The garb
  has to be changed (this is also in the TODO file.) It might be
  desirable to look even deeper into the comment chain so we keep
  *chains* that have additions newer than N days.


* Chop up texts before sending (byers@lysator.liu.se)

  We aren't too good at handling large texts. The problem is that we
  start off by reading the entire thing into memory from disk, then
  shove it all onto an ISC queue. For big texts that means we sit and
  wait for I/O of the text, then we sit and wait while we queue it all 
  up, and while we're doing that, we actually wait for the server to
  write the garbage to the pipe! 

  It would be nice to be able to spread the work around.

  I think the trick would be to not read everything from disk at once. 
  We want to read a bit, send it, read some more, and send that. We
  add a function to ISC that lets us push an ISC packet onto the ISC
  queue that contains no data, just an instruction to call a callback.
  That callback reads some more data from the from the disk and queues 
  hat in ISC too.

  When we want to output a text, we start by enqueueing a callback to
  read the second bit of the text. Then we read the first bit and
  enqueue that. When we call isc_flush, isc will start by calling the
  callback. It will enqueue a callback to read the third bit from
  disk, then read the second bit from disk and enqueue that. Then we
  return to ISC and let ISC write the first bit to the socket. The
  result is that the ISC queue will contain block N of the text, then
  a callback to get N+2, then block N+1.

  The reason for doing it this way is that if we can get the callback
  to do the reads asynchronously, we can have ISC writing a bit of
  data to the socket while we are reading the next bit from disk. If
  we can do async reads, when ISC gets to the callback, the queue will 
  contain a callback for block N+1, then block N of the text. The
  callback will enqueue a callback for block N+2, then initiate an
  async read for block N+1 and return immediately. The server will
  write block N to the socket concurrently with the read of block N+1.
  When the async read returns, its handler (callback, signal handler,
  whatever) enqueues the data in the ISC queue. This should result in
  the server spending much less time in IO wait.

  If we can't do async reads, then we might as well just put the
  callback for block N+1 after the data for block N. Async reads are
  available under Solaris, but not in Linux 2.0. Regrettably the
  interface is probably vendor-dependent.

  A potential problem is that we need to make *sure* that the text
  file is not cleaned when a text is deleted. That is not a problem in 
  the current server, but could be in the future. If we have a pending 
  request for text N, and someone deletes it before we can send the
  whole thing, we have to be able to continue sending it.

  Changes need to be made to ISC (typed queue elements, call the
  callback); prot_a_output and mux.c (need to output texts in a
  special way); plus we have to write the callbacks. This should be
  totally doable.

  NEWS: The aio_ interface is not supported by Linux, at least not
  yet. It is, however, a Posix 4 extension, not just a Solaris thing.
  On Linux we'd have to use poll or select to do asynchronous IO. Is
  it possible to use ISC for this as well?
  


* Sync the database without blocking requests (byers@lysator.liu.se)

  Currently the database is synced in three phases. Phase one creates
  a snapshot of all changed objects by traversing the internal arrays
  and moving the data pointer to the snapshot pointer in the cache
  nodes. The second phase writes objects to disk. The third phase
  frees the snapshot data from memory. The first and last phase block
  the server completely.

  Rewrite the sync algorithm like this. Assume that all objects in the 
  database can be ordered, regardless of type. The server can be in
  one of two states: syncing or not syncing.

  When the server starts syncing, open the target data file and write
  the database header to the file. Write kom_info to the database. Set 
  next_to_sync to zero.

  Periodically, sync a few objects. Examine object next_to_sync. If it 
  has been saved to disk, do nothing. If it has a snapshot, write the
  snapshot to disk. If it is in memory, write it to disk from memory.
  If it is on disk, copy it from the old data file. Clear the
  snapshot. Clear the dirty flag. Clear the saved-to-disk flag.
  Increase next_to_sync and repeat this step until a fixed number of
  objects have been saved.

  When all objects are done, set the state of the server to not
  syncing, set the current data file to the one just synced.

  While the server is syncing, access to objects needs to be done
  carefully. If an object is requested that has not been synced,
  create a snapshot of it before returning it, or write it to disk
  immediately. If an object that has been synced but is not in memory
  is requested, then it must be read from the new data file. 

  Tricky stuff: if something is added while we are syncing, it should
  not be written to disk. If something is deleted while we are syncing 
  is should be written to disk anyway.

  The basic idea here is to make sure that the database file contains
  a copy of the database as it was when the sync was started, no
  matter what happens with it afterwards. We have like an index;
  everything before the index has been synced and can be treated as if 
  we are not syncing at all. Everything beyond the frontier may be
  unsynced, and we need to make sure that it is retained unchanged
  until it gets written to disk.


* Hierarchical Conferences (byers@lysator.liu.se)

  LysKOM conferences should be organized hierarchically.

  Give each conference a parent conference pointer and a children
  conference list. Internally, prepend the parent ID to each
  conference name, so that we can have two conferences with the same
  name contained below different parent conferences. The prepending
  should not be visible to the client.

  Permissions in a conference are based on the permissions in the
  conference closest in child-parent order to the requested conference 
  in which the person is a member (whew!) Some examples: conference A
  is rd_prot and conference B is not. Persons who are members of A can 
  see anything in A and B. Persons who are members in B can see
  anything in B, but not in A. To persons who are members of neither,
  B behaves as if it were rd_prot, since its parent conference is.

  It has to be possible to do name lookup on a subtree of the
  conference tree.

  The small_conf_stat structure should probably mirror the tree
  structure and contain permission information about conferences.
  Don't forget that we want to be able to do sequential traversal of
  all elements in a subtree.

  This should require surprisingly small changes in the server. Mainly 
  small_conf_stat, name lookup and storage, and permissions checks.


* Garbage Collecting Conferences (byers@lysator.liu.se)

  Conferences have an expire field. Conferences that have not
  contained texts for that many days should be removed automatically
  by the garbage collector or by dbck.


* Mark Types (byers@lysator.liu.se)

  Marks should be fixed. Marks should have flags and stuff. Texts
  should know who has marked them. There's a text about this somewhere 
  in LysKOM.


* Multibyte character compatibility (byers@lysator.liu.se)

  The server should be capable of operating in multi-byte
  environments. We'll have to be able to specify a coding system and
  charset for conference and person names. We'll have to make sure the 
  regexp engine can deal with multibyte characters. We'll need a new
  method of separating the subject line from the text body. The name
  matching will have to be more flexible. For eight-bit characters
  we'll need a way of specifying a collate table. For multibyte
  characters, maybe we'll have to be able to specify a procedure to
  determine equivalence of two characters. 



Local variables:
mode: outline
paragraph-separate: "[ \t]*$"
outline-regexp: "[*]+"
end: