innfeed.conf(5)
NAME
innfeed.conf - configuration file for innfeed
DESCRIPTION
This man page describes the configuration file for version
0.10 (beta) of innfeed. This format has changed dramati-
cally since version 0.9.3.
The file innfeed.conf is used to control the innfeed(1)
program. It is a fairly free-format file that consists of
three types of entries: key/value, peer and group. Com-
ments are taken from the hash character ``#'' to the end
of the line.
Key/value entries are a keyword and a value separated by a
colon (which can itself be surrounded by whitespace). For
example:
max-connections: 10
A legal key starts with a letter and contains only let-
ters, numbers and ``_'', ``-''.
There are 5 different type of values: integers, floating-
point numbers, characters, booleans, and strings. Integer
and floating point numbers are as to be expected except
that exponents in floating point numbers are not sup-
ported. A boolean value is either ``true'' or ``false''
(case is not significant). A character value is a single-
quoted character as defined by the C-language. A string
value is any other sequence of characters. If the string
needs to contain whitespace, then it must be quoted with
double quotes, and uses the same format for embedding non-
printing characters as normal C-language string.
Peer entries look like:
peer <name> {
# body ...
}
The word ``peer'' is required. The ``<name>'' is the same
as the site name in INN's newsfeeds file. The body of a
peer entry contains some number (possibly zero) of
key/value entries.
Group entries look like:
group <name> {
# body
}
The word ``group'' is required. The ``<name>'' is any
string valid as a key. The body of a group entry contains
any number of the three types of entries. So key/value
pairs can be defined inside a group, and peers can be
nested inside a group, and other groups can be nested
inside a group.
Key/value entries that are defined outside of all peer and
group entries are said to be at ``global scope''. There
are global key/value entries that apply to the process as
a whole (for example the location of the backlog file
directory), and there are global key/value entries that
act as defaults for peers. When innfeed looks for a spe-
cific value in a peer entry (for example, the maximum num-
ber of connections to set up), if the value is not defined
in the peer entry, then the enclosing groups are examined
for the entry (starting at the closest enclosing group).
If there are no enclosing groups, or the enclosing groups
don't define the key/value, then the value at global scope
is used.
A small example could be:
# Global value for the process
news-spool: /var/news/spool/articles
# Global value applied to all peers that have
# no value of their own.
max-connections: 5
# A peer definition. ``uunet'' is the name used by innd in
# the newsfeeds file.
peer uunet {
ip-name: usenet1.uu.net
}
peer vixie {
ip-name: gw.home.vix.com
max-connections: 10 # override global value.
}
# A group of two peers who can handle more connections
# than normal
group fast-sites {
max-connections: 15
# Another peer. The ``max-connections'' value from the
# ``fast-sites'' group scope is used. The ``ip-name'' value
# defaults to the peer's name.
peer data.ramona.vix.com {
}
peer bb.home.vix.com {
max-connections: 20 # he can really cook.
}
}
Given the above configuration file, the defined peers
would have the following values for the ``max-connec-
tions'' key.
uunet 5
vixie 10
data.ramona.vix.com 15
bb.home.vix.com 20
Innfeed ignores key/value pairs it is not interested in.
Any config file value that can be set via a command line
option, is not used if the command-line option is given.
Config files can be included in other config files via the
syntax:
$INCLUDE filename
There is a maximum nesting depth of 10.
For a fuller example config file, see the supplied innfeed.conf.
GLOBAL VALUES
The following listing show all the keys that apply to the
process as whole. These are not required (compiled-in
defaults are used where needed).
news-spool
This key requires a pathname value. It specifies
where the top of the article spool is. This corre-
sponds to the ``-a'' command-line option.
input-file
This key requires a pathname value. It specifies
the pathname (relative to the backlog-directory)
that should be read in funnel-file mode. This cor-
responds to giving a filename as an argument on the
command-line (i.e. its presence also implies that
funnel-file mode should be used).
pid-file
This key requires a pathname value. It specifies
the pathname (relative to the backlog-directory)
where the pid of the innfeed process should be
stored. This corresponds to the ``-p'' command-line
option.
debug-level
This key defines the debug level for the process. A
non-zero number generates a lot of messages to
stderr, or to the config-defined ``log-file''.
This corresponsds to the ``-d'' command-line
option.
use-mmap
This key requires a boolean value. It specifies
whether mmap'ing should be used if innfeed has been
built with mmap support. If article data on disk is
not in NNTP-ready format (CR/LF at the end of each
line), then after mmap'ing the article is read into
memory and fixed up, so mmap'ing has no positive
effect (and possibly some negative effect depending
on your system), and so in such a case this value
should be false. This corresponds to the ``-M''
command-line option.
log-file
This key requires a pathname value. It specifies
where any logging messages that couldn't be sent
via syslog(3) should go (such as those generated
when a positive value for ``debug-value'', is
used). This corresponds to the ``-l'' command-line
option. A relative pathname is relative to the
``backlog-directory'' value.
backlog-directory
This key requires a pathname value. It specifies
where the current innfeed process should store
backlog files. This corresponds to the ``-b'' com-
mand-line option.
backlog-highwater
This key requires a positive integer value. It
specifies how many articles should be kept on the
backlog file queue before starting to write new
entries to disk.
backlog-ckpt-period
This key requires a positive integer value. It
specifies how many seconds between checkpoints of
the input backlog file. To small a number will mean
frequent disk accesses, to large a number will mean
after a crash innfeed will re-offer more already-
processed articles than necessary.
backlog-newfile-period
This key requires a positive integer value. It
specifies how many seconds between checks for
externally generated backlog files that are to be
picked up and processed.
dns-retry
The key requires a positive integer value. It
defines the number of seconds between attempts to
re-lookup host information that previous failed to
be resolved.
dns-expire
The key requires a positive integer value. It
defines the number of seconds between refreshes of
name to address DNS translation. This is so long
running processes don't get stuck with stale data,
should peer ip addresses change..
close-period
The key requires a positive integer value. It is
the maximum number of seconds a connection should
be kept open. Some NNTP servers don't deal well
with connections being held open for long periods.
gen-html
This key requires a boolean value. It specifies
whether the status-file should be HTML-ified.
status-file
This key requires a pathname value. It specifies
the pathname (relative to the backlog-directory)
where the periodic status of the innfeed process
should be stored. This corresponds to the ``-S''
command-line option.
connection-stats
This key requires a boolean value. If the value is
true, then whenever the transmission statistics for
a peer are logged, then each active connection logs
its own statistics. This corresponds to the ``-z''
command-line option.
host-queue-highwater
This key requires a postive integer value. It
defines how many articles will be held internally
for a peer before new arrivals cause article infor-
mation to be spooled to the backlog file.
stats-period
This key requires a positive integer value. It
defines how many seconds innfeed waits between gen-
erating statistics on transfer rates.
stats-reset
This key requires a positive integer value. It
defines how many seconds innfeed waits before
resetting all internal transfer counters back to
zero (after logging one final time). This is so a
innfeed-process running more than a day will gener-
ate ``final'' stats that will be picked up by log-
file processing scripts.
initial-reconnect-time
This key requires a postive integer value. It
defines how many seconds to first wait before
retrying to reconnect after a connection failure.
If the next attempt fails too, then the reconnect
time is approximately doubled until the connection
succeeds, or max-reconnection-time is reached.
max-reconnect-time
This key requires an integer value. It defines the
maximum number of seconds to wait between attempt
to reconnect to a peer. The initial value for
reconnection attempts is defined by initial-recon-
nect-time, and it is doubled after each failure, up
to this value.
stdio-fdmax
This key requires a non-negative integer value. If
the value is greater than zero, then whenever a
network socket file dcescriptor is created and it
has a value less than this, the file descriptor
will be dup'ed to bring the value up greater than
this. This is to leave lower numbered file descrip-
tors free for stdio. Certain systems, Sun's in par-
ticular, require this. SunOS 4.1.x usually requires
a value of 128 and Solaris requires a value of 256.
The default if this is not specified, is 0.
GLOBAL PEER DEFAULTS
All the key/value pairs mentioned in this section must be
specified at global scope. They may also be specified
inside a group or peer definition. Note that when peers
are added dynamically (i.e. when innfeed receives an arti-
cle for an unspecified peer), it will add the peer site
using the parameters specified at global scope.
article-timeout
This key requires a non-negative integer value. If
no articles need to be sent to the peer for this
many seconds, then the peer is considered idle and
all its active connections are torn down.
response-timeout
This key requires a non-negative integer value. It
defines the maximum amount of time to wait for a
response from the peer after issuing a command.
initial-connections
This key requires a non-negative integer value. It
defines the number of connections to be opened
immediately when setting up a peer binding. A value
of 0 means no connections will be created until an
article needs to be sent.
max-connections
This key requires positive integer value. It
defines the maximum number of connections to run in
parallel to the peer. A value of zero specifies an
unlimited number of maximum connections. In general
use of an unlimited number of maximum connections
is not recommended. Do not ever set max-connections
to zero with dynamic-method 0 set, as this will
saturate peer hosts with connections. [ Note that
in previous versions of innfeed, a value of 1 had a
special meaning. This is no longer the case, 1
means a maximum of 1 connection ].
dynamic-method
This key requires a value between 0 and 3. It con-
trols how connections (up to max-connections) are
opened up to the maximum specified by max-connec-
tions. In general (and specifically, with dynamic-
method 0) a new connection is opened when the cur-
rent number of connections is below max-connec-
tions, and an article is to be sent whilst no cur-
rent connections are idle. Without further
restraint (i.e. using dynamic-method 0), in prac-
tice this means that max-connections connections
are established whilst articles are being sent. Use
of other dynamic-method settings imposes a further
limit on the amount of connections opened below
that specified by max-connections. This limit is
calculated in different ways, depending of the
value of dynamic-method. Users should note that
adding additional connections is not always produc-
tive - just because opening twice as many connec-
tions results in a small percentage increase of
articles accepted by the remote peer, this may be
at considerable resource cost both locally and at
the remote site, whereas the remote site might well
have received the extra articles sent from another
peer a fraction of a second later. Opening large
numbers of connections is considered antisocial.
The meanings of the various settings are:
0 no method
Increase of connections up to max-connec-
tions is unrestrained.
1 maximize articles per second
Connections are increased (up to max-connec-
tions) and decreased so as to maximize the
number of articles per second sent, whilst
using the fewest connections to do this.
2 set target queue length
Connections are increased (up to max-connec-
tions) and decreased so as to keep the queue
of articles to be sent within the bounds set
by dynamic-backlog-low and dynamic-backlog-
high, whilst using the minimum resource pos-
sible. As the queue will tend to fill if
the site is not keeping up, this method
ensures that the maximum number of articles
are offered to the peer whilst using the
minimum number of connections to achieve
this.
3 combination
This method uses a combination of methods 1
and 2 above. For sites accepting a large
percentage of articles, method 2 will be
used to ensure these sites are offered as
complete a feed as possible. For sites
accepting a small percentage of articles,
method 1 is used, to minimize remote
resource usage. For intermediate sites, an
appropriate combination is used.
dynamic-backlog-low
This key takes a value between 0 and 100 and repre-
sents (as a percentage) the low water mark for the
host queue. When the host queue falls below this
level, when using dynamic-method 2 or 3, if 2 or
more connections are open, innfeed will attempt to
drop connections to the host. An IIR filter is
applied to the value to prevent connection flap
(see dynamic-filter). A value of 25.0 is recom-
mended. This value must be smaller than dynamic-
backlog-high.
dynamic-backlog-high
This key takes a value between 0 and 100 and repre-
sents (as a percentage) the high water mark for the
host queue. When the host queue rises above this
level, when using dynamic-method 2 or 3, if less
than max-connections are open to the host, innfeed
will attempt to open further connections to the
host. An IIR filter is applied to the value to pre-
vent connection flap (see dynamic-filter). A value
of 50.0 is recommended. This value must be larger
than dynamic-backlog-low.
dynamic-backlog-filter
This key takes a floating-point value between 0 and
1 which represents the filter coefficient used by
the IIR filter used to implement dynamic-method 2
and 3. The recommended value of this filter is
0.7, giving a time constant of 1/(1-0.7) articles.
Higher values will result in slower response to
queue fullness changes, lower values with faster
response.
max-queue-size
This key requires a positive integer value. It
defines the maximum number of articles to process
at one time when using streaming to transmit to a
peer. Larger numbers mean more memory consumed as
articles usually get pulled into memory (see the
description of use-mmap).
streaming
This key requires a boolean value. It defines
whether streaming commands are used to transmit
articles to the peers.
no-check-high
This key requires a floating-point number which
must be in the range [0.0, 100.0]. When running
transmitting with the streaming commands, innfeed
attempts an optimization called ``no-CHECK'' mode.
This involves not asking the peer if it wants the
article, but just sending it. This optimization
occurs when the percentage of the articles the peer
has accepted gets larger than this number. If this
value is set to 100.0, then this effectivly turns
off no-CHECK mode, as the percentage can never get
above 100.0. If this value is too small, then the
number of articles the peer rejects will get bigger
(and your bandwidth will be wasted). A value of
95.0 is usually pretty good. NOTE: In innfeed 0.9.3
and earlier this value was in the range [0.0, 9.0].
no-check-low:
This key requires a floating-point number which
must be in the range [0.0, 100.0), and it must be
smaller that the value for no-check-high. When run-
ning in no-CHECK mode, as described above, if the
percentage of articles the remote accepts drops
below this number, then the no-CHECK optimization
is turned off until the percentage gets above the
no-check-high value again. If there is small dif-
ference between this and the no-check-high value
(less than about 5.0), then innfeed may frequently
go in and out of no-CHECK mode. If the difference
is too big, then it will make it harder to get out
of no-CHECK mode when necessary (wasting band-
width). Keeping this to between 5.0 and 10.0 less
than no-check-high is usually pretty good.
no-check-filter
This is a floating point value representing the
time constant, in articles, over which the CHECK /
no-CHECK calculations are done. The recommended
value is 50.0 which will implement an IIR filter of
time constant 50. This roughly equates to making a
decision about the mode over the previous 50 arti-
cles. A higher number will result in a slower
response to changing percentages of articles
accepted; a lower number will result in a faster
response.
port-number
This key requires a postive integer value. It
defines the tcp/ip port number to use when connect-
ing to the remote.
no-backlog
This key requires a boolean value. It specifies
whether spooling should be enabled (false, the
default) or disabled (true). Note that when no-
backlog is set, articles reported as "spooled" are
actually silently discarded.
backlog-limit
This key requires a non-negative integer value. If
the number is 0 then backlog files are allowed to
grown without bound when the peer is unable to keep
up with the article flow. If this number is greater
than 0 then it specifies the size (in bytes) the
backlog file should get truncated to when the back-
log file reaches a certain limit. The limit depends
on whether backlog-factor or backlog-limit-high is
used.
backlog-factor
This key requires a floating point value, which
must be larger than 1.0. It is used in conjunction
with the peer key backlog-limit. If backlog-limit
has a value greater than zero, then when the back-
log file gets larger than the value backlog-limit *
backlog-factor, then the backlog file will be trun-
cated to the size backlog-limit. For example if
backlog-limit has a value of 1000000, and backlog-
factor has a value of 2.0, then when the backlog-
file gets to be larger than 2000000 bytes in size,
it will be truncated to 1000000 bytes. The front
portion of the file is removed, and the triming
happens on line boundaries, so the final size may
be a bit less than this number. If backlog-limit-
highwater is defined too, then backlog-factor takes
precedence.
backlog-limit-highwater
This key requires a positive integer value that
must be larger than the value for backlog-limit. If
the size of the backlog file gets larger than this
value (in bytes), then the backlog file will be
dhrunk down to the size of backlog-limit. If both
backlog-factor and backlog-limit-highwater are
defined, then the value of backlog-factor is used.
PEER VALUES
As previously explained, the peer definitions can contain
redefinitions of any of the key/value pairs described in
the GLOBAL PEER DEFAULTS section above. There is one
key/value pair that is specific to a peer definition.
ip-name
This key requires a word value. The word is the
host's FQDN, or the dotted quad ip-address. If this
value is not specified then the name of the peer is
taken to also be its ip-name. See the entry for
data.ramona.vix.com in the example below.
RELOADING
If innfeed gets a SIGHUP signal, then it will reread the
config file. All values at global scope except for ``back-
log-directory'' can be changed. Any new peers are added
and any missing peers have their connections closed.
EXAMPLE
Below is the sample innfeed.conf file.
#
# innfeed.conf file. See the comment block at the
# end for a fuller description.
#
##
## Global values. Not specific to any peer. These
## are optional, but if used will override the
## compiled in values. Command-line options used
## will override these values.
##
news-spool: /var/news/spool/articles
pid-file: innfeed.pid
debug-level: 0
use-mmap: false
log-file: innfeed.log
stdio-fdmax: 0
backlog-directory: /var/news/spool/innfeed
backlog-rotate-period: 60
backlog-ckpt-period: 30
backlog-newfile-period: 600
dns-retry: 900
dns-expire: 86400
close-period: 3600
gen-html: false
status-file: innfeed.status
connection-stats: false
host-queue-highwater: 200
stats-period: 600
stats-reset: 43200
max-reconnect-time: 3600
initial-reconnect-time: 30
##
## Defaults for all peers. These must all exist at
## global scope. Any of them can be redefined
## inside a peer or group definition.
##
article-timeout: 600
response-timeout: 300
initial-connections: 1
max-connections: 5
max-queue-size: 25
streaming: true
no-check-high: 95.0
no-check-low: 90.0
no-check-filter: 50.0
port-number: 119
backlog-limit: 0
backlog-factor: 1.10
backlog-limit-highwater:0
dynamic-method: 3
dynamic-backlog-filter: 0.7
dynamic-backlog-low: 25.0
dynamic-backlog-high: 50.0
no-backlog: false
##
## Peers.
##
peer decwrl {
ip-name: news1.pa.dec.com
}
peer uunet {
ip-name: news.uunet.uu.net
max-connections: 10
}
peer data.ramona.vix.com {
# ip-name defaults to data.ramona.vix.com
streaming: false
}
peer bb.home.vix.com {
ip-name: 192.5.5.33
}
# Blank lines are ignored. Exerything after a '#'
# is ignored too.
#
# Format is:
# key : value
#
# See innfeed.conf(5) for a description of
# necessary & useful keys. Unknown keys and their
# values are ignored.
#
# Values may be a integer, floating-point, c-style
# single-quoted characters, boolean, and strings.
#
# If a string value contains whitespace, or
# embedded quotes, or the comment character
# (``#''), then the whole string must be quoted
# with double quotes. Inside the quotes, you may
# use,t#he standard c-escape sequence
# (,0
# Examples:
# eg-string: "NewConfigfile0
# eg-long-string: "A long string that goes
# over multiple lines. The
# newline is kept in the
# string except when quoted
# with a backslash # as here."
# eg-simple-string: A-no-quote-string
# eg-integer: 10
# eg-boolean: true
# eg-char: 'a'
# eg-ctrl-g: ' 07'
HISTORY
Written by James Brister lt;brister@vix.com for InterNet-
News. This is revision 1.4, dated 1997/07/24.
SEE ALSO
innfeed(1) newsfeeds(5)