NNTP Feed information
NNTP Feeding
Most people feed USENET style news (and ClariNet) news today using the Network News Transfer Protocol (NNTP). Newsfeeding done this way involves a feeding program (at our site) which initiates a TCP/IP session to the NNTP port (119) where it connects to an NNTP server program.
This begins a session where the feeding program offers articles to the server. The server either accepts them and asks that they be sent, or rejects them, indicating it already has them.
This document is not a general introduction to NNTP, however, so if the above was news to you, you may wish to follow some links from this document to learn about NNTP.
People also feed news via modems and UUCP, and in some cases, by UUCP over TCP/IP, which is a bit of a kludge in its way but some think is slightly more firewall safe. It also allows compression and is efficient if it is the only feed.
One of the great things about NNTP, however, is the way that servers reject articles they already have. This lets you have multiple incoming feeds to a server or group of peered servers, and whichever offers the article first sends it to you. This allows redundancy -- one link or server can fail and you still get news.
ClariNet uses this, and generally all customers are fed from 2 or more different feed machines by ClariNet.
Typical Servers
For many people, the program that answers the NNTP port is INN, a single daemon program that handles almost all of USENET news processing. INN is good at handling large numbers of simultaneous incoming feeds efficiently.
There are some extensions to NNTP in INN, including the NNTP streaming mode found in the current "unofficial" patches to INN. We recommend this mode.
Other people use the stand-alone original NNTP daemon called "nntpd." On its own, it is only capable of rejecting duplicates of articles that have made it fully through the news system (which takes 5-10 minutes on many computers.) This is bad if two feeds are truly close to simultaneous, as it won't know to reject the 2nd copy. However, the latest versions support a special daemon caled msgidd (Message-ID daemon) which allows servers to share what messages they have handled in real time.
If you get a ClariNet feed we recommend that you run one of these systems as we will send you two or more feeds that are very close to one another in time.
Setting up for an NNTP feed from ClariNet
Here is a description of how to set up NNTP connections with ClariNet..
Special Feed Program
ClariNet has its own custom feed program, designed to handle the difficulties of maintaining many hundreds of outgoing, redundant NNTP feeds. This program does not maintain a permanent connection, but has feed programs that cycle through destinations which we have news to send to. It watches the status of each feed, including when articles are sent, how fast they are going and any error conditions.
If error conditions persist for too long, the system sends an E-mail message to the contact address at the destination site about the error. Reminder messages are sent if the error persists for a day. In addition, notes usually go out when an error is cured, so that you will know the problem is fixed. (That's very handy if you come in to your email after the problem happened and then was fixed, or fixed itself, so you know you don't have to track it down.)
In addition, the system sends us mail on problems that persist too long, to see if there is anything we can do about it. However, in most cases, feed problems are on the receiving end, and we can't do much more but send E-mail or call. If you get messages about a problem that is on our end, let us know, as we want to train the system not to do that.
We can also tell the system not to send you errors or certain types of errors, but if we do, things may go wrong with your feed without anybody knowing about it for some time, and your ClariNet news may get out of date.
Most readers of ClariNet news are keen that the delivery be fast and up to date, because unlike USENET news material, if our news is late, it becomes "yesterday's news." That's why some customers pay special attention to the needs of getting a reliable ClariNet feed, or even dedicate a special server to it. (ClariNet news is low volume compared to the bulk of USENET, so often a server just for it, which feeds other servers internally, makes sense.)
Finding out your Feed Status
You can find out the status of our feeds to you from all our feeding machines with this form. Enter the fully qualified domain name of the news server that we feed that you wish to query. You need to use the name you gave us to feed -- if the name you think we are using doesn't work, contact us. This program will report the feed status, including error status and time of last error, for all of our sites that feed your site. After you have gotten to the status page, you can bookmark it in your web browser to check status again. See also more information on feed queries.Premium Feed
While we endeavour to keep all feeds as close to live as possible, during heavy load periods or after downtime for our servers, your servers or the network between us, some feeds get a little behind. For sites particularly concerned about keeping their feed as up to date as possible, ClariNet offers a premium feed service for $25 per month. This service bumps your priority in our feeding queues, giving you more time in our feeding slots. Contact sales@clari.net to arrange this service.Messages from our feed system
There are many things that can go wrong with an NNTP feed. Some are configuration errors, some are network errors, and some are system problems that aren't directly related to feeding but stop it from happening.
Some fix themselves from time to time, some need a human's attention to fix. For the ones that fix themselves, we try to not send out a warning too quickly to save your mailbox.
Startup Errors
Some of these errors will come when we first start your feed. Those are most often likely to be configuration errors. When our support staff enter your site to receive a feed from us, the system tries to start feeding you right away. If it finds problems -- and it's not uncommon for there to be problems the first time this is tried -- it reports errors. These aren't things "going" wrong, of course, but things that started off wrong.
Network Errors
Tough problems to diagnose include network errors, which can occur in ClariNet's network, your network, the networks of our respective internet providers and even the internet backbone.Particularly annoying is the error "no route to host." This general TCP/IP error can occur when a network link is down, or when a network link has been deliberately (or accidentally) blocked by a firewall. The former type of error usually clears itself up after a while, and before it does, E-mail is usually pointless anyway. The latter type of error won't clear until a person gets to it. Some sites put their NNTP server behind a firewall and only let certain feeding sites open connections to it. If you block us via your firewall, we can't feed you of course.
Other network problems include connections that fail or time-out due to unknown interruptions. Normally these clear up on their own.
Configuration Errors
These normally only occur when you start up a feed from us, but may occur if somebody changes your server configuration and makes an accidental mistake.
The most common error is for your server to refuse to accept connections from us. See our description of how to set up NNTP connections with ClariNet.
Sometimes your server may be configured to insist on a password from us, or there may be other internal configuration problems.
System Errors
The most common system error is a full disk in the news spool. Most news servers refuse news during this period. Sometimes these go away on their own but often they need attention. This is one reason to consider a special server for ClariNet news.
Another common problem is too high a load. Your system load average can just be too high, or you may have too many other NNTP feeds connected and thus not want any more. It's up to you to set these parameters. If you consider the ClariNet feed to be more important, and you run INN, you may wish to list us in the file hosts.nntp.nolimit, rather than the hosts.nntp file. This lets us in even if there are too many other servers connected.
Just going too slow
Sometimes your system just gets so loaded or your internet connection gets
so loaded that your feed gets behind. This can also happen if you have
a very long-latency internet connection, ie. if you are overseas from your
feed site or have network connections with a very long "ping" time. To
see if you have a latency problem, use a ping program that gives response
time on our site news.clari.net.
In some cases, using "streaming NNTP" will help, particularly the long
latency. Patches for INN to support
this are available via FTP.
Patches for NNTPD are forthcoming.
If you want to have different threshold values for your site on errors,
mail to admin-support@clari.net. In addition, Please make sure
that the contact addresses listed for your site are correct, particularly
the Email contact address for your technical staff. If possible, use
the standard alias usenet and point that at the person who manages
USENET news, and make sure to change it whenever the duty is passed on.
Regularly we have sites who don't change their records or an alias, and then
our system can't report errors to them, which means errors go unfixed for
longer.
Contacting ClariNet
More info on NNTP
|  About Clarinews | Clarinet Info | Support | Info for ISPs | Fun Stuff | Home |