Handling Different Web and News servers

Some sites that want to put up an HTTP (web) interface to ClariNet news or USENET news may face the problem that they don't currently have the two things on the same server. Here are some ways to get around this problem.

Which one you pick depends on load and familiarity with software.

C News on Web Server

The simplest method is to simply install a news server on a system with a web server. We recommend the use of C news, which does not put the huge memory load on a system than INN does. INN was primarily designed for sites with lots of incoming NNTP traffic. It runs a huge daemon to make this efficient, but you don't need that for a single incoming feed of low-volume ClariNews. Even at 9mb/day, a ClariNet feed does not put a serious impact on a machine.

We recommend this because news software like C news was designed exactly for the purpose of mirroring a news tree to another machine, handling expiring it -- the works.

Of course if you are familiar with INN and and don't know C news at all, you may not want to install an unfamiliar package. If you do use INN for this, you will want to tune it for minimum memory usage. (Or buy a bit more RAM -- at under $4 per megabyte, sometimes RAM problems are best solved with money these days rather than sysadmin time.) You can set the flag INN_DBZINCORE to 0 to avoid it keeping a lot of RAM at a cost in efficiency, when you compile it.

Web server on the news server

Another alternative is to put up a web server on the news server for no other purpose than to read news, and run our CGI and news unpacking programs. Again, such a web server would only serve requests for news stories, and as such the load of it would not be high. It doesn't need to be a full production web server.

Which web server is up to you. Apache is free of course, but you may have another choice.

NFS mounting of news directories on system with web server

If you do this, the web server and CGIs can access all the files on the news server, and serve them up. This of course has a cost in NFS overhead. It is however, by far the simplest thing to do administratively. If you have fast NFS servers it may not even be a big problem.

You need to consider whether files will be read frequently or infrequently. If you have a small site, some articles may never be read. There are 3000 each day in a full feed.

In general you would have the news server configured all in one tree, or you would need mounts for the news spool, the news "library" (with history file) and any unpacked pictures and web pages unpacked by our various tools. If you put all those things in one place and use symlinks to give them their "default" names you can do it all with one NFS mount.

NFS mounting of news spools by news server

This is not recommended for a full USENET application, but you could have the news server system NFS mount the directories it will write to from a directory on the web server system. You could NFS mount /usr/spool/news and /usr/lib/news and any directories written to by web unpacking programs.

Then the news files would reside native on the web server machine, and it could access them quickly.

Or you can mix and match. For example you might mount the news spool one direction and the news library another.

NFS mounting of unpacked web trees

If you are unpacking the news to web trees, a good plan is to NFS mount the news spool by the web server, and run the web unpacker on the web server system. It will read the news log and spool files, and generate web pages on your web server for frequent access.

How you mount depends on which way you think traffic will flow. For any large system, we expect that the final files read by the user, be then the news server files or the unpacked web files, should reside native on the system with the web server. The other decisions are ones of taste -- do you want the news server to push files onto the web server via NFS or do you want the web server to pull news files off the news server via NFS.

Special notes

If you use photograb, it needs to be called on the news server. However, the photo cache and included documents can easily be on an NFS mounted file system which really resides on the web server.

If you use ClariCGI you must mount the news server spools by NFS onto the web server. This is not that efficient if news is read a lot, so we recommend the earlier solutions if possible. However, many sites have all news read by NFS -- it is more efficient, for example, than reading by NNTP, so don't take this advice too strongly. If NFS mounts are all you are prepared to do, do them.

File Copying tools

As a last resort, you can arrange to build the files on one system and copy them to the web server. This can be done with a tool like "rdist" or you can easily have the web extractor or other tools make a list of the files they changed and you can feed that list into a program like CPIO to make a stream to somehow send to the other system (via rsh, rcp or even FTP) for unpacking.

If your news server and web server are totally divided by firewalls that block everything but FTP, we strongly recommend you just install C news on your web server or a web server on your news system. These tools were meant to mirror news feeds through firewalls! But if you can't, you can:

1. Modify the web unpacking scripts to generate a list of files they have changed, or
2. Use a tool like "find" to build a list of files that have changed after an unpack, then:
3. Take the list of files and put it into "cpio" or some similar program.
4. Take the output from cpio and FTP it to the other system.
5. On the other system, when the FTP archive arrives, unpack with CPIO into the right tree.
6. Arrange some sort of cleanup that will get rid of old files after a while.