ClariCGI -- USENET and ClariNet News webreading CGI program

1. ClariCGI Configuration/Install guide
2. ClariCGI auto-configuration form
3. ClariCGI template language
4. Guide to ClariCGI predefined template variables
5. Guide to ClariNews standing story slugwords

The ClariCGI is a Common Gateway Interface (CGI) program for use with most web servers that allows improved web access to USENET news articles, and in particular ClariNet news articles. It supports ClariNet's proletext format, which allows ordinary looking plain text articles to contain invisible formatting codes which allow them to be translated to HTML and displayed with nice formatting.

The goal of ClariCGI is to provide easier and better looking access to USENET news articles to people who prefer entirely web based access to information. ClariCGI lets you put up links on web pages that refer to news articles based on patterns. Users do not have to know how to use a newsreader -- even the Netscape built-in newsreader. For articles formatted in proletext, such as ClariNet articles, the change in appearance is also dramatic.

You can think of it as a simple newsreader (though it does not currently support threads or a newsrc) which allows nicely customized looks for groups and articles, however it's also a good general web-to-news interface that makes reliable URLs to refer to articles not just by number or message-id, but by characteristic.

Some things you can do with ClariCGI:

• Provide links to commonly found articles, such as regularly posted FAQs
• Provide links to newsgroups that don't involve calling up a newsreader, but offer the most recent articles to the user.
• Provide links to daily articles, such as ClariNet comic strips. For example a "Today's Doonesbury" link or "Today's Internet Stock Report" link are possible.
• Provide links to the most recent articles in a newsgroup, or the most recent articles matching a pattern.
• Patterns can include subjects, posting users or patterns found in message-ids. Thus one can make a link which fetches the 5 most recent articles in alt.politics.usa.misc posted by a user at whitehouse.gov. (These really exist.)
• Customize the appearance of both menus and articles in any given newsgroup with a template language.

We have some demonstrations of it up on our own web site. For example, you can read the latest photo news story or the biz.clarinet.sample newsgroup with a rotating ClariNet sample, or the ordinary news.software.readers newsgroup.

Since it is a CGI program, it is invoked once for every URL that references a news article. This is not extremely efficient (though a system with a good disk cache helps) but quite satisfactory for low to medium volume use. It is a (relatively) small C program, not a perl script, so it usually loads and runs quickly. It also supports fastcgi for top efficiency.

The CGI offers "stateless" access to newsgroups. This means that the system does not keep track of who the user is, and each request is treated independently. As such it is not a newsreader that keeps track of what you have read and not read, or a personal subscription list. Some of those activities can be simulated, because web browsers usually show links to articles that have already been read in a different colour, and people and webmasters can of course build bookmark pages and other pages with their personal favourite newsgroups and articles.

For those that want true, regular newsreading, a real USENET newsreader is a better idea. There is a decent one built into Netscape and some of the other web browsers. They do not yet support Proletext, however. For long queries, the POST mode may be more suitable.

Special ClariNet Features

The ClariCGI also supports the special format ClariNet currently uses for photo articles, which come as a a pair of multipart-MIME articles. Each article comes twice, once with a small (7K or so) photo and once with a nice large (50K) photo. The ClariCGI does not display the large photo versions in menus. Rather it shows the small ones, and, if they are selected, unpacks them and displays them in a nice format with caption. The small photo is then a link to the large photo. Unpacked photos can be cached in a disk directory to speed access so that only the first user has to unpack the photo.

The ClariCGI contains many facilities to search for specific articles in a newsgroup. These are particularly useful in ClariNet groups, but also have value in ordinary USENET groups. One feature useful for ClariNet groups is the ability to fetch a specific article, but then search for another if the specific article is missing. This allows the handling of regularly updated and superseded articles. If the one you want is missing, it can then hunt for the most recent.

A very powerful feature of ClariCGI is the ClariCGI template language. This programming language lets you control how HTML pages with newsgroup article menus and actual articles are laid out, with control possible on a group by group basis. With this language you can specify the look and feel of most pages generated by ClariCGI and custom configure many items.

Finally ClariCGI has support for the fact that ClariNet Slugwords are embedded in our message-IDs. That's a bit of a kludge, but normally the message-ID is included in the news overview file for fast access, while our slugword normally is not. A slugword is a special keyword which identifies a story.

Operation of ClariCGI

The ClariCGI program needs to be compiled and installed in a location where CGI programs can be executed on the web server. This is often the directory cgi-bin in the web server's directory. Some servers are configured so that the program can exist anywhere if the name ends in .cgi. Install the program anywhere, and note the URL to get to it. (We recommend you look into FastCGI if you anticipate heavy use, since in this case the program runs as a daemon.)

The ClariCGI also needs a configuration file, which by default is found in /usr/local/httpd/conf/claricgi.conf. If you need to locate it elsewhere, you need to change the claricgi.h header file before compiling. Almost all other changes can be made in the claricgi.conf configuration file.

If the configuration file can be located in the above directory or in the current directory of the CGI when it executes, you can simply install the binary of the program, if one is available, and you need not recompile it.

Typically ClariCGI is invoked with a URL that invokes the CGI, and provides a PATHINFO, and optionally QUERY string with the commands for the CGI. This is usually done via a hard-coded URL that includes the URL of the CGI, a slash and question mark, and then the commands for ClariCGI. You can also send the data in "POST" form, which is done by web page forms or by certain specialized web clients, however in POST form the queries are somewhat different and this is not yet officially documented. (In general, we require that a single slash of PATHINFO information be put after the cgi's name if you plan to view article of type text/html that may contain "relative" URLs, which is to say URLs that don't begin with a protocol like http: or a slash. This is because otherwise the browser will not pass the relative URLs back to ClariCGI. However, there must be only exactly one slash, so the use of /? is recommended.) Multiple slashes at the start of the PATH_INFO are removed to deal with URL writers who are not sure how many slashes they need to output.

Read the ClariCGI install and configuration guide for full details.

ClariCGI Query Syntax

Queries to ClariCGI (after the ? in a URL) are presented in one of two ways. Simple queries have their own simple syntax, for example:

is one way to fetch a specific article. This syntax is easy to remember and compatible with the news: URL syntax. In fact the entire news: URL syntax is supported, including the Netscape "?part=2" extension to extract parts of multipart articles.

Complex queries consist of a series of query operators, starting with an initial semicolon, and delimited by semicolons. Query operators can be either:

flag^ or flag+

Turns a flag on, though flag^ is usually used because URL encoding makes it hard to put a plus in a URL in POST mode.

flag-

Turns a flag off

parameter:value

Assigns a specific parameter for the query

object=value

Sets a searching parameter for the query

object/regular-expression

Sets a pattern matching parameter for the query

More on these later.

Complex queries always begin with a semicolon (;) to distinguish them from simple queries and preset queries.

Preset queries consist of a colon (formerly a dollar sign) and a preset query name. They are mapped into complex queries defined in the configuration file. This is a macro facility of sorts, but its real purpose is to allow outside users to do some queries and not others.

Multiple Queries

You can put several queries into one URL. The resulting stories or lists are simply concatenated. Thus you can build multi-story URLs or even mini personal newspapers right inside a URL. Queries can be connected with a double semicolon, as in:

Important URL note

Some characters are difficult to include in a URL, like less-than, greater than, pound-sign, ampersand, space and percent. You need to escape these characters in any hand-coded URLs using the standard %xx hexadecimal escape sequence. Ie. space is %20, and % is %25. Space can also be coded as a plus sign. That means if you use percent, plus sign, space or pound/number sign (#), you should escape them.

Currently ClariCGI does not map "+" to space since it is a legal character in newsgroup names.

ClariCGI Modes

There are several ways you can use the ClariCGI. Several of them work together.

Specific Article Fetch

A specific article fetch calls up a particular article, by newsgroup name and article number, or by message-id. These queries have a simple syntax like that of the news: URL

The message-ID can be provided with or without angle brackets. Providing them is a pain in HTML, to say the least.

A specific article fetch calls up an article, and displays it with nice HTML headers, including links to the newsgroups it is in. The article is displayed in monospace text if it is ordinary text, or in HTML if it is in Proletext or has a MIME Content-Type of text/html.

Fetch by message-id is more portable, but involves slightly more disk I/O.

In the compound syntax, a query like ;group:clari.usa.top;value:1234 returns article 1234 from clari.usa.top. The query ;msgid:msgid@clari.net fetches the article of that specific message-id. In the above two examples, the command name "group" was spelled out in full to make things clear to you, however, almost all query commands and options can be abbreviated to their first letter, so these queries are usually written as, for example, ;g:clari.usa.top;v:1234.

Single Article Find

The most useful feature of ClariCGI, particularly for ClariNet articles is the single article find. This allows you to fetch a single article that matches a query you define. That query can find the first (or Nth) article matching it, or the most recent (or Nth most recent) article.

Thus for example a trivial query (in fact the default one) will return the most recent article in a newsgroup. Most of the useful queries in fact try to find the most recent article of a certain type in the newsgroup.

Thus the plain request of ;group:clari.usa.top does not return the headline menu for the newsgroup (as the simple query does) but rather the most recent article in the group.

You can request articles that have certain patterns in their subject lines, articles from a certain person or site, articles in certain date ranges, and articles that match certain patterns in their message-id (for ClariNet purposes.) You can also match other headers if they are present in the news system's "overview" file.

The query ;group:clari.usa.gov.white_house;subject/Clinton;value:-2 returns the second most recent article from clari.usa.gov.white_house that has the word "Clinton" in the subject line.

USENET news systems today include an "overview" file which lists the important headers for all articles in a newsgroup. The ClariCGI needs this file. Normally only a few headers are present in the file. The administrator of a news system can add more, making them available to newsreaders and ClariCGI queries.

Article List

You can also ask to return a list of articles that match a query. (Or if there is no query, all articles in a range, the earliest or most recent articles, or, all the articles in a newsgroup.)

You can return the list of articles in two ways. One is as a list of headlines (and bylines) with links from the headlines to specific article fetches of the articles in question. This provides a menu of items in a newsgroup, much like a newsreader menu.

You can also ask for the articles to be actually inserted in-line. This can result in several articles being returned as one large web page, concatenated together.

Entire Newsgroup

You can also get an article list for an entire newsgroup, in effect allowing you to read the entire newsgroup. There is no concept of a .newsrc file that knows what you have seen and not seen, you get the entire newsgroup.

This mode has a simple syntax, namely just the newsgroup name as the query. This offers a list of the 20 most recent articles in the newsgroup and a link to the full newsgroup.

Menu of Newsgroups

You can get a menu of newsgroups by providing a newsgroup name that is a regular expression pattern. This pattern will be tested against all newsgroups in the system's active file, however all matches must begin at the start. (You can read this as an implicit ^ at the front of your pattern.)

You can get such a menu by use of the "menu^" tag, or in the simple syntax, by providing a string that contains a star (*), since almost all patterns end up with the .* regular expression in them.

The output of these menus is controlled with three programs in the ClariCGI template language.

You can also get a menu of hierarchies, so that any hierarchies below a certain level are compressed to a single link that will display the next level of the hierarchy.

Personal Newspaper Mode

Not yet fully implemented, this mode outputs a large HTML page with the concatenation of the results of several queries, to in effect build a personal newspaper or collection of USENET news for the reader. Generally this is done by having other software maintain the list of queries for the user's personal newspaper in a local file on the web server.

Right now the personal newspaper page file is a text file. Lines beginning with # are comments. Lines beginning with ";" are complex query lines and their output goes into the personal newspaper. Other lines are lines of plain HTML and they are inserted verbatim into the personal newspaper.

Preset Queries

Usually access by outsiders to USENET news, and particularly ClariNet news, is blocked. However, sometimes sites wish to provide outsiders with access to specific news articles without allowing them general access. It is possible to define preset queries in the configuration file for ClariCGI which can be called by name. No permissions checks are done on these preset queries, so outsiders can access those and only those articles provided by these queries. If the queries result in menus of stories or stories with lists of newsgroups, those queries will fail with a permission denied message.

A preset query looks like:

:localnews:	;g:oursite.local;s/news;v:-1 

Which extracts the most recent article from the newsgroup oursite.local with the word "news" (all lower case) in the subject line.

Complex Query Terms

As noted, complex queries consist of a series of terms delimited by semicolons or or-bars. The delimiter is usually a semicolon. It only has meaning after real querying terms (as opposed to flag and parameter setting terms.)

As noted some terms just set flags and parameters (^, - and : terms) while others are matching terms (those using = and /). For each article tested, the matching terms are tested in order against the article.

If a matching term matches and the operator after it is a semi-colon, the testing of matches continues. If it does not match, the entire query is judged not to match, and the later terms are not tested.

If the operator after a term is an or-bar, then a failure to match does not stop the query, it continues on testing the other terms. If the term matches and is followed by an or-bar, the query is judged immediately as a positive match, and the other terms are not tested. This is sort of a simplified version of short-circuit boolean operation.

Matching terms can also be preceded by an exclamation point (!), which reverses their sense. Thus while f/brad matches articles with "brad" in the From: line, !f/brad matches articles that don't have that pattern.

Below is a description of the terms. While full words are used for mnemonic purposes, currently only the first letter is tested. That may be changed in future, so if you use a string longer than one letter, use the exact string provided.

group:newsgroup

This sets the newsgroup for a query or specific article fetch. All queries, except for message-id based specific fetches and preset queries, must have a newsgroup.

title:string

Sets the title for the returned HTML page. Normally used with article-list queries. A suitable title is created if one is not provided. Note that spaces usually must be expressed as %20 in URLs.

Header:string

Provides this string (HTML) as a header for articles and lists, after the

Tail:string

Provides this string (HTML) at the end of an article or list, for you to put any tail-notes or links you desire. title.

ArticleTemplate:string

Provides the name of a file in the ClariCGI templates directory (if provided) where a ClariCGI template program can be found to control the layout of the article to be displayed or a newsgroup menu to be displayed. Template programs must be predefined and stored in this directory by those configuring ClariCGI -- users are not permitted to write programs in the template language directly.

messageid:id

Sets the message-id for a message-id based specific fetch. All simple syntax queries have a corresponding complex syntax, which is useful in particular to make a query that fetches a specific article but searches if it is not found.

Generic:string

Sets a generic parameter used by some special operators.

part:number

Specifies that a certain component of a multipart article be fetched. The default, 0, fetches the entire article normally. Positive numbers fetch particular parts. A value of -1 fetches the entire article but just the body -- the header is discarded. (This can be used to inline articles with an image Content-type: or HTML articles that you don't want to display normal article headers on.

If the generic parameter is set to a string, and the part number is non-zero, the program will instead search for a multipart that has a "name" attribute set on the Content-Type header. ie. if one of the components has a header of:

Content-type: image/jpeg; name="hello.jpg" 

Then you can extract that with (article-finder);p:1;G:hello.jpg in the query. The part number of 1 is ignored, other than to turn on this search. Naming parts is generally better than numbering them, if you generated the parts.

value:number

Sets the general value field. This field has several meanings. For specific fetches, it is the article number. For article finds, it is which matching article is desired. (Positive numbers find the Nth earlist article, negative numbers find the Nth most recent article. The default of 0 is mapped to -1.) For article lists, it is the number of articles to list, either from the start (positive) or the most recent (negative). If 0, the entire group is listed.

personal:filename

Specifies the system should output a personal newspaper based on the profile in the specified filename, which should reside in the ClariCGI's personal newspapers directory, as configured by the system. The filename must be only alphanumerics or underbars.

The personal newspaper file consists of 3 types of lines. Lines starting with an ';' are complex queries to ClariCGI and their output is put into the html stream. Lines starting with '#' are comments and ignored. All other lines are HTML and inserted into the HTML stream.

Most other options and tags are ignored if a personal newspaper is being generated.

1^

Specifies a specific article fetch. The default type of fetch varies based on other terms and parameters. This insists that a single article is being gone after, not a list, not a find.

Always^

If a request was made for anything but the first match, in either direction, in an article find, and the requested match is not found because there are not enough matches, return the article that is the last match. This assures that article finds that are asking for other than the first match always return some article.

list^

Requests an article headline list.

menu^

Requests that the newsgroup name be treated as a pattern, and that a menu of all newsgroups be provided which match the pattern. Matches must be anchored at the start, so if the pattern matches a string inside the newsgroup name this is not a match, however it is not necessary the pattern match to the end. If you want a pattern that can appear anywhere, start it with .* or ^.* as you wish. To make sure you match to the end, end your pattern with $.

inline^

Requests an inline article list, multiple full-text articles returned. Discouraged except for short lists of a few articles.

whole^

When an article list is made, if the list does not end up listing the entire newsgroup, a link is provided at the bottom which will list the entire newsgroup.

find^

This is an article find query.

summary^

In an article headline list, if there is a Summary: header present on the article and the Summary: header is found in the overview database (which is not the default on most news systems) then the menu of articles will include both headline and summary in the DL/DT/DD HTML tag style, with the link on the headline. (Not yet implemented)

date-

Do not display date/time stamps on menus of story headlines. (These are only displayed if the Subject line does not appear to have its own date in it.)

numbers^

When making a list of links from article headlines, use the newsgroup name and article number rather than the message-id. This is slightly more efficient, but comes at two costs. One, if the user follows a link and then sees the same article crossposted in another group, they won't see the link marked as one that was already-followed. Two, without the slugword in the query, the system will be unable to find a later version of a story that has been deleted and updated.

g^

Tells the system that if the article is not found, it should attempt to extract a slugword from the message-id, and rebuild the query to find the most recent article with this slugword in its own message-id. Works only for ClariNet articles. If the article was fetched by group, or a group parameter was set, the search is in that group. If the article was fetched by message-id, the article is in the first legally accessible group on the message's newsgroup list.

Turns out there is a tradeoff here on message-id fetches, which are the norm for article lists, even though slightly less efficient. The goal of message-id fetches is that if a reader visits a message, they will see that they have been there if the headline appears in another newsgroup. So we can't include the group in the URL or this feature will not work. That means if a message is posted to a group, then updated so as to not be in the group (ie. the group is removed from the update) the update will not be found in a re-query. That's not too bad as there is usually a reason the newsgroup was removed, but in a message-id query, it is the first group that is used, not the group the reader was actually browsing. Group removal is, however, rare.

raw^

Display the article as raw plain text with all USENET headers. Used mostly for debugging, or in the event that one needs to see all headers of an article. Not usually offered overtly to the user but you can add ;raw^ to the end of a query and get it.

In some configurations, a tiny, undocumented link is placed in the header to the raw article.

prole-

cache-

Lower=number

Sets a lower limit on article numbers. Articles below this number will not match.

Upper=number

Sets an upper limit on article numbers. Articles above this number will not match.

before=number

Sets a date limit on matches. Articles posted before the specified time will not match.

Times can be specified in two ways. A positive number is a Unix-style date integer number -- the number of seconds since Jan 1, 1970, midnight GMT.

A negative number is a real number, a number of days prior to the current time.

Note the posting time is used, not the arrival time.

after=number

Sets a date limit on matches, articles posted after the given time do not match. See "before" for more details.

m/pattern

Matches if the pattern (a regular expression) matches the message-id. The message-id does not have angle brackets around it, they are removed if they are present in the overview file (as they usually are.) Matches are case-sensative.

Note that slashes should not be used in queries in the PATHINFO part of a URL to a CGI.

f/pattern

Matches if the pattern (a regular expression) matches the From: line.

s/pattern

Matches if the pattern (a regular expression) matches the Subject: line. Note that this, and all other matches are case-sensative, which is most likely to catch you here on subject lines.

rest/pattern

Matches if the pattern (a regular expression) matches any of the other overview lines defined for the site. Thus if you have defined the Xref line on your site, you could do this:

g/pattern

This pattern matches in the message-ID, starting at the ClariNet slugword part. (Ie. the angle bracket and any preface characters are removed.) If the overview contains the real slugword, that may be used instead.

However, this is just a pointer into the message-ID. The slug will have an upper-case letter after it and the rest of the message-ID, if it is pulled from the message-ID. Thus patterns that want to match all the way to the end should check for an upper case letter or end-of-string ($).

g=string

Requires an exact slugword match. Slugwords consist of upper-case wire identifiers and all-lower case (plus simple punctuation) keyword portions. If you truly want to match a slugword exactly, this is the way to do it.

Usymbol:value

Sets the value of a string Usymbol used in the template language and in other places. Internal symbols can't be over-written with this operation but new symbols can be created that can be known to template programs. For security reasons, all symbols begin with "U" so you can't create builtin template program symbols.

Examples

The best way to see how to use ClariCGI is to look at some example queries. Below we provide only the query part. The full URL usually looks like:

;g:clari.news.front_page

Latest top story. Goes to the front page group and returns the lastest story, whatever it is.

;g:clari.news.photos

Latest photo story. All ClariNet photo stories appear in this newsgroup or the sports.photo group.

;g:clari.tw.stocks;m/interquote

Latest Internet Stock Report. We search for the string "interquote" in the message-id. ClariNet puts story identifiers in message-ids.

;g:clari.usa.top;s/^[Sm

Latest photo story from top U.S. news. Here we test for the subject beginning with "[Sm" which is how the small-version photo stories all do.

;g:clari.living.comics.bizarro;f/bizarro

Today's Bizarro -- this is an example of pulling the most recent comic strip. We insist on the From: line containing "bizarro" so that any monthly postings, advisories etc. are ignored.

;g:clari.living.comics.bizarro;f/bizarro;v:-2

Yesterday's Bizarro. In this case we use the Value of -2 to get the 2nd most recent item.

;g:clari.tw.new_media;f/^C-upi

Most recent UPI item from New Media newsgroup Here we insist that the From: line be from UPI, to pull the most recent item from a newsgroup from a particular source or user.

;g:clari.local.massachusetts;m/Uma-briefs

The lastest Massachusetts news in brief. You could extract this one either with the message-id pattern or just looking for the standard headline in the Subject: line.

;g:clari.usa.gov.white_house;m/clinton

This search is not for a specific slugword, but any story with a slugword (hidden in the ClariNet message-id) that involves Clinton. There will be several of these a day.

;g:clari.usa.gov.white_house;m/clinton; v:-10;l^;t:10%20Recent%20items%20on%20the%20Clintons

Here we see a complex list building call. In this case we look for the 10 most recent items that match our pattern. The pattern is to have "clinton" in the message-id (or slug). The l^ asks for a list, and the t: sets a title for the page when it is presented.

;g:clari.tw.new_media;v:-20;l^;w^

The most recent 20 articles on New Media. In this case we ask as well for a link to the whole newsgroup at the bottom of the page. You could also do this with the simple syntax of just the newsgroup name.

;g:clari.biz.front_page;l^

All the front page business news This is a basic whole-group URL. The simple syntax only gives the most recent 20 items.

;g:clari.tw.computers.apple;l^;v:10;w^

The ten oldest Apple computer stories. In this case the value is positive, providing the 10 oldest.

$oldapple

If there is a line in the configuration file defining $oldapple to be the above query, it will allow any user, not just users in the right domains, to access the items. Normally you will want your ClariCGI just for your own users, and you don't want outsiders to be able to access all your news spools. However, if you have demo pages for outsiders, you can define these preset queries and let them use them. Of course, you should not allow outsiders access to ClariNet articles without permission from ClariNet.

ClariCGI as Proletext Convertor

ClariCGI can also be operated stand-alone with command line arguments as a proletext to HTML translator for text files or RFC1036 style USENET messages or E-mail messages.

If used in standalone mode, the environment variable GATEWAY_INTERFACE must not be defined, because the presence of this variable causes the program to believe it is being run as a CGI. Due to certain factors relating to CGI command line arguments, the use of command line arguments to turn on standalone mode is not practical.

In standalone mode, the program takes file names on the command line or processes stdin if no files are provided.

It also takes the following options:

query=string

For use in testing CGI mode, allows the specification of a QUERY_STRING

+header

Requests that the message be treated as an RFC822/RFC1036 USENET message and that the header be parsed and turned into suitable HTML, and then the body be parsed and proletext translated if necessary.

+ignoreheader

Rquests that the message be expected to have an RFC822/RFC1036 style USENET or E-mail message header, but that this header be stripped and ignored.

+cgi

Forces the program into CGI mode. Mostly for testing.