A BibTeX system in Common Lisp
******************************

CL-BibTeX is a replacement for the BibTeX program, written in
Common Lisp.  The aim is to enable the user to format bibliographic
entries using Common Lisp programs, rather than using the stack
language of BibTeX style files.  

It is released under the terms of version 2.1 of the GNU Lesser
General Public License as published by the Free Software Foundation
(see file COPYING) or any later version, as clarified by the preamble
found in COPYING-preamble.txt.

(Note: run-lisp is included from CLOCC and has its own license.)

Development now takes place on GitHub at

    https://github.com/mkoeppe/cl-bibtex

A project home page is at

    http://www.nongnu.org/cl-bibtex/

(The old development pages at
http://savannah.nongnu.org/projects/cl-bibtex/ are obsolete.)

There is a mailing list `[email protected]'.
You can subscribe to the list at 

    http://lists.nongnu.org/mailman/listinfo/cl-bibtex-users


Reasons
=======

A powerful tool is needed for dealing with citations in scientific
documents.  BibTeX is good for formatting bibliographies, but
customizing the format is a pain because it requires writing/changing
a program in BAFLL (BibTeX Anonymous Forth-Like Language -- 
Drew McDermott in comp.lang.lisp), which looks like this:

FUNCTION {tie.or.space.connect}
{ duplicate$ text.length$ #3 <
    { "~" }
    { " " }
  if$
  swap$ * *
}

Other citation-related processing like producing a list of authors and
citation indices is usually done with a combination of BibTeX, TeX
macros, and ad-hoc scripting with Perl or AWK.

The new CL-BibTeX system aims to replace all this.


Status
======

CL-BibTeX is a compatible and stable re-implementation of the BibTeX
program, which was written by Oren Patashnik in the WEB language.

It consists of the following components:

 * A reader for BibTeX style files (.bst),
 * a reader and a writer for bibliography databases (.bib),
 * a reader for .aux files,
 * an implementation of BibTeX's built-in functions (like for parsing and
   formatting names, etc.),
 * an interpreter for the stack language of BibTeX style files.

Thus CL-BibTeX allows to both use existing BibTeX style files,
and to write new BibTeX styles in Common Lisp.  

In addition to this, CL-BibTeX contains:

 * A compiler that transforms BibTeX style files into comprehensible
   Common Lisp programs (using type analysis).

   The resulting Common Lisp programs can serve as a starting point
   for the customization of the style.

Future directions of CL-BibTeX's development can be found in the TODO
file.


Where to start
==============

Compile and load all Lisp files, using ASDF or bibtex.system.

The function BIBTEX is like the bibtex program.  Pass the file name
STEM; BIBTEX will read STEM.aux, interpret a BST file, and write
STEM.bbl:

	(bibtex-compiler:bibtex "example")

In fact, when the STEM.aux file requests a bibliography style, BIBTEX
first looks whether a Lisp function that implements the style has been
defined using DEFINE-BIBTEX-STYLE.  Otherwise, it tries to find (using
kpathsearch) and load a Lisp file named "STYLE.lbst"; it is supposed
to define the style using DEFINE-BIBTEX-STYLE.  Finally, BIBTEX tries
to find "STYLE.bst" and interpret it.

A BibTeX style implemented in Lisp is supposed to read all
bibliography files in *BIB-FILES* and to write the formatted
bibliography to the stream *BBL-OUTPUT*.  The package BIBTEX-RUNTIME
contains useful routines for reading AUX files, formatting names, etc.
An important function is READ-ALL-BIB-FILES-AND-COMPUTE-BIB-ENTRIES.  

You can invoke the BST-to-Common-Lisp compiler with the function
COMPILE-BST-FILE:

	(bibtex-compiler:compile-bst-file (kpathsea:find-file "amsabbrv.bst")
					  "amsabbrv.lbst")

The resulting Lisp file is a readable Common Lisp version of the BST
file, which can be run on the AUX file to produce a BBL file, without
using the BST interpreter:

        (compile-file "amsabbrv.lbst")
        (load "amsabbrv.lbst")            ; calls DEFINE-BIBTEX-STYLE

	(bibtex-compiler:bibtex "example")	    ; now uses the Lisp style

There is also a little shell script "bibtex" for invoking CL-BIBTEX
from the shell; it uses the "run-lisp" script from CLOCC.


Restrictions of the compiler
============================

The BST->CL compiler works by analyzing the type of the wizard-defined
functions (i.e., those defined by a FUNCTION command in the BST
file).  The compiler tries to find out how many "literals" (values) a
function pops from the stack and how many it pushes and determines
their types (BOOLEAN, INTEGER, STRING).

* If a wizard-defined function tries to pop a function literal that it
  hasn't pushed, or leaves a function literal on the stack, you lose.
  (In other words, you can't define higher-order BST functions.)

* The two branches of an IF$ function must deliver the same net number
  of values.  (As a special exception, the type of IF$ within a WHILE$
  body which occurs in the FORMAT.NAMES function of the standard
  styles is also supported.  This is reported with a warning message.)


On the compilation of a language that no-one can read or write
==============================================================

The language of BibTeX style files (BAFLL) is expected to be read or
written by "wizards" only.  (This is how the original BibTeX calls the
authors of BibTeX style files.)  In fact, when casual users try to
customize a BibTeX style file, they often come up with slightly broken
styles that appear to work but which fail in corner cases.  The reason
is that programs written in BAFLL can hardly be read, written, or even
debugged. 

This causes an extra difficulty for the compiler, because broken
corner cases tend to misbehave on the stack, rendering the function
unanalyzable, hence uncompilable.  The compiler now contains a
heuristic that tries to fix these situations, but it may fail.  In
this case, you should try to understand the error messages, fix the
BibTeX style file accordingly, and try compiling it again.


Implementation dependencies
===========================

It does not run on some 0.7.? version of SBCL because it does not like
(COPY-READTABLE NIL).  SBCL 0.7.9 seems to be fine.

I noticed that CMUCL 3.0.8 18c+ does not like (PEEK-CHAR T STREAM NIL
#\Space); it won't skip over whitespace when the EOF character is
whitespace?  (I don't use this any more in CL-BibTeX.)

CLISP 2.30:
1. When I invoke COMPILE-BST-FILE, CLISP says "Lisp stack
   overflow. RESET"; this seems to be caused by the pprint-dispatch
   function for DEFINE-BIBTEX-STYLE.
2. The format directive ~<...~:>  (logical block) does not seem to work
   properly.
(I have used reader conditionalization to make CL-BibTeX work with CLISP.)



Have fun!

 -- Matthias Koeppe <[email protected]>