CL-WHO - Yet another Lisp markup language


 

Abstract

There are plenty of Lisp Markup Languages out there - every Lisp programmer seems to write at least one during his career - and CL-WHO (where WHO means "with-html-output" for want of a better acronym) is probably just as good or bad as the next one. They are all more or less similar in that they provide convenient means to convert S-expressions intermingled with code into (X)HTML, XML, or whatever but differ with respect to syntax, implementation, and API. So, if you haven't made a choice yet, check out the alternatives as well before you begin to use CL-WHO just because it was the first one you came across. (Was that repelling enough?) If you're looking for a slightly different approach you might also want to look at HTML-TEMPLATE.

I wrote this one in 2002 although at least Tim Bradshaw's htout and AllegroServe's HTML generation facilities by John Foderaro of Franz Inc. were readily available. Actually, I don't remember why I had to write my own library - maybe just because it was fun and didn't take very long. The syntax was obviously inspired by htout although it is slightly different.

CL-WHO tries to create efficient code in that it makes constant strings as long as possible. In other words, the code generated by the CL-WHO macros will usually be a sequence of WRITE-STRING forms for constant parts of the output interspersed with arbitrary code inserted by the user of the macro. CL-WHO will make sure that there aren't two adjacent WRITE-STRING forms with constant strings. CL-WHO's output is either XHTML (default), 'plain' (SGML) HTML or HTML5 (using HTML syntax) — depending on what you've set HTML-MODE to.

CL-WHO is intended to be portable and should work with all conforming Common Lisp implementations. Let us know if you encounter any problems.

It comes with a BSD-style license so you can basically do with it whatever you want.

CL-WHO is for example used by clutu and Heike Stephan.

Download shortcut: http://weitz.de/files/cl-who.tar.gz.


 

Contents

  1. Example usage
  2. Download and installation
  3. Support and mailing lists
  4. Syntax and Semantics
  5. The CL-WHO dictionary
    1. with-html-output
    2. with-html-output-to-string
    3. *attribute-quote-char*
    4. *downcase-tokens-p*
    5. *html-empty-tag-aware-p*
    6. *html-empty-tags*
    7. *html-no-indent-tags*
    8. *prologue*
    9. esc
    10. fmt
    11. htm
    12. str
    13. html-mode
    14. escape-string
    15. escape-char
    16. *escape-char-p*
    17. escape-string-minimal
    18. escape-string-minimal-plus-quotes
    19. escape-string-iso-8859-1
    20. escape-string-all
    21. escape-char-minimal
    22. escape-char-minimal-plus-quotes
    23. escape-char-iso-8859-1
    24. escape-char-all
    25. conc
    26. convert-tag-to-string-list
    27. convert-attributes
  6. Acknowledgements

 

Example usage

Let's assume that *HTTP-STREAM* is the stream your web application is supposed to write to. Here are some contrived code snippets together with the Lisp code generated by CL-WHO and the resulting HTML output.
(with-html-output (*http-stream*)
  (loop for (link . title) in '(("http://zappa.com/" . "Frank Zappa")
                                ("http://marcusmiller.com/" . "Marcus Miller")
                                ("http://www.milesdavis.com/" . "Miles Davis"))
        do (htm (:a :href link
                  (:b (str title)))
                :br)))
Frank Zappa
Marcus Miller
Miles Davis
;; code generated by CL-WHO (simplified)

(let ((*http-stream* *http-stream*))
  (progn
    nil
    (loop for (link . title) in '(("http://zappa.com/" . "Frank Zappa")
                                  ("http://marcusmiller.com/" . "Marcus Miller")
                                  ("http://www.milesdavis.com/" . "Miles Davis"))
          do (progn
               (write-string "<a href='" *http-stream*)
               (princ link *http-stream*)
               (write-string "'><b>" *http-stream*)
               (princ title *http-stream*)
               (write-string "</b></a><br />" *http-stream*)))))
(with-html-output (*http-stream*)
  (:table :border 0 :cellpadding 4
   (loop for i below 25 by 5
         do (htm
             (:tr :align "right"
              (loop for j from i below (+ i 5)
                    do (htm
                        (:td :bgcolor (if (oddp j)
                                        "pink"
                                        "green")
                             (fmt "~@R" (1+ j))))))))))
IIIIIIIVV
VIVIIVIIIIXX
XIXIIXIIIXIVXV
XVIXVIIXVIIIXIXXX
XXIXXIIXXIIIXXIVXXV
;; code generated by CL-WHO (simplified)

(let ((*http-stream* *http-stream*))
  (progn
    nil
    (write-string "<table border='0' cellpadding='4'>" *http-stream*)
    (loop for i below 25 by 5
          do (progn
               (write-string "<tr align='right'>" *http-stream*)
               (loop for j from i below (+ i 5)
                     do (progn
                          (write-string "<td bgcolor='" *http-stream*)
                          (princ (if (oddp j) "pink" "green") *http-stream*)
                          (write-string "'>" *http-stream*)
                          (format *http-stream* "~@r" (1+ j))
                          (write-string "</td>" *http-stream*)))
               (write-string "</tr>" *http-stream*)))
    (write-string "</table>" *http-stream*)))
(with-html-output (*http-stream*)
  (:h4 "Look at the character entities generated by this example")
   (loop for i from 0
         for string in '("Fête" "Sørensen" "naïve" "Hühner" "Straße")
         do (htm
             (:p :style (conc "background-color:" (case (mod i 3)
                                                    ((0) "red")
                                                    ((1) "orange")
                                                    ((2) "blue")))
              (htm (esc string))))))

Look at the character entities generated by this example

Fête

Sørensen

naïve

Hühner

Straße

;; code generated by CL-WHO (simplified)

(let ((*http-stream* *http-stream*))
  (progn
    nil
    (write-string
     "<h4>Look at the character entities generated by this example</h4>"
     *http-stream*)
    (loop for i from 0 for string in '("Fête" "Sørensen" "naïve" "Hühner" "Straße")
          do (progn
               (write-string "<p style='" *http-stream*)
               (princ (conc "background-color:"
                            (case (mod i 3)
                              ((0) "red")
                              ((1) "orange")
                              ((2) "blue")))
                      *http-stream*)
               (write-string "'>" *http-stream*)
               (progn (write-string (escape-string string) *http-stream*))
               (write-string "</p>" *http-stream*)))))

 

Download and installation

CL-WHO together with this documentation can be downloaded from http://weitz.de/files/cl-who.tar.gz. The current version is 1.1.3.

The preferred method to fetch, compile and load CL-WHO is via Quicklisp. Install Quicklisp, then run

(ql:quickload :cl-who)

The current development version of CL-WHO can be found at https://github.com/edicl/cl-who. This is the one to send patches against. Use at your own risk.

Luís Oliveira maintains an unofficial darcs repository of CL-WHO at http://common-lisp.net/~loliveira/ediware/.

You can run a test suite which tests some (but not all) aspects of the library with

(asdf:oos 'asdf:test-op :cl-who)

 

Support and mailing lists

For questions, bug reports, feature requests, improvements, or patches please use the cl-who-devel mailing list. If you want to be notified about future releases subscribe to the cl-who-announce mailing list. These mailing lists were made available thanks to the services of common-lisp.net.

If you want to send patches, please read this first.
 

Syntax and Semantics

CL-WHO is essentially just one macro, WITH-HTML-OUTPUT, which transforms the body of code it encloses into something else obeying the following rules (which we'll call transformation rules) for the body's forms:
 

The CL-WHO dictionary

CL-WHO exports the following symbols:


[Macro]
with-html-output (var &optional stream &key prologue indent) declaration* form* => result*


This is the main macro of CL-WHO. It will transform its body by the transformation rules described in Syntax and Semantics such that the output generated is sent to the stream denoted by var and stream. var must be a symbol. If stream is NIL it is assumed that var is already bound to a stream, if stream is not NIL var will be bound to the form stream which will be evaluated at run time. prologue should be a string (or NIL for the empty string which is the default) which is guaranteed to be the first thing sent to the stream from within the body of this macro. If prologue is T the prologue string is the value of *PROLOGUE*.

CL-WHO will usually try not to insert any unnecessary whitespace in order to save bandwidth. However, if indent is true line breaks will be inserted and nested tags will be indented properly. The value of indent - if it is an integer - will be taken as the initial indentation. If it is not an integer it is assumed to mean 0. Value of *HTML-NO-INDENT-TAGS* controls which tag-contents are excempt from indentation: by default contents of PRE and TEXTAREA tags are not indented to avoid spurious layout changes. (Note: in certain situations additional whitespace may change the layout of tables.)

The results are the values returned by the forms.

Note that the keyword arguments prologue and indent, and the associated variables are used at macro expansion time.

* (with-html-output (*standard-output* nil :prologue t)
    (:html (:body "Not much there"))
    (values))
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html><body>Not much there</body></html>
* (with-html-output (*standard-output*)
    (:html (:body :bgcolor "white"
             "Not much there"))
    (values))
<html><body bgcolor='white'>Not much there</body></html>
* (with-html-output (*standard-output* nil :prologue t :indent t)
    (:html (:body :bgcolor "white"
             "Not much there"))
    (values))
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <body bgcolor='white'>
    Not much there
  </body>
</html>


[Macro]
with-html-output-to-string (var &optional string-form &key element-type prologue indent) declaration* form* => result*


This is just a thin wrapper around WITH-HTML-OUTPUT. Indeed, the wrapper is so thin that the best explanation probably is to show its definition:
(defmacro with-html-output-to-string ((var &optional string-form
                                           &key (element-type ''character)
                                                prologue
                                                indent)
                                      &body body)
  "Transform the enclosed BODY consisting of HTML as s-expressions
into Lisp code which creates the corresponding HTML as a string."
  `(with-output-to-string (,var ,string-form :elementy-type ,element-type)
    (with-html-output (,var nil :prologue ,prologue :indent ,indent)
      ,@body)))
Note that the results of this macro are determined by the behaviour of WITH-OUTPUT-TO-STRING.


[Special variable]
*attribute-quote-char*


This character is used as the quote character when building attributes. Defaults to the single quote #\'. Only other reasonable character is the double quote #\".


[Special variable]
*downcase-tokens-p*


If the value of this variable is NIL, keyword symbols representing a tag or attribute name will not be automatically converted to lowercase. This is useful when one needs to output case sensitive XML. The default is T.


[Special variable]
*html-empty-tag-aware-p*


Set this to NIL to if you want to use CL-WHO as a strict XML generator. Otherwise, CL-WHO will only write empty tags listed in *HTML-EMPTY-TAGS* as <tag/> (XHTML mode) or <tag> (SGML mode or HTML mode). For all other tags, it will always generate <tag></tag>. The initial value of this variable is T.


[Special variable]
*html-empty-tags*


The list of HTML tags that should be output as empty tags. See *HTML-EMPTY-TAG-AWARE-P*. The initial value is the list
(:area :atop :audioscope :base :basefont :br :choose :col :command :embed
 :frame :hr :img :input :isindex :keygen :left :limittext :link :meta :nextid
 :of :over :param :range :right :source :spacer :spot :tab :track :wbr)


[Special variable]
*html-no-indent-tags*


The list of HTML tags that should disable indentation inside them even when indentation is requested. The initial value is a list containing only :pre and :texarea.


[Special variable]
*prologue*


This is the prologue string which will be printed if the prologue keyword argument to WITH-HTML-OUTPUT is T. Gets changed when you set HTML-MODE. Its initial value is
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">"


[Symbol]
esc
[Symbol]
fmt
[Symbol]
htm
[Symbol]
str


These are just symbols with no bindings associated with them. The only reason they are exported is their special meaning during the transformations described in Syntax and Semantics.


[Accessor]
html-mode => mode
(setf (html-mode) mode)


The function HTML-MODE returns the current mode for generating HTML. The default is :XML for XHTML. You can change this by setting it with (SETF (HTML-MODE) :SGML) to pre-XML HTML mode or (SETF (HTML-MODE) :HTML5) to HTML5 mode (using HTML syntax).

Setting it to SGML HTML sets the *prologue* to the doctype string for HTML 4.01 transitional:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
Code generation in SGML HTML is slightly different from XHTML - there's no need to end empty elements with /> and empty attributes are allowed.

Setting it to HTML5 sets the *prologue* to the following doctype string:

<!DOCTYPE html>


[Function]
escape-string string &key test => escaped-string


This function will accept a string string and will replace every character for which test returns true with its character entity. The numeric character entities use decimal instead of hexadecimal values when HTML-MODE is set to :SGML because of compatibility reasons with old clients. test must be a function of one argument which accepts a character and returns a generalized boolean. The default is the value of *ESCAPE-CHAR-P*. Note the ESC shortcut described in Syntax and Semantics.
* (escape-string "<Hühner> 'naïve'")
"&lt;H&#xFC;hner&gt; &#x27;na&#xEF;ve&#x27;"
* (with-html-output-to-string (s)
    (:b (esc "<Hühner> 'naïve'")))
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\"<b>&lt;H&#xFC;hner&gt; &#x27;na&#xEF;ve&#x27;</b>"


[Function]
escape-char character &key test => escaped-string


This function works identical to ESCAPE-STRING, except that it operates on characters instead of strings.


[Special variable]
*escape-char-p*


This is the default for the test keyword argument to ESCAPE-STRING and ESCAPE-CHAR. Its initial value is
#'(lambda (char)
    (or (find char "<>&'\"")
        (> (char-code char) 127)))


[Function]
escape-string-minimal string => escaped-string
[Function]
escape-string-minimal-plus-quotes string => escaped-string
[Function]
escape-string-iso-8859-1 string => escaped-string
[Function]
escape-string-all string => escaped-string
[Function]
escape-char-minimal character => escaped-string
[Function]
escape-char-minimal-plus-quotes character => escaped-string
[Function]
escape-char-iso-8859-1 character => escaped-string
[Function]
escape-char-all character => escaped-string


These are convenience function based on ESCAPE-STRING and ESCAPE-CHAR. The string functions are defined in a way similar to this one:
(defun escape-string-minimal (string)
  "Escape only #\<, #\>, and #\& in STRING."
  (escape-string string :test #'(lambda (char) (find char "<>&"))))

(defun escape-string-minimal-plus-quotes (string)
  "Like ESCAPE-STRING-MINIMAL but also escapes quotes."
  (escape-string string :test #'(lambda (char) (find char "<>&'\""))))

(defun escape-string-iso-8859-1 (string)
  "Escapes all characters in STRING which aren't defined in ISO-8859-1."
  (escape-string string :test #'(lambda (char)
                                  (or (find char "<>&'\"")
                                      (> (char-code char) 255)))))

(defun escape-string-all (string)
  "Escapes all characters in STRING which aren't in the 7-bit ASCII
character set."
  (escape-string string :test #'(lambda (char)
                                  (or (find char "<>&'\"")
                                      (> (char-code char) 127)))))
The character functions are defined in an analogous manner.


[Function]
conc &rest string-list => string


Utility function to concatenate all arguments (which should be strings) into one string. Meant to be used mainly with attribute values.
* (conc "This" " " "is" " " "a" " " "sentence")
"This is a sentence"
* (with-html-output-to-string (s)
    (:div :style (conc "padding:"
                       (format nil "~A" (+ 3 2)))
     "Foobar"))
"<div style='padding:5'>Foobar</div>"


[Generic Function]
convert-tag-to-string-list tag attr-list body body-fn => strings-or-forms


This function exposes some of CL-WHO's internals so users can customize its behaviour. It is called whenever a tag is processed and must return a corresponding list of strings or Lisp forms. The idea is that you can specialize this generic function in order to process certain tags yourself.

tag is a keyword symbol naming the outer tag, attr-list is an alist of its attributes (the car is the attribute's name as a keyword, the cdr is its value), body is the tag's body, and body-fn is a function which should be applied to the body to further process it. Of course, if you define your own methods you can ignore body-fn if you want.

Here are some simple examples:

* (defmethod convert-tag-to-string-list ((tag (eql :red)) attr-list body body-fn)
    (declare (ignore attr-list))
    (nconc (cons "<font color='red'>" (funcall body-fn body)) (list "</font>")))
; Compiling LAMBDA (PCL::.PV-CELL. PCL::.NEXT-METHOD-CALL. TAG ATTR-LIST BODY BODY-FN):
; Compiling Top-Level Form:

#<STANDARD-METHOD CONVERT-TAG-TO-STRING-LIST ((EQL :RED) T T T) {582B268D}>
* (with-html-output (*standard-output*)
    (:red (:b "Bold and red"))
    (values))
<font color='red'><b>Bold and red</b></font>
* (show-html-expansion (s)
    (:red :style "spiffy" (if (foo) (htm "Attributes are ignored"))))

(LET ((S S))
  (PROGN
   NIL
   (WRITE-STRING "<font color='red'>" S)
   (IF (FOO) (PROGN (WRITE-STRING "Attributes are ignored" S)))
   (WRITE-STRING "</font>" S)))
* (defmethod convert-tag-to-string-list ((tag (eql :table)) attr-list body body-fn)
    (cond ((cdr (assoc :simple attr-list))
           (nconc (cons "<table"
                        (convert-attributes (remove :simple attr-list :key #'car)))
                  (list ">")
                  (loop for row in body
                        collect "<tr>"
                        nconc (loop for col in row
                                    collect "<td>"
                                    when (constantp col)
                                      collect (format nil "~A" col)
                                    else
                                      collect col
                                    collect "</td>")
                        collect "</tr>")
                  (list "</table>")))
          (t
            ;; you could as well invoke CALL-NEXT-METHOD here, of course
            (nconc (cons "<table "
                         (convert-attributes attr-list))
                   (list ">")
                   (funcall body-fn body)
                   (list "</table>")))))
; Compiling LAMBDA (PCL::.PV-CELL. PCL::.NEXT-METHOD-CALL. TAG ATTR-LIST BODY BODY-FN):
; Compiling Top-Level Form:

#<STANDARD-METHOD CONVERT-TAG-TO-STRING-LIST ((EQL :TABLE) T T T) {58AFB7CD}>
* (with-html-output (*standard-output*)
    (:table :border 0 (:tr (:td "1") (:td "2")) (:tr (:td "3") (:td "4"))))
<table  border='0'><tr><td>1</td><td>2</td></tr><tr><td>3</td><td>4</td></tr></table>
"</td></tr></table>"
* (show-html-expansion (s)
    (:table :simple t :border 0
            (1 2) (3 (fmt "Result = ~A" (compute-result)))))

(LET ((S S))
  (PROGN
   NIL
   (WRITE-STRING
    "<table border='0'><tr><td>1</td><td>2</td></tr><tr><td>3</td><td>"
    S)
   (FORMAT S "Result = ~A" (COMPUTE-RESULT))
   (WRITE-STRING "</td></tr></table>" S)))


[Function]
convert-attributes attr-list => strings-or-forms


This is a helper function which can be called from CONVERT-TAG-TO-STRING-LIST to process the list of attributes.

 

Acknowledgements

Thanks to Tim Bradshaw and John Foderaro for the inspiration provided by their libraries mentioned above. Thanks to Jörg-Cyril Höhle for his suggestions with respect to attribute values. Thanks to Kevin Rosenberg for the LHTML patch. Thanks to Stefan Scholl for the 'old school' patch. Thanks to Mac Chan for several useful additions.

$Header: /usr/local/cvsrep/cl-who/doc/index.html,v 1.68 2009/03/09 21:54:11 edi Exp $

BACK TO MY HOMEPAGE