At least once per week (my personal estimate) some "newbie" on c.l.l or a Lisp-related mailing list seems to be confused about packages one way or the other. They talk about "loading" packages, "requiring" packages, they wonder why, after loading a system, they still have to use package markers, and so on. This is a pet peeve of mine, and I think it might even be one of the reasons why newcomers find using Lisp libraries harder than it actually is.
As I usually end up trying to write a somewhat helpful explanation and as, naturally, all these explanations are very similar, I finally created this page, so the next time I can just point to it instead of typing the same thing over and over again.
First of all, you probably have to clear your mind. The term "package" is pretty much overloaded. Some Linux distributions like Debian or Gentoo have "packages", programming languages like Java, Perl, or Python have "packages", you name it. It is very likely that you came to Lisp with a preconceived notion of what a "package" is or should be. Forget about it while you're here.
In Common Lisp, a package is a first-class object the semantics of which are clearly defined by the language standard. In fact, from all the terms discussed on this page, it is the only one that (in the context of Common Lisp) has an unambiguous definition. Packages are, loosely speaking, containers for symbols. They are there to help you create separate namespaces within your programs, so to say.
Common Lisp has functions and macros to create, modify, investigate, and delete packages. A very good introduction to packages (and symbols) can be found in chapter 21 of Peter Seibel's excellent Lisp tutorial Practical Common Lisp. The definitive resource is chapter 11 of the (online version of) the ANSI Common Lisp specification.
That's basically all there is about packages. Specifically, you don't
load packages. You
can LOAD
Lisp code which in turns creates a package, but that's something
substantially different.
Also, if your Lisp complains that a specific package could not be
found, that means that the package as a Lisp object isn't in
your image
(i.e. FIND-PACKAGE
would return NIL) because it hasn't been created yet. It
does not mean that the implementation was somehow searching
the file system and failed. (A frequent reason for this kind of
failure is that things happen in the wrong order. More about that
below.)
Systems, as opposed to packages, are something the ANSI standard doesn't even mention. Nevertheless, most experienced Lisp programmers know the term because they will know and use some kind of system definition tool. The most prominent one nowadays is ASDF (used by the majority of open source Lisp libraries); another well-known system definition tool, and a lot older than ASDF, is MK:DEFSYSTEM. Several vendors also distribute their own system definition tools together with their Lisps, see for example LispWorks' Common Defsystem.
A system in this sense is, very loosely speaking, a bunch of code plus some instructions how it should be treated, for example which other systems it depends on, what should be loaded and/or compiled first, and so on. In other words, a system definition tool is similar in purpose to, say, make or Ant.
As an aside, system definition tools can usually do much more than that - Common Defsystem can for example integrate COM type library files, and ASDF is fully extensible and has been used, amongst other things, to compile C files. It is also routinely used to define test suites for the systems it describes.
And, by the way, although ASDF is pretty popular it's not ubiquitous. It comes pre-installed with Lisps like SBCL, OpenMCL, or AllegroCL, and it can probably be loaded into every other Lisp, but that doesn't make it a part of Common Lisp. Basically, it is just a bunch of code with no exact specification of what it's supposed to do and with different release versions which are likely not compatible with each other. Go figure...
REQUIRE, PROVIDE,
and *MODULES*
is that this functionality is deprecated and
implementation-dependent. The fact that it's deprecated shouldn't
bother you much. Every Common Lisp nowadays has these functions, and
the chances that a new ANSI standard will emerge and all the
implementations will suddenly drop these functions are, cough, very
slim. What should bother you, though, is
that REQUIRE might be convenient, but isn't portable. (If
you don't care about portability, that's fine of course.)
For example, in LispWorks you can use
(require "foreign-parser")to load a parser that can read C declarations, but that won't work on, say, OpenMCL. Likewise, you can use
(require :asdf)to load ASDF on OpenMCL, but not on LispWorks.
Some Lisps offer hooks so that you can tweak the
way REQUIRE works, and there are extensions
like common-lisp-controller
that couple REQUIRE with ASDF, but generally a module is
some implementation-dependent thingy you shouldn't confuse with (ASDF)
systems (not to mention
with packages).
There's probably no clear definition for what a library is, but most people will think of it as a collection of code that is supposed to perform one or more specific tasks and is distributed as a whole, usually as some kind of compressed archive that can be downloaded somewhere. Actually, this vague term is I think the most appropriate one if you want to talk about Lisp software. Most Lisp libraries nowadays come with (ASDF) system definitions, but they don't have to. Depending on how you received them, it could be possible that they will pretend to be a module in your Lisp, but they don't have to. Also, a library will usually define one or more Lisp packages, but not necessarily so.
And, due to conventions and maybe lack of imagination, it can and will happen that the library "FOO" comes with a definition for a system "FOO" that you can maybe load as a module named "FOO". And once the code is loaded you'll have a new Lisp package called "FOO". These are four different things which just happen to have the same name! I admit that this is confusing, but I hope that the paragraphs above helped to clarify the situation a bit.
A related issue is that often people complain that they can't compile a Lisp file containing code like this:
;; this line could also be (require :cl-ppcre) (asdf:oos 'asdf:load-op :cl-ppcre) (defun my-simple-number-scanner (string) (cl-ppcre:scan "^[0-9]+$" string))Why is that? And why can I load this file if I can't compile it? And why can I compile it after I've loaded it? Isn't that strange?
No, it's not. The compiler will read the first form (which
is an instruction to compile - if necessary - and load the
CL-PPCRE system), but it won't execute
it. The compiler is only interested in compiling code, after
all. Once it is done with the first form, it'll proceed to the second
form, the function definition. At this point it will eventually
complain as the Lisp reader which tries to read this form
encounters the sequence of characters "cl-ppcre:scan"
which is supposed to denote
an external
symbol of the CL-PPCRE Lisp package, but
there is no CL-PPCRE package yet. The process of loading the
CL-PPCRE system will, amongst other things, create the CL-PPCRE
package, but that hasn't happened yet.
Read chapter 3
of the CLHS for more.
You could
use EVAL-WHEN
here to instruct the compiler to load the
CL-PPCRE system before it reads the
second form. You should rather find a better way to organize
your code, though. The first form is basically a declaration that
your code has a dependency on the
CL-PPCRE system. This shouldn't be in the same
file as your Lisp
code. Write your own
system definition for your code and put the dependencies there.
$Header: /usr/local/cvsrep/weitz.de/packages.html,v 1.13 2008/04/22 09:52:49 edi Exp $