As I've mentioned in my Logs (which are a personal record of what I think about, read, and work on each week), I'm learning Common Lisp from the famous "Gentle book". I can't recommend this book enough, and it's also available for free online.
Anyway, as a break from my studies, I decided to test my rudimentary knowledge of CL by building a static website generator. I also wanted to test the language a bit as well, testing the "code is data, data is code" maxim by putting the website content and styling into the same file as the generator; the source code itself includes the content on each webpage. For obvious reasons* this isn't the best way to make a static site generator -- actually, a lot of things about this program are horrible. But as a CL beginner I learned a lot from making it, so maybe it will help out other beginners.
*Turns out that files are useful abstractions! Also, keeping everything in a single file can cause performance and usability issues, and it's more difficult to organize and include things like images
So before I get started, here are a few disclaimers:
- I'm a BEGINNER programmer, both in Common Lisp and in general
- I took a few shortcuts that you don't want to take in production code
- I've worked through Chapter 6 of Common LISP: A Gentle Introduction to Symbolic Computation, so it's probably easiest to understand this post if you at least get that far. However, I haven't read about file io, macros, and a few other features that I had to use. You should be able to understand how I use them them, because I'll explain what I'm doing very carefully.
Before writing the program, we have to decide what we want it to do.
Basically, we want a program that can define shared elements for our webpages, so that every new page we add will have the same styling, menu, etc. This is what static site generators like Hugo and Jekyll do, except they do it with more complexity and options. For now, let's forget all of that and build an example website.
The example site I built has two pages, "index.html" and "contact.html".
Here is the code for the "index.html" webpage:
<HTML> <HEAD> <TITLE>Site Title</TITLE> </HEAD> <div id="menu"> <ul> <li><a href="contact.html">Contact</a></li> <li><a href="index.html">Home</a></li> </ul> </div> <BODY BGCOLOR="FFF000"> <CENTER><IMG SRC=" https://barkpost-assets.s3.amazonaws.com/wp-content/uploads/2013/05/63.jpg" ALIGN="BOTTOM"> </CENTER> <HR> This is a test homepage.<hr> <B><I>This is the footer. Be sure to <a href="contact.html">Contact Me</a>. </I></B> </BODY> </HTML>
If you compare the two webpages, you'll notice that they are built with the same HTML in the header and the footer. The only difference is the "content", or body of each page. We can use this structure in our program.
Before we start defining the structure of our program, I want to point out one drawback of designing according to this structure. Notice the footer in the webpage? If we were making our own website, we'd probably want to include a slightly different footer for the contact page, because it links to itself, which is silly. This would be easy to accomplish by writing a separate part of the program to build the contact page, but this simple structure (header, per-page content, and footer) won't allow it.
Building a Webpage
Now we're ready to start building the program. The most obvious starting point is to store the header, footer, and webpage content in variables. However, this is frustrating if you don't like escaping every quotation mark found in standard HTML:
<div id=\"menu\"> <ul> <li><a href=\"contact.html\">Contact</a></li> <li><a href=\"index.html\">Home</a></li> </ul> </div> <BODY BGCOLOR=\"FFF000\"> etc...
Instead, let's use something called a reader macro; this is one of those things where I'm going to skip explaining, and just steal someone else's implementation. Let Over Lambda has a simple macro, Sharp Greater Than, that we can use to input HTML into our program as-is. It works like this: instead of typing quotation marks when you're designating a string, type "#>" followed by a phrase that you'll use to signal the end of the string, for example
END-OF-THE-STRING. Here's how I defined the footer in my lisp program:
(defparameter footer #>EOF <hr> <B><I>This is the footer. Be sure to <a href="contact.html">Contact Me</a>. </I></B> </BODY> </HTML> EOF)
I used the phrase "EOF" to designate the start and finish of the string, so that I could type whatever I wanted between them. Of course, you couldn't include EOF anywhere in the webpage or the string would end prematurely, and you'd get errors.
Use the same technique to put the header into a variable called "header". Now create a temporary variable to hold the content for the "index.html" webpage. In a minute, we'll store the content of our pages in a tree so the program can automatically cycle through them and build the website, but for now lets store it in a variable called "index.html".
(defparameter index.html "This is a test homepage.")
Now we have three strings that we need to concatenate and write to a file. Practical Common Lisp has a good chapter on files and file i/o, but if you don't want to get into that stuff yet, Richard Socher provides a simple writetofile function that we can use. It takes two strings
content, and writes
content to the file located at
name in the filesystem. For example, the following code writes "Snow dusts a forest of hoodoos" (and a newline) to the file "hoodoo.txt" in my home directory:
(writetofile "/home/wrycode/hoodoo.txt" "Snow dusts a forest of hoodoos~%")
Now we can test a "manual" build of one of the webpages, before we start abstracting things away in our program. Try building a webpage using "writetofile" and the "concatenate" function:
(writetofile "/home/yourusername/location/for/index.html" (concatenate 'string header index.html footer))
Make sure to modify the "name" argument so that the webpage gets built where you want. Also, notice the
concatenate - that simply tells the function that it's dealing with strings.
Finishing the Program
We've done the hard part now. Opening and writing to a file and finding a good way to save the webpages in our source code were the most difficult problems. Luckily, I had google to help me cheat on them :)
I mentioned earlier that we were going to store the different webpages in a table. In lisp an association list, or alist for short, is really a table. I'm not going to into too much detail about that, because it's covered very well in Common LISP: A Gentle Introduction to Symbolic Computation, Chapter 6.
The following code stores the content of the two webpages in an alist with the keys being symbols that represent the filenames:
(defparameter *pages* '( (index.html "This is a test homepage.") (contact.html #>EOF <H1>This is a Header</H1> <H2>This is a Medium Header</H2> Send me mail at <a href="mailto:firstname.lastname@example.org"> email@example.com</a>. <P> This is a new paragraph! EOF)))
The alist is stored in the global variable
*pages*. Note that
header are also global variables in this program. I used a normal double-quoted string for the content of the homepage, but contact.html had quotes so I used the
#> read macro, just like in the header and footer.
Now, we need to write code to cycle through
*pages* and build and write the code for each webpage using the commands from earlier.
First, a function to build each webpage, which takes the body content of the page as its only argument:
(defun build-page (page-content) (concatenate 'string header page-content footer))
build-page outputs a string containing the entire file, so if we call it with "This is a test homepage." as the argument, it will correctly build the index.html file.
The next function,
build-site is a little messier, but I left it like this so it can be fiddled with. At the end of the post I'll list some simple improvements that could be made to the program.
(defun build-site () (dolist (page *pages*) (let ((page-content (car (rest page))) (page-name (car page))) (writetofile (concatenate 'string *siteloc* (string-downcase (string page-name))) (build-page page-content)) )))
dolist loops through every element in pages. It binds the current element to the variable
page and executes the forms below it before moving to the next element. In our case, the list will only execute twice, because we only have two pages.
let form is for setting the temporary variables
page-name. Note that
page-name is actually a symbol, not a string, because I chose to write the table with symbols as the keys instead of strings. This will matter in a moment.
After the temporary variable bindings,
let proceeds with the
writetofile form. For the
name parameter, we are constructing a filename out of the site's base directory and the page name. Remember how
page-name is a symbol?
string fixes that, and I chose to downcase it as well (it doesn't matter on the World Wide Web, but local file systems are case-sensitive, so the menu links in our header and the contact link in our footer would be broken for local viewing otherwise). Make sure to define
*siteloc* before running the code yourself.
content argument to
writetofile is pretty self-explanatory; it calls the
build-page function we just built with the
page-content variable, and outputs the complete HTML webpage for
writetofile to use.
That's the whole program! I've moved on to my next project, but here are some ideas for changing or extending it:
- The way we're accessing the filesystem is non-portable, consider using
make-pathnameas outlined in Practical Common Lisp
- What if we want multiple types of pages in our website? Modify
build-posts, for two different types of webpages stored in two different alists
- Add a function called
build-blogpagethat cycles through the posts variable and builds a blog page
- Do that, but with pagination?
- Writing blog posts in HTML is a pain (I haven't written a line of HTML by hand in years, even the code in this post was copy-pasted). Integrate a markdown library so that you don't have to.