Now in that blog there was...
Sometimes, in a vast and healthy developer ecosystem of a language like Clojure, it can be difficult to know where to get started when you want to build an application. The target audience of this series is developers who have some Clojure syntax in their hands and want to start building applications.
Over a series of posts we'll build stagehand, a web application for managing an inventory of servers. An inventory is, in the style of Ansible, a collection of data including system and network information.
Specifically, the Servers in our inventory can be:
We'll give it a frontend for common tasks (CRUD, running playbooks) and an API for use by other programs.
This article will cover:
neil to initialize the projectdeps.edn and the Clojure CLIringThese articles will assume readers have some familiarity with the Clojure language. For a quick introduction check out this primer. For a more complete guide checkout the only book for the brave and true.
For every topic I'll provide an introduction and include references to more authoritative or comprehensive sources. By the end readers should have a bit more familiarity and a folder of bookmarks to dig into.
Clojure programmers generally prefer to build applications by composing libraries together rather than using frameworks.
There are Clojure frameworks, they provide a reliable foundation to build on top of. For developers new to Clojure, however, exploring the starter template of a framework feels like figuring out how to get an alien spaceship running.
I believe that a gradual introduction to foundational libraries is a more productive starting point.
To follow along you'll need to install a few things. Links to installation directions here:
bash scripts" resolution bb script to manage your deps.ednFor me, a Clojure dev environment isn't complete without Babashka and neil. These two projects have done a lot in making Clojure more accessible.
Before continuing ensure these commands run without errors:
clojure -M -e '(println "Clojure Online")'
bb -e '(println "Bash? Bash who?")'
neil --version
The choice of text editor is a personal one.
I use Neovim with Conjure. For more detail about my setup check out this post.
For a more complete description of editor options you can check out:
This project will use deps.edn to manage its dependencies, and we'll use neil to manage deps.edn!
neil can:
deps-new templatenreplneil version patchneil version major 3 --forceneil version minor --no-tagMichiel Borkent (@borkdude), the author of babashka, neil, clj-kondo, and many others wrote a great introduction to neil here that goes into more depth.
We'll start by using neil new to initialize the project using a template.
neil new --help
# Usage: neil new [template] [name] [target-dir] [options]
#
# Runs the org.corfield.new/create function from deps-new.
#
# All of the deps-new options can be provided as CLI options:
#
# https://github.com/seancorfield/deps-new/blob/develop/doc/options.md
#
# ...snip...
# The provided built-in templates are:
#
# app
# lib
# pom
# scratch
# template
# ...snip...
The options for deps-new and the default templates can be found here for later reference.
The scratch template includes nearly nothing. Perfect!
Templates can accept options to customize their behavior. We'll use the --scratch option to modify the path of the initial source file the template creates.
neil new scratch stagehand --scratch stagehand/app
# Creating project from org.corfield.new/scratch in stagehand
Let's take a look at our new project:
cd stagehand/
tree
# .
# ├── deps.edn
# └── src
# └── stagehand
# └── app.clj
#
# 2 directories, 2 files
# Not much here. How many lines of code?
wc -l **/*
# 4 deps.edn
# 12 src/stagehand/app.clj
# 16 total
Two files with just sixteen lines of code between them! Might as well include it all here:
;; deps.edn
{:paths ["src"]
:deps {}
:aliases
{:neil {:project {:name stagehand/stagehand}}}}
;; src/stagehand/app.clj
(ns stagehand.app
"FIXME: my new org.corfield.new/scratch project.")
(defn exec
"Invoke me with clojure -X stagehand.app/exec"
[opts]
(println "exec with" opts))
(defn -main
"Invoke me with clojure -M -m stagehand.app"
[& args]
(println "-main with" args))
The docstrings on the functions above show that we can run our new project by either executing a function or by running -main:
clojure -X stagehand.app/exec :name "Rattlin"
# exec with {:name Rattlin}
clojure -M -m stagehand.app Hello World
# -main with (Hello World)
It's working! Those commands are a bit opaque though. The next section will provide some context.
The Clojure CLI is the companion to the deps.edn file. Its main job is to:
classpath so that your source code and libraries are available to the JVMHere are the commands we just ran, with notes on the flags and arguments.
clojure -X stagehand.app/exec :name "Rattlin"
# -X => eXecute
#
# stagehand.app/exec => the `exec` function from the `stagehand.app` namespace
# found on the classpath.
# The function name is not important, though it should
# take a map as a single argument
#
# :name "Rattlin" => `:key "Value"` pairs that are rolled into a map
# and passed to the called function as its only argument.
# In this case that map will look like:
# {:name "Rattlin"}
clojure -M -m stagehand.app Hello World
# -M => Say to yourself, 'Ah, we're using `clojure.main` here.
# So all further options are for `clojure.main`'
#
# -m, --main => Specify a namespace to look for a function named `-main` to
# execute
#
# stagehand.app => The namespace we're going to look for `-main` in
#
# Hello World => Arguments to pass to the `-main` function, as seq of strings
I highly recommend reviewing these resources for a more comprehensive explanation:
clojure.main - Official ReferenceThe scratch template doesn't include a .gitignore file. Let's copy one from the app template:
# assuming you're in the root of the stagehand directory
pushd ..
neil new app the-giver
cp the-giver/.gitignore $OLDPWD
rm -r the-giver
popd
Let's save our game:
git init
git add .gitignore deps.edn src/
git commit -m 'Getting started'
Making this a repo will make it easier to see what the next few commands are adding to our project by using git diff.
One of Clojure's greatest selling points is the REPL, meaning Read Evaluate Print Loop. This allows Clojure programmers to interact with the running application, inspect it, alter its behavior, add functionality, and experiment. Other languages have REPLs, but with Clojure it's a way of life.
Working at the REPL feels like playing with a Rubik's Cube. It's constantly in your hands. The feedback is instant. In comparison, developing compile-and-run languages feels like setting up a bunch of dominos over and over. Though with TDD you can get that loop to look like this:
REPL driven development and TDD are not mutually exclusive. Use your REPL to setup your dominos! Or something!
The Clojure REPL is the gateway to the running program, but we need a way to reach it from our editors. There are a few
ways to do this, but for this project we'll be using nREPL.
We can alter how we start our program so that an nREPL server starts within the same process as the application itself. The server receives messages from nREPL clients, sends the code to the Clojure REPL to be evaluated, and then returns the result back. Many editors are, or have plugins, which allow them to act as nREPL clients.
Refer to this guide from Peter Strömberg aka PEZ, the creator of Calva, for an in-depth explanation.
Certain editors/plugins can inject the nREPL dependency dynamically for you. Calva, for example, starts the process with the following command after selecting Calva: Start a Project REPL and Connect (aka Jack-In)
# line breaks added by me
clojure \
-Sdeps '{:deps {nrepl/nrepl {:mvn/version,"1.0.0"},cider/cider-nrepl {:mvn/version,"0.28.5"}}}' \
-M -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"
# Options:
# -Sdeps EDN From help: Deps data to use as the last deps file to be merged
#
# -M Using clojure.main
# -m nrepl.cmdline Run -main from nrepl.cmdline
#
# --middleware "..." Arguments to the nrepl.cmdline/-main function
If you're using Calva or CIDER's Jack-In process to start your REPL, which has many benefits, feel free to skip ahead.
I generally prefer start the REPL/nREPL server myself and tell my editor connect to the running server. To do that I have to add a dependency on nREPL and add an alias to start the project. Thankfully neil has a command to do just that:
neil add nrepl
{:paths ["src"]
:deps {}
- :aliases
- {:neil {:project {:name stagehand/stagehand}}}}
+ :aliases
+ {:neil {:project {:name stagehand/stagehand}}
+
+ :nrepl ;; added by neil
+ {:extra-deps {nrepl/nrepl {:mvn/version "1.0.0"}}
+ :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}}}
This command has added an nrepl alias to our deps.edn file. Aliases are another feature of the Clojure CLI that enables certain tasks to specify extra dependencies or add another entrypoint to the program.
In this case we see that the nrepl alias specifies an additional dependency on nrepl/nrepl from Maven at version 1.0.0. The :main-opts key is a hint to us that this alias should be run with clojure -M.
clojure -M:nrepl
# Explantion:
# -M => Using `clojure.main` here!
#
# :nrepl => Use the `:nrepl` alias in our deps.edn file so that
# the extra dependency gets loaded, and all the options
# specifed in `:main-opts` get passed to `clojure.main`
For demonstrations of working at the REPL check out:
If we don't add a test runner now we probably never will.
neil add test
tree test/
test
└── stagehand
└── stagehand_test.clj
1 directory, 1 file
Running a git diff will show that neil added an alias to our deps.edn file:
{:paths ["src"]
:deps {}
:aliases
{:neil {:project {:name stagehand/stagehand}}
:nrepl ;; added by neil
{:extra-deps {nrepl/nrepl {:mvn/version "1.0.0"}}
- :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}}}
+ :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}
+
+ :test ;; added by neil
+ {:extra-paths ["test"]
+ :extra-deps {io.github.cognitect-labs/test-runner
+ {:git/tag "v0.5.0" :git/sha "b3fd0d2"}}
+ :main-opts ["-m" "cognitect.test-runner"]
+ :exec-fn cognitect.test-runner.api/test}}}
The test alias adds in the cognitect-labs/test-runner for running our tests. We can run our new test by:
# Using the clojure CLI
clojure -M:test
# or with neil
neil test
We're writing a web application, so we need a way to handle HTTP requests and serve up some HTML. For that we'll use ring.
Ring is the current de facto standard base from which to write web applications in Clojure.
What ring provides:
The ring wiki is great and worth going through end-to-end.
We'll use neil to find the rings, neil to bring them all, and in the deps.edn bind them... ahem
# Neil can help you find libraries with a `search` command
neil dep search ring
# :lib ring/ring-core :version 1.9.6 :description "Ring core libraries."
# :lib ring/ring-codec :version 1.2.0 :description "Library for encoding and decoding data"
# :lib ring/ring-servlet :version 1.9.6 :description "Ring servlet utilities."
# :lib ring/ring-jetty-adapter :version 1.9.6 :description "Ring Jetty adapter."
# :lib ring/ring-devel :version 1.9.6 :description "Ring development and debugging libraries."
# --- snip ---
# We'll start with the minimum set to get off the ground
neil add dep ring/ring-core
neil add dep ring/ring-jetty-adapter
# Let's see how this changes the deps.edn file:
git diff deps.edn
diff --git a/deps.edn b/deps.edn
index 87caaea..4e6b5cd 100644
--- a/deps.edn
+++ b/deps.edn
@@ -1,5 +1,6 @@
{:paths ["src"]
- :deps {}
+ :deps {ring/ring-core {:mvn/version "1.9.6"}
+ ring/ring-jetty-adapter {:mvn/version "1.9.6"}}
:aliases
{:neil {:project {:name stagehand/stagehand}}
With this in place we can start hacking on this application. Start your REPLs!
Remember, if you're using Calva or CIDER's
Jack-Inyou don't need to specify the:nreplalias and can skip to the next section
clojure -M:nrepl
# nREPL server started on port 59171 on host localhost - nrepl://localhost:59171
# nREPL 1.0.0
# Clojure 1.11.1
# OpenJDK 64-Bit Server VM 17.0.4.1+1
# Interrupt: Control+C
# Exit: Control+D or (exit) or (quit)
# user=>
There's a prompt for you to type expressions into, that's the Clojure REPL! We won't be typing much here though. Instead we'll sending code from our text editor as an nREPL client.
As mentioned earlier, starting our program this way causes an nREPL server to start in the same process as our application. As the output mentions, the nREPL server is listening locally on a random port, 59171 in this case. Editors with nREPL support know to look for connect to this server by by referencing the .nrepl-port file, which was created when we ran the previous command.
cat .nrepl-port
# 59171
Refer to your editor specific documentation about managing your connection to the nREPL server and evaluting forms.
Let's get to "Hello World" with ring. Edit src/stagehand/app.clj and type along:
;; file: src/stagehand/app.clj
(ns stagehand.app
"Server Inventory Management"
;; To start working with ring we need a server+adapter
;; Jetty is a good default choice
(:require [ring.adapter.jetty :as jetty]))
;; Adapters convert between server specifics to more general ring
;; requests and response maps. This allows you to change out the server
;; without updating any of your handlers.
;; We'll store the reference to the server in an atom for easy
;; starting and stopping
(defonce server (atom nil))
;; Any function that returns a response is a "handler."
;; Responses are just maps! Ring takes care of the rest
(defn hello
[_request]
{:status 200
:headers {"Content-Type" "text/plain"}
:body "Hello World\n"})
;; note: the `_` in `_request` indicates the argument is unused, while
;; still giving it a useful name
;; Same as above, the 404 handler just returns a map
;; with the "Not Found" status code
(defn not-found
[request]
{:status 404
:headers {"Content-Type" "text/plain"}
:body (str "Not Found: " (:uri request) "\n")})
;; app is the main handler of the application - it'll get
;; called for every request. It will route the request
;; to the correct function.
;;
;; For routing we'll start by just matching the URI.
;; We'll add in a real routing solution in the next blog post
(defn app
[request]
(case (:uri request)
"/" (hello request)
;; Default Handler
(not-found request)))
;; start! the Jetty web server
(defn start! [opts]
(reset! server
(jetty/run-jetty (fn [r] (app r)) opts)))
;; note: the anonymous function used as the handler allows us to revaluate the
;; `app` handler at the REPL to add additional routes / logic without
;; restarting the server or process.
;;
;; Another option is to pass in the handler as a var, `#'app`
;; For a deeper explanation check here:
;; https://clojure.org/guides/repl/enhancing_your_repl_workflow
;; stop! the server and resets the atom back to nil
(defn stop! []
(when-some [s @server]
(.stop s)
(reset! server nil)))
;; -main is used as an entry point for the application
;; when running it outside of the REPL.
(defn -main
"Invoke me with clojure -M -m stagehand.app"
[& _args]
(start! {:port 3000 :join? true}))
;; This is a "Rich" comment block. It serves as a bit of documentation and
;; is convenient for use in the REPL. All the code above is available for use,
;; including our handlers!
(comment
;; Just call the handler by providing your own request map - no need
;; to actually run the server
(app {:uri "/"})
; {:status 200,
; :headers {"Content-Type" "text/plain"},
; :body "Hello World"}
;; For use at the REPL - setting :join? to false to prevent Jetty
;; from blocking the thread
(start! {:port 3000 :join? false})
;; Evaluate whenever you need to stop
(stop!)
;; At the REPL, *e is bound to the most recent exception
*e)
With your nrepl connected editor, evaluate the call to start! in the comment block at the bottom of this file to get the server going. With this we should be able to verify our server is up and our handlers are working as expected:
curl http://localhost:3000
# Hello World
curl http://localhost:3000/bird
# Not Found: /bird
The raw content of an HTTP request looks like:
GET / HTTP/1.1
Host: localhost:3000
User-Agent: curl/7.86.0
Accept: */*
The actual request is easier to write than most plain-text data formats, like JSON or YAML. Unfortunately programs need this to be in a form they can understand. Ring handles translating HTTP requests into Clojure maps.
Let's add a handler to print the request map as our handler sees it.
First we'll add in clojure.pprint to pretty-print the request map:
(ns stagehand.app
"Server Inventory Management"
(:require [ring.adapter.jetty :as jetty]
+ [clojure.pprint :refer [pprint]]))
Add a dump function above app:
(defn dump [request]
{:status 200
:headers {"Content-Type" "text/plain"}
:body (with-out-str (pprint request))})
Update app with an /dump route
(defn app
[request]
(case (:uri request)
"/" (hello request)
+ "/dump" (dump request)
;; Default Handler
(not-found request)))
Reevaluate these functions in your editor/REPL and make a request. Add some extra fields to see how ring handles it:
curl -v 'http://localhost:3000/dump?test=true&something=extra&something=else'
* Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000 (#0)
# Our request
> GET /dump?test=true&something=extra&something=else HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.86.0
> Accept: */*
>
# The response
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sun, 19 Mar 2023 01:28:04 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Server: Jetty(9.4.48.v20220622)
<
{:ssl-client-cert nil,
:protocol "HTTP/1.1",
:remote-addr "127.0.0.1",
:headers
{"accept" "*/*", "user-agent" "curl/7.86.0", "host" "localhost:3000"},
:server-port 3000,
:content-length nil,
:content-type nil,
:character-encoding nil,
:uri "/dump",
:server-name "localhost",
:query-string "test=true&something=extra&something=else",
:body
#object[org.eclipse.jetty.server.HttpInputOverHTTP 0x6ad325e0 "HttpInputOverHTTP@6ad325e0[c=0,q=0,[0]=null,s=STREAM]"],
:scheme :http,
:request-method :get}
The request map Ring produces provides a bit of additional context and breaks out various parts for easy access. There's definitely room for improvement, such as automatically parsing the :query-string. We'll address this in the next section with middleware.
The ring wiki describes the request and response maps in greater detail.
There's a more complete version of this dump handler in the
ring/ring-devellibrary:ring.handler.dump/handle-dump.ring-develhas some very useful functions to aid with development. We'll probably revist this library in a later post.
Middleware offers a way to address cross-cutting concerns across groups of handlers. Middleware can add additional information to a request/response map, or even transform the body of the request.
We'll add some parameter parsing middleware to the entire application. Thankfully ring/ring-core includes middleware to handle this.
First we'll apply the ring.middleware.params/wrap-params to parse any query parameters and form bodies. The full docstring is included here, it's shorter and more complete than anything I could write:
ring.middleware.params/wrap-params [handler] [handler options] ──────────────────────────────────────────────────────────────────────── Middleware to parse urlencoded parameters from the query string and form body (if the request is a url-encoded form). Adds the following keys to the request map: :query-params - a map of parameters from the query string :form-params - a map of parameters from the body :params - a merged map of all types of parameter Accepts the following options: :encoding - encoding to use for url-decoding. If not specified, uses the request character encoding, or "UTF-8" if no request character encoding is set.
The wrap-params middleware above uses string values as keys in the maps it creates. Generally keywords are preferred as keys for easier/faster value access. There's middleware for that too, again here's the docstring:
ring.middleware.keyword-params/wrap-keyword-params [handler] [handler options] ──────────────────────────────────────────────────────────────────────── Middleware that converts the any string keys in the :params map to keywords. Only keys that can be turned into valid keywords are converted. This middleware does not alter the maps under :*-params keys. These are left as strings. Accepts the following options: :parse-namespaces? - if true, parse the parameters into namespaced keywords (defaults to false)
We'll require these namespaces, do a bit of renaming, and then finally apply our middleware using the threading macro.
(ns stagehand.app
"Server Inventory Management"
(:require [ring.adapter.jetty :as jetty]
+ [ring.middleware.params :refer [wrap-params]]
+ [ring.middleware.keyword-params :refer [wrap-keyword-params]]
[clojure.pprint :refer [pprint]]))
----
- (defn app
+ (defn main-handler
[request]
(case (:uri request)
"/" (hello request)
"/dump" (dump request)
;; Default handler
(not-found request)))
+ (def app
+ (-> #'main-handler
+ wrap-keyword-params
+ wrap-params))
Note that #'main-handler is using a var-quote. This makes it so that changes to main-handler are picked up in the REPL.
Middleware is applied in a bottom to top fashion, so first wrap-params does its work, followed by wrap-keyword-params, and then our main-handler.
Reevaluate the file and run that request from earlier:
curl 'http://localhost:3000/dump?test=true&something=extra&something=else'
{ ...omitted...
+ :params {:test "true", :something ["extra" "else"]},
+ :form-params {},
+ :query-params {"test" "true", "something" ["extra" "else"]},
:query-string "test=true&something=extra&something=else",
}
It's working! One thing to note is that query parameters with the same name become a vector in the :params map.
Before writing your own middleware, check the list of standard middleware or thrird party libaries on the ring wiki.
We're off the ground! There's not much of stagehand here yet though. The next article will add a few more libraries and start giving adding in some initial functionality.
The next few articles will cover:
reititThis article was heavily influenced by:
The ClojureDoc guide to Basic Web Development was rewritten by the amazing Sean Corfield shortly before this article was published. It covers a lot more ground, give it a read!
Published: 2023-03-25
Tagged: clojure finding-clojure babashka neil