When I was first learning about UNIX, and learning to use Linux, the most immediately powerful tool that I found was the shell’s pipe operator, ‘|’. Using the commandline (because at that point, linux GUI’s were not so well developed, and the few distros that tried to allow strictly graphical operation usually failed miserably) was at times difficult, and at times rewarding, but it was the pipe that opened up a whole world for me.

I can remember looking through an online student directory in highschool that had names, email addresses, etc. For student government elections it had become popular (if incredibly time consuming) to copy and paste the hundreds of email addresses and send a message to the every student. For me, with my newfound skills, it amounted to something like:

cat directory.txt | grep @ | awk '{print $3}' | perl -pe 's/\n/,/'

It seemed like magic at the time, and in some ways, it still does. What the shell (and UNIX in general) offered was composability - it gave you simple (but powerful) tools, and a standard way of linking them together - text streams. By combining those together, it offered immeasurable power, much more than any single tool. The mathematics of combinations guarantees this.

The more I use graphical interfaces (or anything that does not operate on text streams - commandline curses programs included), the more I am struck by how profound the loss of composability is - each program has to try to implement all the standard things (searching, sorting, transforming) that you might want to do with the information it has, and in that repetition lies inconsistencies and usually plain lack of power. The better ones share common libraries, and gain common functionality, but this only amounts to their least common denominator - two separate programs can not (easily) expose their higher functionality to each other (at least not it compiled languages) in the way that commandline stream processing programs can.

What I realized the other day, is that iOS is the extreme example of that lack of flexibility, taken almost to the point of caricature - the only interaction that is possible is through single applications that for the most part can have no connection to other applications. People rejoiced when copy and paste was added, but that celebration hides a sad loss of the true power that computers have. The existence of files - the only real way that composability is achieved in GUI systems (ie, do one thing, save the file, open with another program, etc) - has been essentially eliminated, and applications must therefore do everything that a user might want to do with whatever data they have or will get from the user.

I’d noticed before how frustrating it was for me to use iOS, but I wasn’t sure until recently exactly why that was, until I realized that it had effectively taken away the one thing that is so fundamental about computers, and why I am a programmer - the ability to compose. Every day I live and breath abstraction, and building things out of different levels of it, and the idea of not being able to combine various parts to make new things is so antithetical to that type of thinking that I almost can’t imagine that iOS was created by programmers. I remember looking at the technical specifications of the most recent iPhone and thinking - that is a full computer, and it’s small enough to fit in a pocket - that is a profound change in the way the world works. But it’s not a computer, it’s just a glorified palm pilot with a few bells and whistles.

The language that I’ve been learning recently is a pure (ie, side-effect free) logic/functional language named Mercury. There is a wonderful tutorial (PDF) available, which explains the basics, but beyond that, the primary documentation is the language reference (which is well written, but reasonably dense) and Mercury’s standard library reference (which is autogenerated and includes types and source comments, nothing else).

Doing I/O in a pure language is a bit of a conundrum - Haskell solved this by forcing all I/O into a special monad that keeps track of sequencing (and has a mythical state of the world that it changes each time it does something, so as not to violate referential transparency). Mercury has a simpler (though equivalent) approach - every predicate that does IO must take an world state and must give back a new world state. Old world states can not be re-used (Mercury’s mode system keep track of that), and so the state of the world is manually threaded throughout the program. A simple example would be:

main(IO_0,IO_final) :- io.write_string("Hello World!",IO_0,IO_1), 
                       io.nl(IO_1,IO_final).

Where the first function consumes the IO_0 state and produces IO_1 (while printing “Hello World!”) and the second function consumes IO_1 and produces IO_final (while printing a newline character).

Of course, manually threading those could become pretty tedious, so they have a shorthand, where the same code above could be written as:

main(!IO) :- io.write_string("Hello World!",!IO), 
             io.nl(!IO).

This is just syntax sugar, and can work with any parameters that are dealt with in the same way (and naming it IO for io state is just convention). It definitely makes dealing with I/O more pleasant.

The task that I set was to figure out how to read in a file. This is not covered in the tutorial, and I thought it would be a simple matter of looking through the library reference for the io library. One of the first predicates looks promising:

:- pred io.read_file(io.maybe_partial_res(list(char))::out, 
                     io::di, 
                     io::uo) is det.

But on second thought, something seems to be missing. The second and third parameters are the world states (the type is io, the mode di stands for destructive-input, meaning the variable cannot be used again, uo means unique output, which means that no other variable in the program can have that value), and the first one is going to be the contents of the file itself. But where is the file name?

The comment provides the necessary pointer:

% Reads all the characters from the current input stream until
% eof or error.

Hmm. So all of these functions operate on whatever the current input stream is. How do we set that? io.set_input_stream looks pretty good:

% io.set_input_stream(NewStream, OldStream, !IO):
% Changes the current input stream to the stream specified.
% Returns the previous stream.
%
:- pred io.set_input_stream(io.input_stream::in, 
                            io.input_stream::out,
                            io::di, io::uo) is det.

But even better is io.see, which will try to open a file and if successful, will set it to the current stream (the alternative is to use io.open_input and then io.set_input_stream):

% io.see(File, Result, !IO).
% Attempts to open a file for input, and if successful,
% sets the current input stream to the newly opened stream.
% Result is either 'ok' or 'error(ErrorCode)'.
%
:- pred io.see(string::in, io.res::out, io::di, io::uo) is det.

With that in mind, let’s go ahead and implement a predicate to read files (much like I was expecting to find in the standard library, and what I put into a module of similar utilities I’ve started, titled, in tribute to Haskell, prelude):

:- pred prelude.read_file(string::in,
                          maybe(string)::out,
                          io::di,io::uo) is det.
prelude.read_file(Path,Contents,!IO) :- 
  io.see(Path,Result,!IO),
  ( Result = ok,
    io.read_file_as_string(File,!IO),
    io.seen(!IO),
    (
      File = ok(String),
      Contents = yes(String)
    ;
      File = error(_,_),
      Contents = no
    )
  ;
    Result = error(_),
    Contents = no
  ).

To walk through what this code is doing, the type says that this is a predicate that does I/O (that’s what the last two arguments are for), that it takes in a string (the path) and give out a maybe(string), and that this whole thing is deterministic (ie, it always succeeds, which is accomplished by wrapping the failure into the return type: either yes(value) or no).

The first line tries to open the file at the path and bind it as the current input stream. I then pattern match on the results of that - if it failed, just bind Contents (the return value) to no. Otherwise, we try to read the contents out of the file and then close the file and set the input stream to the default one again (that is what the predicate io.seen does). Similarly we handle (well, really don’t handle, at least not well) reading the file failing. If it succeeds, we set the return type to the contents of the file.

What is interesting about this code is that while it is written in the form of logical statements, it feels very much like the way one does I/O in Haskell - probably a bit of that is my own bias (as a Haskell programmer, I am likely to write everything like I would write Haskell code, kind of how my python code always ends up with lambda’s and maps in it), but it also is probably a function of the fact that doing I/O in a statically type pure language is going to always be pretty similar - lots of dealing with error conditions, and not much else!

Anyhow, this was just a tiny bit of code, but it is a predicate that is immediately useful, especially when trying to use Mercury for random scripting tasks (what I often do with new languages, regardless of their reputed ability for scripting).

I just started learning a functional/logic language called Mercury, which has features that make it feel (at least to my initial impressions) like a mix between Prolog and Haskell. It has all the features that make it a viable Prolog, but it also adds static typing (with full type inference) and purity (all side effects are dealt with by passing around the state of the world). Since I recently was interested in learning Prolog, but had no desire to give up static typing or purity, Mercury seemed like a neat thing to learn.

While it is not very well known, the language has been around for over 15 years, and has a high quality self-hosting compiler.

Getting to play around with logic/declarative programming is interesting (and indeed the main reason why I’m interested in learning it), but what seems even more interesting with Mercury is how they have incorporated typing to the logic programming (which, unless I’m mistaken, is a new thing). As a tiny code example:

:- pred head(list(T), T). 
:- mode head(in,    out) is semidet. 
:- mode head(in(non_empty_list), out) is det.
head(Xs, X) :- Xs = [X | _].

The first line says that this is a predicate (logic statement) that has two parts, the first is a list of some type T (it is polymorphic), the second is an item of type T.

The fourth line should be familiar to a prolog programmer, but briefly, the right side says that Xs is defined as X cons’d to an unnamed element. head can be seen as defining a relationship between Xs and X, where the specifics are that Xs is a list that has as it’s first element X.

Now with regular prolog, only the fourth line would be necessary, and that definition allows some interesting generalization. Because head([1,2,3],Y) will bind Y to 1, while head([1,2,3],1) will be true (or some truthy value), and if head(X,Y) were used in a set of other statements, together they would only yield a result if X (wherever it was bound, or unified, to a value) had as it’s first value Y, whatever Y was.

Since Mercury is statically typed, it adds what it calls modes to predicates, which specify whether a certain argument (that’s probably not the right word!) is going to be given, or whether it is going to be figured out by the predicate. The other thing it has is specifications about whether the predicate is deterministic. There are a couple options, but the two that are relevant to this example are det, which means fully deterministic, for every input there is exactly one output, and semidet, which means for some inputs there is an output, for others there is not (ie, the unification fails). These allow the compiler to do really interesting things, like tell you if you are not covering all of the possible cases if you declare something as det (whereas the same code, as semidet, would not cause any errors).

What is fascinating about this predicate head is that it has two modes defined, one being the obvious head that we know from Haskell etc:

:- mode head(in,    out) is semidet. 

Which states that the first argument is the input (the list) and the second is the output (the element), and it is semidet because for an empty list it will fail. The next is more interesting:

:- mode head(in(non_empty_list), out) is det.

This says for an input that is a non_empty_list (defined in the standard libraries, and included below), the second argument is the output, and this is det, ie fully deterministic. What this basically means is that failure is incorporated into the type system, because something that is semidet can fail, but something that is det cannot (neat!). Now the standard modes are defined (something like):

:- mode in == (ground >> ground).
:- mode out == (free    >> ground).

Ground is a something that is bound, and the >> is showing what is happening before and after the unification (the analog to function application). So something of mode in will be bound before and after, whereas something of mode out will not be bound before (that’s what free means) and it will be bound afterwards. That’s pretty straightforward.

What get’s more interesting is something like non_empty_list, where inst stands for instantiation state, ie one of various states that a variable can be in (with ground and free being the most obvious ones):

:- inst non_empty_list == bound([ground | ground]).

What this means is that a non_empty_list is defined as one that has a ground element cons’d to another ground element. (I don’t know the syntax well enough to explain what bound means in this context, but it seems straightforward). What this should allow you to do is write programs that operate on things like non-empty-lists, and have the compiler check to make sure you are never using an empty list where you shouldn’t. Pretty cool!

Obviously you can write data types in Haskell that also do not allow an empty list, like:

data NonEmptyList a = NonEmptyList a [a]

And could build functions to convert between them and normal lists, but the fact that it is so easy to build this kind of type checking on top of existing types with Mercury is really fascinating.

This is (obviously) just scratching the surface of Mercury (and the reason all of this stuff actually works is probably more due to the theoretical underpinnings of logic programming than anything else), but seeing the definition of head gave me enough of an ‘aha!’ moment that it seemed worth sharing.

If any of this piqued your interest, all of it comes out of the (wonderful) tutorial provided at the Mercury Project Documentation page. If there are any inaccuracies (which there probably are!) send them to daniel@dbpatterson.com.

Here’s a question: is overemphasis of the Maybe type actually harmful, making it easier for Haskell newcomers to write unreliable code?

In some recent Haskell projects, I relied heavily on the Maybe type. It is simple to understand, Maybe is often the first Monad people learn and one of the first places that people start exploring Haskell’s power (realizing you can use do notation with it was a pretty cool moment for me). It is often the first major focal point to Haskell tutorials. And so it’s not surprising that I’ve used it a lot (I bet many people have).

Now I definitely don’t think that Maybe is not useful sometimes, and here’s a good example: looking for an item in a list. It is either there (Just value) or not there (Nothing). What is important about this example is that it is that both are normal, expected results. But what about the case when you are finding a value in a list but it should definitely be there (let’s say you put it in the list, serialized to disk, read it back in, and are inspecting the list), in that case the two possibilities are not equally likely, and passing back a Nothing value might be hiding some underlying problem.

What I noticed about my code is that I had started using the Maybe monad for failure conditions, or in cases where I really only expected the value to be a Just, but it was so easy to use Nothing that I ended up writing code that type checked (and compiled, and ran), but that provided virtually no information about errors that were occurring, or where they occurred. Part of this ease is the way you can use Maybe as a Monad: comp1 >>= comp2 >>= comp3 is so simple and clean, hiding within it that comp1 can genuinely return either a value or not, but comp2 and comp3 should really only not return a value in the case of something being wrong. If you end up with Nothing at the end of this, you really have no idea what actually went wrong, if anything.

Code written this way is difficult to debug once you find a bug, and good at hiding bugs in the first place (because we don’t know if the the result is Nothing because the item in the database or wherever didn’t exist, or because it was formatted incorrectly or because something else happened that shouldn’t have).

What I realized, which is probably obvious to any experienced Haskell programmer, is that Maybe should not ever be used in cases where an error has occurred. There are (at least) two ways of properly handling errors: the first being the Either type, which is like Maybe if Nothing carried a type with it (so you have either Left error-value or Right success-value), or if it is indeed an error that means things are really messed up (and should not keep going), error - a function that causes a runtime error to be raised (that can be caught, but if not, causes the program to exit).

Especially in web programming (where everything I’ve done recently is), calls to error can (and at least with snap, do) cause the request to terminate and a 500 to be sent to the client, which in the case of an error that can not be recovered from, is probably desired! In most other cases, Either is probably a practical solution, as it allows you to fail in the same way as Nothing, but specify where it happened, and maybe some other details. And it can be used in the Monadic style if you import Control.Monad.Error.

So my conclusion with all of this is to only use Maybe when a value can truly be there or not be there, not when it should be there and it’s absence is an error. And, to be careful about using library functions that return Maybe values if in my case they should only not return values in exceptional cases. I’d be curious to know what more experienced Haskell programmers think about Maybe, and whether they’ve come up with different solutions to the problems I’ve run into.

I’ve been doing a bit of development with the Snap Framework, a web framework in Haskell, and I’m doing this on a Mac development host. Since I am deploying to linux (more specifically, Debian), and prefer to have a similar environment on my development machine as on the production server that it will eventually live on, this meant that just installing the Haskell Platform and other libraries on my laptop wasn’t going to be good enough.

The solution I ended up with is based on Virtualbox - each separate project has a separate virtual machine, that has the same operating system as it will be deployed to (debian, in this case), and all the same libraries. This is also helpful in that it keeps everything separate, so that the libraries of one application won’t conflict with any other (I use this process for non-haskell web development too). The virtual machine is set up as a server, so it will only be accessed remotely or via command line (and that only to set up ssh in the first place).

Then, each virtual machine mounts a folder from the host, which contains the source code for the application. With Virtualbox, this is done with “Shared Folders”, but I’m sure any other virtualization solution would provide something similar. Then once the application is running, set up port forwarding so that the host machine (my development laptop) can access the application running inside the virtual machine. And access it via SSH too, of course.

I’d been doing this for a little while, and keeping an ssh session with gnu screen open to the virtual machine in order to rebuild the application and restart it. But, this was sort of a pain, and I wanted to make it even more transparent that I was using a virtual machine at all (so needing to manually SSH in when I start the machine, and switching to that session to rebuild and relaunch wasn’t ideal).

So the next iteration had two parts: first, use God (the process manager) to maintain the running development web app. This is also duplicating what I have running in production, so this is a good thing! The second step is to make a script that takes care of the process of building and restarting the process, so that it can be easily triggered remotely. The script is very simple:

#!/bin/bash
PATH=/home/dbp/ghc-7.0.4/bin:/home/dbp/.cabal/bin:$PATH
cd /path/to/app
cabal-dev install && sudo god restart app-name

Now this can be triggered remotely with:

ssh -p PORTNUM localhost /path/to/script

Where PORTNUM is the port you have forwarded from the virtual machine for SSH (ie, what on the host corresponds to 22 on the virtual machine).

Now to round out my (at least for now) development environment, I am using Sublime Text 2 (just started, so this is all up in the air), and have set up a custom build system that looks line this:

{
"cmd": ["ssh", "-p", "2222", "localhost", "/path/to/build-script"],
}

What this means is that I can edit my application (all the files are on the host, just shared with the virtual machine, so there is nothing involved in doing that), and then with a shortcut trigger the build. This will compile the application and relaunch it, with the output of the compilation in a little panel so I can see what is going on.

The last parts are version control, testing, and deploying. I use darcs, and Quickcheck according to the guide here: How_to_write_a_Haskell_program#Running_the_test_suite_from_darcs, which allows you to run the tests as a precommit hook (which means commit will fail if the tests do not pass) like:

darcs --record --test

I am constantly pushing patches to a repository at Darcsden, a shared hosting site similar to github but for darcs, which serves as a level of redundancy (and if working with other developers, acts as a common repo we both can push to / pull from). There is also a repository on the production server, which can be pushed to as well.

This repository has a Heroku-inspired (though primitive) post-commit hook to build and deploy the application. This is a part I’d like to improve in the future - building binaries locally and just pushing them is one improvement; another is keeping around the old binaries to make it easy to switch back to a previous version. But for now, this is what the _darcs/prefs/defaults file in the production darcs repo looks like:

apply posthook echo "Rebuilding and deploying...\n" && cabal install && echo "Copying resources...\n" && sudo cp -R resources /path/to/production/root && echo "Restarting App-Name.\n" && sudo /var/lib/gems/1.8/bin/god restart app-name
apply run-posthook

This means that after recording changes locally, to deploy, I run:

darcs push servername:/path/to/production/repo

and select which patches I want to push. It will then build and deploy the application.

Aside from making the deployment more robust, I just need to integrate the version control / deployment into my editor to make this process completely streamlined (ie, being able to do everything from my editor, and just switch between it and a browser to test), which since it has a console pane, should be pretty easy.

disclaimer: This whole post is based on the fact that I was not able to get a certain thing working. Part of the reason to write this is a challenge to someone else to figure out how to do it, and document it. There is extremely little information out there. With that said, I tried pretty hard, and came to the conclusion that it was not possible. If you can figure out how, I will retract my claim that Ocsigen native code is not a viable option for web programming.

edit

This only pertains to native code that is done with static linking. I was able to get ocsigen to link native code dynamically when it was the only library, but was not able to get this working with some external libraries, which only worked with static linking. If dynamic linking was working, all the acrobatics described in this post would be irrelevant. Since I was not able to, this was my experience, but consider this an enormous caveat (and it is terrible that I did not mention this originally. I stopped using ocsigen months ago because of this and other reasons, and wanted to finally get around to posting this, but forgot to mention that critical detail).

edit 2

This was written very negatively. I didn’t intend it to be so - when I first started writing this post it was meant as a guide for someone else who wanted to use statically linked native code with ocsigen. Since it took a bit of work to figure out how to structure the code (and a couple false starts) I wanted to write this down so that it could benefit others. However, partly due to the code organization necessary (and for other reasons), I stopped using Ocsigen for anything but small projects (let’s say, above 800 lines of application code), and I think because of that (and due to re-writing some applications that had reached that limit) I ended up writing this much more negatively than I originally intended. I wanted to write that post, ie, why I stopped using Ocsigen, in another post, but some of it leaked in here unintentionally.

why do I want / need this?

Well, I personally think it is silly to use a language that has a very fast native code compiler and not take advantage of that. But this is a valid point — and my conclusion is that indeed if you are going to use Ocsigen for any even medium sized project, you probably should not use native code.

what I looked at for comparison

The best references were the applications by mfp, particularly ocsiblog. It is a small blog application and also has some even smaller test applications in the repository. It was from these that I got the first native application running.

what it meant I had to change about my app

So the most important thing that you have to keep in mind is that you cannot register any service until runtime. Additionally, you canot register any service twice. These two things serve as somewhat of a death-knell to the cause of native code Ocsigen projects (at least as far as I could tell). The only workaround I could find, which is how the code in the example project by mfp works, is to wrap the entire application (or, all of the application that involves services - non-web libraries can be separate) within a single functor that takes a dummy argument. Then at runtime, you evaluate it once.

Now the reason why this is catastrophic is that it means that all your web code has to be in a single file. Trying to do it any other way, unless you have parts of your application that never interact (ie, never link to one another, never post a form to one another), and you will end up doing multiple evaluation, as far as I could tell. Possibly someone else could make this happen, but the various things I tried did not work. Take this as an open challenge. What would need to happen is that one module would orchestrate loading each one only once and passing them among one another.

interesting echos of haskell purity

Separating code that used services (or more generally, that used to use the server params argument, which is in current ocsigen passed around as Lwt thread data) from that code that is “pure” (ie, doesn’t touch ocsigen), so that the latter could be factored out into separate files (the former all being lumped into one module), was interesting, and reminded me of the isolation that haskell’s IO monad enforces. However, the lack of flexibility in dealing with the “IO” code was pretty limiting.

conclusion

Until this becomes more supported, and someone figures out a way to do this easily and without drastic code reorganization, Ocsigen should be thought of as a byte-code only option. There is enough documentation to make it seem like native code is an option, if only you bother to do it, but I think that is extremely misleading, and it would be worthwhile for the Ocsigen website to make this clear.

This is the first in (hopefully) a series of posts coming out of creating the web application HouseTab with the haskell web framework Snap. All the source for the application is available at http://darcsden.com/position/housetab, and these posts will draw heavily from what is there (linking back where relevant).

One feature that this application has is a tutorial that runs the first time you log on, guiding you through setting up your account. We decided that we did not want this to be something that was separate from the application, but rather a set of dialogs / numbers that appeared within the application, so that not only would the tutorial explain how to set up your account, you would be doing it while going through the tutorial.

If you want to see this in action, feel free to create an account on HouseTab - you can always delete it (permanently and easily) from the Settings page.

There were a few decisions that we made. First, the tutorial would run the first time a new user signed in, but anyone could prompt it to start at any point. Second, the tutorial should be able to remember where you are beyond a page reload (so don’t store state in javascript). The third decision was that the tutorial should follow your actions - recognizing when you have completed a step and automatically proceeding to the next step.

A couple tools made this possible: the Heist templating library and the ajax add-on to it that I released at https://github.com/dbp/heist-async. I also use the snap-auth library to provide session support (and user auth, but that is not covered here). From now on, I’m going to assume familiarity with those, so briefly look over them now if you haven’t already seen them.

To maintain the active status of the tutorial, there is a simple boolean stored with every account. The step in the tutorial is stored in a session variable. Originally I was going to store it permanently, but it seemed that it would be more confusing to log in again days, weeks, or months later and end up half-way through the tutorial. The session variable is a simple number, and a library function is provided that changes it, given what it currently is (ie, set to 3 if it currently is 2):

tutorialStep user old new =
  if not (tutorialActive user) then return () else do
    st <- getFromSession "tutorial-step"
    if st == Just old then setInSession "tutorial-step" new 
                      else return ()

This provides half of the functionality - when a user performs an action (for example, the first step of the tutorial is creating a person on the account), provided they are in the tutorial and at the given step, proceed to the next step. This means that they won’t go backwards if they repeat an earlier step, and can’t accidentally skip ahead by performing another action. An example from the code is:

...
do saveHouseTabPerson $ person' { pHTId = htid}
   nu <- recalculateTotals user
   tutorialStep user "1" "2"
   ... $ renderHT "people/add_success"  
...

The other half is where the tutorial is actually displayed. First, we created a splice (this is Heist terminology, if it doesn’t make sense, read the Heist tutorial linked to above) that displays it’s children if the current tutorial step is that specified in the step attribute (from Views/Account.hs):

import qualified  Data.Text.Encoding as TE
import qualified  Text.XmlHtml as X

tutorialSplice :: Splice Application
tutorialSplice = do 
    node <- getParamNode
    s <- lift $ getFromSession "tutorial-step"
    case X.getAttribute "step" node of
       Just step | Just (TE.encodeUtf8 step) == s 
            -> return (X.elementChildren node)
       _ -> return []

Now for the simple cases, we show the tutorial box on the event of a full page load. This looks, in the page template, like:

...
<tutorial step="1">
  <apply template="tutorial/1"></apply>
</tutorial>
<tutorial step="2">
  <apply template="tutorial/2"></apply>
</tutorial>
  <tutorial step="3">
  <apply template="tutorial/3"></apply>
</tutorial>
...

One of these pages looks like (collapsed the base and template into one for conciseness):

<div-async name="tutorial" id="tutorial">
  <div class="content">
  <p>
    Welcome to the HouseTab Tutorial. 
    Follow these steps to finish setting up your account:
  </p>
  <p>
    <span class="num">1.</span> 
    Add at least one user to your account.
  </p>
  </div>
  <a-async class="end" href="/tutorial/deactivate">
    End Tutorial
  </a-async>
</div-async>

If this were a non-ajax application, this would probably be good enough - when the next page load occurred, the correct step of the tutorial would be selected. To do it with ajax, using heist-async, it isn’t actually much more difficult. First, be sure that the tutorial box is wrapped inside a <div-async> (which it is above), so that it can be replaced be a later page fragment. Then, in the template that is sent down to update the page (in the example above, this template is “people/add_success”), simply conditionally include the next step of the tutorial, which will replace the tutorial box, using the exact same code as it is included for the case of a non-ajax page load:

<tutorial step="2">
  <apply template="tutorial/2"></apply>
</tutorial>

One thing that is important is that this box is not included twice - so the second load should only accur in the special page that is only for ajax responses - in this case, "people/add_success".

Now the last part is the numbered prompts - the idea is, when you get to step 2, there is the box at the top of the page that tells you what to do, but there is also a red circle with a 2 in it that appears on the page where the action you should be performing is. Again, this is simply a matter of conditionally including the proper div’s in the places they belong. They will be positioned and styled with CSS. Since the fragments of the pages are the same templates whether they are loaded partially via ajax or via a full page load, there is nothing special that needs to be done to make it work for ajax. An example is the add person form (which is the first step of the tutorial. form cut down for presentation):

<div-async name="add-person" class="addPerson" id="adduser">
  <form-async action="/people/add" method="POST">
    <h2><label for="name">Add a new user:</label></h2>
    <input name="name" type="text" value="$(name-value)" />
    <button type="submit" title="" class="addform_submit" />
  </form-async>

  <tutorial step="1">
    <div id="tutorial-1">
    </div>
  </tutorial>
</div-async>

That’s all for this short first post!

(this is a code heavy post, with assumed familiarity with javascript, haskell, and the heist templating library, as a warning)

The motivation of the code / method outlined in this post is rather simple: I would like to encode dynamic aspects of the websites I write in the markup, not in Javascript code (corollary to this: I don’t like having lots of code running in javascript, and prefer writing things declaratively to imperatively, preferably with the aid of a type system, but I do want the sites that I build to have dynamic behavior. This solution is an attempt to reconcile these two things).

This may seem impossible, but it actually turns out to be rather simple. The key is to identify a small set of dynamic primitives that should be available, with clearly defined markup that should result in predictable (preferably local, to make it easier to reason about / check correctness) effects.

Then, global event listeners will look for certain events that are triggered on the specified markup, and will perform the needed operation. This means that you will code the effect once in Javascript, which will effectively extend the dynamic abilities of your markup, anywhere where the small javascript primitives are available.

Once the necessary javascript has been described, built, and tested, using Heist I will build Splices (which are special html tags that generate html based on their attributes, children, etc) that will produce the markup that the javascript needs to recognize / carry out the behavior, while exposing a very simple interface to the user (this reduces the chance that the user, ie, me and others working with this code, could generate markup that would screw up the handlers). From the user’s perspective, these special tags ARE the dynamic behavior, and provide a reliable way of ensuring that the effects always work.

The reason for creating this was rather simple: For a project, the designer in my team wanted to have input fields that, when selected, popped up a box that presented options to select by clicking on them. This is the task that usually a select field would fulfill, but it did not fit with the interface, and it was something that had come up before (when we did settle on a plain old select field) so I set out to create a way of consistently creating this pattern. One such end-use case is shown here:

      <box-field name="person-name" value="$(person-name-value)">
        <people>
          <box-option value="$(personId)">
            <personName/>
          </box-option>
        </people>
      </box-field>

The “box-field” and “box-option” are splices that are globally available. They are what encode the idea of this Box Field (a field that presents options in a box to pick between). name is what the field’s name will be when the form is submitted, value is the prefilled value (here dynamic). The “people” splice is what holds the data that is populating the options (and it presents the subSplices “personId” and “personName”) - don’t worry about that for now, as it just as easily could have been a static list of box-options, like:

      <box-field name="person-name" value="$(field-name-value)">
          <box-option value="1">
            Jane
          </box-option>
          <box-option value="2">
            John
          </box-option>
          <!-- ...etc -->
      </box-field>

Since this element might be used in different places, and on fragments of pages returned asynchronously, I did not want to have to make sure I was always attaching the proper handlers, every time the element was created. Having to always attach handlers also runs counter to the idea of encoding the behavior in the HTML - I want declarative dynamic behavior, not imperative.

So after looking a little, I figured what I needed was a way to register a global event listener that would listen for a specific event, match a certain element, and then run a corresponding function on it. I have used the lightweight standalone javascript libraries by @dedfat (of twitter), because they are simple and work well for me, but this could easily be adapted to use jQuery, etc. The function I came up with is “declare” - which takes an event, a selector, a boolean as to whether it should stop propagation (I think the answer will always be yes, but wasnt sure), and then the function to call when this occurs, passing in the element.

/*!
  * declarative.js - copyright @dbp 2011
  * BSD3 License
  */
function declare(event, selector, nopropagate, fun) {
  // bean is an cross-browser standalone event handler 
  // at https://github.com/fat/bean
  bean.add(document.documentElement, event, function(e) {
    e = e || window.event;
    var elem = e.target || e.srcElement;
    // qwery is a cross-browser selector engine 
    // at https://github.com/ded/qwery
    if (!elem || qwery(selector).indexOf(elem) === -1) {
      return;
    }
    fun(elem);

    if (nopropagate) {
      e.stopPropagation();
    }
  });
}

Then I came up with the following HTML that I wanted to govern my box-field, the idea being that the hidden input field would contain the person’s ID when it is selected, and that the display field is what will show the name that is selected (styling done inline here, just to keep it self contained):

<div class="box-field ">
  <input type="hidden" name="person-name" value="">
  <div class="display" 
    style="width: 200px; height:20px; border: 1px solid black;">
  </div>

  <div class="box" style="display:none;">

  <div data-box-value="1" class="option ">
    Jane
  </div>

  <div data-box-value="2" class="option ">
    John
  </div>

  </div>
</div>

Then I wrote up the handlers to make this box-field actually work:

bean.add(document, 'DOMContentLoaded', function () {
  declare("click",".box-field .display",true,function (elem) {
    // bonzo is a cross-browser selector engine 
    // at https://github.com/ded/bonzo
    bonzo(bonzo(elem).next()).show(); // show the box
  });
  declare("click",".box-field .box .option",true,function (elem) {
    bonzo(elem.parentNode).hide();
    d = bonzo(elem.parentNode).previous()[0];
    d.innerHTML = elem.innerHTML;
    bonzo(bonzo(d).previous()[0]).attr("value", 
      elem.getAttribute("data-box-value"));
  });
});

Finally, the splices that turn the markup shown at the beginning of this post into the html that the javascript actually operates on:

import qualified Text.XmlHtml as X
import qualified  Data.Text as T
import Text.Templating.Heist


boxField :: Monad m => Splice m
boxField = do node <- getParamNode
              case X.getAttribute "name" node of
                Nothing -> return [] -- without name, useless
                Just name -> do
                  let klass = T.concat 
                     ["box-field ",
                      (fromMaybe "" $ X.getAttribute "class" node)]
                  let value = fromMaybe "" $ 
                     X.getAttribute "value" node
                  let children = 
                    [ X.Element "input" 
                       [("type","hidden"),
                        ("name",name),
                        ("value",value)] []
                    , X.Element "div" 
                       [("class","display"), 
                        ("style", 
     "width: 200px; height:20px; border: 1px solid black;")] []
                    , X.Element "div" 
                       [("class","box"),
                        ("style","display:none;")] 
                       (X.elementChildren node)
                    ]
                  return [X.setAttribute "class" klass $ 
                    X.Element "div" (filter ((/= "name").fst) $ 
                       X.elementAttrs node) children]

boxOption :: Monad m => Splice m
boxOption = do node <- getParamNode
               case X.getAttribute "value" node of
                 Nothing -> return []
                 Just value -> do
                   let klass = T.concat 
                      ["option ",
                      (fromMaybe "" $ X.getAttribute "class" node)]
                   let attributes = 
                     ("class", klass) : (filter 
                        ((flip notElem ["name","class"]).fst) $ 
                        X.elementAttrs node)
                   return [X.setAttribute "data-box-value" value $ 
                     X.Element "div" attributes 
                       (X.elementChildren node)]

There is obviously a bit of styling that still needs to be done to make the box look like it should, but everything from here on out (and any number of uses of this box) should be outside of the world of javascript - which was my intention in the first place!

PS. I just realized a small flaw in the example I presented. The splice “box-field” should also have a “display” attribute that has predefined what the field should show (to match up with the predefined “value”). This is easy to do, and should not detract from the presentation.

I’ve recently started working with Snap, the Haskell web framework, (http://snapframework.com), and one reason (among many) for my reason to switch from Ocsigen, a web framework written in OCaml (which I’ve written posts about before) was the desire to more flexibly handle ajax based websites. While it seems good in some ways, I eventually decided that Ocsigen’s emphasis on declaring services as having certain types (ie, a fragment of a page, a whole page, a redirect, etc) is in some ways at odds with the way the web works.

After starting to work in Haskell again, and with the Snap team authored templating system Heist, I immediately began looking for ways to work with ajax content more flexibly than I had been doing before. Inspired by the work of Facebook on Primer (provided to the world at https://gist.github.com/376039 ), which is their base-line system for dynamic content - basically, event listeners waiting for onclick events on links that have a special attribute that says it should perform an ajax request, and event listeners for onsubmit events on forms that have a special attribute that indicates the forms should be serialized and submitted asynchronously. But even more interesting than that (to me) was the other half of their system (not, I believe, public, and regardless, written in PHP), which is that the server side response decides what client side div’s it should replace.

At first that sounds a little dirty - it basically entails mixing (conceptually) server code and client code. But then it allows a different sort of methodology - that even with client side modifications, it is the server that ultimately has all control - including what to replace on the client. This is a fascinating idea, because clientside code is notoriously limited be being written in javascript (or with javascript libraries), and thinking about having to maintain clientside and serverside code seems to be a much dirtier solution than having the server, in short, control the client.

Taking this idea, and bringing it into the world of Heist, which is (in my opinion) a fantastic templating system (more info at http://snapframework.com/docs/tutorials/heist ), ended up being quite straightforward, as Heist lends itself to the idea of extending the syntax of html, much like the facebook primer system did.

At first I thought that there should be haskell code that would specify things like “replaceDivsWithSplices …” where div’s would be identified and corresponding splices (things that can be inserted into heist templates) would replace them, and then “replaceDivsWithTemplates”, etc, but the whole solution seemed a little off.

And then I realized that the entire idea could be summed up with a single tag: “div-async”. The idea would be, this would be a special div that could foreseeably be replaced by an asychronous response. A template would have many divs that were marked this way, which in a non-async response would do nothing special, but when an async response came back, all div-async’s would replace corresponding tags on the page.

The only things that remained were the two tags to start the async requests, which I named “a-async” and “form-async”, and a little javascript to make the moving parts work together. And so, heist-async was born. (for the impatient, the code exists at https://github.com/dbp/heist-async , and while I am using this code currently and it seems to work, it could change significantly as things are worked out)

The basics of how this works should be obvious, but I can illustrate a basic example. On a page you have an announcements box. You want the user to be able to click a button and have the announcements box reload without reloading the whole page (new announcements may have occurred). So you have a page template that looks like this:

<html><body><h1>Some page</h1>
  <apply template="announcements></apply>
  <a-async href="/recent_announcements">Reload</a-async>
</body></html>

And an announcements template that looks like this:

<div-async name="announcements">
  <announcements>
    <text/>
  </announcements>
</div-async>

Now to glue this together, all you need to do is serve the original page (with the proper splice set so that the tag actually works), and, at the /recent_announcements url, you just serve the announcements template. Since it is the exact same template, it obviously has the same identifier for the div-async (which is just the attribute “name”), and will therefore replace the current anouncements box with the recently loaded one.

Now that is pretty cool - what it means is that you can have one set of templating code, and the only change you need to do is separate any parts you want to be able to load asynchronously into separate templates, and make sure there is a div-async wrapper around it. (NOTE: since I didn’t mention it before, it might be helpful to now - div-async is just a regular div, so you can set all the regular things, like id, class, etc. Also feel free to take existing div’s and just add -async and set a name).

At this point, I was pretty happy with this, and thought it was working pretty well, but of course the real world is much more complicated, and not everything is so simple - sometimes a single asynchronous request should mean a lot of different things on a page should change. In this case, it is possible that the simple template inheritance will not work, but with the addition of a template that is just for the response, that includes all the templates that should be updated, it seems to work pretty well. An example of one of these could be:

<apply template="announcements"></apply>
<apply template="title"></apply>

In this case, there is still no duplication of formatting code, all that exists now is an explicit list of all the parts of the page that should be replaced by a given request.

Other common things; to hide an element, sending back:

<div-async name="something" style="display:none"></div-async>

Should work. You could also put some empty placeholder div’s like that on a page, and later replace them with ones with actual content.

What I noticed about this is that it makes dynamic page changes very explicit in the templates, which I think is a very good thing - and certainly makes it easier to reason about page changes.

Getting to this point, I started using this to implement a bunch of parts of a new site I’m working on, and I was happily impressed by how it all seemed to be working. Using this, it seems like ajax can be thought of as just an aspect of the templating system - describe what should be replaced, and it will be, without ever having to worry about the clientside code (which is 12k of lightweight libraries and 60 significant lines of code of custom javascript. The 60 lines should easily be able to be translated to depend on common javascript libraries like jQuery, I just didn’t want to make that a requirement).

I’m interested in feedback on the library, and ways that it can be improved. It is still very early software (a week ago, it did not exist), but it is something that I’ve found very powerful, and I’m kind of interested in where it can be taken / what people think about it.