On Code etc.

The Haskell / Snap ecosystem is as productive (or more) than Ruby/Rails.

Thu, 26 Apr 2012 21:06:00 -0400

This may be controversial, and all of the usual disclaimers apply - this is based on my own experience using both of the languages/frameworks to do real work on real projects. Your mileage may vary. Because this is something that has the potential to spiral into vague comparisons, I am going to try to compare points directly, based on things that I’ve experienced. I am not going to say “I like Haskell better” or anything like that, because the point of this is not so much to convince people about the various merits of the languages involved, just to point out that I’ve found that they both are as productive (or that Snap feels more so). For Haskell programmers, this could be an indication to try out the web tools that you have available, especially if you are usually a Rails developer.

As a note - some of this could also apply to other haskell web frameworks (in particular, most of this pertains to happstack, and some pertains to yesod), but since Snap is what I use, I want to keep it based on my own personal experience.

\1. The number one productivity improvement is a smart strong type system. This is less of an issue for small projects, but as soon as you have at least a few thousand lines of code, adding new features or refactoring inevitably involves changes to multiple parts of the codebase. Having a compiler that will tell you all the places that you need to change things is an amazing productivity booster. This can be approximated in some ways with good test coverage, but it is really a different beast - tests often need to be changed as well, and if you aren’t very careful about this it is easy to change them in ways that don’t catch new bugs. Additionally, it is hard (or very tedious, if you do it wrong) to achieve high enough coverage to actually catch all of the bugs introduced in refactoring. This as compared to a compiler that is completely automated and will always be aware of all of the code you have and the ways that it interacts (at least to the extent that you actually use the type system - but if you are a good haskell programmer, you will).

This alone wouldn’t be enough to suggest using Haskell/Snap over Ruby/Rails, as a type system isn’t worth much without supporting libraries, but as I switch between the ecosystems, this is the place where I notice the most drastic improvements in productivity, so I put it first.

\2. Form libraries. There are many different libraries for dealing with forms in Rails, and there is the built in one as well. The general idea is that you define some validations on your models, and then use the DSLs from the form libraries to define forms, and can do validations, etc. In Haskell (in my opinion), the best form library is Digestive-Functors (thanks Jasper!), and the productivity difference is staggering in more complex use-cases. In the sort of vanilla examples that rails has, the validation system works quite well, and dynamic introspection allows you to write really short forms. This begins to break down when you start getting forms that don’t correspond in a simple way to models. I have forms that are sometimes a mix of two models, or forms that are a partial view into a data structure, or any number of other variations.

With Digestive-Functors, I can define the forms that I need, and re-use components between multiple forms (forms are composable), and these validations are on the form, not on the underlying model. It is obviously useful to database level data integrity checks, but I find that having them being the main / only way of doing validations is really limiting - because sometimes there are special cases when you want the validation done one way and other times another.

More generally, it is possible that the business logic of a specific form may have requirements that do not always have to hold for the datastore, and thus should not reside in the integrity checks. Having written a lot of forms (who hasn’t?), I find that getting the first form out is much faster with Rails, but inevitably when I need to change something it starts become difficult fast. Every time I am doing it I keep picturing an exponential curve - sure it starts out really small, but it gets really big really fast! It isn’t that I run into things that are not possible with Rails, but they end up being more difficult, more error prone, and generally reduce my productivity. With Digestive-Functors, I spend a little more time building the forms in the beginning, but I’ve never had requirements for a form that weren’t easily implemented (almost without thinking).

\3. Routing is the next big one. This may be more of an opinion that the previous ones, but I have always thought that great care should be involved in designing the url structure of a site. In this sense, I guess I disagree with the idea of universally using REST - I think it is very useful when writing APIs, but when designing applications for people, I believe the urls should be meaningful to the people, not to machines. Usually, right after modeling the data of an application, I make a site-map - this is a high level view of what the site should look like. Instead, with Rails, I spend time thinking of how I can adapt what I want to the REST paradigm, and usually end up with something that is an incomplete/counterintuitive representation.

More broadly, I think the idea of hierarchical routing is brilliant - the idea that you match routes by pieces. What this allows you to do is easily abstract out work that should be done for many different related requests. In Rails, this is approximated by :before_filters (ie, it a controller for a specific model, you might fetch the item from the id for many different handlers), but it is a poor substitute. For example I often have an “/admin” hierarchy, and to limit this, all I have to do is have one place (the adminRouter or something) that does the required work to ensure only administrators can access, and it can also fetch any data that is needed, and then it can pass back into the route parsing mode. Or if I want to do the rails-style pre-fetching, then I design the routes as “/item/id/action” and have a handler that matches “/item/id”, fetches the item, and then matches against the various actions. If I have nested pieces of data, this is just as easy. I could have “item/id/something/add” which adds a new “something” to the item with id “id”, This would all be in the same hierarchy, so the code to fetch the item would still only exist once.

Not only is this very natural to program, it keeps the flow easy to follow when you are looking back at it, and allows backtracking in a great way: if, in a handler, you reach something that indicates that this cannot be matched, like if the path was “/item/id” but the id did not correspond to an actual item, you can simply “pass” and the route parser continues looking for things that will handle the request. If it finds nothing, it gives a 404.

An example of how you could exploit things in a really clean way - if you are building a wiki-like site, then you first have a route that matches “/page/name” and looks up the page with name “name”. If it doesn’t find it, it passes, and the next handler can be the “new page” handler, that prompts the user to create the page. As with everything else, I’m not saying this cannot be done with Rails, simply that it is much more natural and easy to understand with Snap (and Happstack, where this routing system originated, at least in the Haskell world).

\4. Quality of external libraries. Point 2 was a special case of this, since dealing with forms comes up so much, but I think the general quality of libraries in Haskell is superb. One example that I came up against was wanting to parse some semi-free-form CSV data into dates and times. Haskell has the very mature parsing library Parsec (which has ports into many languages, including Ruby) that makes it really easy to write parsers. I ported an ad-hoc parser to it, and found that not only was I able to write the code in a fraction of the time, but it was a lot more robust and easy to understand.

For testing of algorithmic code, the QuickCheck library is pretty amazing - in it, you tell it how to construct domain data, and then certain invariants that should hold over function applications, and it will fuzz-test with random/pathological data. The first time you write some of these tests (and catch bugs!) you will wonder why you haven’t been testing like that before! I don’t really want to go into it here, but the other point is that many of these libraries are very very fast - there has been, over the last couple years, a massive push to have very performant libraries, with a lot of success. The Haskell web frameworks webservers regularly trounce most other webservers, and there are very high performant json, text processing, and parsing libraries (attoparsec is a version of parsec that is very fast).

\5. Templating. In this, I want to directly compare the experience of using Heist (a templating system made by the Snap team) and Erb/Haml (I mostly use the latter, but in some things, like with javascript, I have to use the former). The first big difference is the idea of layouts/templates/partials in rails. I never really understood why there was this distinction when I first used it, and when comparing it to Heist (which has no distinction - any template can be applied to another, to achieve a layout like functionality, and any template can be included within another, to achieve a partial like functionality) it feels very limited.

The other major difference is that the two templating languages in Ruby allow dynamic elements by embedding raw ruby code, whereas the former allows dynamic stuff by allowing you to define new xml tags (called splices) that you can then use in the templates. I have found this to be an extremely powerful idea, as it allows you to not only do all the regular stuff (insert values, iterate over lists of values and spit out html), but can even allow you to build custom vocabularies of elements that you want to use that are designed to go with javascript (so for example, I built an asynchronous framework on top of this, where I had a “<form-async>” tag and “<div-async>”s that would be replaced asynchronously by the responses from the form posts).

It also adapts to being used with (trusted) user generated input - I’ve used it in multiple CMS systems so that, for example, all links to external sites are set to open in new tabs/windows (by overriding the “<a>” tag and adding the appropriate “target”) or allowing the users to gain certain dynamic stuff for their pages. Compared to this, the situation with Haml always seems hopelessly tied up with ruby spaghetti code - not that it always is (you can always be careful), but the split with Heist both feels like a cleaner separation AND more powerful, which is not something you get often, and I think is a sign that the metaphor that Heist created (which is based on a couple really simple primitives) is really something special.

\6. This is sort of an extension of the first point, and I’m putting it towards the end because it is the most subjective of this already quite subjective comparison - I think that web applications built with Haskell/Snap are much easier to edit / add to than corresponding applications in Ruby/Rails. One of the biggest reasons for this is that there is much more boilerplate/code spread in ruby - some of it is auto-generated, other bits is manually generated, but there ends up being code scattered around. It is pretty easy to add new code, but when you want to edit / refactor existing code, it starts to get hard to figure out where everything is. A bit of this relies on conventions to a degree (which you learn), but there is simply less code in Snap, and usually everything pertaining to a specific function is in one place. This has a lot to do with the functional paradigm - there is no hidden state, so generally all the transformations that occur are very transparent, whereas with Rails it is possible for stuff from the ApplicationController being applied, or just various filters coming into play, or stuff from the model, etc. There is no obvious “starting point” if you want to see how a request travels through your application (candidates include the routes file, the controllers, etc), in the same way where with a Snap application, the code to start the web server is in one of the files you write! You can trace exactly what it is doing from there!

In addition, there is also very little “convention” with Snap. It enforces nothing, which has the consequence (in addition to allowing you to make a mess!) of having the whole application conforming to exactly how you think it should be organized. I’ve found that this actually makes it much easier to add new things or modify existing functionality (fix bugs!), because the entire structure of the application, from how the requests are routed to how responses are generated, is based on code I wrote. This means that making a change anywhere in this process is usually very easy - it feels in some ways like the difference in making a change to an application you wrote from scratch and one that you picked up from someone else. There is also a potential downside to this - the first couple applications I built had drastically different organizational systems

(Side note for anyone reading this who is curious: I’ve converged to the following method: all types for the application lives in a Types module or hierarchy, all code that pertains to the datastore lives in a State hierarchy or module in a small application, code for splices lives in a Splices hierarchy, forms live is a Forms hierarchy, and the web handlers live in a Handlers hierarchy. I also usually have a Utils module that collects some various things that are used in all sort of different places. Everything depends on Types and Utils. Splices, Forms, and State are all independent of one another, and Handlers depends on everything. And then of course there is an Application module and Main, according to the generated code from Snap).

This is a major difference in how Snap even differs from some other Haskell web frameworks, that it seems more like a library with which to build a web application instead of a true framework, but in my experience this is actually a really powerful thing, and makes the whole process a lot more enjoyable, because I never feel like I’m trying to conform to how someone else thinks I should organize things.

\7. I’m bundling the performance, security, etc all at once. Rails is a very stable framework, so lots of work has gone into this. But I think the recent vulnerabilities exposed on a lot of major sites (like GitHub) based on the common paradigm of mass-assignment sort of point out the negative side. Snap is much newer, but it was built with security in mind from the beginning, as far as I can tell, and most libraries that I have used have also mentioned ways that it comes up - the entire development community seems a lot more aware / concerned with it.

I think part of this probably has to do with the host languages - ruby is a very dynamic language that has a history of experimentation (so generally, flexibility is preferred of correctness), whereas Haskell is a language where lots of static guarantees are valued, and security is usually lumped in with correctness. For performance, there is no question that Haskell will win hands down on any performance comparison (and on multithreading). Granted, a lot of web code is disk/database bound so this isn’t a huge deal, but it is nice to know that you aren’t needlessly wasting cycles (and can afford to run on smaller servers).

\8. Now, as a counterpoint, I want to articulate what Rails really has over Snap. Number one, and this is huge, is the size of the community. There are a massive number of developers who know how to use Rails (how many are good at it is another question), and this also means that if you are trying to do something it is much more likely that a prebuilt solution exists. It also means that it will be easier to hire people to work on it, and easier to sell it as a platform to clients/bosses.

The Haskell community is surprisingly productive given its size (and some of the tools it has produced are amazing - examples mentioned in this comparison are Parsec, QuickCheck, Digestive-Functors, etc), but there is some sense where they will always be at a disadvantage. This means that if you are doing any sort of common task with Rails, there will probably be a Gem that does it. The unfortunate part is that sometimes the Gem will be unmaintained, partially broken, incompatible, as the quality varies widely. This is a place where a lot of subjectivity comes in - I have found that most of what I need exists in the haskell ecosystem, and if stuff doesn’t it isn’t hard to write libraries, but this could be a big dealbreaker for some people.

Cheers, and happy web programming.

Using Heist (from the Snap Framework) for CMS functionality.

Tue, 17 Apr 2012 23:06:00 -0400

This is a short post, and it may be totally obvious to people who are familiar with the Heist templating system, but it wasn’t to me at first, so here it is.

As a brief introduction: Heist is a templating system written by the people behind Snap, a Haskell web framework. The basic idea is to write xml, where most of it is regular html, but there are some tags that are special - these are tags that are bound to (essentially) arbitrary Haskell code, that can operate on their children. There are a couple more features, but this the one that really sets it apart: it essentially allows you to extend html with whatever domain logic you want. So for example, if you have an administrator role, you can create a tag <isAdmin> and wrap it around anything you only want displayed to an administrator. You can also create tags that insert text, like a <currentDate/>, and you can have ones that do arbitrarily complex transformations on their children.

But what I want to write about in this post is a simple application that has turned out to be really useful: using it to process the text for a content management system. Now I have only used this when all the people writing are trusted. It would probably be possible to preprocess all the text and only allow a whitelist of tags (and a whitelist of attributes) through, but this is not what I’ve done, so the examples I give may not work for that case (if you do in wrong, you could end up with code injection, DOS attackes etc). Consider yourself warned.

The basic idea is simple: Heist can run just as well on the representation of xml that the XmlHtml library uses (list of nodes) as on files, so we can run our content through Heist with a specific set of splices (what tags are called), which basically define a vocabulary of dynamic things that can be done in any dynamic page. Here’s an example:

Just today, I had a simple request: all the links in the content of the site should open in new tabs if they are external (not within the site). I’m using Markdown as the first pass, which doesn’t have any way to do this (at least in the implementations I know of), so the two options are: make authors write html links and include target=”_blank” (seems unnecessary, and potentially troublesome for non-technical users), add the target attribute some other way. I thought of javascript first, but I prefer sites to work without it, so I started thinking about how to do it serverside. With Heist, the solution was actually obvious - override the “a” tag and have it inspect the href, and if it is external, add the target attribute. To be friendly, we also leave the target alone if it already exists, to allow target=”” to cause nothing to happen.

The code (not the prettiest, but written in just a couple of minutes) is the following:

extLinks = do
 n <- getParamNode
 (href,targets,attrs) <- getHrefTargetAttrs
 stopRecursion
 let n' = case targets of
            (x:xs) -> n -- in this case, they have explicitly specified a target
             [] -> case T.isPrefixOf "http://" href of
               True -> n { X.elementAttrs = ("href", href):("target","_blank"):attrs }
               False -> n
 return [n']
getHrefTargetAttrs = do
 n <- getParamNode
 return $ case n of
   X.Element _ as _ ->
      let href = fromMaybe "" $ lookup "href" as in
      let targets = maybeToList $ lookup "target" as in
      (href, targets, filter ((\e -> e /= "href" && e /= "target") . fst) as)

Another example that I ran into: again, for ease of editing the html/markdown that the users input, all images are accessed relative to the name of the page. This means that, in the source of the page, you only have to write “<img src='something.png'/>” instead of some sort of absolute path. This works fine when the page is being displayed at its proper path, but when summaries are displayed elsewhere, this relative path becomes useless. So again, creating an “img” splice that simply adds the absolute prefix (which can be determined by what page it is in) solves this problem.

I have also found uses where I wasn’t overriding existing tags, but adding new ones. One example was allowing editors to post a list of recent pages by author. So I made a new tag “authorRecentPages” that had an an attribute the authors id, and then the body of the tag is html that shows how to format the individual page.

What I’ve realized is this is a really powerful idea - it allows me to write arbitrary Haskell code and package it up so that anyone who can deal with a little bit of html can use that code on the pages they are working on. There are certainly other ways to do this, but usually it takes quite a bit of work - by deciding that every post is going to have one pass through with Heist, adding new vocabulary is as simple as writing a few lines of code and adding a name for the tag.

Getting started with Snap-Auth

Thu, 29 Mar 2012 17:19:51 -0400

This is a short guide to getting started with the user authentication that comes with the Snap Framework, otherwise known as the auth snaplet. Once the documentation matures (right now there doesn’t seem to be any - just the code), this will probably become irrelevant, but until then, it should help an aspiring snap developer get started quickly.

If you haven’t already, at least check out some of the basic snap documentation, for example, the Quickstart.

There are actually two snaplets you need - sessions and auth. The first provides support for storing data related to sessions, and the second does the actual authentication. They both provide the ability to have multiple backends, each coming with one that is supported.

To add these two snaplets, we first add their respective states to the main state for the application, which in the generated project (created by snap init), is in Application.hs, and looks something like this:

data App = App
    { _heist :: Snaplet (Heist App) }

(or possibly with more records, or not including Heist, etc). We want to add the states for sessions and auth, so our new data type will look like this:

data App = App
    { _heist :: Snaplet (Heist App)
    , _sess :: Snaplet SessionManager
    , _auth :: Snaplet (AuthManager App)
    }

And be sure to add the following includes to the file:

import Snap.Snaplet.Session
import Snap.Snaplet.Auth

Now we need to set up the initializers. In the generated project, this is done in Site.hs. At the bottom of the file is a function called “app”, that starts out:

app :: SnapletInit App App
app = makeSnaplet “app” “Some name” Nothing $ do
   ...

(Or something similar). If you have heist, it probably has a line like this:

h <- nestSnaplet “heist” heist $ heistInit “templates”

To add the initializers for sessions and auth, we need to first import the proper backends:

import Snap.Snaplet.Session.Backends.CookieSession
import Snap.Snaplet.Auth
import Snap.Snaplet.Auth.Backends.JsonFile

Now we add an initializer for each one:

s <- nestSnaplet “sess” sess $ initCookieSessionManager “site_key.txt” “sess” (Just 3600) -- 1 hour login timeout
a <- nestSnaplet “auth” auth $ initJsonFileAuthManager defAuthSettings sess “users.json”

And on the last line, use these to create the App data type:

return $ App h s a

Now you should be able to use any of the functions exported by the Auth snaplet in you handlers, have fun!

iOS is anti-UNIX and anti-programmer.

Thu, 15 Sep 2011 14:03:48 -0400

When I was first learning about UNIX, and learning to use Linux, the most immediately powerful tool that I found was the shell’s pipe operator, ‘|’. Using the commandline (because at that point, linux GUI’s were not so well developed, and the few distros that tried to allow strictly graphical operation usually failed miserably) was at times difficult, and at times rewarding, but it was the pipe that opened up a whole world for me.

I can remember looking through an online student directory in highschool that had names, email addresses, etc. For student government elections it had become popular (if incredibly time consuming) to copy and paste the hundreds of email addresses and send a message to the every student. For me, with my newfound skills, it amounted to something like:

cat directory.txt | grep @ | awk '{print $3}' | perl -pe 's/\n/,/'

It seemed like magic at the time, and in some ways, it still does. What the shell (and UNIX in general) offered was composability - it gave you simple (but powerful) tools, and a standard way of linking them together - text streams. By combining those together, it offered immeasurable power, much more than any single tool. The mathematics of combinations guarantees this.

The more I use graphical interfaces (or anything that does not operate on text streams - commandline curses programs included), the more I am struck by how profound the loss of composability is - each program has to try to implement all the standard things (searching, sorting, transforming) that you might want to do with the information it has, and in that repetition lies inconsistencies and usually plain lack of power. The better ones share common libraries, and gain common functionality, but this only amounts to their least common denominator - two separate programs can not (easily) expose their higher functionality to each other (at least not it compiled languages) in the way that commandline stream processing programs can.

What I realized the other day, is that iOS is the extreme example of that lack of flexibility, taken almost to the point of caricature - the only interaction that is possible is through single applications that for the most part can have no connection to other applications. People rejoiced when copy and paste was added, but that celebration hides a sad loss of the true power that computers have. The existence of files - the only real way that composability is achieved in GUI systems (ie, do one thing, save the file, open with another program, etc) - has been essentially eliminated, and applications must therefore do everything that a user might want to do with whatever data they have or will get from the user.

I’d noticed before how frustrating it was for me to use iOS, but I wasn’t sure until recently exactly why that was, until I realized that it had effectively taken away the one thing that is so fundamental about computers, and why I am a programmer - the ability to compose. Every day I live and breath abstraction, and building things out of different levels of it, and the idea of not being able to combine various parts to make new things is so antithetical to that type of thinking that I almost can’t imagine that iOS was created by programmers. I remember looking at the technical specifications of the most recent iPhone and thinking - that is a full computer, and it’s small enough to fit in a pocket - that is a profound change in the way the world works. But it’s not a computer, it’s just a glorified palm pilot with a few bells and whistles.

Baby steps with Mercury - doing file I/O.

Sun, 11 Sep 2011 18:49:00 -0400

The language that I’ve been learning recently is a pure (ie, side-effect free) logic/functional language named Mercury. There is a wonderful tutorial (PDF) available, which explains the basics, but beyond that, the primary documentation is the language reference (which is well written, but reasonably dense) and Mercury’s standard library reference (which is autogenerated and includes types and source comments, nothing else).

Doing I/O in a pure language is a bit of a conundrum - Haskell solved this by forcing all I/O into a special monad that keeps track of sequencing (and has a mythical state of the world that it changes each time it does something, so as not to violate referential transparency). Mercury has a simpler (though equivalent) approach - every predicate that does IO must take an world state and must give back a new world state. Old world states can not be re-used (Mercury’s mode system keep track of that), and so the state of the world is manually threaded throughout the program. A simple example would be:

main(IO_0,IO_final) :- io.write_string("Hello World!",IO_0,IO_1), 
                       io.nl(IO_1,IO_final).

Where the first function consumes the IO_0 state and produces IO_1 (while printing “Hello World!”) and the second function consumes IO_1 and produces IO_final (while printing a newline character).

Of course, manually threading those could become pretty tedious, so they have a shorthand, where the same code above could be written as:

main(!IO) :- io.write_string("Hello World!",!IO), 
             io.nl(!IO).

This is just syntax sugar, and can work with any parameters that are dealt with in the same way (and naming it IO for io state is just convention). It definitely makes dealing with I/O more pleasant.

The task that I set was to figure out how to read in a file. This is not covered in the tutorial, and I thought it would be a simple matter of looking through the library reference for the io library. One of the first predicates looks promising:

:- pred io.read_file(io.maybe_partial_res(list(char))::out, 
                     io::di, 
                     io::uo) is det.

But on second thought, something seems to be missing. The second and third parameters are the world states (the type is io, the mode di stands for destructive-input, meaning the variable cannot be used again, uo means unique output, which means that no other variable in the program can have that value), and the first one is going to be the contents of the file itself. But where is the file name?

The comment provides the necessary pointer:

% Reads all the characters from the current input stream until
% eof or error.

Hmm. So all of these functions operate on whatever the current input stream is. How do we set that? io.set_input_stream looks pretty good:

% io.set_input_stream(NewStream, OldStream, !IO):
% Changes the current input stream to the stream specified.
% Returns the previous stream.
%
:- pred io.set_input_stream(io.input_stream::in, 
                            io.input_stream::out,
                            io::di, io::uo) is det.

But even better is io.see, which will try to open a file and if successful, will set it to the current stream (the alternative is to use io.open_input and then io.set_input_stream):

% io.see(File, Result, !IO).
% Attempts to open a file for input, and if successful,
% sets the current input stream to the newly opened stream.
% Result is either 'ok' or 'error(ErrorCode)'.
%
:- pred io.see(string::in, io.res::out, io::di, io::uo) is det.

With that in mind, let’s go ahead and implement a predicate to read files (much like I was expecting to find in the standard library, and what I put into a module of similar utilities I’ve started, titled, in tribute to Haskell, prelude):

:- pred prelude.read_file(string::in,
                          maybe(string)::out,
                          io::di,io::uo) is det.
prelude.read_file(Path,Contents,!IO) :- 
  io.see(Path,Result,!IO),
  ( Result = ok,
    io.read_file_as_string(File,!IO),
    io.seen(!IO),
    (
      File = ok(String),
      Contents = yes(String)
    ;
      File = error(_,_),
      Contents = no
    )
  ;
    Result = error(_),
    Contents = no
  ).

To walk through what this code is doing, the type says that this is a predicate that does I/O (that’s what the last two arguments are for), that it takes in a string (the path) and give out a maybe(string), and that this whole thing is deterministic (ie, it always succeeds, which is accomplished by wrapping the failure into the return type: either yes(value) or no).

The first line tries to open the file at the path and bind it as the current input stream. I then pattern match on the results of that - if it failed, just bind Contents (the return value) to no. Otherwise, we try to read the contents out of the file and then close the file and set the input stream to the default one again (that is what the predicate io.seen does). Similarly we handle (well, really don’t handle, at least not well) reading the file failing. If it succeeds, we set the return type to the contents of the file.

What is interesting about this code is that while it is written in the form of logical statements, it feels very much like the way one does I/O in Haskell - probably a bit of that is my own bias (as a Haskell programmer, I am likely to write everything like I would write Haskell code, kind of how my python code always ends up with lambda’s and maps in it), but it also is probably a function of the fact that doing I/O in a statically type pure language is going to always be pretty similar - lots of dealing with error conditions, and not much else!

Anyhow, this was just a tiny bit of code, but it is a predicate that is immediately useful, especially when trying to use Mercury for random scripting tasks (what I often do with new languages, regardless of their reputed ability for scripting).

A functional/logic programming tidbit in Mercury

Sat, 03 Sep 2011 18:23:00 -0400

I just started learning a functional/logic language called Mercury, which has features that make it feel (at least to my initial impressions) like a mix between Prolog and Haskell. It has all the features that make it a viable Prolog, but it also adds static typing (with full type inference) and purity (all side effects are dealt with by passing around the state of the world). Since I recently was interested in learning Prolog, but had no desire to give up static typing or purity, Mercury seemed like a neat thing to learn.

While it is not very well known, the language has been around for over 15 years, and has a high quality self-hosting compiler.

Getting to play around with logic/declarative programming is interesting (and indeed the main reason why I’m interested in learning it), but what seems even more interesting with Mercury is how they have incorporated typing to the logic programming (which, unless I’m mistaken, is a new thing). As a tiny code example:

:- pred head(list(T), T). 
:- mode head(in,    out) is semidet. 
:- mode head(in(non_empty_list), out) is det.
head(Xs, X) :- Xs = [X | _].

The first line says that this is a predicate (logic statement) that has two parts, the first is a list of some type T (it is polymorphic), the second is an item of type T.

The fourth line should be familiar to a prolog programmer, but briefly, the right side says that Xs is defined as X cons’d to an unnamed element. head can be seen as defining a relationship between Xs and X, where the specifics are that Xs is a list that has as it’s first element X.

Now with regular prolog, only the fourth line would be necessary, and that definition allows some interesting generalization. Because head([1,2,3],Y) will bind Y to 1, while head([1,2,3],1) will be true (or some truthy value), and if head(X,Y) were used in a set of other statements, together they would only yield a result if X (wherever it was bound, or unified, to a value) had as it’s first value Y, whatever Y was.

Since Mercury is statically typed, it adds what it calls modes to predicates, which specify whether a certain argument (that’s probably not the right word!) is going to be given, or whether it is going to be figured out by the predicate. The other thing it has is specifications about whether the predicate is deterministic. There are a couple options, but the two that are relevant to this example are det, which means fully deterministic, for every input there is exactly one output, and semidet, which means for some inputs there is an output, for others there is not (ie, the unification fails). These allow the compiler to do really interesting things, like tell you if you are not covering all of the possible cases if you declare something as det (whereas the same code, as semidet, would not cause any errors).

What is fascinating about this predicate head is that it has two modes defined, one being the obvious head that we know from Haskell etc:

:- mode head(in,    out) is semidet.

Which states that the first argument is the input (the list) and the second is the output (the element), and it is semidet because for an empty list it will fail. The next is more interesting:

:- mode head(in(non_empty_list), out) is det.

This says for an input that is a non_empty_list (defined in the standard libraries, and included below), the second argument is the output, and this is det, ie fully deterministic. What this basically means is that failure is incorporated into the type system, because something that is semidet can fail, but something that is det cannot (neat!). Now the standard modes are defined (something like):

:- mode in == (ground >> ground).
:- mode out == (free    >> ground).

Ground is a something that is bound, and the >> is showing what is happening before and after the unification (the analog to function application). So something of mode in will be bound before and after, whereas something of mode out will not be bound before (that’s what free means) and it will be bound afterwards. That’s pretty straightforward.

What get’s more interesting is something like non_empty_list, where inst stands for instantiation state, ie one of various states that a variable can be in (with ground and free being the most obvious ones):

:- inst non_empty_list == bound([ground | ground]).

What this means is that a non_empty_list is defined as one that has a ground element cons’d to another ground element. (I don’t know the syntax well enough to explain what bound means in this context, but it seems straightforward). What this should allow you to do is write programs that operate on things like non-empty-lists, and have the compiler check to make sure you are never using an empty list where you shouldn’t. Pretty cool!

Obviously you can write data types in Haskell that also do not allow an empty list, like:

data NonEmptyList a = NonEmptyList a [a]

And could build functions to convert between them and normal lists, but the fact that it is so easy to build this kind of type checking on top of existing types with Mercury is really fascinating.

This is (obviously) just scratching the surface of Mercury (and the reason all of this stuff actually works is probably more due to the theoretical underpinnings of logic programming than anything else), but seeing the definition of head gave me enough of an ‘aha!’ moment that it seemed worth sharing.

If any of this piqued your interest, all of it comes out of the (wonderful) tutorial provided at the Mercury Project Documentation page. If there are any inaccuracies (which there probably are!) send them to daniel@dbpatterson.com.

data Maybe -- harmful?

Sun, 28 Aug 2011 22:07:25 -0400

Here’s a question: is overemphasis of the Maybe type actually harmful, making it easier for Haskell newcomers to write unreliable code?

In some recent Haskell projects, I relied heavily on the Maybe type. It is simple to understand, Maybe is often the first Monad people learn and one of the first places that people start exploring Haskell’s power (realizing you can use do notation with it was a pretty cool moment for me). It is often the first major focal point to Haskell tutorials. And so it’s not surprising that I’ve used it a lot (I bet many people have).

Now I definitely don’t think that Maybe is not useful sometimes, and here’s a good example: looking for an item in a list. It is either there (Just value) or not there (Nothing). What is important about this example is that it is that both are normal, expected results. But what about the case when you are finding a value in a list but it should definitely be there (let’s say you put it in the list, serialized to disk, read it back in, and are inspecting the list), in that case the two possibilities are not equally likely, and passing back a Nothing value might be hiding some underlying problem.

What I noticed about my code is that I had started using the Maybe monad for failure conditions, or in cases where I really only expected the value to be a Just, but it was so easy to use Nothing that I ended up writing code that type checked (and compiled, and ran), but that provided virtually no information about errors that were occurring, or where they occurred. Part of this ease is the way you can use Maybe as a Monad: comp1 >>= comp2 >>= comp3 is so simple and clean, hiding within it that comp1 can genuinely return either a value or not, but comp2 and comp3 should really only not return a value in the case of something being wrong. If you end up with Nothing at the end of this, you really have no idea what actually went wrong, if anything.

Code written this way is difficult to debug once you find a bug, and good at hiding bugs in the first place (because we don’t know if the the result is Nothing because the item in the database or wherever didn’t exist, or because it was formatted incorrectly or because something else happened that shouldn’t have).

What I realized, which is probably obvious to any experienced Haskell programmer, is that Maybe should not ever be used in cases where an error has occurred. There are (at least) two ways of properly handling errors: the first being the Either type, which is like Maybe if Nothing carried a type with it (so you have either Left error-value or Right success-value), or if it is indeed an error that means things are really messed up (and should not keep going), error - a function that causes a runtime error to be raised (that can be caught, but if not, causes the program to exit).

Especially in web programming (where everything I’ve done recently is), calls to error can (and at least with snap, do) cause the request to terminate and a 500 to be sent to the client, which in the case of an error that can not be recovered from, is probably desired! In most other cases, Either is probably a practical solution, as it allows you to fail in the same way as Nothing, but specify where it happened, and maybe some other details. And it can be used in the Monadic style if you import Control.Monad.Error.

So my conclusion with all of this is to only use Maybe when a value can truly be there or not be there, not when it should be there and it’s absence is an error. And, to be careful about using library functions that return Maybe values if in my case they should only not return values in exceptional cases. I’d be curious to know what more experienced Haskell programmers think about Maybe, and whether they’ve come up with different solutions to the problems I’ve run into.

Haskell web development environment (on a mac)

Sat, 06 Aug 2011 10:12:54 -0400

I’ve been doing a bit of development with the Snap Framework, a web framework in Haskell, and I’m doing this on a Mac development host. Since I am deploying to linux (more specifically, Debian), and prefer to have a similar environment on my development machine as on the production server that it will eventually live on, this meant that just installing the Haskell Platform and other libraries on my laptop wasn’t going to be good enough.

The solution I ended up with is based on Virtualbox - each separate project has a separate virtual machine, that has the same operating system as it will be deployed to (debian, in this case), and all the same libraries. This is also helpful in that it keeps everything separate, so that the libraries of one application won’t conflict with any other (I use this process for non-haskell web development too). The virtual machine is set up as a server, so it will only be accessed remotely or via command line (and that only to set up ssh in the first place).

Then, each virtual machine mounts a folder from the host, which contains the source code for the application. With Virtualbox, this is done with “Shared Folders”, but I’m sure any other virtualization solution would provide something similar. Then once the application is running, set up port forwarding so that the host machine (my development laptop) can access the application running inside the virtual machine. And access it via SSH too, of course.

I’d been doing this for a little while, and keeping an ssh session with gnu screen open to the virtual machine in order to rebuild the application and restart it. But, this was sort of a pain, and I wanted to make it even more transparent that I was using a virtual machine at all (so needing to manually SSH in when I start the machine, and switching to that session to rebuild and relaunch wasn’t ideal).

So the next iteration had two parts: first, use God (the process manager) to maintain the running development web app. This is also duplicating what I have running in production, so this is a good thing! The second step is to make a script that takes care of the process of building and restarting the process, so that it can be easily triggered remotely. The script is very simple:

#!/bin/bash
PATH=/home/dbp/ghc-7.0.4/bin:/home/dbp/.cabal/bin:$PATH
cd /path/to/app
cabal-dev install && sudo god restart app-name

Now this can be triggered remotely with:

ssh -p PORTNUM localhost /path/to/script

Where PORTNUM is the port you have forwarded from the virtual machine for SSH (ie, what on the host corresponds to 22 on the virtual machine).

Now to round out my (at least for now) development environment, I am using Sublime Text 2 (just started, so this is all up in the air), and have set up a custom build system that looks line this:

{
"cmd": ["ssh", "-p", "2222", "localhost", "/path/to/build-script"],
}

What this means is that I can edit my application (all the files are on the host, just shared with the virtual machine, so there is nothing involved in doing that), and then with a shortcut trigger the build. This will compile the application and relaunch it, with the output of the compilation in a little panel so I can see what is going on.

The last parts are version control, testing, and deploying. I use darcs, and Quickcheck according to the guide here: How_to_write_a_Haskell_program#Running_the_test_suite_from_darcs, which allows you to run the tests as a precommit hook (which means commit will fail if the tests do not pass) like:

darcs --record --test

I am constantly pushing patches to a repository at Darcsden, a shared hosting site similar to github but for darcs, which serves as a level of redundancy (and if working with other developers, acts as a common repo we both can push to / pull from). There is also a repository on the production server, which can be pushed to as well.

This repository has a Heroku-inspired (though primitive) post-commit hook to build and deploy the application. This is a part I’d like to improve in the future - building binaries locally and just pushing them is one improvement; another is keeping around the old binaries to make it easy to switch back to a previous version. But for now, this is what the _darcs/prefs/defaults file in the production darcs repo looks like:

apply posthook echo "Rebuilding and deploying...\n" && cabal install && echo "Copying resources...\n" && sudo cp -R resources /path/to/production/root && echo "Restarting App-Name.\n" && sudo /var/lib/gems/1.8/bin/god restart app-name
apply run-posthook

This means that after recording changes locally, to deploy, I run:

darcs push servername:/path/to/production/repo

and select which patches I want to push. It will then build and deploy the application.

Aside from making the deployment more robust, I just need to integrate the version control / deployment into my editor to make this process completely streamlined (ie, being able to do everything from my editor, and just switch between it and a browser to test), which since it has a console pane, should be pretty easy.

Was running into this problem, fixed by disabling that setting....

Wed, 06 Jul 2011 23:09:19 -0400

Was running into this problem, fixed by disabling that setting. Thanks!

leff:

Sorry peeps, still no joy on the “too many redirects” issue. I’ve swapped themes in and out, eliminating my recent theme work as the source. I’m fairly sure it’s a server config issue. Been trying to contact with tumblr support, but only getting unrelated canned responses back so far.

update: there’s a setting: ’Use descriptive URLs’ under the Advanced tab on your Customize screen. Turning this off seems to have fixed the problem. Thanks to Renee at tumblr support for the help.

How to organize Ocsigen projects to compile to a native code binary (and why this is not good).

Wed, 06 Jul 2011 22:53:00 -0400

disclaimer: This whole post is based on the fact that I was not able to get a certain thing working. Part of the reason to write this is a challenge to someone else to figure out how to do it, and document it. There is extremely little information out there. With that said, I tried pretty hard, and came to the conclusion that it was not possible. If you can figure out how, I will retract my claim that Ocsigen native code is not a viable option for web programming.

edit

This only pertains to native code that is done with static linking. I was able to get ocsigen to link native code dynamically when it was the only library, but was not able to get this working with some external libraries, which only worked with static linking. If dynamic linking was working, all the acrobatics described in this post would be irrelevant. Since I was not able to, this was my experience, but consider this an enormous caveat (and it is terrible that I did not mention this originally. I stopped using ocsigen months ago because of this and other reasons, and wanted to finally get around to posting this, but forgot to mention that critical detail).

edit 2

This was written very negatively. I didn’t intend it to be so - when I first started writing this post it was meant as a guide for someone else who wanted to use statically linked native code with ocsigen. Since it took a bit of work to figure out how to structure the code (and a couple false starts) I wanted to write this down so that it could benefit others. However, partly due to the code organization necessary (and for other reasons), I stopped using Ocsigen for anything but small projects (let’s say, above 800 lines of application code), and I think because of that (and due to re-writing some applications that had reached that limit) I ended up writing this much more negatively than I originally intended. I wanted to write that post, ie, why I stopped using Ocsigen, in another post, but some of it leaked in here unintentionally.

why do I want / need this?

Well, I personally think it is silly to use a language that has a very fast native code compiler and not take advantage of that. But this is a valid point — and my conclusion is that indeed if you are going to use Ocsigen for any even medium sized project, you probably should not use native code.

what I looked at for comparison

The best references were the applications by mfp, particularly ocsiblog. It is a small blog application and also has some even smaller test applications in the repository. It was from these that I got the first native application running.

what it meant I had to change about my app

So the most important thing that you have to keep in mind is that you cannot register any service until runtime. Additionally, you canot register any service twice. These two things serve as somewhat of a death-knell to the cause of native code Ocsigen projects (at least as far as I could tell). The only workaround I could find, which is how the code in the example project by mfp works, is to wrap the entire application (or, all of the application that involves services - non-web libraries can be separate) within a single functor that takes a dummy argument. Then at runtime, you evaluate it once.

Now the reason why this is catastrophic is that it means that all your web code has to be in a single file. Trying to do it any other way, unless you have parts of your application that never interact (ie, never link to one another, never post a form to one another), and you will end up doing multiple evaluation, as far as I could tell. Possibly someone else could make this happen, but the various things I tried did not work. Take this as an open challenge. What would need to happen is that one module would orchestrate loading each one only once and passing them among one another.

interesting echos of haskell purity

Separating code that used services (or more generally, that used to use the server params argument, which is in current ocsigen passed around as Lwt thread data) from that code that is “pure” (ie, doesn’t touch ocsigen), so that the latter could be factored out into separate files (the former all being lumped into one module), was interesting, and reminded me of the isolation that haskell’s IO monad enforces. However, the lack of flexibility in dealing with the “IO” code was pretty limiting.

conclusion

Until this becomes more supported, and someone figures out a way to do this easily and without drastic code reorganization, Ocsigen should be thought of as a byte-code only option. There is enough documentation to make it seem like native code is an option, if only you bother to do it, but I think that is extremely misleading, and it would be worthwhile for the Ocsigen website to make this clear.

Housetab 1 - Creating a first run tutorial

Wed, 06 Jul 2011 22:16:00 -0400

This is the first in (hopefully) a series of posts coming out of creating the web application HouseTab with the haskell web framework Snap. All the source for the application is available at http://darcsden.com/position/housetab, and these posts will draw heavily from what is there (linking back where relevant).

One feature that this application has is a tutorial that runs the first time you log on, guiding you through setting up your account. We decided that we did not want this to be something that was separate from the application, but rather a set of dialogs / numbers that appeared within the application, so that not only would the tutorial explain how to set up your account, you would be doing it while going through the tutorial.

If you want to see this in action, feel free to create an account on HouseTab - you can always delete it (permanently and easily) from the Settings page.

There were a few decisions that we made. First, the tutorial would run the first time a new user signed in, but anyone could prompt it to start at any point. Second, the tutorial should be able to remember where you are beyond a page reload (so don’t store state in javascript). The third decision was that the tutorial should follow your actions - recognizing when you have completed a step and automatically proceeding to the next step.

A couple tools made this possible: the Heist templating library and the ajax add-on to it that I released at https://github.com/dbp/heist-async. I also use the snap-auth library to provide session support (and user auth, but that is not covered here). From now on, I’m going to assume familiarity with those, so briefly look over them now if you haven’t already seen them.

To maintain the active status of the tutorial, there is a simple boolean stored with every account. The step in the tutorial is stored in a session variable. Originally I was going to store it permanently, but it seemed that it would be more confusing to log in again days, weeks, or months later and end up half-way through the tutorial. The session variable is a simple number, and a library function is provided that changes it, given what it currently is (ie, set to 3 if it currently is 2):

tutorialStep user old new =
  if not (tutorialActive user) then return () else do
    st <- getFromSession "tutorial-step"
    if st == Just old then setInSession "tutorial-step" new 
                      else return ()

This provides half of the functionality - when a user performs an action (for example, the first step of the tutorial is creating a person on the account), provided they are in the tutorial and at the given step, proceed to the next step. This means that they won’t go backwards if they repeat an earlier step, and can’t accidentally skip ahead by performing another action. An example from the code is:

...
do saveHouseTabPerson $ person' { pHTId = htid}
   nu <- recalculateTotals user
   tutorialStep user "1" "2"
   ... $ renderHT "people/add_success"  
...

The other half is where the tutorial is actually displayed. First, we created a splice (this is Heist terminology, if it doesn’t make sense, read the Heist tutorial linked to above) that displays it’s children if the current tutorial step is that specified in the step attribute (from Views/Account.hs):

import qualified  Data.Text.Encoding as TE
import qualified  Text.XmlHtml as X

tutorialSplice :: Splice Application
tutorialSplice = do 
    node <- getParamNode
    s <- lift $ getFromSession "tutorial-step"
    case X.getAttribute "step" node of
       Just step | Just (TE.encodeUtf8 step) == s 
            -> return (X.elementChildren node)
       _ -> return []

Now for the simple cases, we show the tutorial box on the event of a full page load. This looks, in the page template, like:

...
<tutorial step="1">
  <apply template="tutorial/1"></apply>
</tutorial>
<tutorial step="2">
  <apply template="tutorial/2"></apply>
</tutorial>
  <tutorial step="3">
  <apply template="tutorial/3"></apply>
</tutorial>
...

One of these pages looks like (collapsed the base and template into one for conciseness):

<div-async name="tutorial" id="tutorial">
  <div class="content">
  <p>
    Welcome to the HouseTab Tutorial. 
    Follow these steps to finish setting up your account:
  </p>
  <p>
    <span class="num">1.</span> 
    Add at least one user to your account.
  </p>
  </div>
  <a-async class="end" href="/tutorial/deactivate">
    End Tutorial
  </a-async>
</div-async>

If this were a non-ajax application, this would probably be good enough - when the next page load occurred, the correct step of the tutorial would be selected. To do it with ajax, using heist-async, it isn’t actually much more difficult. First, be sure that the tutorial box is wrapped inside a <div-async> (which it is above), so that it can be replaced be a later page fragment. Then, in the template that is sent down to update the page (in the example above, this template is “people/add_success”), simply conditionally include the next step of the tutorial, which will replace the tutorial box, using the exact same code as it is included for the case of a non-ajax page load:

<tutorial step="2">
  <apply template="tutorial/2"></apply>
</tutorial>

One thing that is important is that this box is not included twice - so the second load should only accur in the special page that is only for ajax responses - in this case, "people/add_success".

Now the last part is the numbered prompts - the idea is, when you get to step 2, there is the box at the top of the page that tells you what to do, but there is also a red circle with a 2 in it that appears on the page where the action you should be performing is. Again, this is simply a matter of conditionally including the proper div’s in the places they belong. They will be positioned and styled with CSS. Since the fragments of the pages are the same templates whether they are loaded partially via ajax or via a full page load, there is nothing special that needs to be done to make it work for ajax. An example is the add person form (which is the first step of the tutorial. form cut down for presentation):

<div-async name="add-person" class="addPerson" id="adduser">
  <form-async action="/people/add" method="POST">
    <h2><label for="name">Add a new user:</label></h2>
    <input name="name" type="text" value="$(name-value)" />
    <button type="submit" title="" class="addform_submit" />
  </form-async>

  <tutorial step="1">
    <div id="tutorial-1">
    </div>
  </tutorial>
</div-async>

That’s all for this short first post!

Turning dynamic web programming on it's head: declarative dynamic html with heist

Mon, 20 Jun 2011 18:38:00 -0400

(this is a code heavy post, with assumed familiarity with javascript, haskell, and the heist templating library, as a warning)

The motivation of the code / method outlined in this post is rather simple: I would like to encode dynamic aspects of the websites I write in the markup, not in Javascript code (corollary to this: I don’t like having lots of code running in javascript, and prefer writing things declaratively to imperatively, preferably with the aid of a type system, but I do want the sites that I build to have dynamic behavior. This solution is an attempt to reconcile these two things).

This may seem impossible, but it actually turns out to be rather simple. The key is to identify a small set of dynamic primitives that should be available, with clearly defined markup that should result in predictable (preferably local, to make it easier to reason about / check correctness) effects.

Then, global event listeners will look for certain events that are triggered on the specified markup, and will perform the needed operation. This means that you will code the effect once in Javascript, which will effectively extend the dynamic abilities of your markup, anywhere where the small javascript primitives are available.

Once the necessary javascript has been described, built, and tested, using Heist I will build Splices (which are special html tags that generate html based on their attributes, children, etc) that will produce the markup that the javascript needs to recognize / carry out the behavior, while exposing a very simple interface to the user (this reduces the chance that the user, ie, me and others working with this code, could generate markup that would screw up the handlers). From the user’s perspective, these special tags ARE the dynamic behavior, and provide a reliable way of ensuring that the effects always work.

The reason for creating this was rather simple: For a project, the designer in my team wanted to have input fields that, when selected, popped up a box that presented options to select by clicking on them. This is the task that usually a select field would fulfill, but it did not fit with the interface, and it was something that had come up before (when we did settle on a plain old select field) so I set out to create a way of consistently creating this pattern. One such end-use case is shown here:

      <box-field name="person-name" value="$(person-name-value)">
        <people>
          <box-option value="$(personId)">
            <personName/>
          </box-option>
        </people>
      </box-field>

The “box-field” and “box-option” are splices that are globally available. They are what encode the idea of this Box Field (a field that presents options in a box to pick between). name is what the field’s name will be when the form is submitted, value is the prefilled value (here dynamic). The “people” splice is what holds the data that is populating the options (and it presents the subSplices “personId” and “personName”) - don’t worry about that for now, as it just as easily could have been a static list of box-options, like:

      <box-field name="person-name" value="$(field-name-value)">
          <box-option value="1">
            Jane
          </box-option>
          <box-option value="2">
            John
          </box-option>
          <!-- ...etc -->
      </box-field>

Since this element might be used in different places, and on fragments of pages returned asynchronously, I did not want to have to make sure I was always attaching the proper handlers, every time the element was created. Having to always attach handlers also runs counter to the idea of encoding the behavior in the HTML - I want declarative dynamic behavior, not imperative.

So after looking a little, I figured what I needed was a way to register a global event listener that would listen for a specific event, match a certain element, and then run a corresponding function on it. I have used the lightweight standalone javascript libraries by @dedfat (of twitter), because they are simple and work well for me, but this could easily be adapted to use jQuery, etc. The function I came up with is “declare” - which takes an event, a selector, a boolean as to whether it should stop propagation (I think the answer will always be yes, but wasnt sure), and then the function to call when this occurs, passing in the element.

/*!
  * declarative.js - copyright @dbp 2011
  * BSD3 License
  */
function declare(event, selector, nopropagate, fun) {
  // bean is an cross-browser standalone event handler 
  // at https://github.com/fat/bean
  bean.add(document.documentElement, event, function(e) {
    e = e || window.event;
    var elem = e.target || e.srcElement;
    // qwery is a cross-browser selector engine 
    // at https://github.com/ded/qwery
    if (!elem || qwery(selector).indexOf(elem) === -1) {
      return;
    }
    fun(elem);

    if (nopropagate) {
      e.stopPropagation();
    }
  });
}

Then I came up with the following HTML that I wanted to govern my box-field, the idea being that the hidden input field would contain the person’s ID when it is selected, and that the display field is what will show the name that is selected (styling done inline here, just to keep it self contained):

<div class="box-field ">
  <input type="hidden" name="person-name" value="">
  <div class="display" 
    style="width: 200px; height:20px; border: 1px solid black;">
  </div>

  <div class="box" style="display:none;">

  <div data-box-value="1" class="option ">
    Jane
  </div>

  <div data-box-value="2" class="option ">
    John
  </div>

  </div>
</div>

Then I wrote up the handlers to make this box-field actually work:

bean.add(document, 'DOMContentLoaded', function () {
  declare("click",".box-field .display",true,function (elem) {
    // bonzo is a cross-browser selector engine 
    // at https://github.com/ded/bonzo
    bonzo(bonzo(elem).next()).show(); // show the box
  });
  declare("click",".box-field .box .option",true,function (elem) {
    bonzo(elem.parentNode).hide();
    d = bonzo(elem.parentNode).previous()[0];
    d.innerHTML = elem.innerHTML;
    bonzo(bonzo(d).previous()[0]).attr("value", 
      elem.getAttribute("data-box-value"));
  });
});

Finally, the splices that turn the markup shown at the beginning of this post into the html that the javascript actually operates on:

import qualified Text.XmlHtml as X
import qualified  Data.Text as T
import Text.Templating.Heist


boxField :: Monad m => Splice m
boxField = do node <- getParamNode
              case X.getAttribute "name" node of
                Nothing -> return [] -- without name, useless
                Just name -> do
                  let klass = T.concat 
                     ["box-field ",
                      (fromMaybe "" $ X.getAttribute "class" node)]
                  let value = fromMaybe "" $ 
                     X.getAttribute "value" node
                  let children = 
                    [ X.Element "input" 
                       [("type","hidden"),
                        ("name",name),
                        ("value",value)] []
                    , X.Element "div" 
                       [("class","display"), 
                        ("style", 
     "width: 200px; height:20px; border: 1px solid black;")] []
                    , X.Element "div" 
                       [("class","box"),
                        ("style","display:none;")] 
                       (X.elementChildren node)
                    ]
                  return [X.setAttribute "class" klass $ 
                    X.Element "div" (filter ((/= "name").fst) $ 
                       X.elementAttrs node) children]

boxOption :: Monad m => Splice m
boxOption = do node <- getParamNode
               case X.getAttribute "value" node of
                 Nothing -> return []
                 Just value -> do
                   let klass = T.concat 
                      ["option ",
                      (fromMaybe "" $ X.getAttribute "class" node)]
                   let attributes = 
                     ("class", klass) : (filter 
                        ((flip notElem ["name","class"]).fst) $ 
                        X.elementAttrs node)
                   return [X.setAttribute "data-box-value" value $ 
                     X.Element "div" attributes 
                       (X.elementChildren node)]

There is obviously a bit of styling that still needs to be done to make the box look like it should, but everything from here on out (and any number of uses of this box) should be outside of the world of javascript - which was my intention in the first place!

PS. I just realized a small flaw in the example I presented. The splice “box-field” should also have a “display” attribute that has predefined what the field should show (to match up with the predefined “value”). This is easy to do, and should not detract from the presentation.

Building Ajax Sites With Snap - Imagining Heist-Async

Wed, 08 Jun 2011 17:11:00 -0400

I’ve recently started working with Snap, the Haskell web framework, (http://snapframework.com), and one reason (among many) for my reason to switch from Ocsigen, a web framework written in OCaml (which I’ve written posts about before) was the desire to more flexibly handle ajax based websites. While it seems good in some ways, I eventually decided that Ocsigen’s emphasis on declaring services as having certain types (ie, a fragment of a page, a whole page, a redirect, etc) is in some ways at odds with the way the web works.

After starting to work in Haskell again, and with the Snap team authored templating system Heist, I immediately began looking for ways to work with ajax content more flexibly than I had been doing before. Inspired by the work of Facebook on Primer (provided to the world at https://gist.github.com/376039 ), which is their base-line system for dynamic content - basically, event listeners waiting for onclick events on links that have a special attribute that says it should perform an ajax request, and event listeners for onsubmit events on forms that have a special attribute that indicates the forms should be serialized and submitted asynchronously. But even more interesting than that (to me) was the other half of their system (not, I believe, public, and regardless, written in PHP), which is that the server side response decides what client side div’s it should replace.

At first that sounds a little dirty - it basically entails mixing (conceptually) server code and client code. But then it allows a different sort of methodology - that even with client side modifications, it is the server that ultimately has all control - including what to replace on the client. This is a fascinating idea, because clientside code is notoriously limited be being written in javascript (or with javascript libraries), and thinking about having to maintain clientside and serverside code seems to be a much dirtier solution than having the server, in short, control the client.

Taking this idea, and bringing it into the world of Heist, which is (in my opinion) a fantastic templating system (more info at http://snapframework.com/docs/tutorials/heist ), ended up being quite straightforward, as Heist lends itself to the idea of extending the syntax of html, much like the facebook primer system did.

At first I thought that there should be haskell code that would specify things like “replaceDivsWithSplices …” where div’s would be identified and corresponding splices (things that can be inserted into heist templates) would replace them, and then “replaceDivsWithTemplates”, etc, but the whole solution seemed a little off.

And then I realized that the entire idea could be summed up with a single tag: “div-async”. The idea would be, this would be a special div that could foreseeably be replaced by an asychronous response. A template would have many divs that were marked this way, which in a non-async response would do nothing special, but when an async response came back, all div-async’s would replace corresponding tags on the page.

The only things that remained were the two tags to start the async requests, which I named “a-async” and “form-async”, and a little javascript to make the moving parts work together. And so, heist-async was born. (for the impatient, the code exists at https://github.com/dbp/heist-async , and while I am using this code currently and it seems to work, it could change significantly as things are worked out)

The basics of how this works should be obvious, but I can illustrate a basic example. On a page you have an announcements box. You want the user to be able to click a button and have the announcements box reload without reloading the whole page (new announcements may have occurred). So you have a page template that looks like this:

<html><body><h1>Some page</h1>
  <apply template="announcements></apply>
  <a-async href="/recent_announcements">Reload</a-async>
</body></html>

And an announcements template that looks like this:

<div-async name="announcements">
  <announcements>
    <text/>
  </announcements>
</div-async>

Now to glue this together, all you need to do is serve the original page (with the proper splice set so that the tag actually works), and, at the /recent_announcements url, you just serve the announcements template. Since it is the exact same template, it obviously has the same identifier for the div-async (which is just the attribute “name”), and will therefore replace the current anouncements box with the recently loaded one.

Now that is pretty cool - what it means is that you can have one set of templating code, and the only change you need to do is separate any parts you want to be able to load asynchronously into separate templates, and make sure there is a div-async wrapper around it. (NOTE: since I didn’t mention it before, it might be helpful to now - div-async is just a regular div, so you can set all the regular things, like id, class, etc. Also feel free to take existing div’s and just add -async and set a name).

At this point, I was pretty happy with this, and thought it was working pretty well, but of course the real world is much more complicated, and not everything is so simple - sometimes a single asynchronous request should mean a lot of different things on a page should change. In this case, it is possible that the simple template inheritance will not work, but with the addition of a template that is just for the response, that includes all the templates that should be updated, it seems to work pretty well. An example of one of these could be:

<apply template="announcements"></apply>
<apply template="title"></apply>

In this case, there is still no duplication of formatting code, all that exists now is an explicit list of all the parts of the page that should be replaced by a given request.

Other common things; to hide an element, sending back:

<div-async name="something" style="display:none"></div-async>

Should work. You could also put some empty placeholder div’s like that on a page, and later replace them with ones with actual content.

What I noticed about this is that it makes dynamic page changes very explicit in the templates, which I think is a very good thing - and certainly makes it easier to reason about page changes.

Getting to this point, I started using this to implement a bunch of parts of a new site I’m working on, and I was happily impressed by how it all seemed to be working. Using this, it seems like ajax can be thought of as just an aspect of the templating system - describe what should be replaced, and it will be, without ever having to worry about the clientside code (which is 12k of lightweight libraries and 60 significant lines of code of custom javascript. The 60 lines should easily be able to be translated to depend on common javascript libraries like jQuery, I just didn’t want to make that a requirement).

I’m interested in feedback on the library, and ways that it can be improved. It is still very early software (a week ago, it did not exist), but it is something that I’ve found very powerful, and I’m kind of interested in where it can be taken / what people think about it.

Making a similar message system as rails' flash with Ocsigen

Tue, 19 Apr 2011 20:28:00 -0400

I’ve recently been working on a project that uses Ruby On Rails, and I was reminded once again how much I like their “flash” functionality. Flash is a session-based temporary message storage facility which is displayed (if it exists) and erased every time a page is displayed, so that you can, for example, set a flash message in a controller (something like, “Successfully saved X”) and it will be shown on whatever the next page is, whether you directly render a template, redirect, etc.

I was thinking that it would be extremely useful to have something similar with Ocsigen (the ocaml web framework, for those unfamiliar with it), especially when used with Actions. Actions are (as far as I know) a uniquely Ocsigen concept - an action is a service that does not live at any given url (rather, when you create a form pointing towards it, Ocsigen uses params to match it up), does not return anything, and after being called simply reloads the current page. This turns out to be incredibly handy, as it means you can put a form pointing to this service on any page, and it will perform whatever action and then reload the current page. Of course you could do the same thing by using redirections and keeping track of urls, but it is so natural with Ocsigen that it definitely changes the way I write web applications. Spefically I’ve noticed that apps I write are much denser, because having a single page be able to have many form that post and reload (instead of the usual - only one) means that what might have in the past been a four or five page application (not for usability but simply for programmer convenience) is now one page.

So getting back to the flash, one of the major problems with actions is that they do not return anything, indeed the original page is reloaded, so unless the change is extremely obvious, the user may not notice anything even happened. Or, more of a problem, if something goes wrong in the action, there is no way of informing the user of the problem. Enter the idea of the flash. In the action, a message is stored in the session data, which is then displayed when the original page reloads.

The two aspects of this are that you need some way of storing session data, which Ocsigen makes very easy (see a previous blog post I wrote about using it for a shopping cart at blog.dbpatterson.com/post/637836183/shopping-carts-with-ocsigen), and then you need to make sure that every page displays this, if it exists, and empties it (so the same message doesn’t keep getting displayed).

The first task is quite easy (note: this, and all subsequent code on this post, is using Ocsigen 1.90):

let notification_state_name = "notification"
let notification_state:string option Eliom_references.eref = 
    Eliom_references.eref ~state_name:notification_state_name 
                            ~scope:`Session None

let notification msg = 
    Eliom_references.set notification_state (Some msg)

This first defines the name of the state, then it defines the state itself (the explicit type is included, which is that it is an eref to what might be a string and might be nothing), and finally we define a helper function to set a message in the notification state.

The second task is to retrieve, empty, and display the message on all page loads. In order to do this, we need a wrapper that we will put around all pages. The way I have done this is to include authentication in the same function, so I end up having services that look like:

(fun () () ->
    require_login 
        (fun user -> ...))

or, in the case of pages that do not need a login, but should still display the messages:

(fun () () ->
    require_none
        (...))

Now these functions are responsible for taking the page output, adding the (if present) message, and then returning them. Now each of these functions is going to have to be specific to the type of page that is going to be displayed, so it makes sense to develop an abstract function that is responsible for getting and setting the notification state, as well as (in my case) checking login credentials. The way this works is I have a function that looks something like this (I’ve simplified for the sake of blog posting, but have included the login state code):

let user_state_name = "session_user"
let user_state:user option Eliom_references.eref = 
    Eliom_references.eref ~state_name:user_state_name
        ~persistent:user_state_name ~scope:`Session None
let failed_login_state = Eliom_references.eref 
        ~state_name:user_state_name ~scope:`Request false 


let gen_require_login (type a) (type b) 
    (template : (string option -> a -> b Lwt.t)) 
    (session_expired : a) 
    (failed_login : (bool -> a)) 
    (logged_in : (user -> a)) =
    Eliom_state.persistent_data_state_status 
        ~state_name:user_state_name () >>= fun status ->
    Eliom_references.get failed_login_state >>= fun failed ->
    Eliom_references.get user_state >>= fun user ->
    Eliom_references.get notification_state >>= fun notification ->
    Eliom_state.close_session 
        ~state_name:notification_state_name () >>= fun () ->
    (template notification (match user, status with
        | Some user, _ -> logged_in user
        | None, Eliom_state.Expired_state -> session_expired
        | _ -> failed_login failed))

This function is a bit more complex, but it is in order to be used flexibly in different situations. Before unpacking how it works in detail, let’s look at it on a high level and see what it can be used for. It takes a series of parameters, some functions and some just plain values that all are or will be evaluated to type a. Often this might be an html page, but it doesnt have to be. The argument template is a function that takes a string option that is the notification message, and the existing page, and returns a (potentially differently typed page) that should have the notification message added to it, all wrapped inside the Lwt monad, which Ocsigen uses to organize threaded execution in a monadic style similar to Haskell’s IO monad.

The way this is written, the caller gets to decide what should happen with all the potential login states and also how the notification should be handled. Indeed, this function can be used in the case where we dont even want to check login status but we do want to display the notification:

let require_none logged_in = 
  let fn = (fun _ -> logged_in) in 
  gen_require_login
   (fun notification body -> 
      Lwt.return (html_page (match notification with
    |Some message -> [div ~a:[a_id "notification"] [pcdata message];
              body]
    |None -> [body])))
    logged_in fn fn

This function uses a helper “html_page” that simply creates an html_page with appropriate headers, and stores the notification in a labeled div at the top of the page, before the main body. It ignores the login state by passing in tho same function for all the possible login states. This means that we actually are checking if they are logged in, we are just giving back the same page regardless.

Another extreme use case is for actions, where we want to check whether they are logged in and not execute otherwise, but we have no use for a current notification:

let action_require_login logged_in =
    let fn = (fun _ -> ()) in
    gen_require_login (fun _ body -> Lwt.return body)
         () fn (fun u -> logged_in u)

What this has is the template function does nothing - it just passes through the body and ignores the notification. It then passes functions that do nothing in the case of all the non-logged in states, and finally performs the action if the user is logged in.

Seeing a few different ways that this function can work, let’s return to the function itself:

let gen_require_login (type a) (type b) 
    (template : (string option -> a -> b Lwt.t)) 
    (session_expired : a) 
    (failed_login : (bool -> a)) 
    (logged_in : (user -> a)) =
    Eliom_state.persistent_data_state_status 
        ~state_name:user_state_name () >>= fun status ->
    Eliom_references.get failed_login_state >>= fun failed ->
    Eliom_references.get user_state >>= fun user ->
    Eliom_references.get notification_state >>= fun notification ->
    Eliom_state.close_session 
        ~state_name:notification_state_name () >>= fun () ->
    (template notification (match user, status with
        | Some user, _ -> logged_in user
        | None, Eliom_state.Expired_state -> session_expired
        | _ -> failed_login failed))

The meaning of all the function arguments should now more or less make sense, so let’s look at how they are put together. Each of these functions does something in the Lwt monad in order to get the value, which means that they have types like:

Eliom_references.get notification_state : string option Lwt.t

The way you chain monadic operations together with Lwt is identical to Haskell - the bind operator:

val (>>=) : 'a Lwt.t -> ('a -> 'b Lwt.t) -> 'b Lwt.t

What this says is, given a value 'a that is in the Lwt monad (in the example above, 'a is string option), and a function that takes a value of type 'a and returns something else also in the Lwt monad and return the second thing. So taking another look at the code above, the following line:

    Eliom_references.get notification_state >>= fun notification ->

Means, get the notification and then pass it off to an anonymous function that takes one argument, namely that notification, inside of which it can be used as a pure value.

Now we’ve now looked at a lot of code, but we still have the problem that nowhere have we actually set a notification message. The action_require_login function above was simplified to the point of missing one original goal of developing this kind of messaging: if something goes wrong (like not being logged in) in an action, a message should be left. So consider instead, the following option for use with actions:

let notification_require_login logged_in =
  (gen_require_login
     (fun _ body -> Lwt.return body)
      "Your session has expired."
     (fun failed -> 
         if failed then "Incorrect username / password." else "")
     (fun u -> logged_in u)) >>= fun msg ->
  notification msg

What this does is run gen_require_login, again with a function that ignore the current notification, and then it chains what comes out of the function into the earlier notification function to set a message. Since the type of msg is string then the type of gen_require_login applied to these arguments must be string Lwt.t, which means that the template function returns that type and must therefore take values of type string. So what this means is that the action function, passed in as logged_in should return a string, which wil be set as the notification message. If the user is not logged in, informative messages are left, which will be displayed when the page that called the action is reloaded.

So this can be used to develop services that have not only the flexibility that actions provide but also a way of communicating with the user. To wrap up, here is a function that demonstrates not only how to use this notification_require_login function, but that it can also be chained on to a redirection:

let add_user_fn =
    (fun () user' ->
      notification_require_login
    (fun user ->
      if add_user user' then
        "Successfully added user!"
      else "Failed to add user.") >>= fun () ->
      Lwt.return index)

Shopping Carts with Ocsigen.

Thu, 27 May 2010 11:18:26 -0400

A pretty common thing for an ecommerce website to have is some sort of shopping cart. There are ways around it - primarily by purchasing one item at a time (or even worse - filling out an order form with the item identifiers!) - but the shopping cart reigns supreme.

To implement a shopping cart, a few things usually need to happen. When someone visits the website, you need to stick a cookie on their browser, and record that on the server, referencing the data (the cart contents). Then this new data store is your cart for the request. Of course you actually first need to check if they already have a cookie - then check if the reference is good, and if so update the expiration time on the cookie and your local data, and then use the cart selected for the request. And this looking for cookies, verifying that the data exists, etc, needs to happen on every request.

Of course, a shopping cart is only one reason why you might want to do something like this - there are many reasons why you might want to store data on a per user basis (but not per login, or not even on a site where logging in is necessary) - recently viewed pages, ‘favorites’, etc.

Ocsigen (well, actually Eliom, but since eliom is distributed with ocsigen, and the former has more name recognition, I’ll use it) abstracts this all out by making it possible (and easy) to create “session data”. You simply name a place to store the data (any data type you want) and then in requests you can get the data, modify it, save it. There is no need to mess with cookies, or even worry about where the data is being stored.

It is interesting to consider this design decision - to not try to avoid the paradigm of the web entirely, as requests still are independent and accessible via sane urls (some continuations based web frameworks will only allow you to enter the site at one point, and then will store the state in a token stored in the url - completely abusing the entire idea of urls and the web). But certain low-level aspects that are tedious to handle and often necessary - like this issue of per-client data - are abstracted out. In this way Ocsigen can be seen not as yet another library to handle dealing with web programming, nor a completely different programming model that is compiled down to something that speaks the language of the web but rather a high-level language in which to write web applications.

But enough digression - here is a shopping cart mechanism from a recent project I was working on, a store to sell MP3s:

type cart = { songs : song list }
let session_data = create_volatile_table ()

let get_cart sp = 
  let sess_data = get_volatile_session_data  ~table:session_data ~sp () in
    match sess_data with
      |Data c -> c.songs
      |Data_session_expired -> []
      |No_data -> []

let mod_cart sp fn =
  let sess_data = get_volatile_session_data  ~table:session_data ~sp () in
  let new_sess = match sess_data with
    |Data c -> {songs = fn (c.songs)}
    |Data_session_expired
    |No_data -> {songs = fn []} in
    set_volatile_session_data  ~table:session_data ~sp new_sess

And that’s it. The action to add to the cart? It’s one line, once you get past the action boilerplate:

let add_to_cart =
  Eliom_predefmod.Action.register_new_post_coservice'
    ~post_params:(user_type song_of_string string_of_song "song")
    (fun sp () song ->
       mod_cart sp (fun c -> if not (List.mem song c) then c@[song] else c);
       return ())

Delete is as simple:

let del_from_cart =
  Eliom_predefmod.Action.register_new_post_coservice'
    ~post_params:(user_type song_of_string string_of_song "song")
    (fun sp () song ->
       mod_cart sp (fun c -> BatList.remove c song);
      return ())

And the last part, to complete the cart mechanism, are the forms to post to the add and del actions. These use one of the more clever features that Ocsigen has - provided you can give it functions to turn native types to and from strings, you can put native types in html forms (as selects, or hidden inputs), you can preapply services with them, and as you saw above, you can use them as parameters to services. One of the most convenient uses of this in my application is a form that is just a button, that has one hidden element - a song. This is written as:

let mk_song_form txt s =
  (fun song_n ->
     [div [user_type_input ~input_type:`Hidden ~name:song_n ~value:s string_of_song ();
       button ~button_type:`Submit [pcdata txt]]])

So then anywhere where I want to put a button to add a song to the cart, where the song is ‘s’, and sp is the conventional name for the server params argument to all services (see the above actions for where it appears), the code is as simple as:

post_form add_to_cart sp (mk_song_form "Add to cart" s) ()

Or to delete:

post_form del_from_cart sp (mk_song_form "Del from cart" s) ()

Pretty simple, eh?

Tussling with Paypal using Ocsigen/OCaml

Tue, 18 May 2010 18:32:19 -0400

Since writing my last post about Ocsigen, detailing the beginnings of a rapidly developed (but very simple) web application with the OCaml web framework, I’ve had occasion to use it in a much more complicated (though still relatively simple) application - a small store to sell mp3s. It took, from start to finish, about 5 days, spending about 8-10 hours per day working on it.

There are many possible posts about this experiment (/experience) that I could write, and I was constantly surprised by how quickly I was able to add features or change large portions of the application (probably the most shocking was writing an entire shopping cart mechanism in under an hour, thanks to Ocsigen/Eliom’s support for session data), but I wanted to focus specifically on the integration with Paypal, which was probably the most difficult part (and wasn’t too hard).

When starting out with a new API (and not being able to use someone else’s bindings, which is a predicament often faced when using less commonly used programming languages), it’s helpful if there is a simple step by step example for a minimum viable integration. Something that shows what every interaction with the bare API looks like, in a language agnostic way. Unfortunately, most of the examples that I could find are instead in PHP, ASP, or ColdFusion, or are spread among many different pages.

Even the closest page I could find seemed to assume you already knew how to use the API, and so ignored many details (which in turn leads to various error messages). At first I wondered how it could be that a company as big as Paypal which has an API that is so widely used could not have stellar documentation, but then I realized if most people are using wrapper written in the programming language they are using, then such an overview is not necessary, or at least broadly needed.

What’s more, the documentation that is important to those writing the wrappers (and therefore familiar with the API) is all the minutiae of various options, which is available, and NOT a brief introduction (which was what I, not having ever used it before and writing in a language that does not have bindings, wanted to read). I still think there must be some page somewhere, but eventually gave up looking!

So, I present a brief overview of a minimum viable integration (with Ocsigen/OCaml used for code examples).

First, some points about actually calling the API. This seems really straightforward once you know what it is, but the error messages weren’t very helpful (in the documentation for errors, very few in any of the ExpressCheckout flow have a recommendation for resolution, they just repeat the error message that the api sent you!)

All api calls are HTTP POSTed to https://api-3t.sandbox.paypal.com/nvp in the testing sandbox, and https://api-3t.paypal.com/nvp in production.
The post-data is in the form PARAMNAME=paramValue&PARAMNAME2=paramValue2 …
The first param Must be METHOD, which determines which API function you are caling. To initiate an expresscheckout, invoke SetExpressCheckout. Otherwise it gives you an error that the method does not exist.
Other required parameters are your API credentials (“USER”, “PWD”, “SIGNATURE”) and the “VERSION” of the api you are using. A recent one seems to be “51.0”, though again, the most recent one seems elusive, as different pages use different versions in their examples.
The method and then credentials are followed by the params for your method, as stated in the documentation for the various methods - this was the one place where documentation seemed to be very good.

On the OCaml end of things there are not a lot of good options for making http requests - and no native ones that worked for me. I tried many options - first, I looked at Ocamlnet’s Http_client, but realized very quickly that it did not support SSL, which made it unusable (aside from that, the interface looked pretty good).

Next was Ocsigen’s http_client library (which was of course available by default), but it didn’t seem to work - and wouldn’t spit out any debugging messages for me, making it frustrating enough to use that I moved on to the next option. I then tried the OCaml bindings to Curl (ocurl), but dynamically linking it into the application with Ocsigen kept giving errors, and since I was on somewhat of a deadline I decided to come back to it if no other options were available.

I think had I taken some more time (or been more familiar with the OCaml language - I’ve only been programming it for about a month) this could have been a good option - but equally there doesn’t seem to be a simple way to use it - it seems very C-like (setting loads of imperative options before telling it to run the call - instead of having a function that you pass a lot of arguments and get back the result of the request).

The final option, and what I ended up using, could be considered the sledgehammer solution - take a program I am familiar with and know works (wget), and call out to it using Unix.open_command. It seems ugly, and indeed I wouldn’t have thought of it if I hadn’t used (and enjoyed) a Haskell library that did the exact same thing.

So, to present a little code in the post, here is a Paypal wrapper (well, that’s giving it a lot more credit than it is due) in 21 lines of OCaml (special thanks to whoever wrote the OCaml section of Rosetta Code for the implementation of syscall - provided this is an efficient way to do that, having such a function available in standard libraries would be great!):

let api_endpoint = "https://api-3t.sandbox.paypal.com/nvp"
let credentials = [("USER", "api_username_from_paypal");
                    ("PWD", "api_password_from_paypal");
                    ("SIGNATURE", "api_signature_from_paypal");
                    ("VERSION", "51.0")]
let syscall cmd =
   let ic, oc = Unix.open_process cmd in
   let buf = Buffer.create 16 in
   (try
      while true do
        Buffer.add_channel buf ic 1
      done
    with End_of_file -> ());
   let _ = Unix.close_process (ic, oc) in
   (Buffer.contents buf)
let wget u pd = syscall ("wget --quiet -O - --post-data=\"" ^     
    (Netencoding.Url.mk_url_encoded_parameters pd) ^ "\" " ^ u)
exception Paypal
let paypal method_ params =
  let ps = ["METHOD", method_]@credentials@params in
    Netencoding.Url.dest_url_encoded_parameters (wget api_endpoint ps)

So, to continue on to actually implementing the ExpressCheckout, we can review from the paypal docs on it that we are supposed to first invoke SetExpressCheckout.

The raw call with wget would look like:

wget -O - --post-data="METHOD=SetExpressCheckout&USER=...&PWD=...\
&SIGNATURE=...&VERSION=51.0&AMT=10.00&RETURNURL=someurl\
&CANCELURL=someurl" "https://api-3t.sandbox.paypal.com/nvp"

In our case setting up the transaction looks like this (as all the credentials are handled by the ‘wrapper’, and the METHOD is the first argument):

paypal "SetExpressCheckout" [("AMT", amount);
                             ("RETURNURL", returnurl);
                             ("CANCELURL", cancelurl)]

Where returnurl is the url that paypal redirects to after the transaction, with “?token=SOMEVAL&PayerID=SOMEVAL” appended to it, and cancelurl is where it sends the customer if they cancel the transaction while at paypal.

Since we are using Ocsigen, we need to use Lwt_preemptive.detach so that this doesn’t block the whole webserver for the duration of the API call (which is going to be quite long, on the scale of a web request). A full service that sets up the paypal express checkout and redirect the person to paypal follows (the documentation says you should pass AMT, RETURNURL, CANCELURL with the redirect - doing so seems to cause it not to work, so I think the documentation is in error or out of date):

let purchase_start =
  Eliom_predefmod.String_redirection.register_new_post_service
   ~fallback:checkout
   ~post_params:unit
   (fun sp () () ->
     Lwt_preemptive.detach 
      (fun () ->
        try
        let chop_params s = 
          BatString.slice ~last:(BatString.find s "?") s in
        let returnurl = 
          chop_params (string_of_uri (make_uri ~absolute:true 
                ~service:(preapply purchase_end ("", "")) ~sp ())) in
        let cancelurl = string_of_uri 
                (make_uri ~absolute:true ~service:checkout ~sp ()) in
        let amount = total (get_shopping_cart sp) in
        let init_paypal = 
          paypal "SetExpressCheckout" [("AMT", amount);
                                       ("NOSHIPPING", "1");
                                       ("RETURNURL", returnurl);
                                       ("CANCELURL", cancelurl)] in
        let token = List.assoc "TOKEN" init_paypal in
        let redir = 
          "https://www.sandbox.paypal.com/webscr?cmd=_express-checkout&" ^ 
          (Netencoding.Url.mk_url_encoded_parameters [("token", token)]) in
          uri_of_string redir
        with _ -> 
          make_uri ~service:checkout ~sp ()) ())

This of course is code that depends on other services - mainly, ‘checkout’ and ‘purchase_end’. The former is simply a place where someone can initiate the checkout process, and purchase_end is what handled what paypal sends back to us. This service also depends on having a shopping cart you can access with get_shopping_cart, but that’s a pretty minor detail. As a side note - for simplicity, a few more details have been omitted - like saving the token before you redirect (to verify when the customer is redirected back).

Also, instead of hard coding in urls for the return and cancel, I used Ocsigen’s make_uri function that can construct them based on the actual services they reference - so if I change the url of the service, it won’t affect this code at all. (Since paypal will be adding the params to the returnurl service, I chop them off of before sending the url).

So then the customer approves the payment, and is redirected back to the url specified in RETURNURL, with the GET params token and PayerID. The API says you can get call GetExpressCheckoutDetails with the token to get their email address, name, etc. This is then simply:

paypal "GetExpressCheckoutDetails" [("TOKEN", token)]

And then once you confirm the payment from them (which is a step not required technically by the API, but the ui in paypal tells them they will confirm the payment in the next step, so it is probably a good idea), you can actually make the payment happen with DoExpressCheckoutPayment, which takes both the token you’ve been using all along, the original AMT, and the PayerID that was passed back in the URL, and specifying that it is a Sale (not an authorization). The code to do this using the very basic wrapper I am using is:

paypal "DoExpressCheckoutPayment" [("TOKEN", token);
                                   ("AMT", amount);
                                   ("PAYERID", payerid);
                                   ("PAYMENTACTION", "Sale")]

That’s all for now - I leave the details of implementation of the other services for subsequent posts or the reader’s imagination. Working on this project has led me to believe that Ocsigen really does offer a new way of looking at web programming - there are still a few areas I’m not familiar with (for example: looking for the equivilant of ‘liftM’ for Lwt’s monads and not finding anything, I ended up just sticking entire sequences of non-cooperative code inside of Lwt_preemptive.detach’s, instead of using the cooperative Lwt_process module - probably not as efficient, but resulted in clean, easy to understand code in little time), but overall, writing web application is very quick and seems very stable - bugs have been rare and easy to fix. Plus, getting to write functional code while doing it is a definite added perk! Look for more posts about writing this application.

0 to application in a day, with Ocsigen/OCaml.

Mon, 26 Apr 2010 12:58:00 -0400

Web frameworks are a dime a dozen these days, and they all promise to deliver enormous productivity boosts without limiting your creativity. The darker side of it is that while they often allow you to create applications very quickly, sometimes they make it pretty hard to deviate from their ideal problem space. And given their origins, that makes sense: Rails was written to make Basecamp, Django was written to make a newspaper website/publishing platform (this one, I believe). If you want to create something that exists within the sphere that the authors thought they would be dealing with, great! Otherwise you are left confronting the fact that creating good frameworks is as hard as creating flexible programming languages, and you picked the wrong one.

I’ve tried out (at least to the point of hello world) at least a dozen frameworks, written a small one myself, and as in the rest of my programming, eventually was drawn to static typing and functional programming languages, as they allow me to more easily map thoughts into code (and have it work, be quick, and understandable later). So I’ve used two different Haskell web frameworks and a hand rolled together collections of libraries for small applications, but still each did not seem to make easier anything outside of the very specific problem space they were designed for: take a regular haskell program and move it onto the web without too much low level muckery. In my case, most recent applications I’ve written are pretty light on the business logic and heavy on the web interaction - web apps that allow clients to modify the sites themselves, or record and process information on a pretty basic (algorithmic) level.

The field of statically typed functional programming languages is pretty small, and the web programming sectors of those is even smaller, so an obvious thing to try after the various Haskell options was Ocsigen/Eliom, a web server/framework written as a research project on continuation based web programming in OCaml. I’d never programming in OCaml before, so this is a brief summary of what the experience of learning OCaml (already knowing Haskell) and using it with Ocsigen/Eliom was like, focusing specifically on how it stacked up when it came to productivity/flexibility (with the added challenge that it is a language that I’m not familiar with).

The application I decided to write was a simple way to log hours and generate timesheets for a part-time job I have tutoring students in (mostly) math. It would replace a set of shell scripts and a carefully maintained directory structure I had been using up until this point, and would result in being able to log hours and generate the timesheets from any computer, not just the one with the specific directory tree/scripts, and be able to more flexibly process the data.

I decided that it would be backed with an sqlite database (as I wanted it to be very simple/portable), and searching around led me to a library called ocaml-orm-sqlite, which presents a simple interface for storing data in the database, no sql (or hand written marshalling code) needed. The other advantage was that the tables it generated automatically were very humanly readable - pretty much exactly what I would have generated if I were doing it by hand. For example, one of the records from my application:

type tutee_session = 
  { year : int; month : int; day : int;
    duration : int; collected : int; tutee : tutee } with orm

Generated the following table in the database:

CREATE TABLE tutee_session (__id__ INTEGER PRIMARY KEY AUTOINCREMENT, 
year INTEGER,month INTEGER,day INTEGER,duration INTEGER,
collected INTEGER,tutee INTEGER);

After figuring out that I now had all the tools I would need, I sketched out the urls that the application would have:

/ - a summary of all the sessions/students, and to record a session.
/timesheet/YEAR/MONTH - a page to generate the timesheet.

Given that I would be the only one using this, and adding students is infrequent whereas recording sessions is frequent, and this was supposed to be a super minimal app, I decided to add students via the sqlite command-line interface, so these were the only pages I would need.

After reading the three part manual for Eliom, I felt pretty ready to start coding. So, the first step was to roughly lay out all the services I would have - mainly the GET services that would live at the two urls above, and the POST handler to actually record a post.

First opening all the libraries that I’ll need, to avoid having to fully qualify everything:

open Lwt
open XHTML.M
open Eliom_services
open Eliom_parameters
open Eliom_predefmod.Xhtml

And then creating a generic html page function, using the static html libraries (from XHTML.M and some from Eliom_predefmod.Xhtml):

let html_page body_html = 
  return
    (html
       (head (title (pcdata "Tutoring App")) [] )
       (body body_html))

This is the first advantage for Eliom - invalid html (for the most part) simply will not compile. If the ‘title’ had been left out, it gives the following error:

Error: This expression has type [> `PCDATA ] XHTML.M.elt
       but an expression was expected of type [< `Base | `Title ] XHTML.M.elt
       The second variant type does not allow tag(s) `PCDATA

Which should be (reasonably) self-explanatory.

Now I’m ready to create two placeholder services, living at / and /timesheet/YEAR/MONTH:

let summary =
  register_new_service
    ~path:[""]
    ~get_params:unit
    (fun _ () () ->
       html_page [h2 [pcdata "Summary!"]])
let timesheet =
  register_new_service
    ~path:["timesheet"]
    ~get_params:(suffix (int "year" ** int "month"))
    (fun _ (year,month) () ->
       html_page 
           [p [pcdata "Year is: "; pcdata (string_of_int year);
               pcdata ", month is: "; pcdata (string_of_int month)]])

For Haskell programmers who don’t know OCaml, this should all look similar enough - a couple of small differences: OCaml has optional parameters to functions - and they are named - that is what the ~path:[""] is doing - setting the optional parameter path equal to [""]. Lambdas are defined with the keyword fun, not \, and values/functions are defined with the let keyword.

Now that we have the urls figured out, we need to create the database backing. This is as simple as:

type tutee = 
  { name : string; type_of_tutoring : string; phone : string;
    rate : int } with orm
type tutee_session = 
  { year : int; month : int; day : int;
    duration : int; collected : int; tutee : tutee } with orm

This uses the orm library mentioned earlier to automatically generate functions to work with the database. Documentation on that can be found in the readme or in the tests.

We’ll stick the sqlite db in /tmp for now, just be keep it simple.

let db_name = "/tmp/tutees.db"

So let’s create the form that allows you to create records.

let record_form =
    (fun (yr_n, (mo_n, (da_n, (dur_n, (col_n, (tutee_n)))))) ->
         [p [pcdata "Year: ";
             int_input ~name:yr_n ~input_type:`Text ();
             pcdata "Month: ";
             int_input ~name:mo_n ~input_type:`Text ();
             pcdata "Day: ";
             int_input ~name:da_n ~input_type:`Text ();
             pcdata "Duration: ";
             int_input ~name:dur_n ~input_type:`Text ();
             pcdata "Collected: ";
             int_input ~input_type:`Text ~name:col_n ();
             pcdata "Tutee: ";
             string_input ~name:tutee_n ~input_type:`Text ();
             string_input ~input_type:`Submit ~value:"Record" ()]])

That’s a bit of code, but most is repetitive (have not completely escaped that yet!) - but it isn’t super complicated, for the most part. Eliom has many many *_input functions for various types of form inputs (including a user_type_input where so long as you can give it functions to marshall to and from strings, you can put native types in selects, hidden input boxes, etc.). According to the documentation, an int_input can take, as it’s input_type, any of [ `Hidden | `Password | `Submit | `Text ] , which should all be pretty straight forward if you’ve ever done any web programing before.

The one part that may be confusing is the ~name:’s being parameters of a lambda function that the entire form is inside. This is because when you actually create the form you create it pointed at a service, which will take parameters identical to those that your form is submitting - and the values for the ~name:’s will be filled in at that point. With that in mind, creating the service that will handle recording values seems to be the next logical step.

let record_action =
  Eliom_predefmod.Action.register_new_post_coservice'
    ~post_params:(int "year" ** (int "month" ** (int "day" ** 
       (int "duration" ** (int "collected" ** (string "tutee"))))))
    (fun _ () (y, (mo, (da, (dur, (c, tname))))) -> 
       let db_tut = tutee_init db_name in
       let tut = tutee_get ~name:(`Eq tname) db_tut in
       let t = { year=y; month=mo; day=da; duration=dur; collected=c;
         tutee=(List.hd tut) } in
       let db = tutee_session_init db_name in
         tutee_session_save db t;
         Lwt.return [])

The first thing about this piece of code is that it is something known as an action, which means it is a service that can be posted to but it does not live at a url and after running the action it reloads whatever page it was posted to from. Which seems pretty much exactly the behavior that we are looking for. Since it is an action it needs to be registered with the functions in Eliom_predefmod.Action, and the documentation indicates that it is register_new_post_coservive' (note the single quote mark) that we want to use. It is also a coservice, or more specifically a non-attached coservice, which means it is a service that can live at the same url as another service, specified only by a state param that Eliom deals with for you. By being non-attached, it means it does not live at any fixed url - it can be attached anywhere you want. This sounds a little complicated, but should make more sense when we actually use it.

The other potentially confusing thing is the way the parameters are written: (int "year" ** (int "month" ** (int "day"... - this is typing to create pairs with pairs within them - I don’t have a clear explanation as to WHY this is necessary, but given that it is, it is pretty straightforward how to write the parameter lists (those aren’t types, they are constructors from Eliom_parameters module.), and how to write the functions to handle them.

Now that we have written the form to enter records, and the handler to process them, it is time to re-write the summary service. It now looks like:

let summary = 
  register_new_service
    ~path:[""]
    ~get_params:unit
    (fun sp () () ->
      let db = tutee_session_init db_name in
      let ss = tutee_session_get db in
        html_page [h2 [pcdata "Summary!"];
          div [post_form record_action sp record_form ()];
          div (List.map (fun s -> 
            p [pcdata s.tutee.name;
                pcdata (String.concat "." 
                  (List.map string_of_int [s.year;s.month;s.day]))]) 
            ss)]

That’s a little bit of a mouthful, but most of it is either stuff we’ve seen before (the register_new_service and ~path stuff), or it is html generation (h2, div, p, pcdata, etc). Probably the most important line is where we integrate our form in, with post_form record_action sp record_form (). post_form is a function from Eliom_predefmod, that takes as arguments a service, the server information (sp, passed to the lambda by the server), a form (well, a function that takes the names of the fields as arguments and returns a form). The last parameter could be used for get params to be passed as well, so says the documentation, but it is just () (ie, unit) in this case. The last couple of lines of the page are to display all the records currently in the database, which we have selected from the database using the tutee_session_get function automatically created for us by the ocaml-orm-sqlite library.

Now all that is left is to write a minimal timesheet handler and we’ll be finished! Following a relatively similar pattern as the summary, this can be written as:

let timesheet =
  register_new_service
    ~path:["timesheet"]
    ~get_params:(suffix (int "year" ** int "month"))
    (fun _ (year,month) () ->
      html_page 
        let db = tutee_session_init_read_only db_name in
        let ss = tutee_session_get ~year:(`Eq year) ~month(`Eq month) db in
           [div (List.map (fun s -> 
                p [pcdata s.tutee.name;
                   pcdata (String.concat "." 
                    (List.map string_of_int [s.year;s.month;s.day]))]) 
                ss)]

The only thing that is new is from the orm library - if you want to filter based on some fields in a record, provided they are built in ones like ints or strings, you can use selectors like those above, ~fieldname:(\Eq value). Otherwise, you can use a~custom:(‘a -> bool)` selector.

With that done, we should be ready to go! On my system (OCaml 3.11.2, Ocsigen 1.2, ocaml-orm-sqlite pulled from github yesterday, sqlite 3.6.22), copying this code, in order (skipping the stuff that is replaced by more complete versions later) will compile with:

ocamlfind c -syntax camlp4o -package ocsigen,lwt,orm.syntax -thread -c tutees.ml

Which will result in a file tutees.cmo, which should then be copied somewhere where the owner of the Ocsigen process (found in /etc/ocsigen/ocsigen.conf) can read it, and then you should add this line to /etc/ocsigen/ocsigen.conf:

<site path="tutees">
  <eliom module="/path/to/tutees.cmo" />
</site>

Note - you might need to comment out some of the tutorial related modules in the config file, as they might take over all the urls.

And then to start ocsigen, run as root:

`ocsigen`

And you should be able to visit the site at http://localhost/tutees/. Once you visit it once, the db file will be initialized, and you can go in and add a entry in the tutee table:

sqlite3 /tmp/tutees.db
SQLite version 3.6.22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> insert into tutee (name, type_of_tutoring,phone,rate) 
               values ("Someone", "Math", "", 50);

Then you should be able to add records with the name Someone, and view them by visiting http://localhost/tutees/timesheet/2010/4 (for example, this month). Notice that you cannot add records without the right types in the fields (and the name of the tutee has to be in the tutee table - though the 500 error could definitely be improved).

Note - if you are using the current Ocsigen 1.3 - you will probably need to change the Lwt.return statements to take () (ie, unit), not [] (an empty list).

Of course this application is still pretty bare-bones, and since I wrote it a couple of days ago, and while writing this post, I’ve added a bit to it, to allow you to delete entries, actually produce useful timesheets, and make the forms use smart select’s now (ie, you pick from one of the available tutees, you do not type in their name and hope you spelled it correctly). You can check out the code for that, if you’d like.

Writing this application in Ocsigen, from conception to the roughly the state detailed in this post, took less than 24 hours, with regular obligations like sleep and work mixed in. In retrospect, that is kind of incredible, as it was not a case of copying someone else’s application and changing certain aspects, but writing it from scratch with only the official tutorial/documentation guiding me. My impression has also been that Ocsigen scales up with complexity - I haven’t yet started experimenting with the more complicated features like sessioning or temporary services, but from what I’ve read, the prospect looks pretty good. And to have that power along with the static typing of a functional programming language and the blistering speed of OCaml - I can think of one bigger project in the near future that I will probably make with Ocsigen.

A (very brief) guide to Ocaml+Ocsigen, for Haskell programmers.

Sat, 17 Apr 2010 22:09:00 -0400

Most of the time I’ve heard the suggestion that people learn OCaml first, then Haskell second. I did the reverse, and in learning (or beginning to learn) OCaml I wanted a guide aimed at Haskell programmers - people who already understand what immutability is, how functional programs are structured, static type inference, pattern matching, but had never touched the language/ecosystem before.

While picking up OCaml and then using it with the web framework Ocsigen over the last few days, here were some very brief notes that I took down of things that were not immediately apparent (and often took a lot of searching around/reading to find). The syntax is pretty easy - just check out a generic comparison or an introduction to ocaml and you should be fine; what was more difficult was getting familiar with the build ecosystem, as the core language seems a little less friendly by default than Haskell (at least with ghc —make and friends). Or perhaps that is just my bias by being more familiar with GHC/Haskell.

first step is making the toplevel more friendly: enter “topfind”!
```
#use "topfind";;
#require "batteries";; (* or whatever *)
```
this allows you to #require any package you want to have access to on the toplevel. to make it automatic whenever you run the toplevel, put the preceding in ~/.ocamlinit. Also, at least on my system, readline is not supported by it. I saw (somewhere, sorry I don’t have the link anymore) that you can start the toplevel within another program to have proper history, line based navigation, etc. I just ran it from within emacs, with tuareg-mode (C-c C-s RET), and so got all of those benefits and more. [edit], mfp mentioned on reddit that two common wrappers for the toplevel are rlwrap and ledit [/edit]
to build (and run) an executable:
```
ocamlfind ocamlc -package batteries -c some.ml
ocamlfind ocamlc -o some -package batteries -linkpkg some.cmo
./some
```
[edit] I’ve since realized (duh, perhaps) that you can also compile straight to native code, using the ocamlopt compiler. An example would be:
```
ocamlfind ocamlopt -package batteries -linkpkg -o some some.ml
```
And it is MUCH faster. (anecdotally, a naive solution to problem 3 on Project Euler ran in about 0.6 seconds via the former method, and about 0.1 seconds via ocamlopt. [/edit]

Also, notice that there is no concept of a ‘main’ function (as far as I can tell) - everything in the file is evaluated in order, so just stick you’re ‘main’ function at the bottom of the file (as anything it depends upon must already have been defined), in something like:
```
let _ = run_program
```
[edit], again, comment from mfp that proper style is to write this as:
```
let () = run_program ()
```
[/edit]
forget polymorphism! (almost) everything is specific. for example, when working with BatBig_int’s, you have to use BatBig_int.(=) to compare them. (this is of course when you are using the batteries big_int library).
when using caml4p (the preprocessor, which I started using to use an sqlite orm library), if you get a Not_found exception, it could be caused by putting -syntax camlp40 instead of -syntax caml4po in the compilation command. seems (painfully, stupidly) obvious, but the error message does not make that clear.
for ocsigen, if you need to link a package, you use something like <extension findlib-package="batteries"/> in the config file. And of course build the .cmo file with -package batteries.
everything is evaluated in order - this means that you can’t use something in a file before you have declared it. The only exception is with mutually recursive functions, which obviously need this. define them on the toplevel like:
```
let rec hello n = goodbye n
and goodbye n = n+1;;
```
Use the batteries-included library (which might already be available for your distribution) . As far as I can tell, it is about comparable (though I haven’t used it extensively) as the standard library for haskell - ie Prelude, Data.List, etc. Great documentation is available at the github page. It is a little confusing sometimes figuring out what functions are added by BatSomething and what already exist in Something. For example, the function List.map is listed in the BatList documentation, when it actually is from List. But, it is pretty easy to figure out - load up batteries in the top level and try both. BatList.map;; will tell you that it does not exist, List.map;; will show the type of the function. [edit], another option, courtesy of mfp, via reddit, is to just open Batteries, which then makes everything available in Something (neat!). This highlights that something I have not touched in this short post is the module system, because I haven’t explored it yet myself! It is reputed to be much more powerful than Haskell’s, so I’m excited to learn about it. [/edit].