Posts tagged ocaml
Posts tagged ocaml
disclaimer: This whole post is based on the fact that I was not able to get a certain thing working. Part of the reason to write this is a challenge to someone else to figure out how to do it, and document it. There is extremely little information out there. With that said, I tried pretty hard, and came to the conclusion that it was not possible. If you can figure out how, I will retract my claim that Ocsigen native code is not a viable option for web programming.
This only pertains to native code that is done with static linking. I was able to get ocsigen to link native code dynamically when it was the only library, but was not able to get this working with some external libraries, which only worked with static linking. If dynamic linking was working, all the acrobatics described in this post would be irrelevant. Since I was not able to, this was my experience, but consider this an enormous caveat (and it is terrible that I did not mention this originally. I stopped using ocsigen months ago because of this and other reasons, and wanted to finally get around to posting this, but forgot to mention that critical detail).
This was written very negatively. I didn’t intend it to be so - when I first started writing this post it was meant as a guide for someone else who wanted to use statically linked native code with ocsigen. Since it took a bit of work to figure out how to structure the code (and a couple false starts) I wanted to write this down so that it could benefit others. However, partly due to the code organization necessary (and for other reasons), I stopped using Ocsigen for anything but small projects (let’s say, above 800 lines of application code), and I think because of that (and due to re-writing some applications that had reached that limit) I ended up writing this much more negatively than I originally intended. I wanted to write that post, ie, why I stopped using Ocsigen, in another post, but some of it leaked in here unintentionally.
Well, I personally think it is silly to use a language that has a very fast native code compiler and not take advantage of that. But this is a valid point — and my conclusion is that indeed if you are going to use Ocsigen for any even medium sized project, you probably should not use native code.
The best references were the applications by mfp, particularly ocsiblog. It is a small blog application and also has some even smaller test applications in the repository. It was from these that I got the first native application running.
So the most important thing that you have to keep in mind is that you cannot register any service until runtime. Additionally, you canot register any service twice. These two things serve as somewhat of a death-knell to the cause of native code Ocsigen projects (at least as far as I could tell). The only workaround I could find, which is how the code in the example project by mfp works, is to wrap the entire application (or, all of the application that involves services - non-web libraries can be separate) within a single functor that takes a dummy argument. Then at runtime, you evaluate it once.
Now the reason why this is catastrophic is that it means that all your web code has to be in a single file. Trying to do it any other way, unless you have parts of your application that never interact (ie, never link to one another, never post a form to one another), and you will end up doing multiple evaluation, as far as I could tell. Possibly someone else could make this happen, but the various things I tried did not work. Take this as an open challenge. What would need to happen is that one module would orchestrate loading each one only once and passing them among one another.
Separating code that used services (or more generally, that used to use the server params argument, which is in current ocsigen passed around as Lwt thread data) from that code that is “pure” (ie, doesn’t touch ocsigen), so that the latter could be factored out into separate files (the former all being lumped into one module), was interesting, and reminded me of the isolation that haskell’s IO monad enforces. However, the lack of flexibility in dealing with the “IO” code was pretty limiting.
Until this becomes more supported, and someone figures out a way to do this easily and without drastic code reorganization, Ocsigen should be thought of as a byte-code only option. There is enough documentation to make it seem like native code is an option, if only you bother to do it, but I think that is extremely misleading, and it would be worthwhile for the Ocsigen website to make this clear.
I’ve recently been working on a project that uses Ruby On Rails, and I was reminded once again how much I like their “flash” functionality. Flash is a session-based temporary message storage facility which is displayed (if it exists) and erased every time a page is displayed, so that you can, for example, set a flash message in a controller (something like, “Successfully saved X”) and it will be shown on whatever the next page is, whether you directly render a template, redirect, etc.
I was thinking that it would be extremely useful to have something similar with Ocsigen (the ocaml web framework, for those unfamiliar with it), especially when used with Actions. Actions are (as far as I know) a uniquely Ocsigen concept - an action is a service that does not live at any given url (rather, when you create a form pointing towards it, Ocsigen uses params to match it up), does not return anything, and after being called simply reloads the current page. This turns out to be incredibly handy, as it means you can put a form pointing to this service on any page, and it will perform whatever action and then reload the current page. Of course you could do the same thing by using redirections and keeping track of urls, but it is so natural with Ocsigen that it definitely changes the way I write web applications. Spefically I’ve noticed that apps I write are much denser, because having a single page be able to have many form that post and reload (instead of the usual - only one) means that what might have in the past been a four or five page application (not for usability but simply for programmer convenience) is now one page.
So getting back to the flash, one of the major problems with actions is that they do not return anything, indeed the original page is reloaded, so unless the change is extremely obvious, the user may not notice anything even happened. Or, more of a problem, if something goes wrong in the action, there is no way of informing the user of the problem. Enter the idea of the flash. In the action, a message is stored in the session data, which is then displayed when the original page reloads.
The two aspects of this are that you need some way of storing session data, which Ocsigen makes very easy (see a previous blog post I wrote about using it for a shopping cart at blog.dbpatterson.com/post/637836183/shopping-carts-with-ocsigen), and then you need to make sure that every page displays this, if it exists, and empties it (so the same message doesn’t keep getting displayed).
The first task is quite easy (note: this, and all subsequent code on this post, is using Ocsigen 1.90):
let notification_state_name = "notification"
let notification_state:string option Eliom_references.eref =
Eliom_references.eref ~state_name:notification_state_name
~scope:`Session None
let notification msg =
Eliom_references.set notification_state (Some msg)
This first defines the name of the state, then it defines the state itself (the explicit type is included, which is that it is an eref to what might be a string and might be nothing), and finally we define a helper function to set a message in the notification state.
The second task is to retrieve, empty, and display the message on all page loads. In order to do this, we need a wrapper that we will put around all pages. The way I have done this is to include authentication in the same function, so I end up having services that look like:
(fun () () ->
require_login
(fun user -> ...))
or, in the case of pages that do not need a login, but should still display the messages:
(fun () () ->
require_none
(...))
Now these functions are responsible for taking the page output, adding the (if present) message, and then returning them. Now each of these functions is going to have to be specific to the type of page that is going to be displayed, so it makes sense to develop an abstract function that is responsible for getting and setting the notification state, as well as (in my case) checking login credentials. The way this works is I have a function that looks something like this (I’ve simplified for the sake of blog posting, but have included the login state code):
let user_state_name = "session_user"
let user_state:user option Eliom_references.eref =
Eliom_references.eref ~state_name:user_state_name
~persistent:user_state_name ~scope:`Session None
let failed_login_state = Eliom_references.eref
~state_name:user_state_name ~scope:`Request false
let gen_require_login (type a) (type b)
(template : (string option -> a -> b Lwt.t))
(session_expired : a)
(failed_login : (bool -> a))
(logged_in : (user -> a)) =
Eliom_state.persistent_data_state_status
~state_name:user_state_name () >>= fun status ->
Eliom_references.get failed_login_state >>= fun failed ->
Eliom_references.get user_state >>= fun user ->
Eliom_references.get notification_state >>= fun notification ->
Eliom_state.close_session
~state_name:notification_state_name () >>= fun () ->
(template notification (match user, status with
| Some user, _ -> logged_in user
| None, Eliom_state.Expired_state -> session_expired
| _ -> failed_login failed))
This function is a bit more complex, but it is in order to be used flexibly in different situations. Before unpacking how it works in detail, let’s look at it on a high level and see what it can be used for. It takes a series of parameters, some functions and some just plain values that all are or will be evaluated to type a. Often this might be an html page, but it doesnt have to be. The argument template is a function that takes a string option that is the notification message, and the existing page, and returns a (potentially differently typed page) that should have the notification message added to it, all wrapped inside the Lwt monad, which Ocsigen uses to organize threaded execution in a monadic style similar to Haskell’s IO monad.
The way this is written, the caller gets to decide what should happen with all the potential login states and also how the notification should be handled. Indeed, this function can be used in the case where we dont even want to check login status but we do want to display the notification:
let require_none logged_in =
let fn = (fun _ -> logged_in) in
gen_require_login
(fun notification body ->
Lwt.return (html_page (match notification with
|Some message -> [div ~a:[a_id "notification"] [pcdata message];
body]
|None -> [body])))
logged_in fn fn
This function uses a helper “html_page” that simply creates an html_page with appropriate headers, and stores the notification in a labeled div at the top of the page, before the main body. It ignores the login state by passing in tho same function for all the possible login states. This means that we actually are checking if they are logged in, we are just giving back the same page regardless.
Another extreme use case is for actions, where we want to check whether they are logged in and not execute otherwise, but we have no use for a current notification:
let action_require_login logged_in =
let fn = (fun _ -> ()) in
gen_require_login (fun _ body -> Lwt.return body)
() fn (fun u -> logged_in u)
What this has is the template function does nothing - it just passes through the body and ignores the notification. It then passes functions that do nothing in the case of all the non-logged in states, and finally performs the action if the user is logged in.
Seeing a few different ways that this function can work, let’s return to the function itself:
let gen_require_login (type a) (type b)
(template : (string option -> a -> b Lwt.t))
(session_expired : a)
(failed_login : (bool -> a))
(logged_in : (user -> a)) =
Eliom_state.persistent_data_state_status
~state_name:user_state_name () >>= fun status ->
Eliom_references.get failed_login_state >>= fun failed ->
Eliom_references.get user_state >>= fun user ->
Eliom_references.get notification_state >>= fun notification ->
Eliom_state.close_session
~state_name:notification_state_name () >>= fun () ->
(template notification (match user, status with
| Some user, _ -> logged_in user
| None, Eliom_state.Expired_state -> session_expired
| _ -> failed_login failed))
The meaning of all the function arguments should now more or less make sense, so let’s look at how they are put together. Each of these functions does something in the Lwt monad in order to get the value, which means that they have types like:
Eliom_references.get notification_state : string option Lwt.t
The way you chain monadic operations together with Lwt is identical to Haskell - the bind operator:
val (>>=) : 'a Lwt.t -> ('a -> 'b Lwt.t) -> 'b Lwt.t
What this says is, given a value 'a that is in the Lwt monad (in the example above, 'a is string option), and a function that takes a value of type 'a and returns something else also in the Lwt monad and return the second thing. So taking another look at the code above, the following line:
Eliom_references.get notification_state >>= fun notification ->
Means, get the notification and then pass it off to an anonymous function that takes one argument, namely that notification, inside of which it can be used as a pure value.
Now we’ve now looked at a lot of code, but we still have the problem that nowhere have we actually set a notification message. The action_require_login function above was simplified to the point of missing one original goal of developing this kind of messaging: if something goes wrong (like not being logged in) in an action, a message should be left. So consider instead, the following option for use with actions:
let notification_require_login logged_in =
(gen_require_login
(fun _ body -> Lwt.return body)
"Your session has expired."
(fun failed ->
if failed then "Incorrect username / password." else "")
(fun u -> logged_in u)) >>= fun msg ->
notification msg
What this does is run gen_require_login, again with a function that ignore the current notification, and then it chains what comes out of the function into the earlier notification function to set a message. Since the type of msg is string then the type of gen_require_login applied to these arguments must be string Lwt.t, which means that the template function returns that type and must therefore take values of type string. So what this means is that the action function, passed in as logged_in should return a string, which wil be set as the notification message. If the user is not logged in, informative messages are left, which will be displayed when the page that called the action is reloaded.
So this can be used to develop services that have not only the flexibility that actions provide but also a way of communicating with the user. To wrap up, here is a function that demonstrates not only how to use this notification_require_login function, but that it can also be chained on to a redirection:
let add_user_fn =
(fun () user' ->
notification_require_login
(fun user ->
if add_user user' then
"Successfully added user!"
else "Failed to add user.") >>= fun () ->
Lwt.return index)
A pretty common thing for an ecommerce website to have is some sort of shopping cart. There are ways around it - primarily by purchasing one item at a time (or even worse - filling out an order form with the item identifiers!) - but the shopping cart reigns supreme.
To implement a shopping cart, a few things usually need to happen. When someone visits the website, you need to stick a cookie on their browser, and record that on the server, referencing the data (the cart contents). Then this new data store is your cart for the request. Of course you actually first need to check if they already have a cookie - then check if the reference is good, and if so update the expiration time on the cookie and your local data, and then use the cart selected for the request. And this looking for cookies, verifying that the data exists, etc, needs to happen on every request.
Of course, a shopping cart is only one reason why you might want to do something like this - there are many reasons why you might want to store data on a per user basis (but not per login, or not even on a site where logging in is necessary) - recently viewed pages, ‘favorites’, etc.
Ocsigen (well, actually Eliom, but since eliom is distributed with ocsigen, and the former has more name recognition, I’ll use it) abstracts this all out by making it possible (and easy) to create “session data”. You simply name a place to store the data (any data type you want) and then in requests you can get the data, modify it, save it. There is no need to mess with cookies, or even worry about where the data is being stored.
It is interesting to consider this design decision - to not try to avoid the paradigm of the web entirely, as requests still are independent and accessible via sane urls (some continuations based web frameworks will only allow you to enter the site at one point, and then will store the state in a token stored in the url - completely abusing the entire idea of urls and the web). But certain low-level aspects that are tedious to handle and often necessary - like this issue of per-client data - are abstracted out. In this way Ocsigen can be seen not as yet another library to handle dealing with web programming, nor a completely different programming model that is compiled down to something that speaks the language of the web but rather a high-level language in which to write web applications.
But enough digression - here is a shopping cart mechanism from a recent project I was working on, a store to sell MP3s:
type cart = { songs : song list }
let session_data = create_volatile_table ()
let get_cart sp =
let sess_data = get_volatile_session_data ~table:session_data ~sp () in
match sess_data with
|Data c -> c.songs
|Data_session_expired -> []
|No_data -> []
let mod_cart sp fn =
let sess_data = get_volatile_session_data ~table:session_data ~sp () in
let new_sess = match sess_data with
|Data c -> {songs = fn (c.songs)}
|Data_session_expired
|No_data -> {songs = fn []} in
set_volatile_session_data ~table:session_data ~sp new_sess
And that’s it. The action to add to the cart? It’s one line, once you get past the action boilerplate:
let add_to_cart =
Eliom_predefmod.Action.register_new_post_coservice'
~post_params:(user_type song_of_string string_of_song "song")
(fun sp () song ->
mod_cart sp (fun c -> if not (List.mem song c) then c@[song] else c);
return ())
Delete is as simple:
let del_from_cart =
Eliom_predefmod.Action.register_new_post_coservice'
~post_params:(user_type song_of_string string_of_song "song")
(fun sp () song ->
mod_cart sp (fun c -> BatList.remove c song);
return ())
And the last part, to complete the cart mechanism, are the forms to post to the add and del actions. These use one of the more clever features that Ocsigen has - provided you can give it functions to turn native types to and from strings, you can put native types in html forms (as selects, or hidden inputs), you can preapply services with them, and as you saw above, you can use them as parameters to services. One of the most convenient uses of this in my application is a form that is just a button, that has one hidden element - a song. This is written as:
let mk_song_form txt s =
(fun song_n ->
[div [user_type_input ~input_type:`Hidden ~name:song_n ~value:s string_of_song ();
button ~button_type:`Submit [pcdata txt]]])
So then anywhere where I want to put a button to add a song to the cart, where the song is ‘s’, and sp is the conventional name for the server params argument to all services (see the above actions for where it appears), the code is as simple as:
post_form add_to_cart sp (mk_song_form "Add to cart" s) ()
Or to delete:
post_form del_from_cart sp (mk_song_form "Del from cart" s) ()
Pretty simple, eh?
Since writing my last post about Ocsigen, detailing the beginnings of a rapidly developed (but very simple) web application with the OCaml web framework, I’ve had occasion to use it in a much more complicated (though still relatively simple) application - a small store to sell mp3s. It took, from start to finish, about 5 days, spending about 8-10 hours per day working on it.
There are many possible posts about this experiment (/experience) that I could write, and I was constantly surprised by how quickly I was able to add features or change large portions of the application (probably the most shocking was writing an entire shopping cart mechanism in under an hour, thanks to Ocsigen/Eliom’s support for session data), but I wanted to focus specifically on the integration with Paypal, which was probably the most difficult part (and wasn’t too hard).
When starting out with a new API (and not being able to use someone else’s bindings, which is a predicament often faced when using less commonly used programming languages), it’s helpful if there is a simple step by step example for a minimum viable integration. Something that shows what every interaction with the bare API looks like, in a language agnostic way. Unfortunately, most of the examples that I could find are instead in PHP, ASP, or ColdFusion, or are spread among many different pages.
Even the closest page I could find seemed to assume you already knew how to use the API, and so ignored many details (which in turn leads to various error messages). At first I wondered how it could be that a company as big as Paypal which has an API that is so widely used could not have stellar documentation, but then I realized if most people are using wrapper written in the programming language they are using, then such an overview is not necessary, or at least broadly needed.
What’s more, the documentation that is important to those writing the wrappers (and therefore familiar with the API) is all the minutiae of various options, which is available, and NOT a brief introduction (which was what I, not having ever used it before and writing in a language that does not have bindings, wanted to read). I still think there must be some page somewhere, but eventually gave up looking!
So, I present a brief overview of a minimum viable integration (with Ocsigen/OCaml used for code examples).
First, some points about actually calling the API. This seems really straightforward once you know what it is, but the error messages weren’t very helpful (in the documentation for errors, very few in any of the ExpressCheckout flow have a recommendation for resolution, they just repeat the error message that the api sent you!)
On the OCaml end of things there are not a lot of good options for making http requests - and no native ones that worked for me. I tried many options - first, I looked at Ocamlnet’s Http_client, but realized very quickly that it did not support SSL, which made it unusable (aside from that, the interface looked pretty good).
Next was Ocsigen’s http_client library (which was of course available by default), but it didn’t seem to work - and wouldn’t spit out any debugging messages for me, making it frustrating enough to use that I moved on to the next option. I then tried the OCaml bindings to Curl (ocurl), but dynamically linking it into the application with Ocsigen kept giving errors, and since I was on somewhat of a deadline I decided to come back to it if no other options were available.
I think had I taken some more time (or been more familiar with the OCaml language - I’ve only been programming it for about a month) this could have been a good option - but equally there doesn’t seem to be a simple way to use it - it seems very C-like (setting loads of imperative options before telling it to run the call - instead of having a function that you pass a lot of arguments and get back the result of the request).
The final option, and what I ended up using, could be considered the sledgehammer solution - take a program I am familiar with and know works (wget), and call out to it using Unix.open_command. It seems ugly, and indeed I wouldn’t have thought of it if I hadn’t used (and enjoyed) a Haskell library that did the exact same thing.
So, to present a little code in the post, here is a Paypal wrapper (well, that’s giving it a lot more credit than it is due) in 21 lines of OCaml (special thanks to whoever wrote the OCaml section of Rosetta Code for the implementation of syscall - provided this is an efficient way to do that, having such a function available in standard libraries would be great!):
let api_endpoint = "https://api-3t.sandbox.paypal.com/nvp"
let credentials = [("USER", "api_username_from_paypal");
("PWD", "api_password_from_paypal");
("SIGNATURE", "api_signature_from_paypal");
("VERSION", "51.0")]
let syscall cmd =
let ic, oc = Unix.open_process cmd in
let buf = Buffer.create 16 in
(try
while true do
Buffer.add_channel buf ic 1
done
with End_of_file -> ());
let _ = Unix.close_process (ic, oc) in
(Buffer.contents buf)
let wget u pd = syscall ("wget --quiet -O - --post-data=\"" ^
(Netencoding.Url.mk_url_encoded_parameters pd) ^ "\" " ^ u)
exception Paypal
let paypal method_ params =
let ps = ["METHOD", method_]@credentials@params in
Netencoding.Url.dest_url_encoded_parameters (wget api_endpoint ps)
So, to continue on to actually implementing the ExpressCheckout, we can review from the paypal docs on it that we are supposed to first invoke SetExpressCheckout.
The raw call with wget would look like:
wget -O - --post-data="METHOD=SetExpressCheckout&USER=...&PWD=...\
&SIGNATURE=...&VERSION=51.0&AMT=10.00&RETURNURL=someurl\
&CANCELURL=someurl" "https://api-3t.sandbox.paypal.com/nvp"
In our case setting up the transaction looks like this (as all the credentials are handled by the ‘wrapper’, and the METHOD is the first argument):
paypal "SetExpressCheckout" [("AMT", amount);
("RETURNURL", returnurl);
("CANCELURL", cancelurl)]
Where returnurl is the url that paypal redirects to after the transaction, with “?token=SOMEVAL&PayerID=SOMEVAL” appended to it, and cancelurl is where it sends the customer if they cancel the transaction while at paypal.
Since we are using Ocsigen, we need to use Lwt_preemptive.detach so that this doesn’t block the whole webserver for the duration of the API call (which is going to be quite long, on the scale of a web request). A full service that sets up the paypal express checkout and redirect the person to paypal follows (the documentation says you should pass AMT, RETURNURL, CANCELURL with the redirect - doing so seems to cause it not to work, so I think the documentation is in error or out of date):
let purchase_start =
Eliom_predefmod.String_redirection.register_new_post_service
~fallback:checkout
~post_params:unit
(fun sp () () ->
Lwt_preemptive.detach
(fun () ->
try
let chop_params s =
BatString.slice ~last:(BatString.find s "?") s in
let returnurl =
chop_params (string_of_uri (make_uri ~absolute:true
~service:(preapply purchase_end ("", "")) ~sp ())) in
let cancelurl = string_of_uri
(make_uri ~absolute:true ~service:checkout ~sp ()) in
let amount = total (get_shopping_cart sp) in
let init_paypal =
paypal "SetExpressCheckout" [("AMT", amount);
("NOSHIPPING", "1");
("RETURNURL", returnurl);
("CANCELURL", cancelurl)] in
let token = List.assoc "TOKEN" init_paypal in
let redir =
"https://www.sandbox.paypal.com/webscr?cmd=_express-checkout&" ^
(Netencoding.Url.mk_url_encoded_parameters [("token", token)]) in
uri_of_string redir
with _ ->
make_uri ~service:checkout ~sp ()) ())
This of course is code that depends on other services - mainly, ‘checkout’ and ‘purchase_end’. The former is simply a place where someone can initiate the checkout process, and purchase_end is what handled what paypal sends back to us. This service also depends on having a shopping cart you can access with get_shopping_cart, but that’s a pretty minor detail. As a side note - for simplicity, a few more details have been omitted - like saving the token before you redirect (to verify when the customer is redirected back).
Also, instead of hard coding in urls for the return and cancel, I used Ocsigen’s make_uri function that can construct them based on the actual services they reference - so if I change the url of the service, it won’t affect this code at all. (Since paypal will be adding the params to the returnurl service, I chop them off of before sending the url).
So then the customer approves the payment, and is redirected back to the url specified in RETURNURL, with the GET params token and PayerID. The API says you can get call GetExpressCheckoutDetails with the token to get their email address, name, etc. This is then simply:
paypal "GetExpressCheckoutDetails" [("TOKEN", token)]
And then once you confirm the payment from them (which is a step not required technically by the API, but the ui in paypal tells them they will confirm the payment in the next step, so it is probably a good idea), you can actually make the payment happen with DoExpressCheckoutPayment, which takes both the token you’ve been using all along, the original AMT, and the PayerID that was passed back in the URL, and specifying that it is a Sale (not an authorization). The code to do this using the very basic wrapper I am using is:
paypal "DoExpressCheckoutPayment" [("TOKEN", token);
("AMT", amount);
("PAYERID", payerid);
("PAYMENTACTION", "Sale")]
That’s all for now - I leave the details of implementation of the other services for subsequent posts or the reader’s imagination. Working on this project has led me to believe that Ocsigen really does offer a new way of looking at web programming - there are still a few areas I’m not familiar with (for example: looking for the equivilant of ‘liftM’ for Lwt’s monads and not finding anything, I ended up just sticking entire sequences of non-cooperative code inside of Lwt_preemptive.detach’s, instead of using the cooperative Lwt_process module - probably not as efficient, but resulted in clean, easy to understand code in little time), but overall, writing web application is very quick and seems very stable - bugs have been rare and easy to fix. Plus, getting to write functional code while doing it is a definite added perk! Look for more posts about writing this application.
Web frameworks are a dime a dozen these days, and they all promise to deliver enormous productivity boosts without limiting your creativity. The darker side of it is that while they often allow you to create applications very quickly, sometimes they make it pretty hard to deviate from their ideal problem space. And given their origins, that makes sense: Rails was written to make Basecamp, Django was written to make a newspaper website/publishing platform (this one, I believe). If you want to create something that exists within the sphere that the authors thought they would be dealing with, great! Otherwise you are left confronting the fact that creating good frameworks is as hard as creating flexible programming languages, and you picked the wrong one.
I’ve tried out (at least to the point of hello world) at least a dozen frameworks, written a small one myself, and as in the rest of my programming, eventually was drawn to static typing and functional programming languages, as they allow me to more easily map thoughts into code (and have it work, be quick, and understandable later). So I’ve used two different Haskell web frameworks and a hand rolled together collections of libraries for small applications, but still each did not seem to make easier anything outside of the very specific problem space they were designed for: take a regular haskell program and move it onto the web without too much low level muckery. In my case, most recent applications I’ve written are pretty light on the business logic and heavy on the web interaction - web apps that allow clients to modify the sites themselves, or record and process information on a pretty basic (algorithmic) level.
The field of statically typed functional programming languages is pretty small, and the web programming sectors of those is even smaller, so an obvious thing to try after the various Haskell options was Ocsigen/Eliom, a web server/framework written as a research project on continuation based web programming in OCaml. I’d never programming in OCaml before, so this is a brief summary of what the experience of learning OCaml (already knowing Haskell) and using it with Ocsigen/Eliom was like, focusing specifically on how it stacked up when it came to productivity/flexibility (with the added challenge that it is a language that I’m not familiar with).
The application I decided to write was a simple way to log hours and generate timesheets for a part-time job I have tutoring students in (mostly) math. It would replace a set of shell scripts and a carefully maintained directory structure I had been using up until this point, and would result in being able to log hours and generate the timesheets from any computer, not just the one with the specific directory tree/scripts, and be able to more flexibly process the data.
I decided that it would be backed with an sqlite database (as I wanted it to be very simple/portable), and searching around led me to a library called ocaml-orm-sqlite, which presents a simple interface for storing data in the database, no sql (or hand written marshalling code) needed. The other advantage was that the tables it generated automatically were very humanly readable - pretty much exactly what I would have generated if I were doing it by hand. For example, one of the records from my application:
type tutee_session =
{ year : int; month : int; day : int;
duration : int; collected : int; tutee : tutee } with orm
Generated the following table in the database:
CREATE TABLE tutee_session (__id__ INTEGER PRIMARY KEY AUTOINCREMENT,
year INTEGER,month INTEGER,day INTEGER,duration INTEGER,
collected INTEGER,tutee INTEGER);
After figuring out that I now had all the tools I would need, I sketched out the urls that the application would have:
/ - a summary of all the sessions/students, and to record a session.
/timesheet/YEAR/MONTH - a page to generate the timesheet.
Given that I would be the only one using this, and adding students is infrequent whereas recording sessions is frequent, and this was supposed to be a super minimal app, I decided to add students via the sqlite command-line interface, so these were the only pages I would need.
After reading the three part manual for Eliom, I felt pretty ready to start coding. So, the first step was to roughly lay out all the services I would have - mainly the GET services that would live at the two urls above, and the POST handler to actually record a post.
First opening all the libraries that I’ll need, to avoid having to fully qualify everything:
open Lwt
open XHTML.M
open Eliom_services
open Eliom_parameters
open Eliom_predefmod.Xhtml
And then creating a generic html page function, using the static html libraries (from XHTML.M and some from Eliom_predefmod.Xhtml):
let html_page body_html =
return
(html
(head (title (pcdata "Tutoring App")) [] )
(body body_html))
This is the first advantage for Eliom - invalid html (for the most part) simply will not compile. If the ‘title’ had been left out, it gives the following error:
Error: This expression has type [> `PCDATA ] XHTML.M.elt
but an expression was expected of type [< `Base | `Title ] XHTML.M.elt
The second variant type does not allow tag(s) `PCDATA
Which should be (reasonably) self-explanatory.
Now I’m ready to create two placeholder services, living at / and /timesheet/YEAR/MONTH:
let summary =
register_new_service
~path:[""]
~get_params:unit
(fun _ () () ->
html_page [h2 [pcdata "Summary!"]])
let timesheet =
register_new_service
~path:["timesheet"]
~get_params:(suffix (int "year" ** int "month"))
(fun _ (year,month) () ->
html_page
[p [pcdata "Year is: "; pcdata (string_of_int year);
pcdata ", month is: "; pcdata (string_of_int month)]])
For Haskell programmers who don’t know OCaml, this should all look similar enough - a couple of small differences: OCaml has optional parameters to functions - and they are named - that is what the ~path:[""] is doing - setting the optional parameter path equal to [""]. Lambdas are defined with the keyword fun, not \, and values/functions are defined with the let keyword.
Now that we have the urls figured out, we need to create the database backing. This is as simple as:
type tutee =
{ name : string; type_of_tutoring : string; phone : string;
rate : int } with orm
type tutee_session =
{ year : int; month : int; day : int;
duration : int; collected : int; tutee : tutee } with orm
This uses the orm library mentioned earlier to automatically generate functions to work with the database. Documentation on that can be found in the readme or in the tests.
We’ll stick the sqlite db in /tmp for now, just be keep it simple.
let db_name = "/tmp/tutees.db"
So let’s create the form that allows you to create records.
let record_form =
(fun (yr_n, (mo_n, (da_n, (dur_n, (col_n, (tutee_n)))))) ->
[p [pcdata "Year: ";
int_input ~name:yr_n ~input_type:`Text ();
pcdata "Month: ";
int_input ~name:mo_n ~input_type:`Text ();
pcdata "Day: ";
int_input ~name:da_n ~input_type:`Text ();
pcdata "Duration: ";
int_input ~name:dur_n ~input_type:`Text ();
pcdata "Collected: ";
int_input ~input_type:`Text ~name:col_n ();
pcdata "Tutee: ";
string_input ~name:tutee_n ~input_type:`Text ();
string_input ~input_type:`Submit ~value:"Record" ()]])
That’s a bit of code, but most is repetitive (have not completely escaped that yet!) - but it isn’t super complicated, for the most part. Eliom has many many *_input functions for various types of form inputs (including a user_type_input where so long as you can give it functions to marshall to and from strings, you can put native types in selects, hidden input boxes, etc.). According to the documentation, an int_input can take, as it’s input_type, any of [ `Hidden | `Password | `Submit | `Text ] , which should all be pretty straight forward if you’ve ever done any web programing before.
The one part that may be confusing is the ~name:’s being parameters of a lambda function that the entire form is inside. This is because when you actually create the form you create it pointed at a service, which will take parameters identical to those that your form is submitting - and the values for the ~name:’s will be filled in at that point. With that in mind, creating the service that will handle recording values seems to be the next logical step.
let record_action =
Eliom_predefmod.Action.register_new_post_coservice'
~post_params:(int "year" ** (int "month" ** (int "day" **
(int "duration" ** (int "collected" ** (string "tutee"))))))
(fun _ () (y, (mo, (da, (dur, (c, tname))))) ->
let db_tut = tutee_init db_name in
let tut = tutee_get ~name:(`Eq tname) db_tut in
let t = { year=y; month=mo; day=da; duration=dur; collected=c;
tutee=(List.hd tut) } in
let db = tutee_session_init db_name in
tutee_session_save db t;
Lwt.return [])
The first thing about this piece of code is that it is something known as an action, which means it is a service that can be posted to but it does not live at a url and after running the action it reloads whatever page it was posted to from. Which seems pretty much exactly the behavior that we are looking for. Since it is an action it needs to be registered with the functions in Eliom_predefmod.Action, and the documentation indicates that it is register_new_post_coservive' (note the single quote mark) that we want to use. It is also a coservice, or more specifically a non-attached coservice, which means it is a service that can live at the same url as another service, specified only by a state param that Eliom deals with for you. By being non-attached, it means it does not live at any fixed url - it can be attached anywhere you want. This sounds a little complicated, but should make more sense when we actually use it.
The other potentially confusing thing is the way the parameters are written: (int "year" ** (int "month" ** (int "day"... - this is typing to create pairs with pairs within them - I don’t have a clear explanation as to WHY this is necessary, but given that it is, it is pretty straightforward how to write the parameter lists (those aren’t types, they are constructors from Eliom_parameters module.), and how to write the functions to handle them.
Now that we have written the form to enter records, and the handler to process them, it is time to re-write the summary service. It now looks like:
let summary =
register_new_service
~path:[""]
~get_params:unit
(fun sp () () ->
let db = tutee_session_init db_name in
let ss = tutee_session_get db in
html_page [h2 [pcdata "Summary!"];
div [post_form record_action sp record_form ()];
div (List.map (fun s ->
p [pcdata s.tutee.name;
pcdata (String.concat "."
(List.map string_of_int [s.year;s.month;s.day]))])
ss)]
That’s a little bit of a mouthful, but most of it is either stuff we’ve seen before (the register_new_service and ~path stuff), or it is html generation (h2, div, p, pcdata, etc). Probably the most important line is where we integrate our form in, with post_form record_action sp record_form (). post_form is a function from Eliom_predefmod, that takes as arguments a service, the server information (sp, passed to the lambda by the server), a form (well, a function that takes the names of the fields as arguments and returns a form). The last parameter could be used for get params to be passed as well, so says the documentation, but it is just () (ie, unit) in this case. The last couple of lines of the page are to display all the records currently in the database, which we have selected from the database using the tutee_session_get function automatically created for us by the ocaml-orm-sqlite library.
Now all that is left is to write a minimal timesheet handler and we’ll be finished! Following a relatively similar pattern as the summary, this can be written as:
let timesheet =
register_new_service
~path:["timesheet"]
~get_params:(suffix (int "year" ** int "month"))
(fun _ (year,month) () ->
html_page
let db = tutee_session_init_read_only db_name in
let ss = tutee_session_get ~year:(`Eq year) ~month(`Eq month) db in
[div (List.map (fun s ->
p [pcdata s.tutee.name;
pcdata (String.concat "."
(List.map string_of_int [s.year;s.month;s.day]))])
ss)]
The only thing that is new is from the orm library - if you want to filter based on some fields in a record, provided they are built in ones like ints or strings, you can use selectors like those above, ~fieldname:(\Eq value). Otherwise, you can use a~custom:(‘a -> bool)` selector.
With that done, we should be ready to go! On my system (OCaml 3.11.2, Ocsigen 1.2, ocaml-orm-sqlite pulled from github yesterday, sqlite 3.6.22), copying this code, in order (skipping the stuff that is replaced by more complete versions later) will compile with:
ocamlfind c -syntax camlp4o -package ocsigen,lwt,orm.syntax -thread -c tutees.ml
Which will result in a file tutees.cmo, which should then be copied somewhere where the owner of the Ocsigen process (found in /etc/ocsigen/ocsigen.conf) can read it, and then you should add this line to /etc/ocsigen/ocsigen.conf:
<site path="tutees">
<eliom module="/path/to/tutees.cmo" />
</site>
Note - you might need to comment out some of the tutorial related modules in the config file, as they might take over all the urls.
And then to start ocsigen, run as root:
`ocsigen`
And you should be able to visit the site at http://localhost/tutees/. Once you visit it once, the db file will be initialized, and you can go in and add a entry in the tutee table:
sqlite3 /tmp/tutees.db
SQLite version 3.6.22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> insert into tutee (name, type_of_tutoring,phone,rate)
values ("Someone", "Math", "", 50);
Then you should be able to add records with the name Someone, and view them by visiting http://localhost/tutees/timesheet/2010/4 (for example, this month). Notice that you cannot add records without the right types in the fields (and the name of the tutee has to be in the tutee table - though the 500 error could definitely be improved).
Note - if you are using the current Ocsigen 1.3 - you will probably need to change the Lwt.return statements to take () (ie, unit), not [] (an empty list).
Of course this application is still pretty bare-bones, and since I wrote it a couple of days ago, and while writing this post, I’ve added a bit to it, to allow you to delete entries, actually produce useful timesheets, and make the forms use smart select’s now (ie, you pick from one of the available tutees, you do not type in their name and hope you spelled it correctly). You can check out the code for that, if you’d like.
Writing this application in Ocsigen, from conception to the roughly the state detailed in this post, took less than 24 hours, with regular obligations like sleep and work mixed in. In retrospect, that is kind of incredible, as it was not a case of copying someone else’s application and changing certain aspects, but writing it from scratch with only the official tutorial/documentation guiding me. My impression has also been that Ocsigen scales up with complexity - I haven’t yet started experimenting with the more complicated features like sessioning or temporary services, but from what I’ve read, the prospect looks pretty good. And to have that power along with the static typing of a functional programming language and the blistering speed of OCaml - I can think of one bigger project in the near future that I will probably make with Ocsigen.
Most of the time I’ve heard the suggestion that people learn OCaml first, then Haskell second. I did the reverse, and in learning (or beginning to learn) OCaml I wanted a guide aimed at Haskell programmers - people who already understand what immutability is, how functional programs are structured, static type inference, pattern matching, but had never touched the language/ecosystem before.
While picking up OCaml and then using it with the web framework Ocsigen over the last few days, here were some very brief notes that I took down of things that were not immediately apparent (and often took a lot of searching around/reading to find). The syntax is pretty easy - just check out a generic comparison or an introduction to ocaml and you should be fine; what was more difficult was getting familiar with the build ecosystem, as the core language seems a little less friendly by default than Haskell (at least with ghc —make and friends). Or perhaps that is just my bias by being more familiar with GHC/Haskell.
first step is making the toplevel more friendly: enter “topfind”!
#use "topfind";;
#require "batteries";; (* or whatever *)
this allows you to #require any package you want to have access to on the toplevel. to make it automatic whenever you run the toplevel, put the preceding in ~/.ocamlinit. Also, at least on my system, readline is not supported by it. I saw (somewhere, sorry I don’t have the link anymore) that you can start the toplevel within another program to have proper history, line based navigation, etc. I just ran it from within emacs, with tuareg-mode (C-c C-s RET), and so got all of those benefits and more. [edit], mfp mentioned on reddit that two common wrappers for the toplevel are rlwrap and ledit [/edit]
to build (and run) an executable:
ocamlfind ocamlc -package batteries -c some.ml
ocamlfind ocamlc -o some -package batteries -linkpkg some.cmo
./some
[edit] I’ve since realized (duh, perhaps) that you can also compile straight to native code, using the ocamlopt compiler. An example would be:
ocamlfind ocamlopt -package batteries -linkpkg -o some some.ml
And it is MUCH faster. (anecdotally, a naive solution to problem 3 on Project Euler ran in about 0.6 seconds via the former method, and about 0.1 seconds via ocamlopt. [/edit]
Also, notice that there is no concept of a ‘main’ function (as far as I can tell) - everything in the file is evaluated in order, so just stick you’re ‘main’ function at the bottom of the file (as anything it depends upon must already have been defined), in something like:
let _ = run_program
[edit], again, comment from mfp that proper style is to write this as:
let () = run_program ()
[/edit]
forget polymorphism! (almost) everything is specific. for example, when working with BatBig_int’s, you have to use BatBig_int.(=) to compare them. (this is of course when you are using the batteries big_int library).
when using caml4p (the preprocessor, which I started using to use an sqlite orm library), if you get a Not_found exception, it could be caused by putting -syntax camlp40 instead of -syntax caml4po in the compilation command. seems (painfully, stupidly) obvious, but the error message does not make that clear.
for ocsigen, if you need to link a package, you use something like
<extension findlib-package="batteries"/> in the config file. And of course build
the .cmo file with -package batteries.
everything is evaluated in order - this means that you can’t use something in a file before you have declared it. The only exception is with mutually recursive functions, which obviously need this. define them on the toplevel like:
let rec hello n = goodbye n
and goodbye n = n+1;;
Prelude, Data.List, etc. Great documentation is available at the github page. It is a little confusing sometimes figuring out what functions are added by BatSomething and what already exist in Something. For example, the function List.map is listed in the BatList documentation, when it actually is from List. But, it is pretty easy to figure out - load up batteries in the top level and try both. BatList.map;; will tell you that it does not exist, List.map;; will show the type of the function. [edit], another option, courtesy of mfp, via reddit, is to just open Batteries, which then makes everything available in Something (neat!). This highlights that something I have not touched in this short post is the module system, because I haven’t explored it yet myself! It is reputed to be much more powerful than Haskell’s, so I’m excited to learn about it. [/edit].