Blogging with F# on GitHub Pages

Introduction

Recently I decided I wanted to move my blog from Wordpress, to a far lighter-weight platform. The platform I chose to host my new blog on was Github Pages. Basically if you have a github account you get a free pages repository where you can host a blog. You simply create a repository with a name that matches the format {username}.github.io and that's basically it. You can now create a index.htm page place it into your repository and away you go.

Unfortunately having a completely static site isn't massively useful for a blog. At the very least, you are going to want a templating engine of some sort. Fortunately Github pages, comes armed with jekyll which is a blog-aware static site generator. Jekyll relies quite heavily on having the correct folder structure, this had me chasing my tail for a moment then I found the superb poole which generates all of the layout a creates a nice looking minimalist blog. Happy Days!

To add a post you simply create a *.md or *.html and save it to the posts directory, push your changes.

Leveraging FSharp.Formatting

FSharp Formatting is a library that enables a form of literate programming, where you can embed markdown directly into a *.fsx script. Then by running a simple command line tool or a script you can convert the script into a HTML or Latex document. When your chosen out put is html you get tool-tips when you hover over terms in your code, exactly like you would in an IDE. For example,

1: 
2: 
3: 

let add a b = a + b

printfn "%d" (add 5 15)

Since Jekyll does not directly support *.fsx files. We need to extend the structure given to us by poole. The first step I took was to include Paket so I can get hold of nuget packages, that I might require for my scripts. This may seem like an odd requirement at first, but because all of my blog posts will be F# script files which are type checked by FSharp.Formatting I effectively have executable / type safe code samples on my blog :). Once paket was installed I ran

./.paket/paket add nuget FSharp.Formatting.CommandTool

This installed the F# command tool, which is a command line wrapper for the FSharp.Formatting library. Next I created a publish.bat so I have a single command to update changes to my blog

@echo off

call .paket\paket restore
call tools\fsformatting.exe literate --processDirectory --lineNumbers true --inputDirectory  "code" --outputDirectory "_posts"

git add --all .
git commit -a -m %1
git push

The script above takes a single parameter which is a commit message, this can be run like so.

./publish.bat "Added post about F# blogging"

At this point all that is left to-do is write some content in the code folder and then run the publish.bat once you have created your master piece. Well that was nearly the case for me. It turns out that jekyll requires a header at the top of each page which looks something like the following

---
layout: page
title: your post title
---

This presented a little bit of a problem as FSharp.Formatting did not have a way of just emitting raw content. Fortunately for me it does have a concept of commands in the documentation. Basically commands allow you to embed the results of computations or hide certain bits of code you may not want in you documentation (more info on this can be found here). All I have done is extend this mechanism slightly by adding an extra command raw. Which allows you to prefix a block of markup that you do not want the formatter to touch. So at the top of each post I now have something like the following,

(*** raw ***)
---
layout: page
title: your post title
----

As of writing this change is not part of the FSharp.Formatting mainline, but there is a PR. However if you deceide that you like this approach, or just want to play, I have created a github repository FsBlog that is this exact blog (minus content).

One more thing is if you are developing in Visual Studio then I highly recommend the Elucidate Extension, so you can visualize your literate scripts as you work on them.

Handling constant type unsupported error for type providers

When writing type providers you are required to define code that will run at run-time with a quotation. This in itself is not a problem, however if you try and pass a none native type to the quotation you will receive the following error,

1: 

Unsupported constant type: xxxx

There is a stack-overflow post here which has an example and a good explanation of the reasons why. A typical work around is to use each field from a record and pass it to a function call in the quotation as an array or as individual parameters. Either way this can end up being quite painful.

So how can we work around this. Well, what we need to do is build a new instance of the object we are trying to pass to the quotation within the quotation itself, and then use the variable that holds this new instance as the parameter in the function call in the Quotation. I have probably not explained that the best but the final code looks like this.

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
9: 

let recordInstance = { Name = "Tester"; DateTime = DateTime.UtcNow }

let providedMethod  = 
     ProvidedMethod("MethodName",
               [(* some parameters *)],typeof<SomeType>, 
               InvokeCode = 
                    QuotationHelpers.quoteRecord 
                         recordInstance 
                         (fun args var ->  <@@ ((%%args.[0] : SomeType).SomeMethod(%%var)) @@>))

Where the args are the original arguments passed by provided method invoke code and var is a quotation that represents our record instance to pass to our method. The implementation of QuotationHelpers is as follows.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 
31: 
32: 
33: 
34: 
35: 
36: 
37: 
38: 
39: 
40: 
41: 
42: 
43: 

module QuotationHelpers = 

    open Microsoft.FSharp.Quotations
    open Microsoft.FSharp.Reflection

    let rec coerceValues fieldTypeLookup fields = 
        Array.mapi (fun i v ->
                let expr = 
                    if v = null then simpleTypeExpr v
                    elif FSharpType.IsUnion (v.GetType()) then unionExpr v |> snd
                    elif FSharpType.IsRecord (v.GetType()) then recordExpr v |> snd
                    else simpleTypeExpr v
                Expr.Coerce(expr, fieldTypeLookup i)
        ) fields |> List.ofArray
    
    and simpleTypeExpr instance = Expr.Value(instance)

    and unionExpr instance = 
        let caseInfo, fields = FSharpValue.GetUnionFields(instance, instance.GetType())    
        let fieldInfo = caseInfo.GetFields()
        let fieldTypeLookup indx = fieldInfo.[indx].PropertyType
        caseInfo.DeclaringType, Expr.NewUnionCase(caseInfo, coerceValues fieldTypeLookup fields)

    and recordExpr instance = 
        let tpy = instance.GetType()
        let fields = FSharpValue.GetRecordFields(instance)
        let fieldInfo = FSharpType.GetRecordFields(tpy)
        let fieldTypeLookup indx = fieldInfo.[indx].PropertyType
        tpy, Expr.NewRecord(instance.GetType(), coerceValues fieldTypeLookup fields)

    and arrayExpr (instance : 'a array) =
        let typ = typeof<'a>
        let arrayType = instance.GetType()
        let exprs = coerceValues (fun _ -> typ) (instance |> Array.map box)
        arrayType, Expr.NewArray(typ, exprs)

    let createLetExpr varType instance body args = 
        let var = Var("instance", varType)  
        Expr.Let(var, instance, body args (Expr.Var(var)))

    let quoteUnion instance = unionExpr instance ||> createLetExpr
    let quoteRecord instance = recordExpr instance ||> createLetExpr
    let quoteArray instance = arrayExpr instance ||> createLetExpr

And thats it. Hopefully this should remove some pain points in developing type providers.

Option Operators

We often have to represent the absence of data within a collection of values. In C# the default type to-go to is an Nullable, F# 3.0 introduced the Microsoft.FSharp.Linq.NullableOperators module to help with comparisons and arithmetic over this type.

The Nullable type can still be somewhat tiresome to use in F# though as it can lead to adding lots of type constraints for example

1: 
2: 
3: 

type NullableSeq<'a when 'a : (new : unit ->'a) 
                    and 'a : struct 
                    and 'a :> ValueType> = seq<Nullable<'a>>

I think a nicer approach is to replace Nullable with Option, to relax some of these type constraints.

1: 

type OptionSeq<'a> = seq<Option<'a>>

however in doing this we have lost the nice operators that are available for the type Nullable<'a>. But this is fairly easy to recreate as the semantics of Nullable and Option are approximately the same. So with a quick copy of the NullableOperators module and some simple find and replace we have the exact same set of operators.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 
31: 
32: 
33: 
34: 
35: 
36: 
37: 
38: 
39: 
40: 
41: 
42: 
43: 
44: 
45: 
46: 
47: 
48: 
49: 
50: 
51: 
52: 
53: 
54: 
55: 
56: 
57: 

module OptionOperators =

    let (?>=) (x : Option<'T>) (y: 'T) = x.IsSome && x.Value >= y

    let (?>) (x : Option<'T>) (y: 'T) = x.IsSome && x.Value > y

    let (?<=) (x : Option<'T>) (y: 'T) = x.IsSome && x.Value <= y

    let (?<) (x : Option<'T>) (y: 'T) = x.IsSome && x.Value < y

    let (?=) (x : Option<'T>) (y: 'T) = x.IsSome && x.Value = y

    let (?<>) (x : Option<'T>) (y: 'T) = not (x ?= y)

    let (>=?) (x : 'T) (y: Option<'T>) = y.IsSome && x >= y.Value

    let (>?) (x : 'T) (y: Option<'T>) = y.IsSome && x > y.Value

    let (<=?) (x : 'T) (y: Option<'T>) = y.IsSome && x <= y.Value

    let (<!--?) (x : 'T) (y: Option<'T>) = y.IsSome && x < y.Value

    let (=?) (x : 'T) (y: Option<'T>) = y.IsSome && x = y.Value

    let (<>?) (x : 'T) (y: Option<'T>) = not (x =? y)

    let (?>=?) (x : Option<'T>) (y: Option<'T>) = (x.IsSome && y.IsSome && x.Value >= y.Value)

    let (?>?) (x : Option<'T>) (y: Option<'T>) = (x.IsSome && y.IsSome && x.Value > y.Value)

    let (?<=?) (x : Option<'T>) (y: Option<'T>) = (x.IsSome && y.IsSome && x.Value <= y.Value)

    let (?<!--?) (x : Option<'T>) (y: Option<'T>) = (x.IsSome && y.IsSome && x.Value < y.Value)

    let (?=?) (x : Option<'T>) (y: Option<'T>) = (not x.IsSome && not y.IsSome) || (x.IsSome && y.IsSome && x.Value = y.Value)

    let (?<>?) (x : Option<'T>) (y: Option<'T>) = not (x ?=? y)

    let inline (?+) (x : Option<_>) y = if x.IsSome then Some(x.Value + y) else None
    let inline (+?) x (y: Option<_>) = if y.IsSome then Some(x + y.Value) else None
    let inline (?+?) (x : Option<_>) (y: Option<_>) = if x.IsSome && y.IsSome then Some(x.Value + y.Value) else None

    let inline (?-) (x : Option<_>) y = if x.IsSome then Some(x.Value - y) else None
    let inline (-?) x (y: Option<_>) = if y.IsSome then Some(x - y.Value) else None
    let inline (?-?) (x : Option<_>) (y: Option<_>) = if x.IsSome && y.IsSome then Some(x.Value - y.Value) else None

    let inline ( ?*  ) (x : Option<_>) y = if x.IsSome then Some(x.Value * y) else None
    let inline ( *?  ) x (y: Option<_>) = if y.IsSome then Some(x * y.Value) else None
    let inline ( ?*? ) (x : Option<_>) (y: Option<_>) = if x.IsSome && y.IsSome then Some(x.Value * y.Value) else None

    let inline ( ?%  ) (x : Option<_>) y = if x.IsSome then Some(x.Value % y) else None
    let inline ( %?  ) x (y: Option<_>) = if y.IsSome then Some(x % y.Value) else None
    let inline ( ?%? ) (x : Option<_>) (y: Option<_>) = if x.IsSome && y.IsSome then Some(x.Value % y.Value) else None

    let inline ( ?/  ) (x : Option<_>) y = if x.IsSome then Some(x.Value / y) else None
    let inline ( /?  ) x (y: Option<_>) = if y.IsSome then Some(x / y.Value) else None
    let inline ( ?/? ) (x : Option<_>) (y: Option<_>) = if x.IsSome && y.IsSome then Some(x.Value / y.Value) else None

F# and Raven DB

Ok so I have blogged about these to before. Previously, F# didn't have brilliant support for Linq. F# Expressions couldn't easily be converted to the Linq expression trees that the RavenDB API required. This caused somewhat of a mis-match between the two which made Raven difficult but not impossible to use from F#. My previous blog introduced a library which is available for Raven clients prior to version 2.0, to bridge this gap, and tried to make Raven more natural to use from F#. However as of Raven 2.0 this library has been removed. The reasons are explained here. I don't disagree with the reasons ayende cut the support, I wouldn't want to support something I had little knowledge of either. However things have changed.... :)

The advent F# 3 and query expressions

So we are now in the era of F# 3.0 and things have changed somewhat. F# is now truly in the data era... Information Rich programming is an awesome feature of F# manifested in the form of Type Providers and Query Expressions. If you haven't read about or don't know what type providers are then I encourage you to check them out here. Type providers are not really applicable for use with RavenDB see it is schemaless so for the rest of thispost we will focus on Query Expressions. It is this feature that means the gap between Linq and F# no longer exists. If you are familiar with

var result = 
    (from x in xs do
     where x.Published >= someDate
     select x.Name).ToArray()

then the query expression syntax shouldn't feel that different,

1: 
2: 
3: 
4: 
5: 
6: 

let publishedOn date xs =
    query {
           for x in xs do
           where (x.Published >= date)
           select x.Title
      } |> Seq.toArray

So What about RavenDB?

Using RavenDB from C# is well documented and things are not that different when using it from F#. The in's and out's are well known and lets face it the API is your safety net. It doesn't let you kill yourself, in fact you have to try very hard to actually do anything really bad in RavenDB. This is I think the best feature of RavenDB.

So, what are the things that we need to be aware of when using RavenDB from F#? First things first, initializing the document store. This can be done pretty much the same as in C#

1: 

let store = new DocumentStore(Url = "http://localhost:8080")

and this is all well and good if we just use straight forward POCO objects. But what if we want to use F# record or Union types? We need to make a few simple changes. Lets first consider F# record types, all we need to do here is declare the Id property as mutable.

1: 
2: 
3: 
4: 
5: 
6: 

type BlogPost = {
    mutable Id : string
    Title : string
    Body : string
    Published : DateTime
}

Simple eh?, but what about Union types, Maps, F# lists? These are a little more complex as Json.NET doesn't do the correct thing to serialize these out of the box. However Json.NET and the internalised Raven counterpart does have an extension point and a UnionTypeConverter or MapTypeConverter as Json.NET implementations can be found here. To use this we need to modify our document store setup a little to include the converter.

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 

let customisedStore =
    let customiseSerialiser (s : Raven.Imports.Newtonsoft.Json.JsonSerializer) =
        s.Converters.Add(new Json.MapTypeConverter())
        s.Converters.Add(new Json.UnionTypeConverter())

    let store = new DocumentStore(Url="http://localhost:8080")
    store.Conventions.CustomizeJsonSerializer <- (fun s -> (customiseSerialiser s))
    store.Initialize()

Querying raven

With the document store setup we can now start querying Raven. As with any Raven query we need to create a session from our document store, then we need to create the query.

1: 
2: 
3: 
4: 
5: 
6: 
7: 

let getPostsAsOf asOfDate = 
    use session = customisedStore.OpenSession()
    query {
        for post in session.Query<BlogPost>() do
        where (post.Published >= asOfDate)
        select post
    }

The above creates a query that is pending execution. To execute this we can run,

1: 
2: 
3: 

let posts = 
    getPostsAsOf (DateTime.Now.AddDays(-2.).Date) 
    |> Seq.toArray

this will give us an array of BlogPosts published from 2 days ago. Notice we had to enumerate to an array for the query to actually be executed. This is the same execution semantics as C#; and it is important to realise that there isn't really any magic going on in the world of F#, it is still just a .NET lanuguage it still compiles to CIL and is fully interoperable with any other .NET library.

Indexes

OK so things haven't got much better here in terms of static indexes. Basically you still need to define them in C# and then you can extend the document initialization process by including the assembly that contains the index definitions. However in Raven 2.0, dynamic indexes and promotion to permanent indexes have been massively improved, which reduces the need to create static indexes.

F# end to end

Introduction

Recently my colleague @simontcousins published an article comparing to similar projects one written in C# one written in F#. The conclusion is that for the two projects the F# one is about 4% of the size. You can read more about this here.

Now normally, a typical project comprises of several languages / technologies.

• Application Code -> C#
• Build -> MSBuild/Nant
• Deploy -> Powershell script/Batch script
• UI (Web) -> HTML/Javascript
• Documentation -> Word/HTML

However the F# project in question here has a profile more like this,

• Application Code -> F#
• Build -> F#
• Deploy -> F#
• UI (Web) -> HTML/CSS/Javascript
• Documentation -> F#

Note the stack is 'almost' completely F# (we actually could have made this completely F# see below). Maybe I'm biased but I think that this is a huge benefit for maintainability I only need to have my F# brain switched on and I can change any part of the application. Sure I have to learn a few libraries, but the playfulness offered by the REPL here makes testing and experimenting easy; this is especially true when it comes writing build and deployment scripts, which I normally find quiet difficult to test.

Full stack F#

So is it possible to use F# for everything Build, Deployment, UI (Web/WPF), Documentation. The answer is simply YES!! There is just a few libraries we need to know about to achieve this. Lets look at each area of a typical application and see which libraries we can use.

Application Logic

Ok this might seem like an obvious one, just write F#, but I thought I'd use this as an opportunity to highlight some libraries that I find useful

• FSharpx - Contains lots of extensions to the F# core modules, many useful data structures, commonly used monads, validation, type providers, async extensions etc.
• FSharp.Data - Contains implementation of various type providers such as CSV, XML and JSON and I'm sure as the library matures we will see a lot more implementations to connect to wide variety of data sources.

Build and Deployment

To build and deploy application with F# we can use FAKE. FAKE allows you to write your build and deployment scripts in F#. A simple script might look something like this

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 

// include Fake libs
open Fake

//Define Targets

Description "Cleans the last build"
Target "Clean" (fun () ->
    trace " --- Cleaning stuff --- "
)

Target "Build" (fun () ->
    trace " --- Building the app --- "
)

Target "Deploy" (fun () ->
    trace " --- Deploying app --- "
)

//Define Dependencies
"Clean"
  ==> "Build"
  ==> "Deploy"

//Start Build
RunParameterTargetOrDefault "target" "Deploy"

The really nice thing about this is you have the full .NET framework and the full power of F# available in your scripts. Additionally FAKE also has an accompanying website, which allows you to manage your deployment agents, so you can easily deploy, upgrade or rollback applications from a central place.

UI

How you tackle writing a UI in F# obviously depends on your choice of technologies. There are others obviously but the two I'm going to consider are WPF and HTML/CSS/Javascript.

WPF

If you choose to write your UI using WPF then things are fairly straight forward, Write XAML markup and then use code behind F#. However wiring up the code behind can be a pain because of the C# code gen involved, typically you have to load the view yourself using a XAML reader, or code the view layout in F# as in these series of examples from John Laio. If however you want to keep XAML markup then FSharpx has a XAML type provider that helps things along here.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 

open System
open System.Windows
open System.Windows.Controls
open FSharpx

type MainWindow = XAML<"libs/MainWindow.xaml">

let loadWindow() =
    let window = MainWindow()
    window.myButton.Click.Add(fun _ ->
        MessageBox.Show("Hello world!")
        |> ignore)
    window.Root

[<STAThread>]
let main(args) = (new Application()).Run(loadWindow())

See Steffan Forkmann's post for more information about this. Lets also not forget about first class events and the asynchronous programming model in F# that makes writing responsive UI's that much easier.

HTML/CSS/Javascript

If you go webby then things get a little more interesting, we have several options available here.

• WebSharper - This is a very complete web framework, I have not personally played with it yet but I understand it does everything you'd want and more. Well worth checking out.
• FunScript - A new project, but it exploits the power of type providers to offer strongly typed access to TypeScript files @thomaspetricek has a nice talk about it. With fun script you still have to write the HTML markup, although providing a Combinator library to emit the markup wouldn't be that difficult. (If some one knows of one I'll add it here), but this isn't application logic anyway. The important stuff is still written in F# and compiled to javascript later on.

Documentation

@thomaspetricek recently released FSharp.Formatting we can generate documentation from an F# script file, using a mix of markdown and F#, for an example see this link. This can then be integrated into your FAKE build script, using a build target something like the following example, Incidentally this is exactly the library that is used to create this blog.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 

open System.IO
open FSharp.Literate

Target "Docs" (fun _ -> 
    let template = Path.Combine(currentDirectory, "template.html")
    let sources = Path.Combine(__SOURCE_DIRECTORY__, "samples")
    let output = "out_dir"
    
    Literate.ProcessDirectory(sources, template, output)
)

Conclusion

Ok I might of over stated that everything can be written in F#, certainly at the moment there is a lack of libraries for emitting XAML and HTML (with the exception of WebSharper of course). But HTML and XAML are really inconsequential when it comes the to the correctness of your application they do not contain any logic which needs to be tested they simply control layout. The important thing to take from the above is the fact that all of your logic whether it is for build, deployment, UI interaction, example documentation can be written in a type safe way using the various libraries and F# feature spoken about above.