I've been building some RESTful application recently and as part of that wanted to look at authentication techniques. Traditional web application will typically use form-based authentication - where the user enters their credentials on a web page and then they are authenticated for whole session. RESTful apps can use this approach, however it is different for every application and relies on server-side state, which is not ideal for scalability, and has the potential for session hi-jacking.

The alternative for RESTful apps is to provide authentication credentials with every call. Many applications have application specific means of providing credentials, which I've implemented in the past, however this time I've been particularly interested in looking at more general solutions.

HTTP supports two standard means of providing authentication information, via the 'Authorization' header: Basic and Digest. The Authorization header is extensible and the Atom community have produced an extension based on the WS-Security specification: WSSE. This has gained quite a lot of acceptance including in the REST community, and so I wanted to see how difficult it would be to implement it in my own application. The answer was that it was pretty straight-forward to get simple mechanism working.

For background on WSSE, read Mark Pilgrim's article on it.

I've spent the last few weeks adding the ability to pay by credit or debit card to an application. Our customer has picked HSBC e-Secure Payments as their payment provider so I needed to integrate with their payment API. HSBC provide two mechanism, an API so that you can capture payment information within your own site and maintain your own branding, or you can redirect to a HSBC site for payment and then they'll redirect the customer back to your own site again once payment is complete. We wanted to use the API since we want customers to remain within our application.

Firstly HSBC do not provide a developer's site with documentation, code or even frequently asked questions so I had a little difficulty getting started. All questions and requests must go to their call centre who, while they are quite helpful, are no substitute for comprehensive documentation. They will email you their specifications - but why not have them on a web site, this is the twenty-first century after all. Secondly, the documentation that they do provide is lacking in a number of areas and incorrect in others (meaning that I have run up quite a phone bill calling their support number). Worst of all, HSBC do not provide a test or integration system.

I was shocked when I realised that I would have to do all my testing in a live environment, and remember that this is for financial transactions!

Stefan Tilkov gave a very interesting talk on the problem of when to choose a specific or a generic solution. While there were no answers provided, after all it's one of those things where there can be no general answer, it was definitely an entertaining and thought-provoking talk.

He started off be describing what he saw as the typical development of an architect. One starts out, fresh from university all keen and eager to build some exciting and cool software. This is the "Enthusiastic Developer" stage. Unfortunately it turns out that what customers want is typically boring, boring, boring - "create a system that allows us to maintain Customers"; "create a system that allows us to maintain Products", and so on. This leads to the "Disillusioned Developer", repeating similar tasks day in and day out. Then, having seen that they are doing something that looks the similar each time, the newly minted architect decides to build a generic solution, a "maintain generic things" engine. Then we can spend just a little time (oh it will only take us a few months to get the framework finished) and then there is only a small amount of specific configuration required, say 20% of the over all total effort. This is the "Enthusiastic Architect".

So they spend 80% of the time building this wonderful generic solution and then then remaining 320% of the project on the rest of it. The problem being that customer's don't really want generic solutions. The form that they want to use to maintain their customers has to behave differently from the one used for products (otherwise they'd use Excel right), it's all those little exceptions that one has to introduce to work around the generic solution, or to extend the generic solution into an all signing and dancing framework that takes the time. We've all been there, well at least all of us who are honest and will admit it. This leads to the "Disillusioned Architect".

He gave a fun little quote: "Some programmers when faced with a problem turn to a generic solution... Now they have two problems."

Following on from yesterday's Clojure session, I was interested to attend Rich Hickey's second one on Persistent Data Structures and Managed References. He'd touched briefly on this topic and it whet my appetite. This talk wasn't specifically on Clojure, but on how one can use immutable data structures, separation of identity and value and managed references to solve many issues that concurrent programs face.

The key to concurrent software design is immutable data structures. If a data structure is immutable then it must always be in a consistent state. Multiple threads can never see a partial update since the values never change. If one considers the number 42, it never changes, it is always 42. Similarly if we think about a composite value, for example the date "12th March 2009", it also never changes. It makes no sense to set the month to be "January". There is another date, "12th January 2009", but this is distinct from the first one.

So, values are immutable, whether they are primitives or composite values.

Functions in functional programming (and indeed really in any form of software) work on values and derive new values. For example, the function "add" takes two immutable values and evaluates to a third one that is the sum of these two. It always produces the same output for the same input, it doesn't matter when it is called, in what order it is called in relation to other functions, it always does the same thing. It is an ideal construct for building highly concurrent programs.

I wanted to see this talk by Guido Schoomheim since Black Pepper has been worked on distributed agile development projects now for a year or so, between sites in the UK and I was interested to hear Xebia experiences. Xebia have a team distributed between their offices in The Netherlands and in India.

Guido started his talk by discussing some interesting statistics regarding the costs of traditional outsourcing. He compared the costs of doing the development locally using a "hyperproductive" agile projects verses standard outsourced waterfall development. In the standard model, when one outsources to a very distant company with a different culture and especially in a different timezone, one has to define exactly what one wants in very detailed specifications which lead directly to a waterfall approach.

His figures suggested that to get the return on investment that one needs for a successful outsourcing project over for example a two year period then the cost of the outsourced team must be about 10% of the alternate local agile team's costs, or even less. However, typically one finds that outsourced teams are about 30% of the cost of local teams. Therefore, he suggested, taking a project that could be done locally with an agile team and cost $2 million, if outsourced to a traditional waterfall team, then that same project would cost around $6 million. Very interesting figures indeed and something that I feel is right and empirically sounds right from my experience of customers who have outsources, however I want to do more research before accepting these figures.

Having laid out some of the well known problems of this traditional approach to distributed teams, especially across continents, he described how his project had addressed them.

This talk was a bit of a surprise. Linda Rising's talk, while thought-provoking, to me didn't really fit an hour's slot in an Agility Organisational Patterns track.

The premise was essentially that we as a society have been taking caffeine since the beginning of the Industrial Age and that life today is different from that of a 19th Century factory worker so we may want to re-evaluate whether the drug caffeine is still useful to us. I'm not really sure how the contents of the talk, which was all about caffeine, fits the title of the talk or the 'agile' track it was in.

She started by highlighting that before the Industrial Revolution, Europeans tended to have beer for breakfast - since water wasn't really safe to drink. However of course that didn't really lend itself to the regimented and time controlled practices required in a factory at the beginning of the Industrial Age. Tea and coffee became popular at about that time, she posited, because it could help us get up with the clock rather than the sun.

The main part of the talk was about the metabolism and concentration of caffeine in the body and discussing some studies that show that while caffeine is good at helping people concentrate on simple and long running tasks (such as those required in a factory), it has no extra benefit in terms of the sorts of tasks that software developers do. In particular she highlighted some of the issues regarding how caffeine effects are based on concentration in the body, and pointed out that drinks often marketed at children can have high levels of caffeine in them: that we "learn" to drink caffeine at an early age where we would be alarmed at giving children other drugs. This was indeed food for thought and I agree with her that perhaps we as a society ought to be re-evaluating how much caffeine we drink.

Simon Wardley from Canonical explored how IT must change in particular in light of multi-core computers and cloud computing. He covered why IT is changing and it impacts IT.

He had a very interesting presentation style and I would recommend that if you get a chance to see one his talks that you do.

The initial theme of his talk was about IT commoditisation, how today's cutting edge and hot topics are tomorrow's boredom. That software (and indeed anything) goes through a process from innovation, bespoke implementations, products and finally ubiquitous utility-like services. All business services exist somewhere on the curve from innovation through to commodity.

This lead him to ask the question of how this impacts IT. Well, in order for one to innovate in the first place then you need build on top of stable components. Without ubiquitous Internet access then one couldn't have a Google. So as we see ubiquitous services, such as Infrastructure as a Service, then we will see innovation that can build on that, such as Sales Force as Software as a Service.

I've done some work with Erlang and was impressed by its concurrency model and I'm completely sold on the idea of immutable objects, however, the JVM is an excellent platform which because of Java's prevalence in the marketplace just keeps getting better and better. So it was with interest that I attended the session on Clojure.

Clojure is a lisp dialect designed to run on the JVM.

I was particularly interested in its use of references to manage concurrency. All data structures are immutable, so they can never be in an inconsistent state. However one needs to be able to "update" state as a program runs. Effectively this means replacing the "value" (as complex as that may be) that is "identified" by the reference. So concurrency in this scenario is managing the process where new instances of the data structures are set as the value when multiple clients want to do this at once. Clojure has four types of references:

Refs, these use software transactional memory to ensure co-ordination of updates of the references. Agents, these are asynchronous processes that defer the update until a convenient point in time. Atoms, these synchronously update the references and repeatedly retries in the event of a conflict. Finally, vars, which are isolated so that they are only accessible to the local process.

This was an interesting session discussing the future of the web as a platform. The concept being that as the technology in the browser matures and grows, one might be able to move away from the plug-in platforms such as Flash and JavaFX to be solely based on browser technologies such as JavaScript and HTML.

Dion Almaer and Ben Galbraith, talked initially about a four "up and coming" technologies that could well transform the web "web platform". The first being Canvass that came out of Safari and has now spread to Firefox, Opera and Chrome. It allows a 2D drawing environment on which one can draw, opening the way for pure web applications to take control of the page, and move away from the document based text, image and rectangle model of 20th century web.

They demonstrated the Bespin in browser editor that is being developed by Mozilla Labs to show the power of Canvass, amongst the other technologies. Like Flash and JavaFX being able to draw arbitrary components natively in the browser means that one can move away from the problems of imposed by the document-centric model of today's applications.

Secondly, they discussed Fast JavaScript, with new JavaScript engines on the horizon one can see that web applications can be transformed. One will be able to do things when JavaScript is blindingly fast that one would not even consider today.

Today I'm at Qcon London, the international software development conference, held at the QEII conference centre in London.

The conference opened with a keynote by renowned computer scientist Tony Hoare. He talked about the science of computing and the engineering of software. The main thrust of his talk was that if computing is a science then software must be engineering and vice versa, and of course he posited that both were.

He made some interesting observations about the difference between science and engineering and that there is a continuum from scientist to engineer.

Science is a long term endeavour, where its theories and proofs have long lasting consequences, once discovered a theory is (potentially) applicable for decades. Whereas engineering is confined to shorter term goals, an engineer is focused on solving a particular problem.