This post follows on from previous posts about the way markup is sometimes created.

In a previous post, I worried about some web frameworks that abstract away the creation of markup, thus preventing (or making it more difficult for) web developers to create what they need — the leaky abstraction.

This post explores one way to overcome this, using XSLT.

As this is a rather long post, here are the key points:

• Creating good quality markup is important for a variety of reasons (quick loading pages, cross browser compatibility, accessibility, etc).
• Some server side frameworks may either generate poor quality markup or use templating systems that are specific to that framework.
• Using XSLT to generate the output can be one way to overcome these limitations, while giving you full control of the markup that you need to be create.
• Markup generation via XSLT may be simpler to unit test as part of a developer’s daily work, which may not be the case (or not necessarily as simple) in other templating systems.
• Being platform agnostic, XSLT can be an effective part of a View in an MVC framework and be applied to different frameworks and platforms, reducing upskilling costs in the long term.
• XSLT may not always be appropriate in all cases, and many are certainly put off by its oddities or perceived limitations, but perhaps it deserves another look?

(Some forth-coming posts will provide some code-examples, in particular of unit-testing XSLT and how it might fit into MVC as well as how it can be written in a reusable manner.)

Poor quality markup from the server side.

A previous post on WYSIWYG editors described how the output quality from editors that hide content markup generation can result in markup that is of poor quality, hard to customize/correct/change, affect accessibility (which is a legal requirement in some countries), result in excess bloat, etc.

Some server-side techniques, tools, and frameworks can contribute to this poor markup. This can happen for number of reasons. For example,

• Popular IDEs may encourage the use of WYSIWYG editors to drop in and drag around HTML controls (or groups of them). As argued in that previous post, WYSIWYG as a paradigm for HTML may actually be inappropriate, (unless you treat WYSIWYG as WYSIWYG-when-css-is-not-enabled).
• Popular frameworks may abstract away the creation of HTML, to do it for you. ASP.NET and GWT are such examples. But, as argued in a previous post, such abstractions are leaky.
• Some popular scripting environments have also encouraged spaghetti code (mixing markup, presentation, and business logic, for example).

And so it seems there is a gap between web standards and server-side output generation that is web standards-friendly.

Web application frameworks help immensely

Various frameworks have been commonplace for a long time now. These help separate out the presentation, business/mid-tier and data access logic.

Most frameworks are not solely about markup generation (if that is their priority at all). Frameworks have been important to address a number of important issues from data access, scalability, performance, transaction management, object-relational mappings, code maintenance, and so on.

These are certainly of value and usually worth using as they have teams of people devoted to them. At their best, they empower the framework user (developers and designers, for example). Yet, on the other hand, some frameworks may take some things too far, thus disempowering framework users.

Some frameworks still produce poor quality markup

Those of you who have used ASP.NET 1.x will know what a mess its controls create in terms of markup output. (ASP.NET 2 is far better, but can still encourage poor markup or make full control trickier than it needs to be.)

While server side programming using the .NET framework, and C# in particular, are extremely powerful, when it comes to creating web sites, that power is used to typically generate poor quality markup (even though it may be maintainable!).

Some of this stems from the fact that although it tries to hide away the markup creation, ASP.NET is a leaky abstraction. (This isn’t just a problem with ASP.NET, but potentially anything that tries to make markup creation a hidden, dark art!)

Abstraction: a good idea but leaky

Abstraction and loose coupling are certainly important, especially between layers such as presentation, business/middle tier logic and data access. It is a common approach to writing software.

Understandably, developers often like the idea of abstracting away markup/output creation, too. In ASP.NET for example, you could create a simple site without almost writing any HTML. Instead, you instantiate/declare controls such as asp:Label, asp:Panel etc, and you trust .NET to create the markup for you(!).

Interesting attempts are even available so you don’t have to write JavaScript and you get the benefit of tool support in IDEs for .NET and Java. For example, Script# enables developers to write code in C# (or any .NET language) and then compile it into JavaScript. Google has also been doing such a thing for a long time with its Google Web Toolkit (GWT) for Java. Such ideas are certainly very interesting and appealing.

While appealing perhaps (and maybe more to back end developers), a web developer also loses the ability to control how the output is created.

Gaining control of how you create the output

Why does it matter?

Being able to control how you create the output ultimately gives you greatest flexibility, especially when you need to create a flexible, rich, yet lean web-based UI.

Using the “web standards” approach (i.e. following W3C recommendations) to achieve these things most efficiently has been well thrashed out for years.

And, just as developers take pride in the quality and readability of their server side code, web developers increasingly take pride in the sites and web apps they create from the code perspective, too.

People from anywhere can easily see your markup; less will see your server side code (though that is not an excuse to write poor server side code!).

Although this is by no means the only factor, when hiring we will often look at sites that people have mentioned in their CVs/resumes (or ask for examples to be sure to see what is theirs) and try to understand how they have coded bits.

Some frameworks give full control but may use specific templating languages

Templating languages that accompany many popular web development frameworks may be open, but use mechanisms specific to that framework.

But there are clearly advantages to this approach, too (else they wouldn’t be so common).

For example, templates based on HTML, combined with the language of that framework (e.g. PHP, or Python, or Ruby, or C#, etc.) means there is some familiarity for developers to get on with the task at hand.

Consultants and contractors moving between frameworks

If you work as a consultant or contractor for numerous clients, you may find yourself moving from project to project and then having to learn different templating languages, some with more limitations than others.

In some cases it may be that you end up limiting yourself to just advising on markup, or only providing XHTML/CSS/JavaScript template files, which other coders then have to reproduce.

I have talked to many who have been frustrated at this process, especially with .NET, it seems, as their nicely validating, clean, accessible markup becomes something completely unrecognisable, when ASP.NET gets its way! Unnecessary trade-offs then result.

At work, we have recently been hiring both permanent and contract web developers mostly for work with .NET platforms. We have found it hard to find people with exceptional skills at both the front end and .NET back-end. We tend to find either good front end skills but a different back end (PHP, Java, etc.) or good ASP.NET but little or no real front end knowledge.

In some instances, this is not a problem, but other times it has been preferable to have .NET and good client side skills.

XSLT as the templating mechanism can perhaps help achieve the best of both worlds:

XSLT — an open standard

XSLT could help achieve both abstraction and full control of the markup.

XSLT is a templating language for transforming XML documents into another format (another XML document or some other text-based format).

For web development this can be useful to output exactly what you want and thus gain full control of the markup output generation.

You can re-use XSLT, and even create a hierarchy of XSLTs, that follow a Method Template design pattern, similar to a class hierarchy in Object Oriented Programming, whereby the base XSLT is equivalent to an abstract base class, and other XSLTs that import the base one can override various templates. (A future post will demonstrate this.)

It does of course assume you can easily (or without much cost) get your data in XML format, so you can transform it.

XSLT in MVC

Web developers are familiar with separation of concerns (semantic HTML markup for good document structure, CSS for style/layout, JavaScript for behaviour) and progressive enhancement (or graceful degradation).

Back-end/mid-tier software developers are also familiar with separation of concerns (n-tier architectures, single responsibility per class, MVC, etc.)

The Model-View-Controller (MVC) pattern for developing user interfaces has been around for a while. It is a decent approach to separating concerns (the model to represent the data, the view to represent the presentation, and the controller to bind them together).

It can also be seen at a high level (a typical n-tier architecture), as well as being used within the presentation tier (the model may interact with the middle tier, the controller will be part of the presentation tier passing the data (model) to the view code to display it).

It is a well-proven pattern in graphical/desktop application development and being successfully applied by many web-based frameworks today.

MVC can also help improve testability. For example, unit testing can sometimes be easier by testing specific views, models and controllers in separation.

Microsoft has recently announced an MVC model for ASP.NET development. I understand one of the many drivers is the unit testability.

From an initial read, this sounds quite good. It encourages better separation. Interestingly it also seems not to support their post-back model which most of their controls are built from. It is therefore not clear to me at the moment if this means you write out more HTML (meaning you get more control of what you output) or if you still use packaged controls (which may risk getting HTML that is not appropriate, or hard to control).

But Microsoft is a bit behind on MVC for web development (though that can also mean it can learn from the earlier frameworks). Other popular frameworks have existed for a while that do this for the web, such as Ruby on Rails, Django, CakePHP, and many more etc. (I am quite impressed with the Python-based Django, for example, even though I have only had a cursory exposure to it — and I usually prefer curly brace type programming languages!)

While these all have their own merits, they all have different ways of constructing the markup.

XSLT, then, could be part of the View part of an MVC framework. (It need not be the only way to create the view, of course.) For example,

• A Controller can invoke a Model to get data the way it needs to
• The Controller can pass the Model to the View
• The View can internally serialize that data into XML and transform it with XSLT to the output.

In this way, the markup generation is still nicely abstracted away, but inside that abstraction itself, you have full control of what you need to output.

Advantages/Benefits of an XSLT-based approach

Here are a few more detailed reasons XSLT can be useful (though, of course, some of these reasons are by no means unique to an XSLT-based approach).

XSLT is an Open Standard

People often talk about the benefits of using open standards on the web client side of things (”web standards”, for example), so why not on the server side?

XSLT 1.0 also has reasonably universal support. (Universal 2.0 support would be even better, though we are not there yet, unfortunately.)

Better Separation of the View from the Model and Controller in MVC

A follow-up blog post will demonstrate this in PHP and, time-allowing, Mono/ASP.NET.

Unit Testable

Many framework templates have been tricky, if not impossible, to fully unit test in isolation from other layers and dependent systems.

ASP.NET’s new MVC model mentioned earlier has unit testing and test driven development as a driver for this approach, but from what I understand, while it makes controllers and models unit testable, the views are still not (please correct me if I am wrong in the comments below!).

From what I have seen of the other View templates in other frameworks, I believe the same constraint applies (but please let me know if I have missed something).

XSLT, however, requiring mostly an XML input (and possibly arguments passed in programmatically), can therefore make it easier to unit test the view, in isolation of a running web server, database, etc. It can therefore be easier to integrate into a developer’s coding process.

XSLT-based unit tests don’t necessarily need a whole web server set up to execute. Such unit tests, which are typically numerous by their nature, would be quick to run for the developer during development. This is important in the adoption of unit testing so it doesn’t make running them feel like watching paint dry after each modification!

Another follow-up blog post will attempt to demonstrate this.

Gain Full Control of the Markup Generation

As explained above, markup is important. XSLT gives you the ability to output the markup as you want, with little or no trade-off. (Some other frameworks also allow this, so this isn’t a unique advantage, by any means.)

Platform and Framework Agnostic

While the suggestion of using XSLT may introduce a problem of yet another technology, it may also enable skills-standardisation across platforms. This would be useful for say web developers that need to work on multiple projects, for different clients, using different technologies.

This of course makes the assumption that XSLT would be widely adopted in this way! If it were, migrating to desirable new frameworks where the View at least is somewhat XSLT-driven may not be as time-consuming (though of course time is still required on the middle-tier and database layers).

I have, for example, migrated an XSLT-driven site based on “classic” ASP, to PHP-driven XSLT, using the same pattern, but where the XSLT hardly changed at all, saving a fair chunk of time.

For a major client of ours at work, their extremely high-traffic .NET 1.1 web site (now moving to .NET 2.0) was architected to by-pass the .NET controls, to avoid all the problems described above, and we transformed XML with XSLT.

Doing this gave us a number of benefits all in one go:

1. We got the lean — and accessible — output needed
2. The “object-oriented” XSLT-based approach also allowed us to create reusable template hierarchies
3. The customer also wanted multiple sub-sites which was achieved with very little effort or duplication of code, while allowing each sub-site to be customised as needed — not just CSS or database-driven data differences, but template level changes such as markup and layout changes. (Had we attempted this with ASP.NET, we would have had to create Server Controls with lots of configuration and then in each sub-site we would have to wire them all up, which would lead to a lot of repetition.)
4. In one case, it wasn’t just separate sub-sites they wanted, but entirely separate web sites/applications (a public facing one, a kiosk-mode one, and a version of the site that could be used by customer support). All this ran off pretty much the same XSLTs with minor adjustments in inherited XSLTs for each web site.
5. The migration to .NET 2 involved no XSLT change (only C# changes to take advantage of things like generics, the improved XSLT processor, etc.)

And the sites comfortably take thousands of orders an hour on very few web servers (though of course performance is a bigger architectural issue; the point here is that XSLT is not a performance hindrance that some believe — see below for more on this).

Portable skill

Being both platform and framework agnostic makes this a portable skill. Of course, XSLT is not limited to just web-based scenarios, either, so it may be quite a good skill to have, generally.

Repeatable

Just as OO techniques and design patterns help with repeatability in different contexts, or just as adopting web standards/CSS-based layout helps to create a repeatable process that can be applied across any web site you wish to develop, using an XSLT-approach can apply to many different projects and in different frameworks.

Development Productivity; Help Inject Key Skills into Project Delivery

I have consulted at a few companies, and read/heard others describe similar scenarios, whereby skilled client side developers are limited to providing templates they throw over to the developers who are not often web developers.

This can result in either extra time explaining the required markup to the developers. Time may also be needed in working out alternatives (and unnecessary trade-offs) because of limitations in the framework of choice, or garbled markup coming out of the development process (ASP.NET, looking at you!).

This past year at work, we have been aggressively hiring for web development positions, looking for a mixture of strong web standards, and server side programming with .NET. We have found it incredibly hard to hire candidates who are exceptional in all these areas (many good web developers seem to avoid .NET for example, and many ASP.NET experts we find know little on web standards).

But we don’t always need everyone to have all those skills, either; the separation afforded by XSLT has allowed us to hire really strong people with parts of these and get them to work in parallel with others. Those well versed in web standards seem to find XSLT easier to adopt, in my experience, especially if full-blown, back-end programming is not their usual area.

Such developers are therefore not excluded from the development process.

Getting the key skills into the project in this way gives a number of benefits, including

• Greater productivity and adaptability: as project requirements change even during development, having more skilled, specialist staff than generalists in all these areas can pay off. (Generalists can often be extremely valuable too, not knocking that at all!)
• Parallel development. We have found it incredibly powerful when development phases are tight and mid-tier and client side development have to start somewhat in parallel. Some of our front-end guys who have not got skills in C# and ASP.NET are nonetheless able to fully contribute to the development process, and the project gains from their in-depth knowledge of the client side.

Encourage Front and Back-end Developers to Work More Closely

During day 1 of @media AJAX, interface designer Mike Stenhouse gave a talk from the perspective of a designer/front end developer having to know more about development, getting more involved in agile development, test-driven, even behaviour-driven development.

Designers understanding more software development processes, getting involved in it – not just throwing templates over the proverbial fence – would help communicate the importance of the “lowly” client side technologies.

At the same time, mid-tier developers get a chance to explain concerns such as architecture, OO, patterns, frameworks etc. This common understanding, even though they need not have to be experts on the other area, would allow them to be more productive.

Of course, Mike Stenhouse’s point isn’t unique to an XSLT-based approach (in fact, I think he was talking about Java or PHP). But one possible advantage an XSLT-based approach may have here is that it might be simpler to learn (controversial statement, maybe!) than some of the framework languages, such as Java, C#, PHP, etc. Where this is the case, XSLT can be a useful way to help bridge the gap.

Coding productivity

Using something like ASP.NET controls and simply dropping them onto a canvas seems really productive at first. I was certainly excited back in 2000/2001 as information about ASP.NET started to come out. I remember being really excited about the date control that ASP.NET had demoed years ago! Such a time saver – or so it seemed.

However, if you factor in the time you often need to sort out the mucky markup output, make things accessible, CSS-friendlier, and so on, the overall time, in my experience doesn’t seem that much better (in some cases worse, some cases better) than an XSLT-driven approach.

If you know the output you need to create, I find writing XSLT to be more liberating as you are in full control of your output destiny!

The switch I described earlier from ASP based XSLT-driven site to an PHP based XSLT-driven was also a switch from Windows/IIS to Linux/Apache (I was finally fed up of IIS’s limited configurability, in a hosted environment).

In just two weeks, a site with about 700 pages was converted, fully tested, and up and running again on a new hosting company. There was virtually no XSLT change, and the design pattern (a kind of Page Controller pattern which I might change slightly to MVC if time allows) was just reapplied to PHP. PHP 5’s object oriented capabilities meant further reduction in server side code. (Come to think of it, it wasn’t two weeks – it was a LOT less, as it was just spare time during some of the evenings.)

For work, most of our customers (one of them being amongst the largest in the world in their industry) use .NET. We often by-pass ASP.NET controls and use XSLT to generate the output instead. C#/.NET are still used on the server side, as .NET itself is quite powerful.

For the generation of the UI, XSLT has allowed us to even create multiple web sites simultaneously, as all the XSLTs can be reused. Where creating a separate sub-site in the past has take 6 months or more, our move to the XSLT-based approach has meant it has taken a fraction of that time (and included SEO, accessibility, web standards, etc.).

As another recent example: For my current project, a colleague (an excellent developer in general, though not well-versed in web-standards, yet) is creating a functionally rich and powerful page as part of the overall app we are developing. It has a complex grid of tabular data, with a folder/tree type browser on the side with lots of dynamic stuff possible on that page. Some controls created by a vendor for ASP.NET 2.0 are being used. While they are rich in functionality, the page produces just under 1MB of output!!!

The output from these controls are plagued with excess divs and unnecessary table-based layout (tables for tabular data of course is fine). Styling some of the specific features also becomes tricky (because the markup created is not very good and can’t be changed).

In addition, on a gut feel, the amount of C# used to prepare the data to bind it to the grid-based controls seems excessive. In the output, there is a lot of unnecessary (and bloated) JavaScript needed to get it to behave as needed, too.

This also (currently) runs like a dog on IE 6 because of all that JavaScript and HTML bloat! I have spent a total of 15 days just trying to get the resizer/splitter control that contains the table and tree to resize properly by battling its JavaScript (to be fair, during that time in the summer, I discovered the excellent jQuery, which helped reduce that effort considerably but I was still using jQuery to do unnecessary stuff)!

In my own opinion, it would have been much more efficient to have written the whole thing with XSLT and use high quality client-side components from people like jQuery, YUI, Dojo, etc. The trees and tabular data on that page are really well suited for XSLT processing. To be fair, the guy writing the page isn’t as well versed in XSLT/web development but an experienced GUI developer, so the control approach makes perfect sense for him and he has certainly made me appreciate that approach more than I used to. (I do still feel that the amount of code needed feels excessive and the experience so far has strengthened my view that while ASP.NET is powerful on the server side, I prefer to use XSLT to create the markup, where possible.)

Drawbacks (or perceived drawbacks) to the XSLT approach

XSLT is certainly not to everyone’s liking. The right tools may not be there in the favored IDE, or it may just look too weird for some people (it is more like functional programming than procedural or object oriented, so can sometimes take a while to get used to). Here are the disadvantages of XSLT I can think of, and where I can, some ways to overcome those problems.

Not everyone likes XSLT!

Many who are used to OO or procedural programming may find this functional programming language strange to get around. Angle bracket coding isn’t to everyone’s taste! And these personal preferences can be important for a team/project. Initially I didn’t find it too intuitive, but after a few days with it, it grew on me.

(And I certainly don’t hold the position that one must only use XSLT. Different people will prefer different technologies, and are of course free to use them.)

Yet another technology to learn

I kind of appreciate this concern, but also in some ways do not. For example, XQuery, or the more generic and powerful LINQ from Microsoft, get many developers excited, and they are willing to learn those if needed. People will invest time in learning CSS, JavaScript, etc.

So, perhaps it could be argued that just like any other technology, if XSLT is determined to be appropriate for the project at hand, then that can be learned too.

XSLT 1.0 is Verbose and Weakly Typed

Some simple things in XSLT 1.0 are a bit verbose to code (though XSLT 1.0 can often be written more cleanly than what many people write).

XSLT (and XPath) are indeed weak on typing — at least 1.0, though given the nature and purpose of XSLT this is not too limiting a factor as it would be if this were a full-blown language. (Some popular languages for the web are also weakly typed, too, e.g. PHP and JavaScript and the “limited” typing can also result in many capabilities, depending how they are used.)

XSLT 2.0 and XPath 2.0 are more powerful and overcome many of these limitations, adding many more powerful capabilities, allowing for leaner code and improved performance.

However, while XSLT 2.0 is an open standard, many have not supported it. (Microsoft, for example, has a decent implementation of XSLT 1.0 in .NET — and MSXML — but has hinted that XSLT 2.0 is not a priority for .NET at the moment — see also this discussion for more information.)

XSLT is a limited language

In a way, if compared to more general purpose languages such as Java or C#, XSLT (especially 1.0) is very limited in what you do.

That can be off-putting straight away.

However, (using extension objects etc aside), the simplicity of the language can also be considered a strength; it may help architect a better separation of data, presentation and logic. I personally have not felt too much pain from its limitation.

Poor Tool Support Compared to Other Languages?

Tool support for XSLT does not seem as good as ones for say C# or Java.

With C#, both Visual Studio in-built refactoring or JetBrain’s excellent Resharper makes refactoring a lot easier. Same with Eclipse and other such IDEs for Java and various other languages.

Need to rename a class that is used throughout your super-massive project? No problem, just right-click the definition and rename; the IDE will correctly update everything for you. Want to implement missing implementations of an interface? Right-click and get it to fill out the missing implementations. Want to move a block of code into a separate method? Highlight it, right click and select an option to extract a method. Want to find all the places where a particular method is used? Right click it and select find all instances. And so on.

In addition, many IDEs and plugins for code assistance goes beyond “intellisense”, advising on good coding practices, code formatting, etc. This is immensely powerful, especially when establishing team standards.

With XSLT, however, the level of sophistication is not as high, as far as I am aware. For example, finding instances of where a given XSLT template is used seems limited. The best editor I have come across for this purpose is oXygen (let me know if there are better ones!). It is has various refactoring and finding capabilities, but finds don’t support searches for XSLT template modes. (I find using moded templates is generally a cleaner way to code many XSLT.)

A developer is therefore reduced to global finds within their text editor of choice. To be honest, I don’t particularly find this a problem, but I know that some developers, especially when used to using tools like JetBrain’s excellent refactoring tools, will find this backward, and I can’t blame them!

Many IDE vendors, such as Microsoft, don’t have their priorities on XSLT. While somewhat understandable, it does contribute to the chicken-egg situation:

• IDE tools are not as good for XSLT;
• XSLT therefore not as widely adopted in that sphere of development;
• IDE tools remain limited for XSLT;
• XSLT development seen as backward from an IDE/tool support perspective so usually not looked at.

As to what IDEs/tools/editors are useful:

• A number of people at work like to use XML Spy for XSLT development, though I personally don’t like it’s text editor.
• I tend to use jEdit, which is a Java-based IDE, so I have used it on Linux, PC and Mac, quite nicely (it is hideous to look at initially, and I have spent a long time customising it! I will explain that in some future blog post!).
• Eclipse and the Aptana plugin looks really useful, and I am beginning to look at that more.
• Many people also swear by oXygen, so I am looking at that a bit more, too.
• But, for my current project, for the past year, at work I have been content with using Visual Studio 2005 (and now 2008) for XSLT. It has an XSLT debugger (and recently a profiler) but I have not needed to use the XSLT debugger yet, so can’t comment on it.

If you know of any other useful XSLT tools, please comment so!

Refactoring in a mixed language environment can be problematic

Suppose you have an XSLT that transforms some XML which comes from a Java or C# serialized object.

If you modify that object in a simple way (rename the class or a public member), in IDEs such as Visual Studio and Eclipse, you can very quickly update all usages of your class/method throughout your project and unit tests, and you are done.

But, imagine you are renaming class members that get serialized in such a way that the name of that class member is the name of the element/attribute in the XML representation. IDEs aren’t going to easily know that and you are likely to have to manually update the XSLTs.

Unit Tested XSLTs may help to some extent to find where changes are needed, but it may still be more work than just working in C# or Java.

Getting data as XML may not be practical

Using XSLT assumes you can easily (or without much cost) get your data in XML format, so you can transform it. This may not be possible in many cases.

In many instances, with in-built XML serialization capabilities, such costs are often minimized or minimal. (You are often doing well if this is your biggest problem in your overall system architecture!)

Even if serialization is not possible, XSLT performance can benefit from well structured XML. Is it easy to create that structure from your object model? Depends on the project at hand.

Perceived Performance Concerns

A common question I often get in discussions of XSLT is concerns of performance.

XSLT performance, amongst other things, is also affected by:

• The XML being transformed – its structure, size, etc.
• The efficiency of the XPath expressions (e.g. using ‘//’ or invoking dynamic scripting languages from within XSLT can be performance killers).
• XSLT itself (e.g. loading, interpreting and executing XSLT in general, whether or not the XSLT Processor is compiled/cached/reused or not, etc.)

As well as addressing the XML and XPath concerns, the XSLT performance concern can also be addressed in many cases.

Using cachable XSLT Processor implementations can dramatically boost performance. (Kind of like XSLT being compiled into the native code, like Java bytecode or .NET IL, etc. and then caching them in memory across sessions, for subsequent reuse). We use this approach on one of UK’s most profitable and biggest online retail web sites, for example.

Also, as a high level language, XSLT has potential to be optimized further at the implementation level resulting in better performance without changes needed to the XSLT itself.

(Extensibility through XSLT extension objects may offer some performance gains if you write extension objects in a compiled language such as Java or C#. A risk may be that you lose some portability — though how much that is a risk depends on your circumstances.)

Fragment Caching is Hard

In addition to the performance boost from using cachable XSLT Processors, the ultimate caching strategy is being able to cache the entire output of a page. Subsequent requests for the same page require very few server resources.

However, there are times where the entire output is not easy or practical to cache, but fragments of the output could be. An example is a page of personalized data combined with general, cachable data.

XSLT in a View is typically used to generate the whole output, so fragment caching could be hard. Ways to mitigate might include implementing caching strategies in Models (which might be worth doing anyway), pipelining XSLTs, where the output of one transformation is fed into the input for another XSLT allowing cached fragments to be saved temporarily in the process.

In this case, specific templating languages such as that from Django, ASP.NET etc may offer better abilities. XSLT 2.0 may also have some more features in this area (it has better support for multiple XML documents, for example). If XSLT were to ever become even more popular for this usage, then I imagine there would be more advances in this area, too.

It implies error-prone XML-based templates

Consider this concern from Django’s design philosophies behind its templating system:

XML should not be used for template languages

Using an XML engine to parse templates introduces a whole new world of human error in editing templates — and incurs an unacceptable level of overhead in template processing.

– Design philosophies, Django, last accessed January 4, 2008

While the above is not specifically geared towards XSLT, human error is certainly an issue with XML-based templates, or XSLT-authoring.

That error, if it comes at template creation time, is typically caught during the development/debug/test cycles, as malformed XML will error early.

Of course, if content comes from external sources (user generated, from third parties, etc) the well-formedness of that XML cannot be guaranteed. However, these are usually not likely to be within the template itself. It may or may not be possible to cleanse such data, depending on the project.

In addition, as addressed earlier, I do not believe the overhead of template processing is that bad. Cached XSLT processors, as well as other caching strategies can help, for example.

A colleague of mine found an ASP.NET data grid control taking some 2 minutes to bind and render a standard .NET dataset. He changed that into an XSLT transformation instead and the data transformed in around 20 seconds — clearly an exceptional case compared to most web needs (2 minutes or 20 seconds implies a LOT of data), but XSLT is not automatically slow.

In addition, depending on the application, the generation of the XML may come entirely through serialization of objects, which in languages such as Java and .NET (not sure about python, which Django is in) is reasonably fast.

In a current project I serialize a very large object (about 200K when saved as XML) and transform it into HTML. The HTML is a very complex form of over 300 form fields! (Fear not, we use a tab-like interface and various other techniques to group them!) Using the cached XSLT Processors on my developer box, the transformation takes just 5-10 milliseconds. The far bigger cost is getting the data from the database in the first place (about 50 to 120 milliseconds, but that is on a developer database on the same box as the rest of the stack). The serialization takes about 5 milliseconds. Even when the XSLT processor is not cached, the transformation times are very small (I think about 150-300 milliseconds depending on the data).

That same design philosophy’s very next point was that you should

Assume designer competence

The template system shouldn’t be designed so that templates necessarily are displayed nicely in WYSIWYG editors such as Dreamweaver. That is too severe of a limitation and wouldn’t allow the syntax to be as nice as it is. Django expects template authors are comfortable editing HTML directly.

– Design philosophies, Django, last accessed January 4, 2008

I would tend to agree with the bit about assuming designer competence, which is actually why I find XML for templates are not as bad as it may initially seem.

As I mentioned earlier, in my experience, when trying to hire front end developers, we have found they are quite capable at adopting XSLT.

It is not that Django’s is wrong in my opinion; I am sure there are many for whom XML-based templates are not appropriate. My point here is that there are also many for whom XSLT will be seen as appropriate, too.

(And this is not a criticism of Django, either. I have only looked at it very, very, very briefly and am already impressed with it!)

Coding Productivity is Perceived to be Poor; Hand writing all that code seems inefficient

A common concern I hear (usually from middle-tier developers who want to avoid writing HTML) is that it is inefficient to hand write all that code. I can certainly understand their concerns; it is something I shared when I used to do more middle-tier development than front-end.

However, I actually find the opposite can be true: After using it for many years, a few colleagues of mine and I find it is far more productive creating web sites this way, because we know what we need to output, and we don’t spend ages battling tools that generate HTML to make them do stuff they don’t easily want to. We also help minimize, or completely avoid the excessive bloat that many tools create.

In addition, past experiences suggest that many of our XSLT projects seem to have less code than would be needed in say a C#/ASP.NET template (like the tree/grid example mentioned earlier).

Unlikely to get vendor buy-in if they have vested interest in other ways

Vendor buy-in may be limited if there is an inherent interest in vendor lock-in.

This is a mixed one too. Take .NET for example. Microsoft is unlikely to do away with its control-based approach and cause more upheaval to its target developer audience by doing this. It also reduces the dependency on Microsoft!

I would think that if this XSLT idea has any large-scale merit, open source platforms/frameworks are more likely to adopt it, though that is not necessarily a given, either.

XSLT for web-based Views hasn’t taken off yet, so it must be a bad choice

I think this point can have merit. However, I think there may be a number of different reasons why XSLT has not taken off.

For example, it may not have been appropriate in the past because XSLT tools were not as mature. While the XSLT tools were maturing, other technologies came through.

In addition, not all data is always easy to represent as, or convert into, XML, and the tools built into languages these days such as serialization may not have been around sufficiently in the past.

But the chicken-egg scenario mentioned above about IDEs not supporting XSLT as much so XSLT not getting as much traction in this area, could also be a factor. And, as hinted above, some vendors may have interests in promoting their own ways of doing things as an open standard may make it easier for someone to move to another platform, if needed.

Of course, developers not liking the language may also be a factor. (It seems to be quite big in the Java world, from what I can tell, though I don’t have much hands-on experience with Java these days – if anyone has some more insights here, please let me know!)

We have used this XSLT approach very successfully on a number of projects, one of which is one of the most prominent online retail web sites in the UK, if not the world. We haven’t really had a chance to write about it. Maybe there are some (many?) others who have also not had a chance to write about similar approaches? The BBC, I understand from some colleagues/friends that have worked there in the past, use it but in a more off-line scenario to pre-generate certain pages/sub-sites. So it can’t be all bad!

In addition, valid (or invalid) reasons to reject it in the past do not always automatically apply in for now or the future, especially in a dynamically changing environment such as the Internet. Times and technology change, so maybe it is at least worth re-validating (not adopting blindly of course!).

In conclusion

Despite the concerns, or perceived concerns, I feel the potential advantages make it worth considering.

There are certainly XSLT-based frameworks out there, though not as popular. (Apache’s Cocoon project comes to mind, for example.)

I would think that just as advocates of web standards have pushed hard to get open standards adopted for client side web development, maybe there is a case for XSLT on the server side to aide in that process.

But it also depends on personalities, priorities, and specific requirements, too. Some people really don’t like XML-based things, which is fair enough as sometimes there are legitimate reasons for it.

Others may not be convinced by the need for controlling markup, thinking it is unimportant or that their favourite vendor will sort it all out for them!

I am not arguing that XSLT should be the only way of doing things. I think it can be useful in many situations, and I have certainly found it to be so. Following the ideas of using web standards on the client side, using XSLT on the server side (at least for the generation of output) appeals to me too. What I therefore hope to do in coming posts is demonstrate some possibilities.

Some examples

To explore this further, as this post is already quite long, in the next few days or weeks (whenever I get the time!) I will add posts to demonstrate

• Unit testable XSLT
• “Object-oriented” XSLT
• XSLT as the View part of MVC, or as part of a Page Controller pattern.

[You might, rightly, wonder why this is a WordPress blog, and not something I have written using say PHP with XSLT! Fair, but lack of spare time is the main problem: It has taken me two years just to get round to starting this blog! One day I will convert, time allowing!]

In the meanwhile, if you have got this far, thanks for bearing with me!

Do you have similar concerns, or agree/disagree?

Image credits

“XSL Transformers” reposted with kind permission from NaGraDim from flickr. See the original image at http://www.flickr.com/photos/nagradim/750087828/.

PHP 5 has the usual things, such as classes, interfaces, abstract classes, inheritance, etc, but some useful programming constructs have been missing, though PHP 6, under development, aims to rectify that.

However, it seems that many of those features are going to be brought forward to the up-coming PHP 5.3 (which may make it more likely that it will get installed by web hosting companies sooner than they would likely go for PHP 6).

Sitepoint has an excellent summary of the features. The list of features include:

• Namespaces
• The use keyword to use namespaces!
• Namespace aliases
• Class constants
• Namespaced functions
• The Global namespace
• Autoloading namespaced classes
• Late static binding
• Variable static calls
• MySQL native driver
• Additional OpenSSL functions
• Improved command line parameter support
• XSLT profiling
• New Error Levels
• and more…

But visit the sitepoint article for more details.

If you want, there are PHP snapshot builds of PHP 5.3 (and 6.0).

(Those of you that have never coded in PHP may be surprised to see some of these features only surface now, yet, PHP has still managed to drive some of the biggest web sites out there…! I have only used it for about a year myself, in spare time, but it seems quite powerful even without these things. Can’t knock it too much then, I guess )

The native MySQL driver will be a good boost for performance, and I will be looking forward to the XSLT profiling in code, while namespaces will be an obvious winner too.

But a lot of PHP programmers are often using it for small scripts and may not often need or want the full power of an OO language. Personally, I prefer OO, so am looking forward to this. Do you use PHP? If so, do you prefer the OO approach and like what is coming?

ASP.NET has been problematic for many web developers/designers trying to do the “right thing”, such as

• Use web standards
• Output semantically meaningful markup
• Develop to standards, rather than browsers for cross-browser development
• Use progressive enhancement concepts to cater for as many user preferences and technology limitations as possible
• Create sites that are as accessible as possible without unnecessary trade-offs with the visual design
• Create sites that are friendly for SEO
• Create lean and optimal markup for quicker download
• etc.

(ASP.NET 1 was far worse, and version 2 allows many of the above to happen, but not all, and not always easily.)

Describing the problems caused by the poor quality and bloated markup and the weird/wonderful/complex/ugly/hacky ways to work around this seems to take ages. Some fans of ASP.NET don’t like to hear it when going into these details, and react defensively. And, to be fair, these problems are not unique to ASP.NET, either.

But recently, I rediscovered a summary explanation to help explain why ASP.NET is often problematic from a web developer’s perspective:

ASP.NET is a leaky abstraction:

That term, “leaky abstraction”, was coined some time ago (end of 2002) by Joel Spolsky, a prominent writer on software development and he was noting the problems with ASP.NET way back then.

In teaching someone about ASP.NET programming, it would be nice if I could just teach them that they can double-click on things and then write code that runs on the server when the user clicks on those things. Indeed ASP.NET abstracts away the difference between writing the HTML code to handle clicking on a hyperlink (<a>) and the code to handle clicking on a button. Problem: the ASP.NET designers needed to hide the fact that in HTML, there’s no way to submit a form from a hyperlink. They do this by generating a few lines of JavaScript and attaching an onclick handler to the hyperlink. The abstraction leaks, though. If the end-user has JavaScript disabled, the ASP.NET application doesn’t work correctly, and if the programmer doesn’t understand what ASP.NET was abstracting away, they simply won’t have any clue what is wrong.

– Joel Spolsky, The Law of Leaky Abstractions, November 11, 2002

I think this is a decent summary description that can be useful for constructive discussion on what to do about it (if anything).

Abstraction without understanding = problems

Of course, this is not just an ASP.NET issue, but a general problem in software development, as Spolsky added:

The law of leaky abstractions means that whenever somebody comes up with a wizzy new code-generation tool that is supposed to make us all ever-so-efficient, you hear a lot of people saying “learn how to do it manually first, then use the wizzy tool to save time.” Code generation tools which pretend to abstract out something, like all abstractions, leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don’t save us time learning.

And all this means that paradoxically, even as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder.

– Joel Spolsky, The Law of Leaky Abstractions, November 11, 2002

This helps explain:

Why non-web developers or IE-only web developers like ASP.NET

It is an abstraction, so you don’t worry about it

Why web developers/designers despise ASP.NET

The abstraction leaks, horribly at times, and you want to worry about what is being abstracted

What is still interesting is that as a software developer (not necessarily a web specialist), Joel has understood some of the problems web developers and designers will have with .NET even though many Microsoft converts and ASP.NET fans do not!

Many software developers look down on web developers, thinking things like HTML and CSS are beneath them (its “just” the UI, or that JavaScript is not a real programming language, etc).

While those attitudes are generally changing, many software/mid-tier/non-web developers I come across still hold some of these views and don’t want to write anything in these languages, preferring the safety and familiarity of their IDE! (To be fair, some of these concerns are quite valid, too, and the principle for supporting the abstraction is actually often good; it is just that with web technology at the level it is at now, some of that abstract necessarily has to leak…)

Negative perception of IE and ASP.NET for Web Developers

It is interesting to note at various conferences like @media, @media AJAX, and on various blogs by noted web developers and designers, Microsoft gets a lot of stick for its IE web browser (as it is recognized as the poorest in quality of all the modern browsers, with the most bugs, biggest divergence from web standards and causing the most headaches for modern web development).

Indeed, since winning the “browser war” IE has stagnated and because of its dominance, web technologies have hardly changed in the last 10 years (we are still on HTML 4.01/XHTML 1.0, CSS 2.1 etc). As I posted earlier, IE has slowed down web development progress in recent years.

Combine that with the poor quality markup typically generated by ASP.NET, there has been a lot of criticism from the web development community aimed at Microsoft for either not caring about standards (or, as with ASP.NET 2 and IE 8) caring about it as an after thought or when threatened by rivals.

Here is quite a telling quote summarizing some of the implications:

I sympathize with your situation, but you desperately need to get your seniors and peers to understand what’s happening, and to assign your team way more resources. IE is a first-impression MSFT product for most web developers, and yet you all continue to underestimate the dramatic spillover effect this poor developer experience has had and will continue to have on your other products and services.

Let me drive this point home. I am a front-end programmer and a co-founder of a start-up. I can tell you categorically that my team:

- Won’t download and play with Silverlight
- Won’t build a Live widget
- Won’t consider any MSFT search or ad products in the future

And the reason is because of IE – because MSFT disregards its most important relationship with us. Until this relationship is repaired, nothing else stands a chance.

Please fight the good fight and drive this point home in your org. Good luck.

– Anonymous commenter to The First Year of IE7, November 30, 2007

It may be seem a bit harsh, as some developers and technical managers at Microsoft certainly seem to “get it”, but for whatever reason it has happened and left a sour taste in the mouths of many web designers and developers.

Leaky abstractions lead to productivity loss and unnecessary trade-offs

As I blogged earlier, the WYSIWYG paradigm that Visual Studio and other such IDEs encourage, is often inappropriate for web development as it encourages bad practice such as non-semantic markup.

The perceived productivity gains of such editors vanish when needing to deal with typical web related concerns (semantic markup, accessibility, progressive enhancement, SEO, etc). In fact, what often occurs is unnecessary trade-offs. E.g.

• A back-end developer uses an ASP.NET control that offers quick development with sophisticated capabilities just by dragging it onto a canvas and setting a few properties.
• A web developer then claims that control is poor from all the web development perspectives and lists a number of things to change.
• The back-end developer resists (especially if it comes from a “lowly” web developer!).
• Trade-offs are then made (or not).
• Reworking in full (if embraced at all) takes a long time.

Even though ASP.NET 2.0 is far better than 1.x, the abstractions still leak; it is too easy to create sites that either

• Rely on JavaScript
• Violate, or easily encourage ignoring, basic accessibility
• Encourage further leaky abstractions because of the ease at which various ASP.NET controls can be created (the idea is good, but the fundamental leakage means that it will carry through to various controls).

So what can people do?

As an overall thought: by its nature, abstracting away HTML is almost guaranteed to result in a leaky abstraction.

Instead, it may be better to embrace the reasons for the leak, and provide cleaner separation while empowering people to work with the markup, given it is so crucial from so many perspective.

So here are some examples of a few ways to deal with the challenge, all with their merits for different purposes. Your mileage with each option will vary according to need. In addition, it is not a complete list; there may be other ways to achieve what you need:

Strict discipline with the controls you choose to use

The controls idea is certainly nice. ASP.NET and others (e.g. Java Server Faces) have put a lot of resources into this approach, and is certainly appealing from various perspectives (packaging components, reusability, productising components, etc).

Despite the problems, trying to stick with the principle of controls based abstraction may still be worth trying to pursue.

If so, I think it would require being VERY careful about third party controls you use, or purchase.

The other problem I see with this is in the case of medium/large teams; this strict discipline is unlikely to hold for too long, and eventually a developer with less experience on web development (or no inclination to do the web bit) will insist on using some poor quality controls that offer perceived productivity gains (“they can fix the problems you highlight to them!”, or “it looks okay”, or “people with accessibility needs don’t use the web, so it doesn’t matter!”).

Your mileage on this option will vary, depending on your team and project, etc.

Inherit and override problematic controls

Don’t like the way asp:label outputs its HTML? Override its Render() method. Same with any other control.

Sounds nice and easy enough, but this option is likely to run into trouble when someone invariably will want to use or integrate a composite, possibly third party, control written elsewhere that cannot be overridden (easily). So ASP.NET adaptors may be a better alternative than this one.

ASP.NET Control Adaptors

Also known as CSS Adaptors (which reveals a fundamental misunderstanding; developers want semantic markup for all sorts of reasons, not just creating CSS-based layouts).

This approach tries to give developers the opportunity to clean up the HTML output by overriding the default implementation.

The kind of problems I see with this approach:

• Code examples I have seen that look awful — lots of string concatenations, obscure configuration settings, etc.
• It may not always be easy to deploy a quick change after initial development (need to recompile a .NET Assembly, for example).

Despite the potential problems, it may still be a useful option if there aren’t any other alternatives.

(Does this approach help prove the abstraction fails?)

Do nothing (wait for Microsoft to fix it)

It may seem odd to entertain an option of “do nothing”, but I often think this is an option that should be considered in many situations, if not just for completeness but because pondering on this may let you think out of the box, so to speak.

Why should this option be entertained for ASP.NET? Well, you could take the approach of:

• It’s not my problem; I just want to create cool apps
• Microsoft will always fix it; we can just wait
• If we get sued (accessibility compliance) or our bosses/managers complain, we can just blame Microsoft
• The abstraction might be leaky now, but that doesn’t mean it always will — look at the improvement of ASP.NET 2 vs ASP.NET 1.x
• I’m so fed up I have lost the will to live and do anything else other than just be assimilated
• Problems? What problems? (Already assimilated a long time ago )

And so on.

Needless to say, I would disagree with many of these perspectives. For example,

• I feel as a web developer, it is out responsibility to use the tools available to create sites that are as accessible to as wide an audience as possible.
• Claiming ignorance, or actively avoiding issues you know are there somehow does not feel right.
• Microsoft may certainly fix these issues over time. Reading blogs and interviews of key people in ASP.NET and Internet Explorer teams certainly gives some hope for the future, but it depends on whether you think you can wait or not (and whether you have enough information on when they will release specific fixes that are important to you).
• The controls paradigm might always be leaky regardless of what is done. I think it would still be important to know the output that is being created in order to fully understand the implications of what you are creating. If there are problems with the site, you are likely to have to sort it out, not Microsoft (or any other vendor/framework you go for).

For many web developers who know exactly what they need, having a tool do it for you, and to a lower quality, will be infuriating. Knowing it can be done better (not just for one’s own ego, but because it is good for the user/customer) can be compelling enough to want to do something about it.

Indeed, a principle behind a number of open source frameworks has been to try and make the web better, and keep it open because it is felt that open standards and related principles will lead to greater choice, diversity and vibrancy of the web. (Indeed, a lot of “Web 2.0″ and the social web has come about because of open source technologies and ideas.)

Use an open source alternative

Those loyal to Microsoft will not like to hear this, but this is a decent option in many scenarios.

As Simon Willison, co-creator of the popular Django web framework said a few years ago, one of the key benefits of the open source approach is you are in charge of your own destiny.

(I’m not going to debate the open vs closed source paradigms here; that is not the purpose of this blog. I will just say that I didn’t use to buy the arguments of open source in the past, but in the last 3 or 4 years I have come to respect it a LOT more — I have not regretted moving to Linux at home recently, one bit, for example, and looking at the types of projects flourishing is quite encouraging.)

Don’t use ASP.NET, but don’t throw the baby (.NET) out with the bath water (ASP.NET)

Okay, so for some scenarios, completely changing platforms may not be an option.

In addition, I have to admit, programming in .NET (C# 2, etc) is nice.

Although Microsoft may be losing credibility with web developers because of the poor quality Internet Explorer browser and because of the problems of ASP.NET from a web designer’s perspective, .NET more generally is actually quite good.

Unlike on the Internet Explorer side of things, the MSDN blogs seem to be more active regarding on-going developments on .NET. Scott Guthrie, for example, has an immensely useful and popular blog on all things about Visual Studio, ASP.NET etc. (Scott Guthrie is General Manager of development teams that build the .NET CLR and the core .NET Base Class Libraries, ASP.NET, Silverlight, IIS 7.0, Visual Studio Tools for ASP.NET, WPF and Silverlight.)

For some projects we have successfully used .NET but by-passed the ASP.NET controls, using XSLT instead.

XSLT — an open standard

Using XSLT to generate the output can be one way to overcome limitations of proprietary frameworks where you lack control of the output (ASP.NET) or where other frameworks may be open source but you have to learn their templating conventions.

XSLT can also make the markup generation simpler to unit test as part of a developer’s daily work, which is not always the case (or not necessarily as simple) in other templating systems, because you don’t have to be dependent on a running web server and all the set up that would come with it (thus making unit tests quick to run as well, increasing the likelihood of them being used and maintained).

Being platform agnostic, XSLT can be an effective part of a View in an MVC framework and be applied to different frameworks and platforms, reducing up-skilling costs in the long term.

If you consult or contract for a number of customers/clients, it would be nice if there was one common skill that is portable across all these frameworks (just like HTML/CSS/JavaScript typically is on the client side). XSLT could be such a thing, potentially, all the while giving you full control of the markup that you need to be create.

In my experience, and those of some of my colleagues at work, using XSLT to by-pass .NET controls has actually been more productive from an overall development perspective because we save time that would otherwise be needed battling the controls trying to force or work alternative markup options, building accessibility, etc. (And we have architected/developed sites for one of the largest online retailers in the world, so the strategies are not merely wishful thinking, but reasonably proven).

XSLT may not always be appropriate in all cases, and many are certainly put off by its oddities or perceived limitations. But, perhaps XSLT deserves another look?

(Indeed, a follow-up post does just that, starting by looking at XSLT in server side web frameworks.)

Do you find these problems with ASP.NET for your sites? How do you overcome them, or do you just stick with it?

Image credits

“Mona Unmasked” reposted with kind permission from gravityx9 from flickr. See the original image at http://www.flickr.com/photos/gx9/354387041/.