Hawksworx

A little while ago, inspired by Simon Willison’s demo of Nodejs at the FullFrontal conference, I felt the urge to make something, anything, using Nodejs.

Nodejs gives lightening fast, event driven IO with Javascript running server-side in the rather nippy V8 Javascript engine. Until Nodejs, I hadn’t seen much need to bring Javascript to the server other than for testing and as a bit of a novelty.

Nodejs changed all of that. Built by Ryah Dahl with a strict philosophy that none of its procedures should ever perform a blocking operation, it has a single threaded architecture and relies heavily on the use of a single event loop. Anyone with a good knowledge of Javascript or even just a familiarity with jQuery will be familiar with the pattern of event handlers, callbacks and closures needed to build some really rather powerful things, quickly and easily with Nodejs.

For my part, I just wanted to make something to explore the ease in which Nodejs could support comet-style long polling for some real-time, collaborative task. The Web is already getting flooded with examples of chat rooms and IRC servers built with Nodejs, but I really wanted to make something that felt like a real web application, not an example.

To that end, I resurrected an old idea which I first built circa 1999 for randomly choosing someone from a list to make the tea. Not earth-shattering, but a bit of fun and simple enough for me to see through to completion. For this incarnation, dubbed Teafr.com, I piggybacked on the lists feature of Twitter to allow people to create and manage their lists of potential tea-makers somewhere they are already comfortable. My app would allow a user to recall a list of their choosing and then perform a lottery, selecting a winner (or loser) from the list at random. To add a little suspense (in the loosest possible sense) I added a basic animation which gradually got slower until the winner was revealed.

I captured a couple of videos showing teafr.com doing its stuff, one of them on an iPhone.

The fun part of this for me was in allowing this lottery and the animation to be visible to everyone viewing the same list from wherever they are. Nodejs makes it trivially simple to handle long-lived http requests so that all clients could listen out for the initiation of a new lottery event. Since this is all just lightweight HTTP, HTML, CSS and Javascript, it also runs pretty nicely on both my iPhone and Palm Pre. It is rather satisfying to stand next to someone who is viewing the site on a desktop machine, and then start their screen animating with a quick tap on your handset.

Cheap thrills, I know.

It seems to me though, that the cheapness of the thrills is what is so exciting. I’m no rocket scientist, and I managed to create this entire application, complete with real-time online views in a couple of evenings work. The short hop and low cognitive friction for a Javascript developer to be able to build these kind of things from end to end is incredibly liberating. I’m bursting at the seams with ideas for things I want to build, and am now capable of building with a new, powerful and yet familiar tool set.

The only detail which kept my new toy from the masses was having a suitable place to host it. I’ve been using Dreamhost for a while now and have to say that I’m pretty happy with them, but even with the great level of control and access to your server that they give you, you need a little bit more control to compile and run Nodejs. The easiest and cheapest way that I found to host Nodejs powered site was with a dedicated virtual server. There is no shortage of providers in this space and as Web developers get more sophisticated, the use of this kind of solution is getting more commonplace. I opted for Linode which is working out rather nicely for me, but equally, I could have plumped for a similar offering from Slicehost or Amazon Web Services. I’ll soon migrate all of my sites onto the single Linux instance that they host for me.

For now though, I need to resist the temptation to spend all of my spare time (ha!) noodling away with Nodejs, MongoDB, jQuery and various other fun things which ’speak Javascript’ that smart people on the Web keep on creating and selflessly sharing with the world for free.

No really! Read that title again. I’m about to argue against placing so much blame for painful Web development at the door of Internet Explorer 6. I’m not going try and claim that IE6 isn’t a huge pain in the arse of all good Web developers. Let’s face it, Web developers love to bitch about it, but I do think that it’s time for some accountability.

The problem with IE6 is simple. It has poor support for Web standards. It doesn’t always correctly render valid HTML and CSS, and requires nudging, tweaking and finagling in order to display the content in the desired way. Its interpretation of Javascript and interactions with the DOM are also less than perfect. That’s a pain, but does it really warrant such massive hatred from Web developers? As a Web developer, my first instinct would be to answer, ‘well, yes’. The cost of developing for IE6 is extraordinarily high, even when compared to the benefit from its market share, because it is such a horrible and time consuming slog.

But hang on a moment. The Web is something of a meritocracy. People tend to vote with their feet (or mice) and will abandon the weak experience in favor of the strong. Doesn’t this concept stretch to the browser too? Surely if IE6 was really that bad, people would have abandoned it and adopted a better, newer, more well behaved Web browser. Right? Yes and no. Many people have done exactly that. They have upgraded to IE7, IE8, Firefox, Safari, Opera, Chrome or something else. Let’s be honest, it isn’t hard to find a better browser than IE6 these days. But it isn’t quite that simple.

First of all, not everyone who uses the Web really understands what A Web browser is. Many people just know that they click an icon to get to “The Internet” (by which they often mean Google) or directly to their email (by which they mean one of the many Web based email services). Understanding that they could choose a different application which accesses hese services to perform these tasks is something of a cognitive leap.

Others realised that they could use a better browser and have upgraded the browser on their home computers, but at work they are locked in to using whatever the company IT department dictates. Often, this means IE6. Why is that?

This is my beef. It’s not that IE6 is out-dated and substandard (although it is), but that large organisations have needed to lock users in to using an approved Web browser. Having worked in a few large organisations, I’m no stranger to that scenario, and it’s immensely frustrating to be a user forced to use something so sub-par, and to be required to support it when you are trying to build the Webs next new hotness. So why do they do that? There can be many reasons, but the largest, and most difficult to dispute is actually of our own creation. We (the Web developers of the world) built expensive, bespoke Web applications for large enterprises and (naturally) ensured that they worked on the most popular browser of the time. Critically though, this was done without thought to how the code might perform in future, standards compliant browsers. In many cases, the code just wasn’t future proof.

As a result we have applications and systems which depend being viewed in IE6 in order to function. This represents a huge risk to many large companies who invested so heavily in their development, and have come to depend on the applications. It’s little wonder that they force their employees to use IE6.

Remember that this isn’t just the fault of IE6. There have been many poor browsers over the years, but thankfully, natural selection has let them disappear and allowed better browsers to take over. IE6 suffers from its own success. That is, it’s market penetration. Combine that with bad habits from developers and you have a browser which despite being left behind by evolution, is artificially kept alive by large enterprises as if it were on a magical life-support machine. I imagine that it would rather be left to die in peace.

Surely then, we Web developers have learned our lesson. We’re not going let something like that happen again are we? Staggeringly, after all the grumbling and pain, I see evidence that many of us haven’t learned our lesson at all. We still write code which detects which browser is being used, before going on to specify behaviour accordingly. What we should be doing is testing for the support of our chosen implementation, then implementing accordingly. Think object detection, not browser detection. That way, when a browser improves its support, or the user starts using a new browser, the application can enjoy the benefits, rather than just falling over.

I was particularly aggravated to learn how IE8 can be set to render as if it were IE7. With a meta tag to provide “Internet Explorer 7 Compatibility mode”.

<meta http-equiv="X-UA-Compatible" content="IE=7">

Why would this feature exist? Microsoft explain that it is so that developers can opt for “backwards compatibility with Internet Explorer 7 JavaScript and layout behavior”. In other words, it exists to try and get around the fact that so many developers wrote code to satisfy the quirks of IE7 without a thinking about how their code would fail when the quality of the browser is improved (have better conformance with W3C standards for HTML and CSS).

Microsoft didn’t want people to be resistant to upgrading to IE8 because of so many sites failing to render correctly in their shiny, new, more compliant browsers, so they provided a way to force the browser to continue to mis-interpret the code. I’m afraid I have to call “bull-shit” on this practice. I think that all this does is create another wave of Web sites and applications which are likely to get stuck in non-compliance land and require old rendering engines and legacy browsers to be kept alive to server them.

Only by developing for the rising tide of web standards, rather than the various quirks of specific browsers, can we hope to avoid locking our users in to a given browser. It’s time to make sure that we don’t get into this situation ever again. So next time you are tempted to grumble about IE6, remember that you can help avoid a similar mess in future by developing with standards in mind.

The recent release of TiddlyWiki v2.5 included something rather exciting for me: JQuery, the popular Javascript library is now part of the TiddlyWiki core.

This is exciting for a number of reasons.

1. The TiddlyWiki core functions can now use jQuery to perform all manner of DOM inspection and DOM manipulation. We can refactor a ton of code to benefit from jQuery’s blistering Sizzle engine and pass the burden of maintaining lots of utility functions over to those clever jQuery bods. All of which will simplify the TiddlyWiki codebase and ultimately make it easier to read and easier to maintain.
2. TiddlyWiki plugin developers will now be able to make use of jQuery in their plugins. That’s great news for both hardcore plugin developers and people dabbling for the first time. jQuery is elegantly expressive, powerful, and superbly documented. All of this lowers the barriers to entry for a would be developer and smooths the way for exisiting developers.
3. There is a huge wealth of jQuery plugins which can now be utilised by TiddlyWiki. The quality of many of these plugins is tremendously high. Bring ‘em on!

But how does a TiddlyWiki plugin developer get started? How can we bring a jQuery plugin into TiddlyWiki and make it available via a Macro? Let’s take a look at an example.

Fred, my colleague at Osmosoft stumbled upon a plugin which creates a nice navigation structure from an HTML List element. The plugin has lots of examples and documentation and seemed like a good contender for bringing something useful into TiddlyWiki.

To include the plugin, all we need to do is copy it into a tiddler and then tag the tiddler with systemConfig. After saving the file and reloading, the plugin is available for us to use.

With the jQuery plugin availble, we can use it to provide a snazzy UI for any UL or LI elements on the page. The Javascirpt syntax for that is nice and simple:

// Turn the list element with an ID of 'myList' into a funky ListNav
$('#myList').listnav();

This is great, but we want people to be able to call this from TiddlyWiki without having to write Javascript. If we create a TiddlyWiki macro to call this for us, people can easily make a nice NavList anywhere. Let’s make a macro which turns the contents of a tiddler into a NavList like this:

<<listnav TiddlerTitle>>

To make that available, we should create a TiddlyWiki plugin which delivers this macro. We do this in a new tiddler. I created a tiddler called ‘ListNavPlugin’ and once again tagged it with systemConfig so that it becomes code that TiddlyWiki knows to interpret. To start with, let’s just create the macro and ensure that we can invoke it.

 
// create macro object
config.macros.listnav = { 
 
  // Add a handler function to be invoked by <<listnav TiddlerTitle>> 
  handler: function(place, macroName, params, wikifier, paramString, tiddler) {
 
    // do some magic...
    alert("I'm gonna make a funky listnav!");
 
  }
};

To see it working, I created two more new tiddlers. MyFruityList holds a list of items, and ExampleListNav holds a call to the new macro.

Saving the TiddlyWiki file and reloading will make the macro available to call. If we open the ExampleListNav tiddler, we should see our macro called like this:

All good, but not exciting yet. Let’s flesh out our TiddlyWiki macro a bit. We’ll use a TiddlyWiki function to get the text from our desired tiddler and then make a list from it. Where we might once have used some TiddlyWiki utility functions to help us create the HTML elements (like createTiddlyElement), we can now use jQuery’s helper functions to append elements to the document.

 
// create macro object
config.macros.listnav = {
 
  // Add a handler function to be invoked by <<listnav TiddlerTitle>> 
  handler: function(place, macroName, params, wikifier, paramString, tiddler) {
 
    // target tiddler passed in as macro parameter
    var title = params[0];
 
    // read list items from tiddler contents
    var text = store.getTiddlerText(title);
    if(text) {
 
      // generate list
      var items = text.split("\n");
      var list = jQuery("<ul />").attr("id", "listnav").appendTo(place);
      jQuery.each(items, function(i, itm) {
        jQuery("<li />").text(itm).appendTo(list);
      });
 
    }
  }
};

Now we’re getting somewhere. We’ve grabbed each line from our MyFruityList tiddler and used jQuery to turn them into an unordered list.

We can now call the jQuery plugin to render our list as a NavList. The plugin requires a little extra HTML so we can create that, and then call the listnav plugin.

 
// create macro object
config.macros.listnav = {
 
  // Add a handler function to be invoked by <<listnav TiddlerTitle>> 
  handler: function(place, macroName, params, wikifier, paramString, tiddler) {
 
    // target tiddler passed in as macro parameter
    var title = params[0];
 
    // read list items from tiddler contents
    var text = store.getTiddlerText(title);
    if(text) {
 
      // generate nav bar
      jQuery("<div />").attr("id", "listnav-nav").appendTo(place);
 
      // generate list
      var items = text.split("\n");
      var list = jQuery("<ul />").attr("id", "listnav").appendTo(place);
      jQuery.each(items, function(i, itm) {
        jQuery("<li />").text(itm).appendTo(list);
      });
 
      // apply listnav
      list.listnav();
    }
  }
};

When we save our plugin, reload the page and open up our ExampleListNav tiddler, we see the navlist. Cool! Hang on though, it looks a bit rubbish. Not like the examples we saw earlier. We have to include the CSS to style our new navlist. Easy enough, let’s use our TiddlyWiki plugin to create a sylesheet for our new HTML by adding this:

 
// add default styles (adapted from http://www.ihwy.com/labs/downloads/jquery-listnav/2.0/listnav.css)
config.shadowTiddlers.StyleSheetListNav = "/*{{{*/\n" +
  "#listnav-nav { margin: 20px 0 10px; }\n" +
  ".ln-letters { overflow: hidden; }\n" +
  ".ln-letters a { font-size: 0.9em; display: block; float: left; padding: 2px 6px; border: 1px solid #eee; border-right: none; text-decoration: none; }\n"+
  ".ln-letters a.ln-last { border-right: 1px solid #eee; }\n" +
  ".ln-letters a:hover, .ln-letters a.ln-selected { background-color: #eaeaea; }\n" +
  ".ln-letters a.ln-disabled { color: #ccc; }\n" +
  ".ln-letter-count { text-align: center; font-size: 0.8em; line-height: 1; margin-bottom: 3px; color: #336699; }\n" +
  "/*}}}*/";
store.addNotification("StyleSheetListNav", refreshStyles);

And we’re done. The jQuery plugin is turning our boring old list into a dynamic indexed list with navigation. Cool! For extra credit, we can make our jQuery a little more concise. Notice that we call jQuery by name rather than with the common idiom of $. We could simply replace each jQuery with $ in our code and that would work, but beware! There are other Javascript libraries that use the $ shortcut. we don’t want to introduce the possibility of a clash. Luckily, there is a simple way to get around that. All we need to do is wrap our TiddlyWiki plugin in a closure and pass jQuery in as an argument (as explained on the jQuery documentation site). Then we can use whatever shortcut for jQuery in our code that we like. The final code looks like this:

 
(function($) { // set up alias
 
  // create macro object
  config.macros.listnav = {
 
  // Add a handler function to be invoked by <<listnav TiddlerTitle>> 
  handler: function(place, macroName, params, wikifier, paramString, tiddler) {
 
    // target tiddler passed in as macro parameter
    var title = params[0];
 
    // read list items from tiddler contents
    var text = store.getTiddlerText(title);
    if(text) {
 
     // generate nav bar
     $("<div />").attr("id", "listnav-nav").appendTo(place);
 
     // generate list
     var items = text.split("\n");
     var list = $("<ul />").attr("id", "listnav").appendTo(place);
     $.each(items, function(i, itm) {
       $("<li />").text(itm).appendTo(list);
     });
 
     // apply listnav
     list.listnav();
    }
  }
};
 
  // add default styles (adapted from http://www.ihwy.com/labs/downloads/jquery-listnav/2.0/listnav.css)
  config.shadowTiddlers.StyleSheetListNav = "/*{{{*/\n" +
  "#listnav-nav { margin: 20px 0 10px; }\n" +
  ".ln-letters { overflow: hidden; }\n" +
  ".ln-letters a { font-size: 0.9em; display: block; float: left; padding: 2px 6px; border: 1px solid #eee; border-right: none; text-decoration: none; }\n"+
  ".ln-letters a.ln-last { border-right: 1px solid #eee; }\n" +
  ".ln-letters a:hover, .ln-letters a.ln-selected { background-color: #eaeaea; }\n" +
  ".ln-letters a.ln-disabled { color: #ccc; }\n" +
  ".ln-letter-count { text-align: center; font-size: 0.8em; line-height: 1; margin-bottom: 3px; color: #336699; }\n" +
  "/*}}}*/";
  store.addNotification("StyleSheetListNav", refreshStyles);	
 
})(jQuery);

You can explore the finished example in a TiddlyWiki.

Recently I saw Tim Stevens post on Twitter about a slide show he had built using his Liveloom application. His slide show grabs some photos from Flickr tagged with ‘Osmosoft’ and then renders them with all manner of visual effects using Flex.

While the visual effects available via Tim’s app are impressive, I’m a big advocate of open web standards and enjoy making things from HTML and Javascript, rather than using tools like Flex. My first encounter with the slide show made by Liveloom was on an iPhone where it couldn’t run. The temptation to cobble together a simple Flickr mashup which could operate in any Javascript capable browser was too much to resist.

Let’s not pretend, that there aren’t already slide show mashups galore, or that I was going to be able to rival some of the advanced visual effects that Liveloom can offer, but making a simple, attractive slide show which could run on lots of devices seemed like a fun and valuable exercise. I was also keen to see just how quickly and easily I could produce this using jQuery. I’m forever going on about how good jQuery is at manipulating the DOM, so this seemed like a good chance to demonstrate that.

I shared the result my efforts as the ‘Poloaroiderizer’ and after getting some great feedback, promoted it to its very own domain. Check it out over at polaroiderizer.com

As it turns out, the task was incredibly simple and took just a few hours one evening. The interface is rendered with HTML and CSS. As such, it will work in any modern browser. I used jQuery to request JSONP from the Flickr API. JSPONP gets you around the cross domain scripting restrictions and delivers JSON to the browser which is a snap to parse and render. There is no logic on the server. All of my code is run in the browser.

The animations are provided by CSS and DOM manipulation, made simple by jQuery’s animation functions. The display needs Javascript to render, so without it Polaroiderizer is a bit toothless, but that doesn’t mean we can forget about those without Javascript. If you submit a flickr tag while you don’t have Javascript capabilities, the form will submit your tag to the standard flickr interface and you’ll find yourself at the flickr search results page. This is another example of progressive enhancement. A topic close to my heart that I tried to demonstrate recently on Unobtrusify.

Another little bonus of building this slide show app this way, is that it is super-easy to share the results. I made sure that the tag entered was echoed into the address of the page as the page fragment identifier. Manipulating and interrogating the frag id can be useful in Ajax applications as my colleague Michael Mahemoff explained recently. When the page loads, it looks for a tag stored in the frag id as its starting point. That means you can bookmark slide shows and share them as easily as email a URI. Here are a few:

http://polaroiderizer.com/#PhilHawksworth http://polaroiderizer.com/#TheWebIsAgreement http://polaroiderizer.com/#TED2009

Recently Jon Lister, a colleague of mine at Osmosoft showed me a website made by his friend Joshua Bradley. The site, used some of the Javscript code from TiddlyWiki’s animation engine to create some nice visual effects. I loved the design, but could see some room for improvement in the implementation. I’m a big advocate of Unobtrusive Javascript and Progressive Enhancement and so I set about producing a quick demo of how a similar result could be achieved in the most Web-kind and accessible way available using jQuery for the behaviors.

The result has been published as Unobtrusify.com.

The aim.

• Create a similar effect to that on Josh’s site, but make sure that the page is readable without Javascript.
• Use images to make the headings look snazzy, but make sure that they are not required in order for the content to make sense.
• Use only unobtrusive Javascript and keep the HTML as clean as possible.
• Reduce the number of http requests required to as few as possible in order to improve performance.

The approach.

First of all, I wrote the text for the page. I chose a simple statement and tried to structure it such that it would make sense regardless of which sections were expanded.

Then I used the simplest HTML markup I could to logically represent the content with its various headings. This is how the page would look to text-only browsers search engines, web-crawlers and screen-readers.

I then used a well-known CSS technique to replace the text in the headings with images. This would ensure that the text would remain for non-human consumers of the site, while the images would be presented to those able to appreciate them. The technique is simple. You prevent the browser from scrolling the content of your element with overflow:hidden and then scoot the text out of the way with text-indent. Now that the way is clear, you can display an image with background-image. Be sure to specify the dimensions of your desired image as the background-image property will not resize your element to the correct size automatically. The CSS would look something like this:

#myHeading {
	text-indent: -9999px;
	overflow: hidden;
	background-image: url(myImage.gif);
	width: 380px;
	height: 123px;
}

My content had 6 headings to render in this way. I also wanted to have a mouseover effect to give some affordance for the click-ability of the headings so this would require another 6 images. Rather than having 12 images to download (which would require 12 separate HTTP requests), I combined all of these images into a single image. This would have a number of effects. Firstly, combining the 12 images into one meant that the total download would be a bit smaller due to the way that the file was compressed. (A tiny saving, but every little helps.) Secondly, there is an overhead with making HTTP requests so when it comes to performance, the fewer the better. This method cuts out 11 HTTP requests. Score! Thirdly, as the browser uses the same image for the original heading images and their associated mouseover images, there is no need to preload the alternate images to avoid that nasty pause when mousing over. The image is already downloaded and ready to display. A nice bonus.

In order to use this ‘image sprite’ for each and every heading, I just needed to specify the background-position for each one. Some attributes would be common to each one so I could save some code like this:

h1 {
	text-indent: -9999px;
	overflow: hidden;
	background-image: url(images/unobtrusive_sprite.gif);
	width: 380px;
}
 
h1#uj {
	background-position: 0 0;
	height: 123px;
}
 
h1#cmh {
	background-position: 0 -123px;
	height: 150px;
}
...

At this point our page looks like this. This is exactly how we want things to appear for those without Javascript. There is no ability to toggle the display of the various sections, the content is shown in full, and there is no mouseover behavior on the headings to suggest that they can be clicked (since they cannot). This is the essence of Progressive Enhancement. We have a perfectly serviceable web page (albeit a simple one) which we can now enhance for those with Javascript enabled.

Using jQuery to easily and unobtrusively add behavior to elements on the page, we can now hide all of the expanded sections. We do this with a simple jQuery statement like this:

$('#wrapper > div').hide();

This hides all of the div elements which are a direct descendent of the element with an ID of wrapper. (my chosen HTML structure).

Headings are not by default clickable so we can add some behavior to suggest that the clicking a heading will have a effect by changing the cursor for them like this:

$('#wrapper h1').addClass('clickable');

A CSS class of ‘clickable’ specifies the cursor with cursor: pointer;

We also use jQuery to show the hover image by just repositioning the background image when we hover with the mouse and also to show the hidden div element when we click a heading. Remember, none of this will happen unless Javascript is available.

I also use an additional trick to prevent a flash of unstyled content or FOUC (gratifyingly pronounced ‘FOOOOOOK’ by John Hicks) while the Javascript is being downloaded. This trick is very well explained by Karl Swedberg on the excellent learningjquery.com site.

For a better picture of exactly what is going on, why not swing by Unobtrusify.com and exercise your right as a citizen of the Web to view the source. Hitting View Source is so often the best way to learn how things are done. Go on! Go and get your hands dirty!