May 15, 2008

A Way To Unit Test ASP.NET IHttpHandler Implementations

If you find yourself writing simple HTTP handler code that produces and consumes structured data (for instance, some RESTful application), you may wonder how to test it without fiddling with IIS or configuration files. Here's a trick to write pure unit tests that verify your IHttpHandler implementation does what you expect. By "pure unit tests", I mean test code that:

works without configuration files (like web.config);
needs no servers (like Cassini, IIS, or your own mock HTTP server with full ASP.NET pipeline that you were about to write and debug just before you stumbled upon this article);
doesn't access file system (like ashx files);
avoids globals (like HttpContext.Current).

The Unit Tests In Action

Pretend we have an IHttpHandler implementation, TagHandler, that lets browsers retrieve a list of tags (using GET requests) and post new tags (using POST requests). The handler responds to GET requests with an array of tag objects serialized as JSON. It also parses submitted data (JSON) in POST requests and creates tags based on that.

Here are the unit tests that verify the POST behavior. Note the use of a helper class, HttpHandlerTest, that does all the tricky bits such as setting up HttpContext instance, creating a fake request body, the session state, and more.

[TestFixture]
public class TagHandlerTestFixture {
    private Dictionary<String, Tag> tags;

    [SetUp] public void SetUp() {
        // the tags object is always empty at the start of each test case!
        tags = new Dictionary<String, Tag>();
    }

    [Test, ExpectedException(typeof (ArgumentException))]
    public void PostTagWithInvalidRequestContentType() {
        HttpHandlerTest http = HttpHandlerTest.ImitatePost("tags.ashx", "{}", Encoding.UTF8);
        // Won't like text/xml, it wants application/json!
        http.Context.Request.ContentType = "text/xml";
        http.Execute(new TagHandler(tags));
    }

    [Test] public void PostTag() {
        HttpHandlerTest http = HttpHandlerTest.ImitatePost("tags.ashx", 
            "{\"name\":\"foo\",\"description\":\"a a a\"}", Encoding.UTF8);
        http.Context.Request.ContentType = "application/json";
        http.Execute(new TagHandler(tags));
        Assert.IsTrue(tags.ContainsKey("foo"));
        Assert.AreEqual("a a a", tags["foo"].Description);
    }
    
    // ... more tests
}

You start off with setting up either a fake GET or POST request with HttpHandlerTest.Imitate[Get/Post]() method, then you set up additional preconditions in the initialized HttpContext object accessible as a property on HttpHandlerTest instance. Then you pass the IHttpHandler instance to the Execute() method. Your handler does its work. And then you can verify your post-conditions using NUnit assertions.

Here are some more tests from the same fixture that exercise the GET behavior:

    [Test] public void GetTagsResponseIsApplicationJson() {
        HttpHandlerTest http = HttpHandlerTest.ImitateGet("tags.ashx");
        http.Execute(new TagHandler(tags));
        Assert.AreEqual("application/json",
                        http.Context.Response.ContentType);
    }

    [Test] public void GetNoTags() {
        HttpHandlerTest http = HttpHandlerTest.ImitateGet("tags.ashx");
        http.Execute(new TagHandler(tags));
        Assert.AreEqual("{}", http.Output);
        Assert.AreEqual(Encoding.UTF8.GetType(),
            http.Context.Response.ContentEncoding.GetType());
    }

HttpHandlerTest Internals

Under the hood, HttpHandlerTest creates an instance of HttpContext initialized with a custom version of SimpleWorkerRequest. ImitateGet gives you an instance of HttpHandlerTest geared for testing GET requests, which do not have any content in the request body. ImitatePost is for simulating POST requests that do have content in the request. It should be easy to add support for other HTTP methods like DELETE, HEAD, and PUT.

public class HttpHandlerTest
{
    // Methods
    protected HttpHandlerTest(string requestMethod, string requestBody,
        Encoding requestEncoding, string page, string query);
    public virtual void Execute(IHttpHandler testSubject);
    public static HttpHandlerTest ImitateGet(string virtualPath);
    public static HttpHandlerTest ImitatePost(string virtualPath,
        string requestBody, Encoding requestEncoding);

    // Properties
    public virtual HttpContext Context { get; }
    public virtual string Output { get; }
}

Links

Full source code for HttpHandlerTest with the example web app. Note: the example web app uses AjaxPro.JSON.2.dll and nunit.framework.dll which are bundled in the "lib" directory. You can get everything by: svn co http://codebackpack.googlecode.com/svn/tags/how-to-test-httphandlers
Of course once I checked that nobody else found solution just as easy as the one presented here, and wrote this post, I was immediately shown ("Thank you" GeorgeR) that there was already a post about this by Phil Haack and my solution looks very similar :( Oh well, at least this proves that this is not such a bad idea after all.
Lutz Roeder's Reflector has proven very valuable in figuring out which methods of SimpleWorkerRequest to override.

Posted by Pavel Repin in .NET and C# Development, Web Development | Permalink | Comments (0) | TrackBack (0)

April 22, 2008

JavaScript Error Tracking: Why window.onerror Is Not Enough

Cozi builds family organization software, including a free online calendar, shopping list and family journal. In order to provide a richer experience for the users of our web application, over the past year we have been transitioning more and more of our code from C# running in an ASP.NET application to JavaScript running in the browser. We are using the jQuery JavaScript library for event handling, DOM manipulation, effects, styling and AJAX calls. In the process of this migration, we lost our ability to easily track errors and exceptions using an error log on the application server. Not knowing what errors our users might be encountering when they use our product made us nervous, so we set out to regain complete error tracking capabilities. Our first attempt to solve the problem was to attach a custom JavaScript handler to window.onerror. This handler called back to the application server to log an error; an example handler might look like this:

window.onerror = function(msg, url, lineNo) 
{
  trackError(msg, url, lineNo);
}

This attempt largely worked, but as we started paying close attention during development, we noticed that not all JavaScript errors were successfully reported to the application to be logged. Specifically, we were missing some Firefox errors. Investigating further, we discovered that an error thrown once the page was loaded and the browser event loop took over was reported and tracked for IE(6/7), but it was not reported and tracked for Firefox(2/3). You can see evidence of this in the example page. This page throws two errors when it loads: one during the load itself and the other in the jQuery document.ready handler. A third error is thrown when the button is clicked. The custom window.onerror handler shows an alert with the error message and any other details that are available. Try it in Firefox and again in IE. You'll see only one alert in Firefox, but you'll see all three in IE.

I googled and googled, but I found nothing that suggested what the problem might be. It was either one of those situations where you had to know the answer to craft the query that would find the answer, or else no one else had struggled with this problem. Since we use jQuery as the glue in our event-driven application, I sought help from the jQuery Development group. It took some time, but help arrived. (Thanks, dhtmlkitchen!) As it turns out, in Firefox, an error thrown by code in an event handler attached using addEventListener does not make it to the window.onerror handler. There is an existing tracking issue for this bug at Mozilla: issue #312448. The jQuery bind function uses addEventListener in Mozilla and attachEvent in IE.

That explained it. All of our application behavior once the page loads is triggered by user interaction, and we use jQuery bind to attach handlers to the appropriate JavaScript DOM events. In short, most of our application's interesting code, i.e. the code most likely to cause errors, runs as a result of an event handler being triggered. These errors were arriving at window.onerror for IE, but due to the Mozilla bug, not for Firefox.

On then to the fix. It was clear that wrapping code with try/catch blocks was necessary, but there was no way we were going to successfully do that in every handler function. It took some fiddling, but overriding jQuery bind with a version that wraps the provided handler function in try/catch blocks does the trick:

// override jQuery.fn.bind to wrap every provided function in try/catch
var jQueryBind = jQuery.fn.bind;
jQuery.fn.bind = function( type, data, fn ) { 
  if ( !fn && data && typeof data == 'function' )
  {
    fn = data;
    data = null;
  }
  if ( fn )
  {
    var origFn = fn;
    var wrappedFn = function() { 
      try 
      {
        origFn.apply( this, arguments );
      }
      catch ( ex )
      {
        trackError( ex );
        // re-throw ex iff error should propogate
        // throw ex;
       }
     };
     fn = wrappedFn;
   }
   return jQueryBind.call( this, type, data, fn );
};

The bind override is inelegant in that it needs to know too much about how the original jQuery function shuffles parameters around, but if and when it breaks we'll just need to fix it in one place. Since bind function is used under the covers by short-cut functions like click, hover, etc, this one-time fix takes care of tracking almost all application errors. Note that to be completely thorough, we should also override the jQuery one function.

To catch errors that are thrown during page initialization, we also wrap document.ready functionality in try/catch blocks:

$(document).ready( function() {
  try
  {
    initializeEverything();
  }
  catch ( ex )
  {
    trackError( ex );
    // re-throw ex iff error should propogate
    // throw ex;
  }
} );

Try out the fixed version of the example page. Like the first example page, this page throws two errors when it loads: one during the load itself and the other in the jQuery document.ready handler. A third error is thrown when the button is clicked. The window.onerror handler shows an alert with the error message and any other details that are available. Try it in Firefox and again in IE. Now you'll see all three alerts in both Firefox and IE.

What About Safari?

Safari does not yet support window.onerror, which means that without the steps described above, we wouldn't be able to track any errors from Safari. The try/catch additions to document.ready and jQuery bind will allow us to track most, but not all, errors arising in Safari.

April 08, 2008

Combining JavaScript and CSS files for Improved Performance

Summary

Cozi has developed an ASP.NET control to combine multiple JavaScript or CSS files into a single reference, thus improving performance. The control handles problems of cache expiry dates and versioning, is easy to use, and can be switched on and off for debugging and testing.

Background

We’d like to organize our JavaScript and CSS files logically, without having to worry about performance. For instance, as we write more and more object-oriented JavaScript, we might want to put each JavaScript class into its own file. However, doing so would increase the number of round trips between the browser and the Web server.

This performance penalty is especially bad for JavaScript files since each one may contain a document.write which could change the sense of everything after it—so the browser can only download one at a time.

Unfortunately each extra JavaScript file adds about 200ms to page load time—the exact time varies by a dizzying array of factors (DSL vs Cable, how far it is from the browser to the server, etc, etc).

CSS files are not quite as bad as page parsing does not have to stop whilst they are being downloaded; however, the page cannot be correctly rendered until they have all been received.

Caching can help mitigate this delay, although it has its own problems.

Caching

To get maximum end-user performance, a Web site needs to set expiration headers on as many files as possible, especially static web content like images, CSS, JavaScript, etc. These headers tell the browser that it does not need to check back with the server for an updated version of a given file for some defined period.

Once that period has expired (or if no expiration is set), the browser will typically send a 'conditional get' giving the 'last-modified-time' of the file. If the file has not changed, the Web server can respond with a '304 Not Modified' response and not send down the data again.

Unfortunately the browser still needs to wait for the response before continuing. Therefore, it is best if the browser has a copy of the file AND it knows that it can be used without referring back to the Web server.

Versioning

Caching can also interfere with file versioning.

The easiest way to enabling caching is to tell your Web server that a whole directory of files can be cached on the client for a week (or day, or whatever). Unfortunately when you edit one of the cached files and deploy the new version, you have a problem because existing clients will run with their locally cached copy of the file until they reach the end of the expiration period.

How we solved these problems at Cozi

At Cozi we are developing an increasingly rich Web-based tool for making life simpler for families, including a shared family calendar, shopping list management, and more.

Our engineers need to have quite a few JavaScript and CSS files so that several of us can work on the site without trampling on each other’s code. However, this created the performance problems described above.

Part of our solution was to combine CSS and JS files into groups that get downloaded as one blob, saving many round trips.

Our requirements for the combining tool were:

1. It had to be easy to use.

2. You must be able to switch it on and off for testing and debugging scenarios (on a page-view-by-page-view basis).

3. It must support long cache expiration dates on these combined files.

4. It must solve the versioning problem.

To implement the solution, we wrote an ASP.NET control that wraps around our <link> and <script> HTML statements.

For example here’s a simplified version what the CSS references for our home page looks like:

<client:CombinerControl runat="server">

...

</client:CombinerControl>

The combiner outputs a reference to an Http Handler which will serve the combined file (I've put in some line breaks to make this a little more readable):

As far as the browser is concerned, this is just a reference to a CSS file. It neither knows nor cares that it is programmatically generated.

A few parameters are passed to the handler (Combiner.ashx). They are:

Parameter	Meaning
ext	The file extension of the files, this is to save having to pass the file extension for each file separately.
ver	This is how we handle versioning the CSS files. The combiner calculates the ‘version’ of the combined file by mushing together the last-modified-time of all the constituent files. If any one of the files changes, this parameter will change and the browser will request a new combined file.
type	This is the Content-Type the handler should return.
files	This is the list of files to be combined. It is somewhat compressed to reduce the length of the URL generated. ! means the following token is a new directory that applies until the next change of directory is seen. * means a file to be combined. ' replaces / for legibility, as / would have to be encoded as %2f

Here’s a similar code snippet combining JavaScript code:

<WebClientCode:CombinerControl ID="CombineScript" runat="server">

</WebClientCode:CombinerControl>

The combiner outputs a reference to an Http Handler which will serve the combined file:

The combiner control meets our requirements thus:

1. It had to be easy to use

It is, you just wrap the files you want: <client:CombinerControl runat="server">

2. You must be able to switch it on and off for testing and debugging scenarios

By passing a parameter on the Query String you can turn this behaviour off for testing and the control will merely output the list of files as though the control itself didn’t exist.

3. It must support setting long cache expiration dates on these combined files

The handler that serves the combined file automatically sets a very long expiration on the combined file, no configuration is needed.

4. It must solve the versioning problem

Because browsers cache based on the full, case-sensitive URL of the resource, you can set ‘infinite’ expiration, as if any file changes, a different reference will be written out and the browser will request a new file.

Gotchas

As always, we ran into a few gotchas:

You can’t combine CSS files with different media tags; all the CSS files must have the same media type.
You must preserve the order of the files; both for script and CSS files, order can matter.
You must remove UTF-8 BOM marks. It would be OK to have a BOM mark at the very start but many of our JS files had BOM marks at the start and when we combined them, these BOMs ended up in the middle of the combined output and caused the browsers problems as they were interpreted as strange characters.
It’s very hard to combine script files that take query string parameters; luckily it is quite unusual for static script files to take parameters so this wasn’t much of a limitation.

Further work

Our combiner also manipulates the contents of the CSS and JS files for other reasons, unrelated to main work of combining the files. For example, we might blog about how we use the combiner to load balance and edge-cache our images in another post.

April 01, 2008

Multiple Firefox Profiles: Run Firefox 2 and 3 Side-By-Side, and More

I find it useful to have multiple Firefox profiles for developing and testing. A clean profile for testing allows you to replicate most users' environments, who don't install extensions. Running a development profile in a separate profile lets you restart the browser without messing with your default environment. You can also run Firefox 2 and Firefox 3 side-by-side in separate profiles.

I currently have the following profiles:

default: I browse the web in this profile. Outlook opens links in this profile.
dev: I do most of my development here. Firebug gets a heavy workout.
test: A clean profile. I only install the ResizeIT extension here.
Firefox3: The latest beta of Firefox 3. The other profiles are Firefox 2-based.

Creating a new profile

Follow the instructions here to create test and dev profiles. Essentially, run:

 "%ProgramFiles%\Mozilla Firefox\firefox.exe" -profilemanager -no-remote

and use Create Profile to create profiles called dev and test

Your profile.ini should now look something like this:

 [General]
 StartWithLastProfile=1

 [Profile0]
 Name=default
 IsRelative=1
 Path=Profiles/y1mds144.default
 Default=1

 [Profile1]
 Name=dev
 IsRelative=1
 Path=Profiles/k034naef.dev

 [Profile2]
 Name=test
 IsRelative=1
 Path=Profiles/f2akbryu.test

Running the new profiles

If you want to run two profiles side-by-side, you need to run firefox.exe with the -no-remote and -P <profile> options. This launches each profile in a separate process.

Either create a batch file that looks like this:

 @start "Firefox Test" "%ProgramFiles%\Mozilla Firefox\firefox.exe" -no-remote -P "test"

Or create a Desktop Shortcut where the Location looks like:

 "C:\Program Files\Mozilla Firefox\firefox.exe" -no-remote -P "dev"

More information: Complete list of Firefox command-line options.

Running Firefox 2 and Firefox 3 side by side

By default, Firefox 2 installs into %ProgramFiles%\Mozilla Firefox, while Firefox 3 Beta 4 installs into %ProgramFiles%\Mozilla Firefox 3 Beta 4.

They can be installed on the same computer, but you must take care to use different profiles. Not all Firefox extensions have been updated yet for Firefox 3.

When you install Firefox 3, make sure that you do not launch it automatically at the end of the installation. Instead, run:

 "%ProgramFiles%\Mozilla Firefox 3 Beta 4\firefox.exe" -profilemanager -no-remote

and use Create Profile to create a profile called Firefox3. Then, follow the instructions above to create a batch file or a desktop shortcut, adjusting the path to firefox.exe.

Posted by George V. Reilly in Web Development | Permalink | Comments (2) | TrackBack (0)

March 24, 2008

Porting C# code to C++

Cozi has a downloadable PC client written in .NET which includes an awesome photo collage screen saver. We've had many requests to make the screen saver available as a separate download; however the .NET 2.0 dependency makes it a large download for many people, plus we'd like to port it to other platforms in the future. So we decided to convert the screen saver to C++ (using WTL/ATL as the class library).

Before we started, it seemed like quite a big task as the layout engine in our photo collage screen saver is pretty sophisticated. However, so far it's turned out to be essentially a mechanical translation.

This was helped by a few factors:

.NET uses GDI+ under the hood to provide nearly all the System.Drawing functionality. GDI+ is also available in the unmanaged world and (luckily) .NET had changed very little of the fundemental interfaces.
GDI+ has a nice class-based model so you're spared all the COM tedium of QueryInterface/Release.
ATL provides a reasonable set of collection classes that mapped quite well onto the C# collection classes.
We just didn't bother with CPP files; there's only 1 in the whole project, and this clearly helps translate C# as otherwise you'd spend hours declaring members then defining them — this is definitely the most tedious bit about C++. Of course, there is some point at which it would get too inefficient to do that for all classes, but our project seems to compile pretty quickly so far.
Expectations have changed. C# has dimmed the appetite for micro-optimizations, and there are many cases in the converted code where classes are being allocated and copied unnecessarily. In the past, we might have optimized these away but now we're happy to leave it and fix any problems that show up during performance testing. Similarly we use collection classes where before we'd have converted them to arrays.

Class library

As we have a certain amount of UI within the screen saver (for the configuration dialogs) we wanted a class library to help with that. The main choices were WTL/ATL and MFC; we chose WTL as it had a lower download size and we didn't need the sophistication of MFC. This then dictated using ATL for strings (CString), smart pointers (CAutoPtr) and collections. We could have used STL or another library too, but I know ATL so I preferred to stick with the library I knew.

Mechanical transformations

I ported the code over class by class and the first things I did were just mechanical changes.

Replace Rect with Rectangle (for some reason .NET renamed this class and we use it a lot).
Add a ; at the end of the class definition (this caught me almost every time).
Replace "String" and "string" with "CString".
Remove the "public" from the front of the class definition.
Remove IDisposable if defined
Move any code from Dispose() to the destructor.
Change static (and enum) references from Class.Member to Class::Member.
Add a : after public/private/protected declarations of members.
Change Debug.Assert to ATLASSERT
Map collection classes, usually this was just changing List<blah> to CAtlArray<blah>
CAutoPtr<> any allocations.
CAutoPtr<> any "usings", or just declare an object and let its destructor sort it out.
Change properties into GetXXX and SetXXX methods.
Pass classes either as references or as pointers.
Remove "this." (you could change to this-> to lower the risk of unexpected consequences but it looks ugly).
Change "const int" class members to enum {};
Replace null with NULL (or #define null NULL).

Notes

Doing this conversion has given me some perspective on the strengths and weakness of the two languages.

Pro for C++: Destructors are good. For code that needs deterministic cleanup (and as it happens this code needs a lot because of all the images) C++ is way better, IDispose is OK but look at the amount of code compared to C++.
Pro for C#: Properties are good. C# properties are a vast improvement over getter and setter methods.
Pro for C#: Field initialization is great. Having been able to initialize fields right at the point of declaration is far superior to C++ where you have to declare in one place and initialize in the constructor. This extends to consts which you can't declare within the class for some strange reason in C++.
Pro for C#: foreach is great. I'm sure I could STL or similar to get some of the same benefits, but foreach seems so natural.

Another thing I noticed is that garbage collection is not helping as much as I'd imagined. Whilst it's great not to worry about it, so far as long as I'm religious about using CAutoPtr and related auto-cleanup classes memory leaks don't seem to be the problem they were.

Example

(Comments and error handling have been removed.)

namespace Cozi.ScreenSaver
{
  using System;
  using System.Drawing;
  using System.Diagnostics;
  using System.IO;

  public class CollageImage : IDisposable
  {
    private const int c_resizeImagesOverMegaPixels = 9;

    Bitmap m_bmp;
    bool m_disposed = false;

    public CollageImage( string filePath )
    {
      if ( File.Exists( filePath ) )
      {
        this.m_bmp = new Bitmap( filePath );
      }
      else
      {   
        // Create an dummy bitmap if the path doesn't exist
        // (i.e. net path disconnected)
        this.m_bmp = null;
        return;
      }

      int megaPixels = (this.m_bmp.Width * this.m_bmp.Height) / 1000000;

      if (megaPixels > c_resizeImagesOverMegaPixels)
      {
        int width = System.Windows.Forms.Screen.PrimaryScreen.Bounds.Width / 2;
        int height = width * this.m_bmp.Height / this.m_bmp.Width;
        Image resized = new Bitmap(this.m_bmp, new Size(width, height));
        this.m_bmp = (Bitmap)resized;
      }
    }

#region IDisposable Members
    public void Dispose()
    {
      Dispose(true);
      GC.SuppressFinalize(this);
    }

    ~CollageImage()
    {
      Debug.WriteLine("LEAK: Failed to Dispose of CollageImage");
    }

    protected virtual void Dispose(bool disposing)
    {
      if (!this.m_disposed)
      {
        if (disposing)
        {
          if (this.m_bmp != null)
          {
            this.m_bmp.Dispose();
            this.m_bmp = null;
          }
        }

        this.m_disposed = true;
      }
    }
#endregion


    public Size Size
    {
      get 
      {
        if (this.m_bmp == null)
        {
          return new Size(1, 1);
        }

        return this.m_bmp.Size; 
      }
    }

    public void Draw(Graphics g, Rectangle bounds)
    {
      if (this.m_bmp == null)
      {
        return;
      }

      g.DrawImage(this.m_bmp, bounds);
    }
  }
}

Becomes:

#pragma once
#include "stdafx.h"

class CollageImage
{
  CAutoPtr<bitmap> m_bmp;

  enum
  {
    c_resizeImagesOverMegaPixels = 8
  };

  public: CollageImage( const CString& filePath )
  {
    if (::GetFileAttributes(filePath) != INVALID_FILE_ATTRIBUTES)
    {
      m_bmp.Attach(Bitmap::FromFile(filePath));
    }
    else
    {   
      return;
    }

    int megaPixels = (m_bmp->GetWidth() * m_bmp->GetHeight()) / 1000000;

    if (megaPixels > c_resizeImagesOverMegaPixels)
    {
      Image* resized = null;

      int width = GetSystemMetrics(SM_CXSCREEN) / 2;
      int height = width * m_bmp->GetHeight() / m_bmp->GetWidth();

      Image* resized = new Bitmap(width, height);
      Graphics g(resized);
      g.DrawImage(m_bmp, Rect(0, 0, width, height));

      m_bmp.Free();
      m_bmp.Attach((Bitmap*)resized);
    }
  }

  public: Size GetSize()
  {
    if (m_bmp == null)
    {
      return Size(1, 1);
    }

    return Size(m_bmp->GetWidth(), m_bmp->GetHeight()); 
  }

  public: void Draw(Graphics* g, Rect bounds)
  {
    if (m_bmp == null)
    {
      return;
    }

    g->DrawImage(m_bmp, bounds);
  }
};

Posted by Mark Atherton in .NET and C# Development | Permalink | Comments (0) | TrackBack (0)

March 19, 2008

Debugging JavaScript in IE from Visual Web Developer 2008 Express Edition

I use the timebombed Internet Explorer Virtual PC images from Microsoft to test the Cozi web client under IE6 on Vista. The VPC images come preloaded with the horrible, ancient Script Debugger, which hasn't received any developer love since periwigs were in fashion. Not recommended unless you can't find anything better.

But there is something better: Visual Web Developer 2008 Express Edition. VWD is free and it comes with a great JavaScript debugger and intellisense.

Unfortunately, it's not at all obvious how to actually use VWD to debug JavaScript running in IE. I eventually figured it out, with some help from Jaro, a former colleague of mine in IIS, who managed to find a short video explaining what to do for VWD 2005.

Here, then, are the necessary steps in VWD 2008.

Install Visual Web Developer 2008. You can turn off all the optional extras. If you're doing this inside Virtual PC, use CD > Capture ISO Image....

In Internet Explorer (6 or 7):

Tools > Internet Options > Advanced:
- Clear Disable Script Debugging (Internet Explorer)

In Visual Web Developer:

Create an empty website project, which defaults to C:\...\WebSite1\:
- File > New Web Site > ASP.NET Web Site
Modify Website > Start Options and set the URL to a good starting point in your application:
- Start action: Start URL: http://foo/bar/
- Server: Use Custom Server: http://foo/bar/
- Debuggers: Disable ASP.NET debugger
File > Save C:\...\WebSite1 to save the website project

Use Debug > Start Debugging (or hit F5) to launch Internet Explorer. Set breakpoints in your code by clicking in the left margin of some source code.

At one point (and I forget the details), I had to uninstall the old Script Debugger, then Repair VWD in Add or Remove Programs.

On Vista, I think it's probably necessary to use Run as Administrator when starting VWD.

Posted by George V. Reilly | Permalink | Comments (1) | TrackBack (0)

March 13, 2008

Daylight Savings Time and JavaScript

Run the following code in IE6 or Firefox 2.0.0.12:

 The time is <span id='now'></span>.
 <script type="text/javascript">
 document.getElementById('now').innerHTML = new Date();
 </script>

If you're in the US or Canada, you should see that the script claims that the time is an hour earlier than it actually is.

The picture here shows a piece of the Cozi web client running under Firefox 2 (wrong) and IE7 (correct).

The problem is due to the new Daylight Savings Time rules which went into effect last year in the United States. Formerly, DST began on the first Sunday in April; now it begins on the second Sunday in March.

Programs compiled with Visual Studio 2005 will not honor the new DST rules, not even if you've installed SP1. You have to install an additional hotfix and rebuild before your code will handle the new rules. (I confirmed this by building Vim without and with the hotfix last night.) This only seems to be a problem in C/C++ code. The .NET runtime correctly handles the new DST rules.

I've reported this problem in Bugzilla as JavaScript Date() off by an hour after DST change. As you can see from the bug report, if I roll the system clock back to before the new DST cutover or forward past the old cutover, the JavaScript date is once again correct.

Update 2008/03/14: Mea culpa. This is not a widespread problem. It is caused by the presence of set TZ=PST8PDT in my C:\AutoExec.bat. Per KB932590, the existence of the TZ environment variable will cause the CRT to use the old DST rules. (I can't remember why I set TZ several years ago. It's part of the accumulated mess of files that I bring everywhere with me.)

Posted by George V. Reilly in Cross-Browser Issues | Permalink | Comments (0) | TrackBack (0)

March 10, 2008

Transparent PNGs Can Deadlock IE6

Summary: Internet Explorer 6 does not support transparency in PNG images. The best-known solution is to use the DirectX AlphaImageLoader CSS filter. It's less well known that using AlphaImageLoader sometimes leads to a deadlock in IE6. There are two workarounds. Either wait until after the image has been downloaded to apply the filter to the image's style, or use the little-known transparent PNG8 format instead of the filter.

The First Set of Hangs

Back in September 2007, some of my colleagues at Cozi complained that IE6 would occasionally hang while loading the homepage of our web application. I took a look at some of these IE6 processes in Process Explorer and I could see that thread 0 (the UI thread) was stuck in MsgWaitForMultipleObjects.

I attached WinDbg to these hung IE6 processes and found a callstack like this:

 0:020> .symfix+
 No downstream store given, using C:\Program Files\Debugging Tools for Windows\sym
 0:020> .reload
 ................................................................................
 Loading unloaded module list
 .............
 0:020> ~0k
 ChildEBP RetAddr  
 0013bfec 7c90e9ab ntdll!KiFastSystemCallRet
 0013bff0 7c8094e2 ntdll!ZwWaitForMultipleObjects+0xc
 0013c08c 7e4195f9 kernel32!WaitForMultipleObjectsEx+0x12c
 0013c0e8 7e4196a8 user32!RealMsgWaitForMultipleObjectsEx+0x13e
 0013c104 7e211e76 user32!MsgWaitForMultipleObjects+0x1f
 0013c168 7e200b6b urlmon!CTransaction::CompleteOperation+0x140
 0013c1a4 7e1ef557 urlmon!CTransaction::Start+0x52c
 0013c228 7e1ef1b4 urlmon!CBinding::StartBinding+0x4d8     <<<<---
 0013d2c0 7e1ef07e urlmon!CUrlMon::StartBinding+0x1d8
 0013d2f8 6bdd9abe urlmon!CUrlMon::BindToStorage+0x67
 0013d55c 6be43a78 dxtrans!CDXTransformFactory::LoadImageWrapW+0xdc
 0013e600 77135d81 dxtmsft!CDXTAlphaImageLoader::put_Src+0x1b5
 0013e61c 77136390 oleaut32!DispCallFunc+0x16a
 0013e6ac 6be29b0b oleaut32!CTypeInfo2::Invoke+0x234
 0013e6dc 6be29c9a dxtmsft!ATL::CComTypeInfoHolder::Invoke+0x45
 0013e724 6be16448 dxtmsft!ATL::CComDispatchDriver::PutProperty+0x78
 0013e75c 6be210eb dxtmsft!CComPropBase::IPersistPropertyBag_Load+0x9b
 0013e770 6bde9513 dxtmsft!ATL::IPersistPropertyBagImpl<CDXTAlphaImageLoader>::Load+0x2a
 0013e788 6bde51a3 dxtrans!CDXTFilter::Load+0x1f
 0013e834 6bdea876 dxtrans!CDXTFilterBehavior::AddFilterFromBSTR+0x424
 0:020> du (0013c228 + 48)
 0013c270  "http://cozicentral.cozi.com/imag"
 0013c2b0  "es/clock/clock_2.png"

(0013c228 is the ChildEBP address beside urlmon!CBinding::StartBinding+0x4d8. I was eventually to learn that du (<StartBinding's ChildEBP> + 48) would give me the URL that was being requested. It took probing with dds to figure this out. Spending 10 years in the Windows Division at Microsoft left me with advanced skills in debugging weird failures.)

This happened intermittently, but it was more likely when the browser's cache had been flushed. The exact URL would vary, but it was always one of the PNG images used to draw the digital clock. We had a lot of transparent PNGs, but it always seemed to happen to the clock digits, which are loaded on demand. This image was always available: it was not a server problem. The hang would eventually time out, after 20 or 30 minutes. Until it did, you could do nothing with IE. It wouldn't repaint. You couldn't close the app. All you could do was kill it in Task Manager. When the UrlMon call finally did time out, the requested image would not have shown up, but otherwise the browser was fine. This happened only with IE6, never with IE7 or Firefox. This was very strange, and we really didn't know what to make of it.

It recurred sporadically, and not having any better ideas, we decided to replace the PNGs for the clock digits with transparent GIFs in the IE6-specific CSS. The PNGs use alpha transparency, to get antialiasing and the translucent reflection below the baseline. We were using filter:progid:DXImageTransform.Microsoft.AlphaImageLoader in IE6-specific CSS to get alpha transparency. Unlike modern browsers, IE6 does not support alpha transparency. Fully transparent pixels are rendered as an opaque blueish-gray. Hence, our use of the well-known AlphaImageLoader hack.

We couldn't bake the background into the clock digit images: we have cobranded editions with different backgrounds, so the background underlying the digits can vary with both cobrand and window size. GIFs may be transparent, but they have no notion of alpha transparency, so we had to remove the translucent reflection in the GIFs. Not as pretty, but acceptable.

Hangs, Redux

The problem went away for several weeks, but in late October, it came back with a vengeance, recurring multiple times a day. Some machines seemed immune, some could repro it with almost 100% success. There was no obvious commonality; it reproed on several builds of IE6. It had to be fixed: we could not ship a web application that sometimes hung IE6 hard.

I formed the hypothesis that the two-connection limit was contributing to the problem, since it looked like DirectX was bypassing WinInet and using UrlMon to fetch the image. At the time, we were making a lot of Ajax requests to fetch data for different portions of the homepage, and that somehow this was tying up WinInet.

Sure enough, raising IE's connection limit made the problem go away -- as far as we could tell. This wasn't an acceptable workaround. We could hardly tell our users, "By the way, our app tends to hang on Internet Explorer 6. Please make the following changes in your registry."

Commenting out the AlphaImageLoader filter from the CSS also made the problem go away, but the results looked horrible.

We weren't enthusiastic about converting all of our transparent PNGs to GIFs. Our design relies heavily on alpha blending and many of the softly blended border images, for example, would look bad without alpha transparency. GIFs are limited to a 256-color palette and a few of our images are very colorful. Moreover, there are a lot of images and maintaining two sets would be a pain. At that point, we were still trying to make our web application look just as good on IE6 as in IE7 and Firefox.

I had, of course, Googled extensively to see if anyone else knew what was going on. I didn't find anything that was of any use. I'm amazed that so few people seem to have encountered this. I don't have a really simple repro case, but what we were doing does not seem especially odd for a Web 2.0 application.

Enter Microsoft

So we contacted Microsoft.

Several of us are Microsoft veterans, so initially we made direct contact with the Internet Explorer team. They were able to reproduce the problem intermittently and asked us to open an official support incident.

Fairly quickly, they came up with a workaround of loading the images first and then applying the filter. This ensures that the image is in the browser's cache:

 <img src="…" onload="iePNGLoader.loadThis(this);" alt="…" width="…" height="…" />

 var iePNGLoader =
 {
   loadThis: function(img)
   {
     if (navigator.userAgent.indexOf("MSIE") >-1 && parseInt(navigator.appVersion) <= 6)
     {
       var pSrc = img.src;
       img.onload = null;
       img.src = "/…/shim.gif";
       img.style.filter = "progid:DXImageTransform.Microsoft.AlphaImageLoader(" +
               "enabled=true, src='" + pSrc + "')";
     }
   }
 };

Unfortunately, that's not adequate for our web application. We make heavy use of background-images in CSS; onload events fire only for IMG tags. Also, I could never get img.onload events to fire after windows.onload, and we insert a number of images into the DOM as a result of Ajax calls.

We needed a JavaScript workaround, because even if Microsoft were to hotfix the bug, there's no way that we could ensure that all of our users installed it.

To make a long story slightly shorter, I spent a lot of time in November, with help from the IE CPR engineer, trying to preload images for IE6. To be completely safe, it was necessary to preload all of our images, even though most of them would never be used. I ended up loading them into 1x1 divs with a negative z-index obscured behind our toolbar. When windows.onload fired, I walked through all the transparent PNGs, setting their style.backgroundImage attributes (or src, if node.tagName == 'IMG') to a transparent 1x1 GIF (shim.gif, above), and applying the AlphaImageLoader filter. Inserting all of these images into the DOM added a very noticeable overhead, but we were prepared to live with it, if it gave us a reliable workaround.

Satzansatz has a comprehensive list of the shortcomings of AlphaImageLoader. For example, we use the background-repeat attribute to vertically or horizontally tile border images, allowing our bordered regions to resize themselves to the page's dimensions. Filtered images do not tile. At best, you can stretch them with sizingMethod='scale'. This happened to look okay with our border tiles, but it certainly wouldn't in general.

Anyway, preloading worked flawlessly on our intranet. Then we deployed to a test server in the datacenter, and it all fell apart. Apparently, the increased network latency exposed a bug in my preloading implementation. It turns out that if you dynamically insert images into the DOM, they are loaded after windows.onload fires -- even if the script itself executes before windows.onload. (Statically declared images are loaded before windows.onload fires, of course.) Hence, I was applying the filter before the image had necessarily loaded.

PNG8 to the Rescue

We were on the verge of switching over to GIF images, when one of my colleagues pointed out the PNG8 format to us, which we had never heard of before.

The SitePoint article on PNG8 contains a good explanation and examples. Briefly, however, typical PNG images contain arbitrary RGBA values, while PNG8 images contain an indexed palette of at most 256 colors, each of which can have 8-bit Red, Green, Blue, and Alpha values. This is similar to GIF's indexed palette, but PNG8 allows 256 degrees of opacity in each palette color, while GIF's transparency is all or nothing.

IE6, it turns out, does support transparency in PNG8 images, but it's a binary transparency, just like GIFs.

Here, then are three different logos, each rendered three different ways.

Here are the logos as PNG32s in IE6, without the AlphaImageLoader filter. Transparent pixels are opaque. Evidently unacceptable.

Here are the PNG8s in Firefox. Note the alpha transparency around the Cozi logo on the left. Look at the smooth edges in the logos. You can see the subtle alpha blending around the bordered box above the left Cozi logo and Mr Potato Head.

And here are the PNG8s in IE6. All alpha transparency is rendered as complete transparency. Note the jaggies in the logos, and the unsubtle borders around the box.

Clearly, PNG8 gives visually inferior results on IE6, but it looks great on modern browsers.

Kevin Freitas also covers the PNG8 technique. The comments in his article indicate that both the GIMP and Photoshop are capable of producing indexed PNG images.

The Aftermath

Googling again in early 2008, I did find an explanation from mid-2006. (I don't know why I never found this before. My google-fu is excellent.)

To understand the problem, here is an explanation from the IE team (thanks Peter Gurevich!):

Each IE 6 window is a UI thread.

The HTML page and any script you wrote for the page run in the UI thread. Therefore filters in your page or in script will download on the UI thread

IE’s implementation of the AlphaImageLoader filter downloads images synchronously

Synchronous loading of an image or successive images on the UI thread has the potential to hang the browser and adversely affect the user experience.

They recommend using transparent GIFs as a workaround. We recommend using PNG8s.

I don't know why the AlphaImageLoader deadlock hasn't been a problem for more people. If they're out there, they haven't been vocal. The AlphaImageLoader hack is well known. The IE CPR engineer knew of only one other case like ours.

We have since decided that we will no longer make heroic efforts to get our web application looking just as good on IE6 as it does in modern browsers. Quite apart from the extraordinary amount of pain we endured in the Affair of the Transparent PNGs, fighting with IE6 has been a huge timesink for us, especially when it comes to working around its CSS bugs.

A significant fraction of our users are still using IE6, so we have no choice but to support it. However, we no longer aim to achieve full parity.

Alas, the recent announcement that Microsoft will be making IE7 an auto-update turns out not to be a panacea. IE7 will replace IE6 on intranets, at the discretion of IT organizations, but not on the Internet at large.

Acknowledgements. Despite my well-founded distaste for IE6, I'd like to thank the IE team members who helped us out. The amazing deadlock photo at the top of this article is taken from Ned Batchelder, who apparently got it from Reddit.

Posted by George V. Reilly in Web Development | Permalink | Comments (4) | TrackBack (0)

March 08, 2008

The Cozi Tech Blog

At Cozi, we're all about simplifying family life though great software. And respect for families permeates our corporate culture, too. For example Cozi employees have a solid work/life balance, we hold weekly "family meetings," we offer four weeks of vacation a year, we meet in the "family room," etc.

We're starting this Cozi Tech Blog to give back to another important family: the technical community. The technical community has given many things to Cozi, including MySQL, jQuery , NHibernate, and other open source technologies, which have saved us a lot of money and have made our jobs easier. We hope we can begin to return the favor by sharing what we're learning as we build Cozi.

Unlike the Cozi main blog, this blog is written primarily for software engineers. We look forward to your comments and suggestions. Who knows—one day this blog may tempt you to join our family.

Posted by Will Friedman in About | Permalink | Comments (0)

	FAQ Get Cozi Tell a Friend Privacy Policy Terms of Use Press Contact Us Cozi Blog Discuss Cozi
	© 2008 Cozi Group, Inc. Cozi is a trademark of Cozi Group, Inc. in the USA.