Douglas Stockwell's Weblog

F# in Visual Studio 2008 for free!

Douglas Stockwell — 2007-11-26T13:15:45+10:00

Although there will presumably be a Visual F# Express at some point in the future, at the moment the F# Visual Studio addin requires a full version of Visual Studio since the other express versions do not allow addins.

It just so happens that with the release of Visual Studio 2008 we can now download a "full" version for free, it's called Visual Studio 2008 Shell and is designed specifically for language addins. The only difference is that it comes naked, without support for C#, VB.NET or C++.

Just install Visual Studio 2008 Shell (Integrated mode) then F#. If the F# addin does not function at first, launch "devenv.exe /setup".

Update: Paste From Visual Studio for Windows Live Writer

Douglas Stockwell — 2007-11-24T14:00:08+10:00

I just released a new version of my Windows Live Writer plugin Paste From Visual Studio.

The new version sports:

More compact HTML output
Support for non-ASCII characters
Automatic stripping of leading whitespace

Sample output:

public string CssColor(int i)
{
    if (i >= 0 && i < colors.Count)
    {
        string name = null;
        Color color = colors[i];

        if (namedColors.TryGetValue(color.ToArgb(), out name) && name.Length <= 7)
            return name.ToLower();
        else
            return string.Format("#{1:x2}{2:x2}{3:x2}", i, color.R, color.G, color.B);
    }
    else
        return "black";
}

Corresponding HTML:

<pre class="code"><span style="color:blue">public string </span>CssColor(<span style="color:blue">int </span>i)
{
    <span style="color:blue">if </span>(i &gt;= 0 &amp;&amp; i &lt; colors.Count)
    {
        <span style="color:blue">string </span>name = <span style="color:blue">null</span>;
        <span style="color:maroon">Color </span>color = colors[i];

        <span style="color:blue">if </span>(namedColors.TryGetValue(color.ToArgb(), <span style="color:blue">out </span>name) &amp;&amp; name.Length &lt;= 7)
            <span style="color: blue">return </span>name.ToLower();
        <span style="color:blue">else
            return string</span>.Format(<span style="background:yellow;color:#a31515">"#{1:x2}{2:x2}{3:x2}"</span>, i, color.R, color.G, color.B);
    }
    <span style="color:blue">else
        return </span><span style="background:yellow;color:#a31515">"black"</span>;
}</pre>

The Imagine Cup pays off: RikPVR

Douglas Stockwell — 2007-11-14T15:20:26+10:00

After the 2004 Imagine Cup [pdf], Richard and I walked away with memories and a new wardrobe of Microsoft T-Shirts. In 2008 Readify is offering paid work placements to the top finalists. Times have changed, but the core of the competition remains the same - building amazing software.

T-Shirts aside we also built a lot of IP, which Richard and his passion for recording copious amounts of Television has slowly been able to turn into a viable commercial application.

On the 20th of October, Richard launched RikPVR!

This is the evolution, or perhaps revolution of what was once the "TV Prognosticator": n-thousand lines of code that flew us both half way around the world. Rewritten, tuned and tested, but the premise remains the same:

Putting you in control of your TV viewing by making recording simple.

All you need is a Windows PC (XP or better) and a Digital TV Tuner (of the DVB-T kind) and you will be able to:

Get setup in just a few minutes with the setup wizard.
Pull in guide data from any XMLTV source.
Record n-channels with n-tuners in n-clicks.
Check on recordings, stop early or add time if they're running late.

Richard is continuing to innovate, over on the RikPVR blog he mentions that in the future it should be possible to record multiple co-channels on a single tuner, there's also a rumor that viewing live TV and recordings in progress is not far off.

Minimal CAPTCHA unleashes unfounded levels of genuine comments on 11011.net

Douglas Stockwell — 2007-11-14T12:56:08+10:00

Joseph was recently disgraced after his learnwpf CAPTCHA was h4cked by websecurity.com.ua. That got me thinking about my own highly secure implementation.

About 18 months ago the spam on my blog reached the point where I had to do something about it. I weighed my time and spent about 30s implementing the following.

And... I haven't had a single spam since. With relatively low traffic and a unique CAPTCHA it's easy enough to slide under the spammers radar. However by requiring even the most minimal CAPTCHA spammers have damaged the usability of my site. The extra effort required to click that extra checkbox is obviously the main deterrent for genuine commenters.

Until today! Announcing the "Magical Disappearing Minimal CAPTCHA" (MDMCAPTCHA).

I've also spiced things up by adding live preview and validation - fewer clicks!

Observable LINQ

Douglas Stockwell — 2007-10-16T09:33:06+10:00

So I have been neglecting my blog, Observable LINQ Part 3 never made it and it now looks like Paul Stovell will beat me to release with his SyncLINQ project. Head over to his blog for a nice screencast on the technology. This is very similar to my implementation which is already alive in RikReader. But not to worry, Observable LINQ may one day see the light of day, I still have a few tricks up my sleeve ;)

Thanks for using Microsoft Connect!

Douglas Stockwell — 2007-08-21T12:11:04+10:00

Product/Technology - Visual Studio and .NET Framework
Feedback ID - 284008
Feedback Title - Valuetypes are boxed in Array.BinarySearch(T[], T), Array.Sort(T[]), Comparer.Default and EqualityComparer.Default
The following fields or values changed:
Field Resolution changed from [Not Set] to [Not Reproducible]
Field Status changed from [Active] to [Resolved]

Are the testers incompetent or is it all in my head?

If anyone with an x64 machine would like to help confirm my sanity, could you compile and run the following program?

static class Test
{
    static void Main()
    {
        int[] ints = new int[] { 1 };

        while (true)
        {
            System.Array.BinarySearch<int>(ints, 1);
        }
    }
}

Fire up the "Performance Monitor" and add counters for (.NET CLR Memory):

Gen 0 Collections - a lot
% Time in GC - I'm seeing about 0.7%
Allocated Bytes/sec - about 400mb/s

I'm running Windows Vista Ultimate x64 on an E6600 with the x64 CLR 2.0.50727.

My theory for why it doesn't reproduce on x86? "secret magical JIT optimisation".

RikReader 1.6

Douglas Stockwell — 2007-07-18T13:08:52+10:00

1.5 was a silent release mostly dealing with performance improvements.

1.6 adds annotations: text/ink tagging, and highlighting. Text annotations are indexed and can be searched using the "tag:" prefix. Handwriting recognition will be added for Ink annotations in a future release. Annotations can be added using some new icons at the bottom left of the two-pane reader. Text annotations can also be added by CTRL-T.

I uncovered a indexing problem where some items were not being removed from the index when deleted. 1.6 addresses this by rebuilding the entire index, so please be aware that this process can take several minutes on the first run.

Head over to the main RikReader page for download links.

JScriptConverter fixed

Douglas Stockwell — 2007-07-04T15:32:46+10:00

I've been chasing a bug in RikReader ever since people started sending error reports back to me. This particular one is curious because it's dependent on the Operating System's UI Language, so I get the reports back in an assortment of languages.

Todo: I could probably serialise these in English initially, but that's not the problem at the moment.

FileNotFoundException:

Невозможно загрузить файл или сборку "...y88nnzzx.dll" или один из зависимых от них компонентов. Не удается найти указанный файл.
Não foi possível carregar arquivo ou assembly '...7uwqjtre.dll' ou uma de suas dependências. O sistema não pode encontrar o arquivo especificado.
Impossible de charger le fichier ou l'assembly '...qxpulisn.dll' ou une de ses dépendances. Le fichier spécifié est introuvable.
Det går inte att läsa in filen eller sammansättningen ...pyq_nlzi.dll eller ett av dess beroenden. Det går inte att hitta filen.
無法載入檔案或組件 '...5gxdnvh4.dll' 或其相依性的其中之一。系統找不到指定的檔案。
Die Datei oder Assembly ...6dokpecz.dll oder eine Abhängigkeit davon wurde nicht gefunden. Das System kann die angegebene Datei nicht finden.
ファイルまたはアセンブリ '...gnxsmxbd.dll'、またはその依存関係の 1 つが読み込めませんでした。指定されたファイルが見つかりません。

Which I think means "Unable to load assembly <file.dll> or one of its dependencies. The file cannot be found."

It was caused by this seemingly innocent code in JScriptConverter:

CompilerParameters cp = new CompilerParameters();
cp.GenerateInMemory = true;

CompilerResults results = 
    new Microsoft.JScript.JScriptCodeProvider()
    .CompileAssemblyFromSource(cp, source);

Assembly result = results.CompiledAssembly;

I have the Japanese MUI set up on another account, so I was able to find out that CompileAssemblyFromSource was setting an error status which I didn't check. That error being the even more cryptic:

error JS2999: Microsoft.Vsa.VsaException: ItemNameInUse (0x80133024)

I don't know what that means, nor why it is dependent on MUI. So I gave up searching and just rewrote JScriptConverter to set up the evaluation from C# instead of generating a seperate assembly.

private static VsaEngine engine = VsaEngine.CreateEngine();
private static JSLocalField[] fields = new JSLocalField[]
    { new JSLocalField("values", typeof(object[]).TypeHandle, 0) };

private static object Evaluate(string code, object[] values)
{
    StackFrame.PushStackFrameForMethod(new object(), fields, engine);
    ((StackFrame)engine.ScriptObjectStackTop()).localVars[0] = values;
    object result = Eval.JScriptEvaluate(code, "unsafe", engine);
    engine.PopScriptObject();
    return result;
}

Which turned out simpler than expected. The full implementation of JScriptConverter is here.

Comparer Boxing Workaround

Douglas Stockwell — 2007-06-24T19:24:29+10:00

Following on from my previous post, I've created some wrappers around System.Collection.Generics.Comparer<T>.Default and System.Collections.Generic.EqualityComparer<T>.Default which return specialised versions for dealing with ValueTypes without boxing.

For the previous example, we can eliminate the boxing like this:

int[] ints = new int[] { 1 };
Array.BinarySearch<int>(ints, 1, Utils.Collections.Comparer<int>.Default);

Or for a Dictionary<K,V>.

new Dictionary<int, int>(Utils.Collections.EqualityComparer<int>.Default);

The code is here.

Update: Actually, it seems Dictionary<K,V> is broken too, since all the methods which take a Key (Add, Contains, TryGetValue, Remove) still end up boxing by checking the key against null.

BCL Generics Boxing Bug?

Douglas Stockwell — 2007-06-24T17:47:26+10:00

Here's another challenge, "Binary search an array of int without boxing", I'll even give you the wrong answer "Use the generic Array.BinarySearch<T>(T[], T) method".

int[] ints = new int[] { 1 };
int index = Array.BinarySearch<int>(ints, 1);

Try running the second line 100,000 times in the CLR Profiler and you'll see a good 2.3mb or so of heap allocations. The cause as shown by the CLR Profiler lies in GenericArraySortHelper<T>. BinarySearch(T[], Int32, Int32, T). The corresponding C# probably looks something like this:

if (array[mid] == null)
    cmp = (value == null) ? 0 : -1;
else
    cmp = array[mid].CompareTo(value);

In IL, that comparison against null looks like this:

ldarg 'value'
box !T
brfalse 'somewhere'

The box instruction is something you really don't want to see in a code path that accepts ValueTypes. One way to avoid it would have been a separate ArraySortHelper for dealing with value types - but for now no such class exists.

The BCL authors of this specific area seem to have been unaware of the boxing, at least that's what these comments from the SSCLI/2.0 seem to indicate:

// This function is called when the user doesn't specify any comparer.
// Since T is constrained here, we can call IComparable<T>.CompareTo here.
// We can avoid boxing for value type and casting for reference types.

On further examination Array.Sort<T>, Comparer<T>.Default and EqualityComparer<T>.Default seem to be equally broken for value types. This is a pretty serious issue, even more so since Array.Sort/BinarySearch is used internally by List<T>. IEquatable<T> and thus a default EqualityComparer<T> is used by List<T>.Contains and much of Dictionary<T,U>. I'm surprised EqualityComparer<T> now seems to be boxing since I vaguely remember using IEquatable<T> to solve boxing in a Dictionary once before.

I have filed a bug here on connect.microsoft.com.

There are workarounds, but they require writing your own code, the easiest is to pass your own IComparer<T>.

Update (11 July): Maybe we can breathe a little easier, it looks like the problem may only occur on the x64 CLR. I guess the x86 JIT is smart enough to skip over the box/brfalse instructions!

C# Closure Quiz

Douglas Stockwell — 2007-06-20T15:19:59+10:00

void Test(out SomeDelegate d1, out SomeDelegate d2)
{
    object a = new ObjectGraph();
    d1 = delegate { return a; };

    object b = new ObjectGraph();
    d2 = delegate { return b; };
}

This contrived method is the potential root of some undesired characteristics in a program.

What is the problem? (hint: ildasm is your friend)
How can we refactor the method to avoid it?

RikReader has expired again!

Douglas Stockwell — 2007-06-02T00:40:08+10:00

As usual I managed to miss the time-bomb expiry date, sorry! A new build is up now with some small fixes and a new expiry date. Launch RikReader twice or Click Here to update.

Visual Studio to HTML (vs2html.exe)

Douglas Stockwell — 2007-05-18T15:57:31+10:00

My Windows Live Writer plugin, "Paste from Visual Studio" has been doing quite well, with almost 1000 downloads since last November.

But not everyone uses Writer and there far are more scenarios for converting source code to HTML than just blog posting. As a result, I've created a little utility which takes the current contents of the clipboard and replaces it with the converted HTML.

The utility, vs2html.exe, has no user-interface, no options, and no error handling. I find it's best used by adding it to the quick launch menu - then just CTRL-C in VS, hit the icon, and CTRL-V wherever you need the HTML.

You can configure vs2html to use a specific font and size instead of the default (Consolas, 12). To do this just to rename the executable using the format "vs2html.consolas.12.exe" (vs2html.font.size.exe).

Download vs2html.exe (24kb) vs2html.zip (6kb)

Sample:

[ContentProperty("Child")]
public class AdornerPopup : Adorner, IAddChild

Output:

<pre style="font-family:consolas;font-size:12">[<span style="color:rgb(43,145,175)">...
<span style="color:Blue">public</span> <span style="color:Blue">class</span> <span s...

Observable LINQ, Part 2

Douglas Stockwell — 2007-05-14T20:17:04+10:00

In Part 1, I introduced StackedList<T>, a virtual list of observable sub-lists.

More recently I've been working on a cross between a ListBox and a TreeView (putting hierarchical data inside a list - the exact opposite of ATC's TreeListView). This has some nice benefits: multiple selection, filtering, virtualisation, sorting (although to be honest, sorting an entire TreeView proves an interesting, but perhaps not so useful experience).

To convert a Tree into a List, we have to flatten the data structure. In a tree, a single node is an entity with a value and a list of children. After conversion to a list, a node becomes a list where the head of the list is the value, and the tail is the list of descendent values.

Here's the magic method that kicks this all off:

public static IList<TreeListItem> CreateList(object value)
{
    return new TreeListItem(value).items;
}

The constructor, and items are both private, so perhaps this seems bizzare at first. But remember, the head of the resultant list is the TreeListItem that corresponds to the value that was passed in:

this.items = new StackedList<TreeListItem>(
    new TreeListItem[] { this });

TreeListItem extends ListBoxItem and pushes some bits around to wire up HierarchicalDataTemplates like the real TreeView does. At some later point in time, the item is expanded, and the children ItemsSource is bound.

private void PopulateChildren(Collections.IList newValue)
{
    ...

    this.flattenedChildren =
        new FlattenedList<TreeListItem>(
            new ConvertedList<object, IList<TreeListItem>>(
                Utils.IList.UntypedToTyped<object>(newValue),
                delegate(object value)
                {
                    return new TreeListItem(value).items;
                }));

    this.items.AddCollection(flattenedChildren);
}

Tada! More virtual lists.

FlattenedList<T> uses a StackedList<T> internally to convert an IList<IList<T>>, a list of lists, into IList<T>. So, FlattenedList is used to flatten a list of child lists into a single list - which then becomes the tail of the list for the current node.

ConvertedList<TInput, T> uses a conversion delegate to convert an IList<TInput> to an IList<T>. ConvertedList is used here to take the list of child nodes of type object, promote these into TreeListItems and then return the list corresponding to each node. So this ConvertedList takes an IList<object> and presents an IList<IList<TreeListItem> - which is then flattened into the parent nodes as described above.

To be as generic as possible, my TreeListItems support binding to child lists of the non-generic type IList or some generic IList<T>, in future I'll probably extend this support to any type that at least implements IEnumerable.

Utils.IList.UnTypedToTyped<T>(IList list) dynamically promotes a non-generic list into a read-only strongly typed IList<T>. This is supported by a couple of additional collection wrapper classes: UpcastList and DowncastList for dealing with IList<T> of some other T, and ObjectList for dealing with a non-generic list.

All the classes mentioned here virtualise the INotifyCollectionChanged events raised by child collections. The end result is that we can flatten an entire hierarchical data structure of observable lists into a single, virtual, observable list.

Stay tuned for part 3, where we will harness the power of virtual lists together with the syntactic goodness of LINQ.

Observable LINQ, Part 1

Douglas Stockwell — 2007-05-14T18:10:46+10:00

Here's a simple LINQ query in C# 3.0:

var people = new ObservableCollection<Person>();
var friends = new ObservableCollection<string>();

var myFriendsAges = 
    from p in people
    join f in friends on p.Name equals f
    select p.Age;

Using ObservableCollection<T> from .NET 3.0 doesn't buy us anything that List<T> wouldn't. We end up creating an ordinary sequence, IEnumerable<int>. But, it doesn't have to be this way, LINQ is easily extended. Read on.

For use in RikReader's hierarchical view I implemented a new collection class, StackedList<T>. This is similar to System.Windows.Data.CompositeCollection in that I can flatten multiple collections into a single list. But StackedList is far more generic, it will directly flatten any IList<T>, and should the child list implement INotifyCollectionChanged, the collection change events are remapped such that StackedList is able to provide a complete read-only illusion of a virtual collection.

StackedList<int> list = new StackedList<int>();
Console.WriteLine(list.Count); // -> "0"

ObservableCollection<int> childList = new ObservableCollection<int>();
childList.Add(1);
list.AddCollection(childList);
Console.WriteLine(list.Count); // -> "1"

childList.Add(2);
Console.WriteLine(list.Count); // -> "2"
Console.WriteLine("{0}, {1}", list[0], list[1]); // -> "1, 2"

If we tried to write the same program using CompositeCollection we'd probably end up with something like this:

CompositeCollection composite = new CompositeCollection();
CollectionView view = CollectionViewSource.GetDefaultView(composite) as CollectionView;
Console.WriteLine(view.Count); // -> "0"

ObservableCollection<int> childList = new ObservableCollection<int>();
childList.Add(1);
CollectionContainer container = new CollectionContainer();
container.Collection = childList;
composite.Add(container);
Console.WriteLine(view.Count); // -> "1"

Even though CollectionView has a Count property, it doesn't implement IList, it actually exposes a new list like interface ICollectionView which is IEnumerable. CollectionView is also non-generic, but notwithstanding, it is a similar approach.

With StackedList I was easily able to flatten a hierarchy of Feeds and Folders into a single, observable list of Items - which was then directly bindable for visualisation.

Stay tuned for Part 2, where we will explore the application of some other virtual lists.