XamlXaml.com

My career has taken quite a few turns in the last year or so. I now find myself in the mobile phone industry. Software development for mobile devices is a whole other ball of wax than that for the desktop. While the purpose of this blog was originally to focus on XAML and WPF I’d like to expand it to include mobile development as well as the occasional post that has nothing (gasp!) to do with technology. Apologies to all for the now-not-quite-concise domain name

My work in WPF image manipulation lately has made me question just how good an idea it is to have WPF measurements not tied to pixels, but rather device-independent-units. I understand the theoretical benefits of the design such as controls automatically being able to scale to the correct size regardless of screen resolution, but when trying to manipulate images this all becomes another layer of abstraction which makes everything more difficult. 

The primary issue is the translation between what image is being displayed on the screen and the actual image size. For example, if a bitmap is 300×300 pixels the ActualHeight and ActualWidth of the Image object (System.Windows.Controls.Image) that the bitmap is associated with could be anything…from 15×15 to 1000×1000 device-independent-units. Yet, for the purposes of image manipulation the original image size always has to be used, because even if an image is displayed at 1000×1000 device-independent-units, the actual number of pixels contained in the BitmapSource is still 300×300. Great.

So, now we are left with having to manipulate the original bitmap and resizing/scaling it up to the display size. This is probably only a minor headache when dealing with 1 original image and 1 manipulated image, but when dealing with 1 original image decomposed into 1000’s of composite images it becomes problematical.

This became apparent while working on my current project, the Infinite Image Engine. The framework takes a source image, decomposes it into hundreds or thousands of swaths of average color, and then recomposes it using a database of images matching each swath to an image of similar average color. It is fairly straightforward when the source image size displayed matches the actual image size, but decidedly not so otherwise.

The primary problem is that the most convenient image manipulation handles are found in the Image’s BitmapSource (Image.Source). This has methods to copy pixels and modify pixel values, but it is all based on the original bitmap’s pixel sizes and does not reflect the display size. This is by-design, but that doesn’t leave developers with any built-in options to manipulate on-screen images. Instead we are left to create an abstraction layer to do the translation between the display size and the original image size. While this is certainly doable, it wouldn’t be my first or even second choice for how to get things done. If we are going to be stuck with device-independent-units in WPF, and overall I think it is a good design, then the measurement should be ubiquitous throughout the system, including at the BitmapSource level.

Michael G. Emmons

I wanted to clear up a previous post. As noted in a comment on the original post the DownloadCompleted and DownloadFailed events on BitmapImage objects only fire if the images are not being loaded from the internal cache. This explains the (correct) behavior that on the second iteration of bitmap loading in my test application no events fired–the single image that was being loaded was already cached.

To determine if the image has already been loaded simply check the IsDownloading property. If this is false directly after assigning the Source as a URI then the download events will not fire.

 Thanks to MS for responding quickly.

 -MGE

I have been having issues with the DownloadCompleted and DownloadFailed events failing to fire correctly from the BitmapImage object. This is a huge issue as the asynchronous processing of images is central to many types of WPF apps. Having events failing to fire correctly will likely have a rather large negative impact on these apps.

The problem is easily recreated. Iterate through any number of BitmapImage creations, subscribe to the DownloadCompleted and DownloadFailed events, and track the number of DownloadCompleted + DownloadFailed events which fire vs the number of events subscribed to. Those numbers should always add up.

In my testing the events only fire correctly on the first full iteration. After the first iteration they will never fire again. Ever. The event subscriptions are simply going into a black hole. Verify for yourself. You can create your own app or just use my test app.

I’d be interested to hear the results of everyones testing.

-MGE

The framework is coming along nicely. In the screenshots above you can see the before and after for my test image. The after image is composed of 800 photos pulled from flickr.

I’ve been focusing on the color matching algorithms, which I’m now pretty happy with. Although in the processed image you will notice that the halo around his head does not quite match, that is the result of settting a fairly high color tolorance of 20 rather than problems with color matching. Setting to a lower tolorance will result in closer matches (and also more difficulty in finding matches).

My next focus will be to improve rendering. For example, the overall look of the composite image will be a closer match if the same replacement image is not used more than once–or at least used only a very few times and not directly next to each other. Using the same images in succession results in too much of a pattern appearing in the image, as is the case with the processed image above. Less pattern is better. Think pointillism, like Matisse’s “La Maison verte, Venice” and you will have a good idea of what I’m striving for.

It’s interesting to note just how closely the composite will often match the original image. For example, the boy’s nose in the processed image is clearly visible. His ears and hair shading are distinct as well as the shadow under his chin. Some of the feather patterns on the left wing are even rendered…although in this case the wing pattern may be an example of the resuse of images working in favor of the composite image.

That’s it for now. More to come soon!

-MGE

I’ve always been fascinated by how our minds perceive visual data  (may I call that the MGPU for Mental Graphical Processing Unit? *cough*geek*cough*). What I find interesting is that what we see, mostly, is not an accurate reflection of reality. It is simply how our mind filters and manipulates the data it is presented with. Even when we are aware of exactly how we are being misled by misinterpreted visual data we still are not able to force our minds to interpret it correctly.

Case in point is this optical illusion:

Believe it or not the tiles marked A and B are exactly the same color. It’s impossible to see in this visual context, but take a piece of paper and cover up everything except those two squares and it becomes obvious.

One piece of optical tomfoolery we rely on everyday is the lowly pixel. A pixel is the smallest unit of measure on an LCD or CRT display. Pixels allow us to be fooled into seeing a color that isn’t really there by taking three phosphors, one red, one blue, one green, and manipulating them. Our eyes see a whole range of colors..even though the image is only made up of red, blue, and green phosphors. That’s the faulty wiring in our MGPU.

This rather long segway is an introduction to my current project: the Infinite Image Engine. You may have seen the coffee table books that have well-known, easily recognizable pictures that have been manipulated so that each image is made up of hundreds or even thousands of smaller pictures. If you look at it closely, all you see are a random group of images. If you look at it from a distance, you will see the larger image.

The Infinite Image Framework gets its name from the fact that it is possible to zoom infinitely on any image. As you zoom closer the image is resolved into smaller images. As you zoom into the smaller images these are comprised of yet smaller images, ad infinitum.

The framework I am designing will allow you to select an image and have that image be transformed into a collage of smaller images while still being recognizably the same. In its current testing incarnation you can choose keywords for the types of images you want to have included. The images will be pulled off of flickr and matched as appropriate.

The tricky parts of the framework are done and I’m just hammering out the tweaks that will allow maximum flexibility.

I will hopefully have some screenshots posted in the next few days and some apps/code posted shortly thereafter.

Cheers!

Michael G. Emmons

Here is the updated source code for the PostItBoard WPF example app so that it compiles and runs on the latest NET 3.0 / WPF RC1. Or if you are only inerested in the binaries you can grab those too. In addition, apologies to everyone who asked me about the source code for the flickr assemblies that were missing from the previous release. This release has all the source code and projects.

Please drop me a line if you have any problems running this WPF example application.

Note: I get quite a few emails asking me if the image above is an actual screenshot of the application or if it was photoshopped. The image was in no way photoshopped. It is the application as it appears at runtime.

Cheers!

Michael G. Emmons

Click the image to run the WPF ScrollingCanvas example xbap or you can get the source code for it. This example creates a custom user control based on the Canvas container. This ScrollingCanvas allows you to scroll any FrameworkElement: text, video, shapes, or even whole scenes. Moreover, you can scroll multiple FrameworkElement objects at the same time in a single scroll container.Both speed and direction of the scroll is controllable via dependency properties. Simply drop the ScrollingCanvas control on the design surface or add it in XAML, add any FrameworkElements you want to scroll to the ScrollingCanvas container and call the ScrollBegin() method. All object scrolling is handled through local animations in the C# codebehind.