Cory Plotts’ Blog

I’ve been toying with and working on this blend mode library of mine for some time now (see posts 1, 2, 3, 4, 5). However, it is now only with WPF 4.0 (currently Beta 2) … that I can actually implement all the blend modes.

Hue, Saturation, Color, and Luminosity: Where Are You?

If you were watching closely, you will have noticed that I have not (to date) implemented the Hue, Saturation, Color, and Luminosity blend mode effects.

Why is that? Whenever fxc.exe would compile the .fx file (through the Shader Effects BuildTask and Templates), it would give me the following two errors:

• error X5608: Compiled shader code uses too many arithmetic instruction slots (100). Max. allowed by target (ps_2_0) is 64.
• error X5609: Compiled shader code uses too many instruction slots (102). Max. allowed by the target (ps_2_0) is 96.

Those errors mean that you’ve exceeded the limitations of Pixel Shader 2.0 … which is what WPF 3.5 SP1 targets for pixel shader effects.

So, what can you do about this? You could optimize the HLSL so that your effect code goes under the limits … but if you notice in the errors above (these particular errors are for the Saturation effect) … you will see that I am significantly above the number of allowed arithmetic instruction slots (100 versus 64). I would have to optimize significantly.

WPF 4.0 to the Rescue

Another option is to utilize WPF 4.0!

WPF 4.0 increases what you can do with effects because it now allows you to target Pixel Shader 3.0. This wikipedia article has a great chart on the differences between the different shader models. With PS 3.0, we shouldn’t hit any limits.

Realizing this, as soon as Visual Studio 2010 and .NET 4 Beta 2 came out (PS 3.0 is only available in Beta 2), I installed it and gave it a whirl. There was really only one gotcha in getting this to work. The Shader Effects BuildTask only compiles to PS 2.0 … and there is no option to toggle between PS 2.0 and PS 3.0.

This means that, for now, probably until release, you will have to manually run fxc.exe (targeting ps_3_0) and add the compiled shader output file (the .ps file) to the project as a Resource.

Caveats

There are some caveats to using PS 3.0 in WPF 4.0. Most notably, there is no software fallback if your hardware or situation (remote desktop, forced software rendering) don’t support it. This means that your effect code will simply not run and not appear.

Some good news is that Microsoft has extended the RenderCapability API so that you can determine what level of pixel shader support is currently available. They’ve updated the RenderCapability.IsPixelShaderVersionSupported method and have added two additional methods: RenderCapability.IsPixelShaderVersionSupportedInSoftware and RenderCapability.MaxPixelShaderInstructionSlots.

(The above information and more is available in David Teitlebaum’s PDC09 session titled Mastering WPF Graphics and Beyond. Jamie Rodriguez also has some great posts covering what’s new in WPF 4.0 (Beta 1, Beta 2).)

Another caveat that bears mentioning is that Silverlight (both 3.0 and 4.0) only support PS 2.0. So, if you are creating and using effects across both technologies, you will not be able to use PS 3.0. This also means that the Hue, Saturation, Color, and Luminosity blend modes (and the attached code) are WPF only blend modes. Sorry Silverlight! I love you, but that’s how the chips fall.

Hue, Saturation, Color, and Luminosity: Here They Are!

Alright … enough with the caveats! Let’s see the effects!

First, let me refer you back to Angie Bowen’s post where she covers what each blend mode does … including the Hue, Saturation, Color and Luminosity effects.

To show off what each effect does, I’m going to go ahead and use her reference images … first showing how they blend out of Photoshop … and then showing how they blend with my library … for comparison purposes. Why? Well, it seems as if my blend mode library does not always match what is coming out of Photoshop … sometimes it’s pretty darn close and at other times it’s more than just a little bit off.

I have been through and through the math (and HLSL) for these effects and I honestly can’t find anything wrong there. If I could speculate what is going on, I would say that I’m bumping up against some float versus double precision issues (where Photoshop is using doubles for greater precision) during the conversion from RGB to HSL and back. If you can figure out what I’ve done wrong (if anything) please let me know.

However, for the most part, they work great!

Hue Blend Mode

The result is the hue of the blend layer and the saturation and luminance of the base layer.

Photoshop
My Blend Mode Library

Photoshop
My Blend Mode Library

Saturation Blend Mode

The result is the saturation of the blend layer and the hue and luminance of the base layer.

Photoshop
My Blend Mode Library

Photoshop
My Blend Mode Library

Color Blend Mode

The result is luminance of the base layer and the hue and the saturation of the blend layer.

Photoshop
My Blend Mode Library

Photoshop
My Blend Mode Library

Luminosity Blend Mode

The result is hue and the saturation of the base layer and the luminosity of the blend layer.

Photoshop
My Blend Mode Library

Photoshop
My Blend Mode Library

The Binaries and the Source Code

Here is the source code for the library and here is the assembly.

Again, remember that this is a WPF-only release as Silverlight does not yet have support for Pixel Shader 3.0.

Also, all of these new blend modes are opacity aware … meaning that you can pull the opacity out of the upper layer to lessen the effect of the blend mode (see my last post on the matter).

Enjoy!

p.s.

At one point, I thought that the Shader Effects BuildTask and Templates was built into Visual Studio 2010. It appears I was mistaken about that. I have updated this post to indicate that.

Blend Modes, Pixel Shader Effects, WPF

I had the opportunity to attend my first PDC this year and it was great! The content was top-notch and it was extremely fun to meet a lot of the Microsoft guys (and non-Microsoft guys) that I’ve worked with over the years (albeit virtually).

Here is my recap of the event.

The basic story was around Windows 7, Windows Azure (cloud computing), Visual Studio 2010, .NET 4 (WPF 4), and Silverlight 4.

If you weren’t able to go, you can still take in all the content yourself at your leisure … as Microsoft has made it publicly available at http://microsoftpdc.com/Videos.

During the event, I also live tweeted most things I attended. Check out http://twitter.com/cplotts if you want a …  ‘as it happened’ feel for the event.

Sessions That I Attended

 

Session Code	Session Title	Presenters	Notes
WKSP04	Getting the Most out of Silverlight 3	Ian Griffiths, Richard Griffin	This workshop was tag team with Ian Griffiths covering basic Silverlight concepts and then how to go about hooking a Silverlight application up to some services. Richard took over and talked about using MVVM and MEF with Silverlight and ended the day showing how to use Blend to turn a screen shot into a living breathing application.
KEY01	Day 1 Keynote	Ray Ozzie, Bob Muglia	This keynote was all about Windows Azure and cloud computing. Of particular note was the announcement for Microsoft PinPoint … a new marketplace for online services … and the announcement for Microsoft Codename ‘Dallas’ which is Microsoft’s effort to treat data as a service (which they will make available through PinPoint).
CL09	How Microsoft Visual Studio 2010 Was Build with WPF 4	Paul Harrington	This was a great session where Paul described how Microsoft rewrote Visual Studio using WPF. This was a major undertaking and they had to actually add functionality to the WPF API in order to make it happen. Also of note was the many keyboard interop issues that they tackled and how they did so. A great WPF session, especially for those doing interop.
CL30	Microsoft Expression Blend 3 for Developers: Tips, Tricks, and Best Practices	DoRon Motter	This was a quick lunch session where DoRon went through 9 quick tips on using Expression Blend.
CL14	Advanced Graphics Functionality Using DirectX	Michael Oneppo	This was the 2D DirectX session that I attended. Most of it went right over my head, but I would like to dive in and understand DirectX since WPF is built on top of it.
CL11	Advanced WPF Application Performance Tuning and Analysis	Eric Harding, Bart De Smet	In this session Eric and Bart go through some general performance truths and then show you how to use vmmap, .NET Memory Profiler, Perforator, Visual Profiler, Process Explorer, and xPerf to solve performance problems in the areas of memory, cold start, warm start, and runtime. This is a must watch session for software engineers using WPF.
KEY02	Day 2 Keynote	Steven Sinofsky, Scott Guthrie, Kurt DelBene	This was the big keynote. Steven Sinofsky started it by talking about Windows 7. Then Scott Guthrie talked about Silverlight 4 and the new functionality it brings to the table. Finally, Kurt DelBene wrapped it up with Office 2010 and Sharepoint 2010. A lot of stuff happened in between too: demos of hardware accelerated text rendering in IE 9, demos of SketchFlow, demos of cool Silverlight 4 goodness, and more. And of course I have to mention that they gave away a free laptop to all attendees! Awesome with a capital A!
CL31	Mastering WPF Graphics and Beyond	David Teitlebaum	This was the session I was looking forward to the most. David rocks and I am very interested in WPF and in graphics. David is the WPF Graphics Program Manager. In this session, he covered the new WPF 4 graphics features (new text rendering, layout rounding, animation easing, pixel shader 3.0 support, cached composition, and more). Another must watch WPF session, especially geared towards those working in graphics. They should’ve given David a two hour slot, it was like drinking from a fire hose.
CL15	Modern 3D Graphics Using Windows 7 and Direct3D 11 Hardware	Michael Oneppo	This was the 3D DirectX session that I attended. Good stuff, mostly over my head. It whetted my appetite though. There was a great slide on the D3D10 pipeline showing where each shader fits in.
CL20	Improving and Extending the Sandbox with Microsoft Silverlight 4	Joe Stegman	This was a surprise session as this wasn’t announced until PDC itself and it was all about the work Microsoft has done to extend the Silverlight sandbox, and just in general to crack it open. Of particular note is the new elevated trust out of browser stuff … that allows you to do almost anything via COM Automation. This is a game changer and likely will prove beneficial to our own Silverlight efforts.
CL22	Attended CL22, Advanced Topics for Building Large Scale Applications with Microsoft Silverlight	John Papa	I have never heard John Papa speak, but he is excellent. And although the topic doesn’t suggest it … this talk was all about MVVM (the Model-View-ViewModel pattern). In the talk he talks about two main variations on the MVVM pattern and how to use Prism as a ‘gap filler’ where MVVM doesn’t provide any answers. This is a must watch session if you are doing WPF or Silverlight. The MVVM pattern is the industry de-facto standard and John explains it well and concisely. I myself will be going over this one again.
CL02	Microsoft Silverlight 3: Advanced Performance and Profiling Techniques	Seema Ramchandani	I was a little late to this session but the parts I caught were solid gold. Seema covered Silverlight graphics and performance and this talk fits right alongside Eric Harding’s and Bart De Smet’s WPF performance talk. Another must-see.
CL10	WPF 4 Plumbing and Internals	Blake Stone	This talk was probably the best of the conference. Ok, maybe that is unfair to the other great content, but Blake Stone did an amazing job describing some of the internals of WPF and many of the things that trip developers up. Here is a link to the slides and sample code. Needless to say, this WPF session is a must-watch.
FT24	Building Extensible Rich Internet Applications with the Managed Extensibility Framework	Glenn Block	This session is an introduction to MEF (Managed Extensibility Framework) which Microsoft is using internally in its own products. This is a great technology and Glenn always gives a good presentation.
CL35	Custom Behaviors for Advanced Microsoft Silverlight UI Effects	Pete Blois	Pete Blois is the guy who created Snoop. So, you have to love him. In this session, he shows the audience how to create custom Expression Blend behaviors, making it easy to drag, drop, and create compelling user experiences. If you have the Blend love, you’ll enjoy this session.
CL24	XAML Futures in Microsoft .NET Framework, Microsoft Silverlight, and Tools	Rob Relyea, Michael Shim	Rob Relyea and Michael Shim closed out the conference for me by talking about the new  and upcoming features in XAML 2009 (XAML 2006 is what is currently out there) and what has been going on in this space lately. Unfortunately, neither Silverlight 4 nor WPF 4 nor any of the tools take advantage of XAML 2009 yet.

 

Sessions That I Missed but Want to Watch

 

Session Code	Session Title	Presenters	Category
CL13	Windows Touch Deep Dive	Reed Townsend	Touch/Multi-Touch Development
CL27	Multi-Touch on Microsoft Surface and Windows 7 for .NET Developers	Anson Tsao, Robert Levy	Touch/Multi-Touch Development
CL23	SketchFlow: Prototyping to the Rescue	Christian Schormann	SketchFlow
CL01	Microsoft Silverlight 4 Overview	Karen Corby	Silverlight
CL06	Networking and Web Services in Microsoft Silverlight	Yavor Georgiev	Silverlight
CL21	Building Amazing Business Applications with Microsoft Silverlight and Microsoft .NET RIA Services	Brad Abrams	Silverlight
CL03	DirectX 11 DirectCompute	Chas Boyd	DirectX
FT35	Microsoft Visual C# IDE Tips & Tricks	DJ Park	C#
FT11	Future Directions for C# and Visual Basic	Luca Bolognese	C#
FT31	Dynamic Binding in C# 4.0	Mads Torgersen	C#
FT51	Future of Garbage Collection	Patrick Dussud	C#
FT07	The State of Parallel Computing	Burton Smith	Parallel Computing
FT21	PLINQ: LINQ, but Faster!	Ed Essey, Igor Ostrovsky	Parallel Computing
FT54	Power Tools for Debugging	Sandeep Karanth, Kapil Vaswani	Diagnostics, Debugging, Testing
FT16	Advanced Diagnostics, IntelliTrace, and Test Automation	Habib Heydarian	Diagnostics, Debugging, Testing
CL16	Optimizing for Performance with the Windows Performance Toolkit	Michael Milirud	Diagnostics, Debugging, Testing
VTL30	Managing Development to Inspire Innovation and Create Great Software Experiences	Scott Stanfield, Bill Crow	UX
VTL05	A New Approach to Exploring Information on the Web	Gary Flake	UX

 

What a great conference! Hope this recap helps you decide what to dive in on yourself and provides for some easy links to do so.

Enjoy!

General PDC09

 

My employer has graciously sent me to PDC this year and I’m stoked!

In fact, I’m here right now … about to walk down to the convention center, get registered, and attend one of the pre-cons (I’m going to the Silverlight one with the two Griffs (Ian Griffiths and Richard Griffin)).

If you see me there … don’t hesitate to say hello.

General PDC09

After I wrote my trio of blog posts (1, 2, 3) about blend modes using pixel shader effects, I was toying around with it …

 

The Problem

… and noticed that it didn’t handle an opacity changes on the upper layer (B).

In fact, this was brought to my attention by a great blog post by Angie Bowen. In it she explains how the blend modes work and, she says:

Remember that to get better results you can also adjust the opacity of the upper layer.

Trying it out, revealed that some of the blend modes were okay, but most were not. Most of the blend modes would simply result in a black square if you pulled all the opacity out of the upper layer (B). This was obviously wrong, for if you pull all the opacity out of the upper layer (B), you should get the lower layer (A).

Argh!

 

The Solution

So, I dove back in on the blend mode math, trying to figure out what I needed to do to make these blend modes … opacity aware. That’s got a nice ring to it, doesn’t it!?

I started at the top of the list and got the NormalEffect working:

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 inputColor;
    inputColor = tex2D(input, uv);

    float4 blendColor;
    blendColor = tex2D(blend, uv);

    inputColor.rgb = (1 - blendColor.a) * inputColor.rgb + blendColor.rgb;

    return inputColor;
}

Ok, the above math made sense. When the opacity of the upper layer (blendColor.a) was 1 (opaque), the result was just blendColor. Otherwise when the opacity of the upper layer was 0 (transparent), the result was inputColor.

So, I then started to tackle the darken blend modes (Darken, Multiply, …) and quickly ran into problems. It was at that point, that I ran into this post in the WPF Forum. My blend modes were rendering full black (not white), but it provided a crucial piece of knowledge.

Namely, WPF uses pre-multiplied alpha everywhere for performance reasons.

What does that mean? Well, it means that the RGB values for inputColor are already multiplied by the alpha value for inputColor and that the RGB values for blendColor are already multiplied by the alpha value for blendColor.

Ah! Do you see it? This explains why the blend modes were going to a black square when pulling the opacity out. Take the Multiply blend mode. In the above HLSL, it would be:

// R = Base * Blend
resultColor.rgb = inputColor.rgb * blendColor.rgb

So, if the alphas were pre-multiplied in and you were pulling opacity out of the blend (upper) layer … then blendColor.rgb would go to zero … which would cause resultColor to go to zero … which would cause the gradient square to go to black!

Thinking about this … brought about the general solution for making these blend modes opacity aware. I needed to simply:

1. Un-pre-multiply the blend layer alpha value out.
2. Apply the blend mode math.
3. Then re-multiply the blend layer alpha value in again.

Here is the HLSL for the opacity aware Multiply blend mode:

float4 main(float2 uv : TEXCOORD) : COLOR
{
    float4 inputColor;
    inputColor = tex2D(input, uv);

    float4 blendColor;
    blendColor = tex2D(blend, uv);

    float4 resultColor;
    resultColor.a = inputColor.a;
    // un-premultiply the blendColor alpha out from blendColor
    blendColor.rgb = clamp(blendColor.rgb / blendColor.a, 0, 1);

    // apply the blend mode math
    // R = Base * Blend
    resultColor.rgb = inputColor.rgb * blendColor.rgb;

    // re-multiply the blendColor alpha in to blendColor
    // weight inputColor according to blendColor.a
    resultColor.rgb =
        (1 - blendColor.a) * inputColor.rgb +
        resultColor.rgb * blendColor.a;

    return resultColor;
}

A few comments about the above code. Notice that I am clamp(ing) when I un-premultiply (i.e. divide) the alpha out. This assures that the RGB values will be between 0 and 1 (where 0 is black and 1 is white … HLSL operates in ScRGB color space). This is necessary since dividing by values close to 0 (blendColor.a) can yield large numbers or even positive infinity … which throws off the math.

Secondly, when I re-multiply the blend layer alpha value back in … I need to also properly weight the inputColor … just like I did in the NormalEffect above.

Finally, notice that I really don’t have worry about the opacity on the lower layer (A). I just pass its value off to the result by setting resultColor.a equal to inputColor.a.

Applying this general formula worked in all cases!

 

The Gradient Contour Test Harness

In order to verify that I was doing math correctly, and to see the effect of pulling the opacity out of the blend modes … I have built a new gradient test harness. I have called it the gradient contour test harness since it not only shows the A + B = R gradient squares but it also shows the R gradient square with contours … just like Paul Dunn’s post does when you mouse over the R squares.

It is extremely interesting (to me at least) watching the contours as you pull out the opacities.

For example, Here are three gradient contour squares for the Pin Light blend mode at opacity values of 1.0, 0.5, and 0.0:

I have also included a button labeled ‘Swap’ which swaps the A and B layers … since not all blend mode effects are commutative.

The gradient contours were made possible via Dwayne Need’s library. Check out the code (the class Grayscale4Bitmap) and see this post for more info.

The Image Test Harness

I’ve also put opacity sliders in the image test harnesses. Let’s take a look at the Pin Light effect at opacity values of 1.0, 0.5, and 0.3.

Images	Opacity
	1.0
	0.50
	0.3

As you can see … pulling out the opacity … lessens the effect that the upper layer/texture has on the lower layer … and proves the truth of what Angie Bowen was saying earlier about using the opacity of the upper layer to achieve better results.

 

The Binaries and the Source Code (aka The Goods)

So, as I’m fond of saying … without further adieu … here is source code for the Blend Mode library and here are the library binaries … now opacity aware!

I also have updated the Silverlight test harness (as always you will need the Silverlight 3.0 runtime).

p.s.

The gradient contour test harness is WPF only … you won’t find that on the Silverlight side. Maybe someday I’ll get my Silverlight test harnesses up to parity with what’s in the WPF test harnesses … but I can’t see when. Any one want to do it for me? Bueller? Bueller?

Blend Modes, Pixel Shader Effects, Silverlight, WPF