How do I avoid using fx_5_0?

Jul 11, 2014 at 5:49 PM
With the recommendations around Effects 11 to move away from using it, there's a lot of questions of how you are supposed to use HLSL without fx_5_0. For example, here's a question I got today:
I’ve been compiling my effect files to bytecode and storing them in my own file format for my engine. I’m currently using the deprecated fx_5_0 target for now when compiling. The reason for this is so that I can later create vertex, geometry, pixel, ect. shaders respectively. When compiling a hlsl file with D3DCompile and you specify one of the recommended targets (e.g. vs_5_0, ps_5_0, ect.), I get an error (D3DCompile: pEntrypoint pointer is invalid). In the docs say that D3DCompile ignores the pEntrypoint parameter, so specifying NULL is recommended. Am I missing something? Also, when specifying one of the recommended targets, is D3DCompile compiling the whole effect file to bytecode or just the specific portion of the file for the target and entry point? Thanks!
Q: How does the legacy fx_5_0 profile do anyhow?

A: When you run a .fx file through the HLSL compiler with the fx_5_0 profile, it does essentially two major functions. First it generates a bunch of metadata from the various elements in the .fx file like SAS annotations, state blocks, etc. and the technique/pass groups. The second is that it executes the HLSL compilation for each specific shader specified in the technique/passes: once for each compile statement. So for example, in the BasicHLSLFX11 sample you can find this technique:
technique11 RenderSceneWithTexture1Light
    pass P0
        SetVertexShader( CompileShader( vs_4_0_level_9_1,
            RenderSceneVS( 1, true, true ) ) );
        SetGeometryShader( NULL );
        SetPixelShader( CompileShader( ps_4_0_level_9_1,
            RenderScenePS( true ) ) );

        SetDepthStencilState( EnableDepth, 0 );
This creates metadata about "RenderSceneWithTexture1Light" and "EnableDepth" and then internally invokes the HLSL compiler twice:
  1. Compile the .fx file with shader profile "vs_4_0_level_9_1" and entry-point "RenderSceneVS" with some static parameters "1, true, true".
  2. Compile the .fx file with shader profile "ps_4_0_level_9_1" and entry-point "RenderScenePS" with the static parameters "true".
All the compiled shader blobs and metadata are put into the output for consumption by the Effects11 runtime.

Q: How do I build the individual shaders without fx_5_0?

A: So instead of letting the HLSL compiler generate a bunch of magic shaders, you can use the same .fx file and build each shader combination yourself. Because of the uniform variables, you need to modify the file to add a specific entry-point for each combination
VS_OUTPUT RenderSceneWithTexture1LightVS( float4 vPos : POSITION,
                         float3 vNormal : NORMAL,
                         float2 vTexCoord0 : TEXCOORD )
    return RenderSceneVS( vPos, vNormal, vTexCoord0, 1, true, true );
Then you compile it yourself:
fxc /Tvs_4_0_level_9_1 /ERenderSceneWithTexture1LightVS BasicHLSLFX11.fx
When using these compiled shaders, you have to load each compiled shader blob, set it to the device, and any state or other blocks have to be created by your code as well.

Q: Why does MSDN say you should pass null for pEntrypoint?

A: The MSDN doc page for D3DCompile is a little confusing. It should say:
The name of the shader entry point function where shader execution begins. When you compile using a fx profile (i.e. fx_4_0, fx_5_0, etc.), D3DCompile ignores pEntrypoint; in which case we recommend that you set pEntrypoint to NULL because it is good programming practice to set a pointer parameter to NULL if the called function will not use it. For all other shader profiles, a valid pEntrypoint is required.