We will hunt down and depict the bug!

My nVidia GPU fails to divide 33,37,41,47,55,66,74,75,82 or 94 by itself

Just when i finished debugging a generative shader, that renders a texture for proper addressing of a hexagonal neighbourhood (see previous post), i ran into that glitch for edges of 21, 24, 28, 31 and probably others.

I started media player classic, opened it's nice little pixel shader editor and reduced the code to the fllowing lines:

(Sorry my bLog has no highlighting for HLSL yet.)

sampler s0 : register(s0);
float4 p0 :  register(c0);

float4 main(float2 tex : TEXCOORD0) : COLOR {

    // Map x to numbers 0 to 100:
    float n = floor(tex.x * 100);
    
    // This expression should always result 1.0, but due to rounding errors
    // for n in {33,37,41,47,55,66,74,75,82,94} it's resulting below 1.0:
    float err = trunc(1.01 * n ) / n;
    
    // Let's visualize it
    float4 c = 1.0;
    // Original error in red: 
    c.r = frac( err );    
    // adding this epsilon prevents the rounding error in green:
    c.g = frac( err + 0.000001);
    // just a little less will not prevent the error in blue:
    c.b = frac( err + 0.00000099);
    
    // So, red is the error and green prevents it while blue shows up again, right?
    // I get ten magenta stripes badly hidden in 90 black ones:
    return c;
}

Obviously, the line in charge is the division by itself. To prevent the compiler from optimizing, i left one multiplication that should in any case lead to the original integer after the truncation:

float err = trunc(1.01 * n ) / n;

And that is what my media player shows up (open some dummy video file to reproduce it):