I think I finally finished with the base of the terrain texturing!
I refactored the terrain shaders, using a very modular approach with tons of function calls and a clean design. I sure hope that the shader compiler is really good. If not, a final version of the shader might have to be written someday that flattens out the implementation and uses all manner of optimizations.
I also started using branching heavily. I am using both good and bad kind of branching. The good one relies on constants passed to the pixel shader body and I am pretty sure the compiler does compile time evaluation of the constants and removes unnecessary branches from the code. The bad kind of branching is the use of run-time "if"s in shaders. GPU really don't love branching. Pixels are evaluated in a clustered fashion and branches can cause the entire cluster to wait for a sync. This can be mitigated if there is a high probability that all the parallel shaders executed for the cluster will take the same path. I done some testing and the results are inconclusive, maybe tending to go a little bit toward having lower performance if I use branching, even thought the body of the branch that is skipped is more expensive.
Using these methods I created two shader implementations, one very basic for the low quality and one that handles higher quality rendering. These are further parameterized with compile time flags to create all variants that I need. I also managed to greatly optimize the implementation, giving a 10-15 FPS increase on weak hardware. On strong hardware I can't tell, because currently I am bus capped.
I also implemented adaptive detail mapping, allowing you to specify a radius for detail mapping. Medium quality setting use this, not for the performance, but because it reduces repeating patterns in terrain somewhat. On high I am not using it because the small performance gain is not worth it when compared to the quality loss. It is a high quality setting for a reason.
The final step was to do something about the view distance. I determined that the landscape looks the best when I use a very distant and aggressive fog. The farther the fog start is, the larger the terrain seems. The fog is exponential and does a good job (but not a perfect one) of hiding polygons entering though the far plane. This small pop-in is so minor that you won't notice it unless you are really looking for it.
One thing that I need to dos till is make the view distance adjustable at run-time.
Using all the above I finished my hardest task: out of the dozens of permutations, choose only 5 quality settings. This was ridiculously hard because all were tough compromises. Just now I changed the spherical harmonics computation just like that and I'm not sure which one I like better. Anyway, there are 3 quality settings: low, medium and high. You can also choose to have enhance the harmonics for better quality, but this does not work for low quality, thus giving 5 quality levels instead of 6. I am fairly happy with these setting. I also made sure that they have comparable color warmness and intensity, but some minor differences are present.
Here is a video showing a 64 square kilometer map with small view distance at maximum terrain quality, large item density using 8xMSAA and SMAA while the character is running at very high speed traversing the map not quite diagonally (I wanted to go from corner to corner but I messed up :) ):
Now that the terrain shaders are finished (I hope) I need to add day and night cycles to it and see about those lights.
For the rest of the post let me entertain you with some very interesting shader variants I managed to produce:
These are not photoshoped or using any other textures than the one from the video. Just a shader variant that produces strange colors a more wet look:
If I ever need an alien looking landscape I know where to start. I did not manage to produce workable shaders out of this method because the output is too noisy and weird in lot of places. It also has pretty bad temporal aliasing.