Huber loss: If your loss isn't nice, you have to Huberize
There is a standard remedy for some of the nastiness of the quantile loss: You can Huberize it, i.e., replace the of the function near 0 with a smoothed version (a quadratic function). Figure 1 shows the difference between the standard quantile loss and the Huber quantile loss, with an example of standard quantile loss in the upper part of the figure, and the Huberized version below. This solves the first problem, but unfortunately not the second, bigger problem: The second derivative is still not very helpful, as it is still zero apart from a small region around 0. Hence, the Huber quantile loss won't solve our problem. It is still helpful in many other situations, and if you want to learn more about Huber loss for quantile regression, I recommend the paper by Aravkin et al.