The problem is that you actually need to combine the results of the heuristic with the cost so far. So if just using the square of the heuristic you’d have f(x) = g(x) + h(x)*h(x), which for heuristic values greater than 1 would make the search tend to a best-first (and the heuristic is non-admissible).
Taking the square of the cost so far doesn’t fix it either, because g(x)*g(x) + h(x)*h(x) still doesn’t have the same metric properties as g(x) + h(x).