It isn’t wrong… but it might not be the best technique for your situation. Certainly the various Math methods make perfect sense as static methods… it would be dumb for example to construct a Double object just so you could call cos() on it…
You likely have a case where some static utility mehtods make sense, but in your one example… what defines the frustum? Should you have a Frustum object with a clipTo(Polygon3D) method?
You have near and far clipping planes and field of view… either they will need to be arguments to the clipToFrustum method, or you will need a single static frustum defined in the class, make the clip method part of the Polygon3D object, in which case it could take a Frustum argument, or there could be a single Frustum defined as a class variable, or do what I mentioned above and have the clipTo method an instance method of a Frustum class.
Lots of options and it isn’t always a matter of getting right or wrong… you have to imagine what will be best for the overall design.
Look at some of the cases where Sun has used static methods (SwingUtilities, Math, etc…) and try to reason why they wre done as static methods. you should likely have the same kind of reasons for your Polygon3Dutility methods
edit
P.S. Speed should not be a concern at all when choosing to implement something as static vs. instance methods. (optimization generally comes later - unless you know from experience and have are confident that one choice will definately be TOO slow.)