glass shader question

I am trying to create a refraction effect by distorting the pixels in the back ground. I use a gray scale image for this purpose, gradients at each point will correspond to the displacement in the x,y direction. My question is, how will the amount of distortion change as the view point move away?

For example, when view is at 10 units away from the glass, the displacement at a particular point is (3, 6), if the view move to 100 units away, should the displacement at
that point change to (3/10, 6/10)?