After SVGA (800 x 600) the next popular size was XGA (1024 x 768). These were 4:3 aspect ratio. After that things became a fit more varied. I used SXGA (1280 x 1024) quite a bit, but it has a different aspect ratio. Then came all the widescreen resolutions in 16:9 aspect ratio. HD 720 (1280 x 720) seems a reasonable minimum for those. So unless you can change the aspect ratio of your game, it is hard to use all the screen area.
I like 800 x 600 for embedded java and flash, as it leaves plenty of room for sidebars on a modern monitor, which is probably at least 1280 x 720. It would also be useable on 1024 x 768 with a bit of window scrolling, provided the user doesn’t have a zillion toolbars enabled in the browser.
For a pop-up window, you need to leave room for the borders. If we assume a minimum of 1024 x 768 and then consider that a HD screen may only have 720 vertically, then 1024 x 720 less window borders is about the top. Someone mentioned 950 x 650 which seems a reasonable maximum. I still have a soft spot for 800 x 600 as it leaves some room for OS toolbars etc.
For full screen, one really needs to go for the largest native resolution, since most displays these days are LCD and do a poor job with anything else. Thus coping with arbitrary aspect ratios needs to be considered. If it looks ok on 1024 x 768 and 1280 x 720, thats probably ok.