Your GPU has 512 MB of VRAM, and that’s where your VBO data is stored. Remember that this memory is shared with everything else running just like system RAM, so don’t expect all of those 512MB to be unused. For example, Windows Aero uses about 100-125MB of VRAM constantly. If you disable it, it drops to ~25 if I recall correctly. I recommend you try out GPU-Z, a program that can monitor memory usage, at least on NVidia cards.
512 MB is however not the limit of how much VBO data you can have. If the graphics card runs out of memory, it will start swapping VRAM to system memory, similar to how your computer swaps system memory to the hard drive. This is obviously not very good for performance, but it is actually not that bad either. The memory manager is intelligent enough to swap out unused things, and the PCI-E bus is fast enough to handle it pretty well. What happens when you overwhelm your VRAM depends on with what you overwhelm it with. If you have 1GB of unused (cached or preloaded or something) VBOs lying around you won’t see much of a performance drop (I estimate it to less than 10% at least). If you overwhelm it with textures, things get much worse, as the whole texture is needed in memory for a much longer time (a VBO is only read ones per draw and can then be swapped out again). Source of info: My own game. I load chunks of the game world into VBOs and only draw the ones that pass the frustum culling test. I used about 2GB of VRAM, and my card only has 1.5GB. xD
Fun fact: If you overwhelm your VRAM with a framebuffer (render target) you will freeze your computer. Yeah, the mouse and everything. Swapping memory that is so commonly used simply freezes the game. What? You wonder why I know that? Who wouldn’t want to play a game with 32SxAA (= 2x2 driver supersampling + 8xMSAA) on a 2x2 supersampled RGBA_32F render target? I mean, come on! That’s just 16x ordered grid supersampling with 8xMSAA and my 1.5GB VRAM ran out?!