How do you come up with ideas for games?

It seems like the hardest part of making a game is coming up with the idea. It seems to be the part that takes the longest, for me at least. So, how do you personally do it? I just kind of sit there for a while and when something comes to my mind, I write it down, or if I like it a lot, I making the game around it.