Both the shaft and the balls need to be inside the cock ring for it to work properly. Sizes run 1-3/4", 1-7/8", 2", 2-1/8", and 2-1/4", with most guys being (I assume) in the 1-7/8" to 2" range. The ring should feel snug but not constricting when you're soft and pleasantly constricting (but not painfully so) when you're hard.
I'd recommend against getting a ring with too small a gauge, especially one of those cheap chrome-plated steel numbers -- they tend to cut into the skin in an uncomfortable way.
I have two relatively thick ones composed of silicone and manufactured by Tantus (and widely available) that I like and my wife finds comfortable when I'm thrusting:
The tear-drop shaped (cock sling) version is an interesting variation because it stimulates a guy's perineum (the place between balls and anus), too.
[Link]
