
Go to the System blade, select Network Settings and then Test Xbox Live Connection. You'll be able to see what sort of NAT your system is behind, and if it's set to Moderate or Strict then you will only be able to connect to certain other users.
To fix this, set your NAT to Open by opening the following ports on your router: 88 (UDP), 3074 (UDP), and 3074 (TCP).
[Thanks, silvernome]