The sshd configuration file allows you to list the IP:
# nano /etc/ssh/sshd_config
Locate the line "ListenAddress 0.0.0.0". If that line begins with a '#' (without quotes), remove this character so it is enabled. Set the IP address to the IP you would like it to listen on.
Once this is done you will need to restart the ssh service:
# service sshd restart
Make sure you test the new port logging in via SSH. That way if it's not working, it will prevent you from consoling the server by another means.