After adding the public key to your Jenkins user profile you can execute jenkins-cli with your ssh key:

$ java -jar jenkins-cli.jar -s http://hudson.hh.arvm.de/ -i ~/.ssh/jenkins_rsa who-am-i
Authenticated as: clsa1
Authorities:
  jenkins_admin

If you are using cygwin you have to transform the path like this:

java -jar jenkins-cli.jar -s http://hudson.hh.arvm.de/ -i $(cygpath -aw "/cygdrive/c/cygwin64/home/CLSA1/.ssh/jenkins_rsa")