keychain: Set Up Secure Passwordless SSH Access For Backup Scripts on Linux

We establish connections to remote systems without supplying a password. However, I do not want to store my password-less keys ( passphrase-free keys) on my servers. The ssh-agent, takes care of keys with a passphrase, which allowing me to have a ssh-agent process per system per login session easily. How do I dramatically reduces the number of times I have to punch my passphrase from once per new login session to once every time my local server is rebooted? How do I use keychain utility for all my backup scripts for secure passwordless login?

OpenSSH offers RSA and DSA authentication to remote systems without supplying a password. keychain is a special bash script designed to make key-based authentication incredibly convenient and flexible. It offers various security benefits over passphrase-free keys.

How does keychain make it better than a keyless passphrase?

If an attacker manages to log into the server with passphrase-free keys, all other your servers/workstation on which keys used are also the security risk. With keychain or ssh-agent attacker will not be able to touch your remote systems without breaking your passphrase. Another example, if your laptop or hard disk stolen, an attacker can simply copy your key and use it anywhere as a passphrase does not protect it.

The keychain act as a manager for ssh-agent, typically run from ~/.bash_profile. It allows your shells and cron jobs to share a single ssh-agent process. By default, the ssh-agent started by keychain is long-running and will continue to run, even after you have logged out from the system. If you want to change this behavior, take a look at the --clear and --timeout options, described below. Our sample setup is as follows:

peerbox.nixcraft.net.in => Remote Backup Server. Works in pull only mode. It will backup server1.nixcraft.net.in and server2.nixcraft.net.in.
vivek-desktop.nixcraft.net.in => My desktop computer.
server1.nixcraft.net.in => General purpose remote server.
server2.nixcraft.net.in => General purpose remote web / mail / proxy server.

You need to install keychain software on peerbox.nixcraft.net.in so that you or scripts can log in securely to other two servers for backup.

Install keychain on CentOS / RHEL / Fedora Linux

RHEL/CentOS Linux user type the following command to first enable psychotic repo and install keychain package on CentOS 7.x:
##[*** Install psychotic repo **]##
$ sudo rpm --import http://wiki.psychotic.ninja/RPM-GPG-KEY-psychotic
$ sudo rpm -ivh http://packages.psychotic.ninja/6/base/i386/RPMS/psychotic-release-1.0.0-1.el6.psychotic.noarch.rpm
##[*** install keychain from psychotic repo **]##
$ sudo yum --enablerepo=psychotic install keychain

Sample outputs:

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.fibergrid.in
 * epel: epel.mirror.angkasa.id
 * extras: mirror.fibergrid.in
 * updates: mirror.fibergrid.in
Resolving Dependencies
--> Running transaction check
---> Package keychain.noarch 0:2.8.0-3.el7.psychotic will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=============================================================================
 Package       Arch        Version                      Repository      Size
=============================================================================
Installing:
 keychain      noarch      2.8.0-3.el7.psychotic        psychotic       44 k
 
Transaction Summary
=============================================================================
Install  1 Package
 
Total download size: 44 k
Installed size: 97 k
Is this ok [y/d/N]: y
Downloading packages:
keychain-2.8.0-3.el7.psychotic.noarch.rpm               |  44 kB   00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : keychain-2.8.0-3.el7.psychotic.noarch                     1/1 
  Verifying  : keychain-2.8.0-3.el7.psychotic.noarch                     1/1 
 
Installed:
  keychain.noarch 0:2.8.0-3.el7.psychotic                                    
 
Complete!

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.fibergrid.in
* epel: epel.mirror.angkasa.id
* extras: mirror.fibergrid.in
* updates: mirror.fibergrid.in
Resolving Dependencies
–> Running transaction check
—> Package keychain.noarch 0:2.8.0-3.el7.psychotic will be installed
–> Finished Dependency Resolution Dependencies Resolved =============================================================================
Package Arch Version Repository Size
=============================================================================
Installing:
keychain noarch 2.8.0-3.el7.psychotic psychotic 44 k Transaction Summary
=============================================================================
Install 1 Package Total download size: 44 k
Installed size: 97 k
Is this ok [y/d/N]: y
Downloading packages:
keychain-2.8.0-3.el7.psychotic.noarch.rpm | 44 kB 00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : keychain-2.8.0-3.el7.psychotic.noarch 1/1
Verifying : keychain-2.8.0-3.el7.psychotic.noarch 1/1 Installed:
keychain.noarch 0:2.8.0-3.el7.psychotic Complete!

Fedora Linux user type:
$ sudo dnf install keychain

Install keychain on Debian / Ubuntu Linux

To add the package:
$ sudo apt-get update
$ sudo apt-get install keychain

Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  gnupg-agent ssh-askpass
The following NEW packages will be installed:
  keychain
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 27.4 kB of archives.
After this operation, 81.9 kB of additional disk space will be used.
Get:1 http://mirrors.service.networklayer.com/ubuntu xenial/universe amd64 keychain all 2.8.1-0.1 [27.4 kB]
Fetched 27.4 kB in 0s (0 B/s) 
Selecting previously unselected package keychain.
(Reading database ... 81414 files and directories currently installed.)
Preparing to unpack .../keychain_2.8.1-0.1_all.deb ...
Unpacking keychain (2.8.1-0.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up keychain (2.8.1-0.1) ...

Reading package lists… Done
Building dependency tree
Reading state information… Done
Suggested packages:
gnupg-agent ssh-askpass
The following NEW packages will be installed:
keychain
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 27.4 kB of archives.
After this operation, 81.9 kB of additional disk space will be used.
Get:1 http://mirrors.service.networklayer.com/ubuntu xenial/universe amd64 keychain all 2.8.1-0.1 [27.4 kB]
Fetched 27.4 kB in 0s (0 B/s)
Selecting previously unselected package keychain.
(Reading database … 81414 files and directories currently installed.)
Preparing to unpack …/keychain_2.8.1-0.1_all.deb …
Unpacking keychain (2.8.1-0.1) …
Processing triggers for man-db (2.7.5-1) …
Setting up keychain (2.8.1-0.1) …

Install keychain on FreeBSD

To install the port:
# cd /usr/ports/security/keychain/ && make install clean
To add the package use pkg as follows:
# pkg install keychain

Install keychain on OpenBSD

To add the package use pkg_add as follows:
# pkg_add -v keychain

How do I setup SSH keys with passphrase?

Simply type the following commands:
$ ssh-keygen -t rsa
OR
$ ssh-keygen -t dsa
Assign the pass phrase when prompted. See the following step-by-step guide for detailed information:

  1. Howto Linux / UNIX setup SSH with DSA public key authentication (password less login)
  2. Howto use multiple SSH keys for password less login

How do I Use keychain?

Once OpenSSH keys are configured with a pass phrase, update your $HOME/.bash_profile file which is your personal initialization file, executed for login BASH shells:
$ vi $HOME/.bash_profile
Append the following code:

### START-Keychain ###
# Let  re-use ssh-agent and/or gpg-agent between logins
/usr/bin/keychain $HOME/.ssh/id_dsa
source $HOME/.keychain/$HOSTNAME-sh
### End-Keychain ###

### START-Keychain ###
# Let re-use ssh-agent and/or gpg-agent between logins
/usr/bin/keychain $HOME/.ssh/id_dsa
source $HOME/.keychain/$HOSTNAME-sh
### End-Keychain ###

Now you’ve keychanin configured to call keychain tool every login. Just log out and log back in to server from your desktop to test your setup:
$ ssh [email protected]
Sample Output:

Fig.01 – Keychain in Action

keyhcain is up and running. Now, all you have to do is append your servers key file $HOME/.ssh/id_dsa.pub to other UNIX / Linux / BSD boxes:
# scp $HOME/.ssh/id_dsa.pub server1.nixcraft.net.in:~/pubkey
# scp $HOME/.ssh/id_dsa.pub server2.nixcraft.net.in:~/pubkey
# ssh server1.nixcraft.net.in cat ~/pubkey >> ~/.ssh/authorized_keys2; rm ~/pubkey
# ssh server2.nixcraft.net.in cat ~/pubkey >> ~/.ssh/authorized_keys2; rm ~/pubkey
# ssh [email protected]
# ssh [email protected]

Task: Clear or delete all of ssh-agent’s keys

# keychain --clear

Security Task: Make sure intruder cannot use your existing ssh-agent’s keys (only allow cron jobs to use password less login)

The idea is pretty simply only allow backup shell scripts and other cron jobs to allow password-less login but all users including an intruder must provide a passphrase key for interactive login. It is done by deleting all of ssh-agent’s keys. This option will increase security, and it still allows your cron jobs to use your ssh keys when you are logged out. Update your ~/.bash_profile as follows:

/usr/bin/keychain --clear $HOME/.ssh/id_dsa

/usr/bin/keychain –clear $HOME/.ssh/id_dsa

If you are using RSA, use:

/usr/bin/keychain --clear $HOME/.ssh/id_rsa

/usr/bin/keychain –clear $HOME/.ssh/id_rsa

Now, just log in to remote server box once :
$ ssh [email protected]
Log out (only grant access to cron jobs such as backup)
# logout

Task: Use keychain with backup scripts for password-less login via cron job

Add the following before your rsync, tar over ssh, or any other network backup command:

source $HOME/.keychain/$HOSTNAME-sh

source $HOME/.keychain/$HOSTNAME-sh

Here is a sample rsync script:

#!/bin/bash
# Remote Server Rsync backup Replication Shell Script
# Local dir location
LOCALBAKPOINT=/iscsi
LOCALBAKDIR=/backups/server1.nixcraft.net.in/wwwroot
# Remote ssh server setup
SSHUER=root
SSHSERVER=server1.nixcraft.net.in
SSHBACKUPROOT=/wwwroot
 
# Make sure you can log in to remote server without a password
source $HOME/.keychain/$HOSTNAME-sh 
 
# Make sure local backup dir exists
[ ! -d ${LOCALBAKPOINT}${LOCALBAKDIR} ] && mkdir -p ${LOCALBAKPOINT}${LOCALBAKDIR}
 
# Start backup 
/usr/bin/rsync --exclude '*access.log*' --exclude '*error.log*' -avz -e 'ssh ' ${SSHUER}@${SSHSERVER}:${SSHBACKUPROOT} ${LOCALBAKPOINT}${LOCALBAKDIR}
 
# See if backup failed or not to /var/log/messages file
[ $? -eq 0 ] && logger 'RSYNC BACKUP : Done' || logger 'RSYNC BACKUP : FAILED!'

#!/bin/bash
# Remote Server Rsync backup Replication Shell Script
# Local dir location
LOCALBAKPOINT=/iscsi
LOCALBAKDIR=/backups/server1.nixcraft.net.in/wwwroot
# Remote ssh server setup
SSHUER=root
SSHSERVER=server1.nixcraft.net.in
SSHBACKUPROOT=/wwwroot # Make sure you can log in to remote server without a password
source $HOME/.keychain/$HOSTNAME-sh # Make sure local backup dir exists
[ ! -d ${LOCALBAKPOINT}${LOCALBAKDIR} ] && mkdir -p ${LOCALBAKPOINT}${LOCALBAKDIR} # Start backup
/usr/bin/rsync –exclude ‘*access.log*’ –exclude ‘*error.log*’ -avz -e ‘ssh ‘ ${SSHUER}@${SSHSERVER}:${SSHBACKUPROOT} ${LOCALBAKPOINT}${LOCALBAKDIR} # See if backup failed or not to /var/log/messages file
[ $? -eq 0 ] && logger ‘RSYNC BACKUP : Done’ || logger ‘RSYNC BACKUP : FAILED!’

If you are using rsnaphot backup server (see how to setup RHEL / CentOS / Debian rsnapshot backup server) add the following to your /etc/rsnapshot.conf file

# Get ssh login info via keychain
cmd_preexec    source /root/.keychain/hostname.example.com-sh

# Get ssh login info via keychain
cmd_preexec source /root/.keychain/hostname.example.com-sh

A note about keychain and security

  • Cracker with an advanced attacking with deadly coding skills can still get key from memory. However, keychain makes it pretty difficult for normal users and attackers to steal your keys and use it.
  • OpenSSH sshd server offers two additional options to protect abuse of keys. First, make sure root login disabled (PermitRootLogin yes). Second, specify which user accounts on the server are allowed to be used for authentication by adding AuthorizedKeysFile %h/.ssh/authorized_keys_FileName. See sshd_config man page for further details.
Suggested readings:

Posted by: SXI ADMIN

The author is the creator of SXI LLC and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.