softdist@ubuntu01:~/test> cat ~/.ansible.cfg
[defaults]
allow_world_readable_tmpfiles=True
[ssh_connection]
retries = 3
ssh_args = -o ControlPersist=1200s -o ControlMaster=auto
scp_if_ssh=True
We have a test ansible playbook, it sleep 10 seconds.
softdist@ubuntu01:~/test> cat sleep.yml
—
– name: ‘Test playbook to sleep for a specified interval’
hosts: all
gather_facts: false
vars:
sleep_interval: 10
tasks:
– name: sleep for a specified interval
command: sleep ‘{{ sleep_interval }}’
change sleep_interval to 1300 and rerun
We test the sleep 10 seconds, the ansible playbook runs without issue:
softdist@ubuntu01:~/test> ansible-playbook -vvvv -i ubuntu-02 ./sleep.yml
Using /home/softdist/.ansible.cfg as config file
PLAYBOOK: sleep.yml ************************************************************
1 plays in ./sleep.yml
PLAY [Test playbook to sleep for a specified interval] *************************
TASK [sleep for a specified interval] ******************************************
…
changed: [ubuntu-02] => {
“changed”: true,
“cmd”: [
“sleep”,
“10”
],
“delta”: “0:00:10.005255”,
“end”: “2018-08-30 10:04:11.815449”,
“invocation”: {
“module_args”: {
“_raw_params”: “sleep ’10′”,
“_uses_shell”: false,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“warn”: true
},
“module_name”: “command”
},
“rc”: 0,
“start”: “2018-08-30 10:04:01.810194”,
“stderr”: “”,
“stdout”: “”,
“stdout_lines”: [],
“warnings”: []
}
PLAY RECAP *********************************************************************
ubuntu-02 : ok=1 changed=1 unreachable=0 failed=0
We change the sleep 10 seconds to sleep 1300 seconds, now the playbook fails:
fatal: [ubuntu-02]: UNREACHABLE! => {
“changed”: false,
“msg”: “Failed to connect to the host via ssh: OpenSSH_6.6.1, OpenSSL 0.9.8j-fips 07 Jan 2009\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 25: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \”/home/softdist/.ansible/cp/ansible-ssh-ubuntu-02-22-softdist\” does not exist\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to ubuntu-02 [10.18.6.32] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9999 ms remain after connect\r\ndebug1: identity file /home/softdist/.ssh/id_rsa type -1\r\ndebug1: identity file /home/softdist/.ssh/id_rsa-cert type -1\r\ndebug1: identity file /home/softdist/.ssh/id_dsa type -1\r\ndebug1: identity file /home/softdist/.ssh/id_dsa-cert type -1\r\ndebug1: identity file /home/softdist/.ssh/id_ecdsa type -1\r\ndebug1: identity file /home/softdist/.ssh/id_ecdsa-cert type -1\r\ndebug1: identity file /home/softdist/.ssh/id_ed25519 type -1\r\ndebug1: identity file /home/softdist/.ssh/id_ed25519-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_6.6.1\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1\r\ndebug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug3: load_hostkeys: loading entries for host \”ubuntu-02\” from file \”/home/softdist/.ssh/known_hosts\”\r\ndebug3: load_hostkeys: found key type ECDSA in file /home/softdist/.ssh/known_hosts:8\r\ndebug3: load_hostkeys: loaded 1 keys\r\ndebug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r\ndebug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: none,zlib@openssh.com,zlib\r\ndebug2: kex_parse_kexinit: none,zlib@openssh.com,zlib\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr\r\ndebug2: kex_parse_kexinit: hmac-ripemd160,hmac-ripemd160-etm@openssh.com,hmac-sha1,hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com,umac-128@openssh.com,umac-64-etm@openssh.com\r\ndebug2: kex_parse_kexinit: hmac-ripemd160,hmac-ripemd160-etm@openssh.com,hmac-sha1,hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com,umac-128@openssh.com,umac-64-etm@openssh.com\r\ndebug2: kex_parse_kexinit: none,zlib@openssh.com\r\ndebug2: kex_parse_kexinit: none,zlib@openssh.com\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: mac_setup: setup umac-64-etm@openssh.com\r\ndebug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none\r\ndebug2: mac_setup: setup umac-64-etm@openssh.com\r\ndebug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none\r\ndebug1: sending SSH2_MSG_KEX_ECDH_INIT\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ECDSA 36:4a:ba:ac:f8:90:3b:ad:e7:bb:b4:76:ac:71:bc:ea [MD5]\r\ndebug3: load_hostkeys: loading entries for host \”ubuntu-02\” from file \”/home/softdist/.ssh/known_hosts\”\r\ndebug3: load_hostkeys: found key type ECDSA in file /home/softdist/.ssh/known_hosts:8\r\ndebug3: load_hostkeys: loaded 1 keys\r\ndebug3: load_hostkeys: loading entries for host \”10.18.6.32\” from file \”/home/softdist/.ssh/known_hosts\”\r\ndebug3: load_hostkeys: found key type ECDSA in file /home/softdist/.ssh/known_hosts:8\r\ndebug3: load_hostkeys: loaded 1 keys\r\ndebug1: Host ‘ubuntu-02’ is known and matches the ECDSA host key.\r\ndebug1: Found key in /home/softdist/.ssh/known_hosts:8\r\ndebug1: ssh_ecdsa_verify: signature correct\r\ndebug2: kex_derive_keys\r\ndebug2: set_newkeys: mode 1\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug2: set_newkeys: mode 0\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: SSH2_MSG_SERVICE_REQUEST sent\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug2: key: .ssh/softdist.pem (0x7f512040f840),\r\ndebug2: key: /home/softdist/.ssh/id_rsa ((nil)),\r\ndebug2: key: /home/softdist/.ssh/id_dsa ((nil)),\r\ndebug2: key: /home/softdist/.ssh/id_ecdsa ((nil)),\r\ndebug2: key: /home/softdist/.ssh/id_ed25519 ((nil)),\r\ndebug1: Authentications that can continue: publickey\r\ndebug3: start over, passed a different list publickey\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: .ssh/softdist.pem\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Server accepts key: pkalg ssh-rsa blen 279\r\ndebug2: input_userauth_pk_ok: fp dc:fb:b7:a2:77:a2:01:f7:bd:1f:18:2a:48:19:38:7a [MD5]\r\ndebug3: sign_and_send_pubkey: RSA dc:fb:b7:a2:77:a2:01:f7:bd:1f:18:2a:48:19:38:7a [MD5]\r\ndebug1: Authentication succeeded (publickey).\r\nAuthenticated to ubuntu-02 ([10.18.6.32]:22).\r\ndebug1: setting up multiplex master socket\r\ndebug3: muxserver_listen: temporary control path /home/softdist/.ansible/cp/ansible-ssh-ubuntu-02-22-softdist.nctVg3GTWQYeJNAF\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug3: fd 4 is O_NONBLOCK\r\ndebug3: fd 4 is O_NONBLOCK\r\ndebug1: channel 0: new [/home/softdist/.ansible/cp/ansible-ssh-ubuntu-02-22-softdist]\r\ndebug3: muxserver_listen: mux listener channel 0 fd 4\r\ndebug2: fd 3 setting TCP_NODELAY\r\ndebug3: packet_set_tos: set IP_TOS 0x08\r\ndebug1: control_persist_detach: backgrounding master process\r\ndebug2: control_persist_detach: background process is 19675\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug1: forking to background\r\ndebug1: Entering interactive session.\r\ndebug2: set_control_persist_exit_time: schedule exit in 1200 seconds\r\ndebug1: multiplexing control connection\r\ndebug2: fd 5 setting O_NONBLOCK\r\ndebug3: fd 5 is O_NONBLOCK\r\ndebug1: channel 1: new [mux-control]\r\ndebug3: channel_post_mux_listener: new mux channel 1 fd 5\r\ndebug3: mux_master_read_cb: channel 1: hello sent\r\ndebug2: set_control_persist_exit_time: cancel scheduled exit\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4\r\ndebug2: process_mux_master_hello: channel 1 slave version 4\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4\r\ndebug2: process_mux_alive_check: channel 1: alive check\r\ndebug3: mux_client_request_alive: done pid = 19677\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 277\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: process_mux_new_session: channel 1: request tty 1, X 0, agent 0, subsys 0, term \”xterm\”, cmd \”/bin/sh -c ‘/usr/bin/python /home/softdist/.ansible/tmp/ansible-tmp-1535637899.23-78087440439237/command.py; rm -rf \”/home/softdist/.ansible/tmp/ansible-tmp-1535637899.23-78087440439237/\” > /dev/null 2>&1 && sleep 0’\”, env 1\r\ndebug3: process_mux_new_session: got fds stdin 6, stdout 7, stderr 8\r\ndebug2: fd 7 setting O_NONBLOCK\r\ndebug2: fd 8 setting O_NONBLOCK\r\ndebug1: channel 2: new [client-session]\r\ndebug2: process_mux_new_session: channel_new: 2 linked to control channel 1\r\ndebug2: channel 2: send open\r\ndebug2: callback start\r\ndebug2: client_session2_setup: id 2\r\ndebug2: channel 2: request pty-req confirm 1\r\ndebug1: Sending environment.\r\ndebug1: Sending env LANG = en_US.UTF-8\r\ndebug2: channel 2: request env confirm 0\r\ndebug1: Sending command: /bin/sh -c ‘/usr/bin/python /home/softdist/.ansible/tmp/ansible-tmp-1535637899.23-78087440439237/command.py; rm -rf \”/home/softdist/.ansible/tmp/ansible-tmp-1535637899.23-78087440439237/\” > /dev/null 2>&1 && sleep 0’\r\ndebug2: channel 2: request exec confirm 1\r\ndebug3: mux_session_confirm: sending success reply\r\ndebug2: callback done\r\ndebug2: channel 2: open confirm rwindow 0 rmax 32768\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: channel_input_status_confirm: type 99 id 2\r\ndebug2: PTY allocation request accepted on channel 2\r\ndebug2: channel 2: rcvd adjust 2097152\r\ndebug2: channel_input_status_confirm: type 99 id 2\r\ndebug2: exec request accepted on channel 2\r\ndebug1: channel 0: free: /home/softdist/.ansible/cp/ansible-ssh-ubuntu-02-22-softdist, nchannels 3\r\ndebug3: channel 0: status: The following connections are open:\r\n #2 client-session (t4 r0 i0/0 o0/0 fd 6/7 cc 1)\r\n\r\ndebug1: channel 1: free: mux-control, nchannels 2\r\ndebug3: channel 1: status: The following connections are open:\r\n #2 client-session (t4 r0 i0/0 o0/0 fd 6/7 cc 1)\r\n\r\ndebug1: channel 2: free: client-session, nchannels 1\r\ndebug3: channel 2: status: The following connections are open:\r\n #2 client-session (t4 r0 i0/0 o0/0 fd 6/7 cc 1)\r\n\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug3: fd 0 is not O_NONBLOCK\r\ndebug3: fd 1 is not O_NONBLOCK\r\ndebug1: fd 2 clearing O_NONBLOCK\r\ndebug2: Control master terminated unexpectedly\r\nConnection to ubuntu-02 closed by remote host.\r\nShared connection to ubuntu-02 closed.\r\nTransferred: sent 3748, received 1652 bytes, in 850.1 seconds\r\nBytes per second: sent 4.4, received 1.9\r\ndebug1: Exit status -1\r\n”,
“unreachable”: true
}
to retry, use: –limit @/home/softdist/test/sleep.retry
PLAY RECAP *********************************************************************
ubuntu-02 : ok=0 changed=0 unreachable=1 failed=0
Solution is changing to asyn and it works:
softdist@ubuntu01:~/test> cat sleep.yml
—
– name: ‘Test playbook to sleep for a specified interval’
hosts: all
gather_facts: false
vars:
sleep_interval: 1300
tasks:
– name: sleep for a specified interval
command: sleep ‘{{ sleep_interval }}’
async: 1301
poll: 30
it will poll every 30 seconds and keeps the ssh connection
<ubuntu-02> SSH: EXEC ssh -vvv -o ControlPersist=1200s -o ControlMaster=auto -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/softdist/.ansible/cp/ansible-ssh-%h-%p-%r -tt ubuntu-02 ‘/bin/sh -c ‘”‘”‘/usr/bin/python /home/softdist/.ansible/tmp/ansible-tmp-1535643038.95-204278054602748/async_status.py; rm -rf “/home/softdist/.ansible/tmp/ansible-tmp-1535643038.95-204278054602748/” > /dev/null 2>&1 && sleep 0′”‘””
changed: [ubuntu-02] => {
“ansible_job_id”: “677019101514.19432”,
“changed”: true,
“cmd”: [
“sleep”,
“1300”
],
“delta”: “0:21:40.004719”,
“end”: “2018-08-30 11:30:17.227022”,
“finished”: 1,
“invocation”: {
“module_args”: {
“_raw_params”: “sleep ‘1300’”,
“_uses_shell”: false,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“warn”: true
},
“module_name”: “async_status”
},
“rc”: 0,
“start”: “2018-08-30 11:08:37.222303”,
“stderr”: “”,
“stdout”: “”,
“stdout_lines”: [],
“warnings”: []
}
PLAY RECAP *********************************************************************
ubuntu-02 : ok=1 changed=1 unreachable=0 failed=0