This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author icycle
Recipients icycle
Date 2019-01-14.15:52:23
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1547481144.26.0.447593759473.issue35737@roundup.psfhosted.org>
In-reply-to
Content
We are seeing a crash apparently in crypt.py when invoked via SaltStack and have narrowed it down to some change in the Linux kernel introduced by this security update: https://access.redhat.com/errata/RHSA-2018:3083

Linux kernel 3.10.0-862.14.4.el7.x86_64 works fine
Linux kernel 3.10.0-957.el7.x86_64 and later show this error:

2018-11-28T16:35:13.302740+00:00 ip-10-128-152-49 cloud-init: [INFO    ] Executing state cmd.script for [setup-secondary-ips]
2018-11-28T16:35:13.494523+00:00 ip-10-128-152-49 cloud-init: [ERROR   ] An exception occurred in this state: Traceback (most recent call last):
2018-11-28T16:35:13.497189+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/state.py", line 1889, in call
2018-11-28T16:35:13.500053+00:00 ip-10-128-152-49 cloud-init: **cdata['kwargs'])
2018-11-28T16:35:13.502780+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1839, in wrapper
2018-11-28T16:35:13.505822+00:00 ip-10-128-152-49 cloud-init: return f(*args, **kwargs)
2018-11-28T16:35:13.508537+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/states/cmd.py", line 1118, in script
2018-11-28T16:35:13.511297+00:00 ip-10-128-152-49 cloud-init: cmd_all = __salt__['cmd.script'](source, python_shell=True, **cmd_kwargs)
2018-11-28T16:35:13.514308+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py", line 2114, in script
2018-11-28T16:35:13.517107+00:00 ip-10-128-152-49 cloud-init: fn_ = __salt__['cp.cache_file'](source, saltenv)
2018-11-28T16:35:13.520171+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/modules/cp.py", line 474, in cache_file
2018-11-28T16:35:13.523112+00:00 ip-10-128-152-49 cloud-init: result = _client().cache_file(path, saltenv)
2018-11-28T16:35:13.526199+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 188, in cache_file
2018-11-28T16:35:13.529055+00:00 ip-10-128-152-49 cloud-init: return self.get_url(path, '', True, saltenv, cachedir=cachedir)
2018-11-28T16:35:13.532046+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 494, in get_url
2018-11-28T16:35:13.535280+00:00 ip-10-128-152-49 cloud-init: result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
2018-11-28T16:35:13.538335+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 1145, in get_file
2018-11-28T16:35:13.541621+00:00 ip-10-128-152-49 cloud-init: data = self.channel.send(load, raw=True)
2018-11-28T16:35:13.544750+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/utils/async.py", line 65, in wrap
2018-11-28T16:35:13.548071+00:00 ip-10-128-152-49 cloud-init: ret = self._block_future(ret)
2018-11-28T16:35:13.551304+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/utils/async.py", line 75, in _block_future
2018-11-28T16:35:13.554546+00:00 ip-10-128-152-49 cloud-init: return future.result()
2018-11-28T16:35:13.557950+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
2018-11-28T16:35:13.561205+00:00 ip-10-128-152-49 cloud-init: raise_exc_info(self._exc_info)
2018-11-28T16:35:13.564478+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
2018-11-28T16:35:13.568139+00:00 ip-10-128-152-49 cloud-init: yielded = self.gen.throw(*exc_info)
2018-11-28T16:35:13.571683+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 312, in send
2018-11-28T16:35:13.575103+00:00 ip-10-128-152-49 cloud-init: ret = yield self._crypted_transfer(load, tries=tries, timeout=timeout, raw=raw)
2018-11-28T16:35:13.578736+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
2018-11-28T16:35:13.582255+00:00 ip-10-128-152-49 cloud-init: value = future.result()
2018-11-28T16:35:13.585869+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
2018-11-28T16:35:13.589636+00:00 ip-10-128-152-49 cloud-init: raise_exc_info(self._exc_info)
2018-11-28T16:35:13.593537+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
2018-11-28T16:35:13.597250+00:00 ip-10-128-152-49 cloud-init: yielded = self.gen.throw(*exc_info)
2018-11-28T16:35:13.604695+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 284, in _crypted_transfer
2018-11-28T16:35:13.608535+00:00 ip-10-128-152-49 cloud-init: ret = yield _do_transfer()
2018-11-28T16:35:13.612022+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
2018-11-28T16:35:13.615530+00:00 ip-10-128-152-49 cloud-init: value = future.result()
2018-11-28T16:35:13.619175+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
2018-11-28T16:35:13.622702+00:00 ip-10-128-152-49 cloud-init: raise_exc_info(self._exc_info)
2018-11-28T16:35:13.626336+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 879, in run
2018-11-28T16:35:13.629839+00:00 ip-10-128-152-49 cloud-init: yielded = self.gen.send(value)
2018-11-28T16:35:13.633372+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 271, in _do_transfer
2018-11-28T16:35:13.636931+00:00 ip-10-128-152-49 cloud-init: data = self.auth.crypticle.loads(data, raw)
2018-11-28T16:35:13.640794+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 1316, in loads
2018-11-28T16:35:13.644318+00:00 ip-10-128-152-49 cloud-init: data = self.decrypt(data)
2018-11-28T16:35:13.647997+00:00 ip-10-128-152-49 cloud-init: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 1296, in decrypt
2018-11-28T16:35:13.651578+00:00 ip-10-128-152-49 cloud-init: raise AuthenticationError('message authentication failed')
2018-11-28T16:35:13.655231+00:00 ip-10-128-152-49 cloud-init: AuthenticationError: message authentication failed
2018-11-28T16:35:13.658805+00:00 ip-10-128-152-49 cloud-init: [INFO    ] Completed state [setup-secondary-ips] at time 16:35:13.491356 duration_in_ms=196.894


This is very reproducible and we originally reported it here: https://github.com/saltstack/salt/issues/50673 but it does not appear to be related to SaltStack so I am trying this as the next place to file. 

Please advise what additional info may be needed.
History
Date User Action Args
2019-01-14 15:52:26icyclesetrecipients: + icycle
2019-01-14 15:52:24icyclesetmessageid: <1547481144.26.0.447593759473.issue35737@roundup.psfhosted.org>
2019-01-14 15:52:24icyclelinkissue35737 messages
2019-01-14 15:52:23icyclecreate