Symptoms

https://saltexploit.com/

- very high CPU usage, sometimes only on 1 core, sometimes on all
- Fan spin! (Physical hardware only, of course.)
- Mysterious process named salt-minions is CPU intensive, stemming from /tmp/salt-minions or /usr/bin/
- Additional binary in /var/tmp/ or /usr/bin named salt-store or salt-storer
- Firewalls disabled
- Most applications crashing or taken offline

...

1. Stop Salt services. e.g. salt-master, salt-minion, salt-api, etc.

2. Here you are a shell script. (Original code: https://gist.github.com/itskenny0/df20bdb24a2f49b318a91195634ed3c6#file-cleanup-sh)
   Run to your machine.
   

   Code Block
   language py
   # remove for root crontab -l | sed "/$i/d" | crontab - # check other users crontabs for f in $(ls /var/spool/cron/crontabs/*); do sed -i "/$i/d" $f done # remove ssh public key added by v5 sed -i '/PbNwmJNcFwSLF12fFBoF\/$/d' /root/.ssh/authorized_keys for i in $(ls /home/*/.ssh/authorized_keys); do sed -i '/PbNwmJNcFwSLF12fFBoF\/$/d' $i done # newer versions seem to set themselves immutable chattr -i /tmp/salt-store chattr -i /var/tmp/salt-store chattr -i /tmp/salt-minions chattr -i /usr/bin/salt-store # remove all the trash rm /tmp/salt-minions rm /tmp/salt-store rm /var/tmp/salt-store rm /usr/bin/salt-store rm /etc/selinux/config rm -rf /tmp/.ICE* rm -rf /var/tmp/.ICE* rm /root/.wget-hsts # create apparmor profiles to prevent execution echo 'profile salt-store /var/tmp/salt-store { }' | tee /etc/apparmor.d/salt-store apparmor_parser -r -W /etc/apparmor.d/salt-store echo 'profile salt-minions /tmp/salt-minions { }' | tee /etc/apparmor.d/salt-minions apparmor_parser -r -W /etc/apparmor.d/salt-minions # reenable nmi watchdog sysctl kernel.nmi_watchdog=1 echo '1' >/proc/sys/kernel/nmi_watchdog sed -i '/kernel.nmi_watchdog/d' /etc/sysctl.conf # disable hugepages sysctl -w vm.nr_hugepages=0 # enable apparmor systemctl enable apparmor systemctl start apparmor # kill processes and reenabler kill -9 $(ps faux | grep /tmp/.ICE | grep -v grep | awk '{print $2}') ps aux | grep ICEd | grep -v grep | cut -c5-15 | xargs -n 1 kill -9 kill -9 $(pgrep salt-minions) kill -9 $(pgrep salt-store) # fix syslog touch /var/log/syslog service rsyslog restart

3. Update SaltStack to patch version(2019.2.4)
   

   Code Block
   language py
   $ sudo yum check-update salt (omit) salt.noarch 2019.2.4-1.el7 salt-py3-2019.2 $ sudo yum list salt (omit) Installed Packages salt.noarch 2019.2.2-1.el7 @salt-py3-2019.2 Available Packages salt.noarch 2019.2.4-1.el7 salt-py3-2019.2 $ sudo yum update salt (install all dependencies) ==================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================== Updating: salt noarch 2019.2.4-1.el7 salt-py3-2019.2 9.4 M Updating for dependencies: salt-api noarch 2019.2.4-1.el7 salt-py3-2019.2 19 k salt-cloud noarch 2019.2.4-1.el7 salt-py3-2019.2 22 k salt-master noarch 2019.2.4-1.el7 salt-py3-2019.2 2.9 M salt-minion noarch 2019.2.4-1.el7 salt-py3-2019.2 38 k salt-ssh noarch 2019.2.4-1.el7 salt-py3-2019.2 20 k salt-syndic noarch 2019.2.4-1.el7 salt-py3-2019.2 19 k Transaction Summary ==================================================================================================================================================================================================== $ sudo yum list salt (omit) Installed Packages salt.noarch 2019.2.4-1.el7 @salt-py3-2019.2

4. ACL for 4505-4506 ports by Firewall(Recommendation)
   <Sample>
   

   Code Block
   language py
   $ firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="xxx.xxx.xxx.0/24" port port="4505-4506" protocol="tcp" accept' $ firewall-cmd --reload

References

https://saltexploit.com/

https://github.com/saltstack/salt/issues/57057

https://docs.saltstack.com/en/latest/topics/releases/2019.2.4.html

https://gist.github.com/itskenny0/df20bdb24a2f49b318a91195634ed3c6#file-cleanup-sh