Below I detail how I patched over 800 AIX LPAR’s that were exposed by CVE-2014-6271 [1] and CVE-2014-7169 [2], also known as shellshock, using the NIM server.
From everything that I’ve been reading on IBM’s Knowledge Centre, creating an LPP source containing only RPM’s isn’t possible. To patch my AIX environment, I decided to use the “script” resource available to the NIM master, along with the pre-existing NFS mounts that I had configured.
NIM master NFS configuration.
NIM:Emre# cat /etc/exports /export/nim/images -ro,anon=0 NIM:kristian# showmount -e export list for NIM: /export/nim/images (everyone)
Location of patched bash RPM on NIM master.
NIM:Emre# ls -l /export/nim/images/bash_CVE-2014-6271-7169 total 3448 -rw-r----- 1 root system 1765643 Sep 30 08:22 bash-4.2-17.aix5.1.ppc.rpm
Script to patch bash (Name: install_bash_CVE-2014-6271-7169)
#!/bin/ksh
#
# Script to install new version of bash to
# patch CVE-2014-6271 and CVE-2014-7169
#
#
# Get NIM master hostname
NIM_MASTER_HOSTNAME=`grep NIM_MASTER_HOSTNAME /etc/niminfo | awk -F = '{ print $2 }'`
# Create temporary mount location
mkdir /install_bash_CVE-2014-6271-7169
# NFS mount patch
mount ${NIM_MASTER_HOSTNAME}:/export/nim/images/bash_CVE-2014-6271-7169 /install_bash_CVE-2014-6271-7169
# Install patch
rpm -Uvh /install_bash_CVE-2014-6271-7169/bash-4.2-17.aix5.1.ppc.rpm
# Unmount NFS mount
umount /install_bash_CVE-2014-6271-7169
# Remove temporary mount location
rm -r /install_bash_CVE-2014-6271-7169
exit
Now that we have the location of the RPM on the NIM master, and the script that will be run on the NIM client to patch bash, we can now define a NIM script resource.
NIM:Emre# nim -o define -t script \ -a server=master \ -a location=/export/nim/patches/install_bash_CVE-2014-6271-7169 \ -a comments="bash fix for CVE-2014-6271 and CVE-2014-7169" bash_CVE-2014-6271-7169 NIM:Emre# lsnim -l bash_CVE-2014-6271-7169 bash_CVE-2014-6271-7169: class = resources type = script comments = bash fix for CVE-2014-6271 and CVE-2014-7169 Rstate = ready for use prev_state = unavailable for use location = /export/nim/patches/install_bash_CVE-2014-6271-7169 alloc_count = 0 server = master
We will now create a NIM machine group that will contain all the NIM clients that we will update. I find the easiest way to do this is by listing out all the NIM client definitions in the format required for the group define command. An example is shown below.
NIM:Emre# for i in `lsnim -t standalone | awk '{ print $1 }'`; do echo "-a add_member=$i \\"; done
-a add_member=aix1 \
-a add_member=aix2 \
-a add_member=aix3 \
-a add_member=aix4 \
-a add_member=aix5 \
-a add_member=aix6 \
Define the NIM group
NIM:Emre# nim -o define -t mac_group \ -a add_member=aix1 \ -a add_member=aix2 \ -a add_member=aix3 \ -a add_member=aix4 \ -a add_member=aix5 \ -a add_member=aix6 PROD_LPARS
The next thing I do is validate that the NIM master can actually talk to all the NIM clients in the machine group. The below method of checking was modified from a post originally written by Brian Smith [3] to work with NIM groups, instead of the standalone clients.
NIM:Emre# for srv in `lsnim -g PROD_LPARS | grep member | grep -v EXCLUDED | awk '{ print $3 }' | sed 's/;//' | sort`; do printf "%-20s" $srv; nim -o lslpp $srv >/dev/null 2>&1; [ "$?" == 0 ] && echo OK || echo "Problem"; done
aix1 OK
aix2 OK
aix3 OK
aix4 OK
aix5 Problem
aix6 OK
For any NIM client that returns “Problem”, I exclude them from the NIM group operation
NIM:Emre# nim -o select -a exclude=aix5 PROD_LPARS
We’re now in a position to execute the patch across all the NIM clients listed in the group definition.
Patch all NIM clients via NIM master
NIM:Emre# nim -o cust -a script=bash_CVE-2014-6271-7169 -a concurrent=10 PROD_LPARS
+-----------------------------------------------------------------------------+
Concurrency Control
+-----------------------------------------------------------------------------+
Processing will begin with the first 5 machines from the group...
+-----------------------------------------------------------------------------+
Initiating "cust" Operation
+-----------------------------------------------------------------------------+
Allocating resources ...
Initiating the cust operation on machine 1 of 5: aix1 ...
Initiating the cust operation on machine 2 of 5: aix2 ...
Initiating the cust operation on machine 3 of 5: aix3 ...
Initiating the cust operation on machine 4 of 5: aix4 ...
Initiating the cust operation on machine 5 of 5: aix6 ...
+-----------------------------------------------------------------------------+
"cust" Operation Summary
+-----------------------------------------------------------------------------+
Target Result
------ ------
aix1 INITIATED
aix2 INITIATED
aix3 INITIATED
aix4 INITIATED
aix6 INITIATED
Note: Use the lsnim command to monitor progress of "INITIATED"
targets by viewing their NIM database definition.
+-----------------------------------------------------------------------------+
Concurrency Control
+-----------------------------------------------------------------------------+
The first 8 machines have been processed. As machines finish
installing processing will resume with the remaining members
of the group, one at a time.
+-----------------------------------------------------------------------------+
Concurrency Control: "cust" Operation Summary
+-----------------------------------------------------------------------------+
Target Result
------ ------
aix1 COMPLETE
aix2 COMPLETE
aix3 COMPLETE
aix4 COMPLETE
aix6 COMPLETE
Once the process has completed, you can validate the version of bash installed across all NIM clients by running the following command.
NIM:Emre# for i in `lsnim -g PROD_LPARS | grep member | grep -v EXCLUDED | awk '{ print $3 }' | sed 's/;//' | sort`; do echo $i; nim -o lslpp -a lslpp_flags=-Lc -a filesets=bash $i | grep bash | awk -F : '{ print $2 }'; echo ""; done
aix1
bash-4.2-17
aix2
bash-4.2-17
aix3
bash-4.2-17
aix4
bash-4.2-17
aix6
bash-4.2-17
One final clean up task, is to ensure you include all NIM members back into the group if you excluded them previously.
NIM:Emre# nim -o select -a include_all=yes PROD_LPARS
