Spacewalk installation, configuration, and usage:

Title:

spacewalk: how-to

Author:

Douglas O’Leary <dkoleary@olearycomputers.com>

Description:

Spacewalk installation, configuration, and usage:

Date created:

03/28/15

Date updated:

08/11/15

Disclaimer:

Standard: Use the information that follows at your own risk. If you screw up a system, don’t blame it on me…

Overview:

I’ve recently started in on a familiarization exercise in Spacewalk in order to support a patch management effort at my job. I found lots of good references on the net for installation, but fewer for initial configuration and pretty much none that covered patch management.

This page will cover lessons learned first, followed by useful commands and URLs of note and, finally, the procedures.

Lessons learned:

  • Log rotation needed for /var/log/rhn/reposync

  • rm /etc/syscnfig/rhn/systemid makes a system believe it is no longer registered to a spacewalk server. System still has to be removed from the Spacewalk UI.

  • Bug in rhnChannel.py that prevents remote command line channel additions. Details of the bug are in the spacewalk archives and the actual patch is available on github as 1122438 I don’t know how to patch the file from github so I just edited the file and removed the extraneous comma at the end of line 1726. File location is: /usr/lib/python2.6/site-packages/spacewalk/server/rhnChannel.py Restart spacewalk for the change to take affect.

  • osad required to push patches to a system immediately. If it’s not installed/running, the patches will eventually get pushed; but, it’ll happen when the next system poll happens.

  • An alternative to osad is to execute rhn_check on the target system.

  • Logs located under /var/log/rhn. Repo sync logs located under /var/logs/rhn/reposync.

  • Only one instance of spacewalk-repo-sync can run at a time. This makes multiple cron jobs to sync repos a bit of a challenge, hence my global reposync script.

  • I need to patch to the latest point release available. Reason is Redhat’s policy basically stating that they only support the latest point release. You can run previous versions, but if you come up with a bug and report it, Redhat will check to see if it’s fixed in the current point release. If so, they’ll tell you to upgrade. If not, they’ll try to fix the bug but then only apply it to the current point release at which time, they’ll tell you to upgrade. Makes sense, if you think about it. The only differenc between rhel6.5 and rhel6.6 is a kernel version and some of those kernel versions change due to security issues.

  • Required ports (inbound -> client):

    • 80: bi-directional

    • 443: bi-directional

    • 4545: outbound for monitoring

    • 5222: inbound (push actions to a client)

    • 5269: inbound, push actions to a spacewalk proxy

  • Ensure hosts resolve FQDN first and that hostname on the spacewalk server shows the fqdn. This will save you a boat-load of ssl problems later in the process.

Useful commands:

rhn_check:

Run on client to check the spacewalk server for any pending actions

rhn-channel –list

Displays channels the client can see.

rhn-channel -a -c ${channel} -u ${user} -p ${pwd}

Run on a target system, this command subscribes the system to the ${channel}. If ${usr} and ${pwd} aren’t specified, the command will request them.

spacewalk-common-channels

Automates the process of creating base and child channels as well as creating the repo definitions

spacewalk-repo-sync

CLI to sync repos. Using the –parent-channel option, can be used to sync the repos that belong to all child repos as well.

rpm -q gpg-pubkey –qf ‘%{name}-%{version}-%{release} –> %{summary}n’

Displays all gpg keys used by rpm:

# rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'
gpg-pubkey-4bd6ec30-4ff1e4fa --> gpg(Puppet Labs Release Key (Puppet Labs Release Key) <info@puppetlabs.com>)
gpg-pubkey-c105b9de-4e0fd3a3 --> gpg(CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>)
gpg-pubkey-863a853d-4f55f54d --> gpg(Spacewalk <spacewalk-devel@redhat.com>)
gpg-pubkey-0608b895-4bd22942 --> gpg(EPEL (6) <epel@fedoraproject.org>)
gpg-pubkey-066e5810-53cfcd27 --> gpg(Spacewalk <spacewalk-devel@redhat.com>)
rpm -e gpg-pubkey-066e5810-53cfcd27

Removes rpm gpg key

gpg –with-fingerprint ${key_file}

Displays the key ID and fingerprint required for spacewalk software channels. In the example bleo, the key ID is 0608B895:

# gpg --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
pub  4096R/0608B895 2010-04-23 EPEL (6) <epel@fedoraproject.org>
      Key fingerprint = 8C3B E96A F230 9184 DA5C  0DAE 3B49 DF2A 0608 B895
db-control backup ${dir}

Backs up the embedded postgres dabase

db-control examine ${dir}

Examines a backup of the postgres database.

db-control verify ${dir}

Verifies a postgres db backup.

Interesting URLs:

https://fedorahosted.org/spacewalk/wiki/UserDocs

Main wikidocs for spacewalk

https://fedorahosted.org/spacewalk/wiki/SpacewalkWithLDAP

Spacewalk installation w/users/groups in ldap

http://yum.spacewalkproject.org/

Binary RPMS

Yum repos:

Installation:

Install spacewalk server:

  • Ensure hosts resolve FQDN first and that hostname on the spacewalk server shows the fqdn. This will save you a boat-load of ssl problems later in the process.

  • Ensure OS on spacewalk server is up to date.

  • Set selinux permissve (need to get this working) on spacewalk server

  • Ensure appropriate space is available to the appropriate directories and/or they’re linked to sufficiently sized filesystems. For my test environment, 20g postgres db filesystem and 50g repo storage area will results in ~50% allocation when centos6, epel, and rpmforge repos are fully synced.

    • Create 20g /var/lib/pgsql

    • Create 50g /var/satellite

  • Set firewal rules on spacealk server:

    # show input
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

for p in 5222 443 80
do
    iptables -I INPUT 5 -m state --state NEW -p tcp \
        -m tcp --dport ${p} -j ACCEPT
done

# show input
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
6    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443
7    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5222
8    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

service iptables save

  • Configure requisite repos:

    • Download/install epel and spacewalk yum repo:

      yum -y install http://yum.spacewalkproject.org/2.2/RHEL/6/x86_64/spacewalk-repo-2.2-1.el6.noarch.rpm
yum -y install http://dl.fedoraproject.org/pub/epel/6/`uname -i`/epel-release-6-8.noarch.rpm

    • Create jpackage repo:

      cat > /etc/yum.repos.d/jpackage-generic.repo << EOF
[jpackage-generic]
name=JPackage generic
#baseurl=http://mirrors.dotsrc.org/pub/jpackage/5.0/generic/free/
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0
enabled=1
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
EOF

  • Install spacewalk:

    yum -y install spacewalk-postgresql \
    spacewalk-setup-postgresql-2.2.2-1.el6.noarch \
    spacewalk-utils-2.2.25-1.el6.noarch \
    spacewalk-remote-utils-2.2.2-1.el6.noarch \
    spacewalk-dobby # needed for db-control cmd

  • Set selinux contexts. Required even though we set the selinux policy to permissive above.

    restorecon -Rv /var/lib/pgsql restorecon -Rv /var/satellite

  • Configure spacewalk: spacewalk-setup --disconnected USE FQDN for spacewalk server

  • Configure the primary admin account by accessing the system’s web server:

    https://${fqdn}

  • Create the default channels:

    spacewalk-common-channels -v -u ${user} -p '${pwd}' -a x86_64 \
-k unlimited 'centos6*' 'spacewalk-nightly-client*'

  • Set up epel, rpmforge and spacewalk client channels:

    • Use the repo files in /etc/yum.repos.d for the required information. $basearch can be determined by running uname -m

    • spacewalk client repo, at least for me, is required because I can’t get the spacewalk-nightly-client-centos6-x86_64 to work. Every time I try to load a package from it, it errors out saying its unsigned.

    • URLs to note:

    • Process:

      1. Channels -> manage s/w channels -> manage repositories

      2. create new repository. Fill in required menu items, then create repository.

      3. Channels -> manage s/w channels, then create new channel. I don’t remember where, but I did read a recommendation to use only one repo for each channel. That makes sense to me. Fill in the menu items:

        • channel name: human readable channel name

        • channel label: label for use w/spacewalk-repo-sync

        • Find a url for the repo gpg signing key. Paste it in gpg key url:. Google, the repo url, and/or the yum repo file are all good places to look.

        • ID key id and fingerprinter. Paste information into appropriate boxes. See gpg command above.

        • create channel

      4. On that same menu (Channels -> manage s/w channels), select the repositories tab. Check the radio box for the repository that you created in #2. Update repositories.

      5. Plan on syncing the repo either through a global script or in this same menu. Note that only one spacewalk-repo-sync command can run at one time.

        On that same menu (Channels -> manage s/w channels), select the sync tab. Specify an appropriate time for automatic sync. Select schedule. Then, if so desired, select sync now. Monitor progress on the satellite server at /var/log/rhn/reposync/${channel}.log

  • Sync the repos:

    • Command: spacewalk-repo-sync --parent-channel  centos6-x86_64

    • This will take a very long time. In excess of 24 hours… Ensure you run the command via tmux or an at job such that it can’t be interrupted. Global reposync started at 07:44:34 on 03/12/15

      Started:

      07:44:37

      03/12/15

      Ended:

      15:49:11

      03/13/15

      Total time:

      32:04:34

    • You can track progress by examining the appropirate log file under ${satellite}:/var/log/rhn/reposync

  • Create a activation key that clients will use for their initial regsitration.

    • Systems -> activation keys -> create new key

    • Enter appropriate information. Select universal default

    • In the child channels tab, select the appropriate channels. Ensure epel is selected as it’ll be needed later. Also, ensure the spacewalk client we manually entered is selected and not the default spacewalk client(n) for centos..

Install and register spacewalk clients:

Execute the following process on all existing clients:

  • Install spacewalk client repo. See Yum repos: above for urls

  • Install epel client repo: See Yum repos: above for urls

  • Install spacewalk client packages:

    yum install osad rhn-client-tools rhn-check \
rhn-setup rhnsd m2crypto yum-rhn-plugin

  • Install spacewalk server’s CA cert:

    yum -y install \
http://${fqdn}/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm

  • Backup and move old repos:

    cd /etc/yum.repos.d
mkdir backup && mv *.repo backup

  • Register client:

    rhnreg_ks --serverUrl=https://${fqdn}/XMLRPC \
--sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT \
--activationkey=1-centos6-x86_64

  • yum clean all

  • Verify connectivity by executing yum repolist or``rhn-channel –list`` on each client.

  • Configure osad. Basically, identify the osa_ssl_cert. This perl one-liner is in my kickstart scripts:

    perl -i -ple s'/(^osa_ssl_cert).*/\1 = \/usr\/share\/rhn\/RHN-ORG-TRUSTED-SSL-CERT/g' \
    /etc/sysconfig/rhn/osad.conf

Patching:

Assuming you followed these directions, you now have a nice shiney new Spacewalk server with at least centos6, epel, and rpmforge related child repos. You should have at least one client registered as well. The spacewalk server itself can - and should - be registered as a client.

Grouping systems:

Systems can be grouped together either permanently (more or less) or temporarily.

The permanent option is under Systems -> System groups. In the right corner is a selectable link titled create a new group. Enter a name and description, then select create group. When fully rolled out, I will have groups like the following:

${application-X}-dev
${application-X}-QA
${application-X}-TRN
${application-X}-PRD
#---------------------
${application-Y}-dev
${application-Y}-QA
${application-Y}-TRN
${application-Y}-PRD

This will bring up a new page. Along the top row of the main box you’ll see Details, Systems, Target System, Errata, Admins, and Probes. Select Target Systems then check the radio box for the clients that should be part of this group.

System Set Manager:

I may just be a little thick but it took me a bit to figure out how to work with groups of systems even temporarily. The system set manager (SSM) is the temporary grouping mechanism. If you’re going to patch a set of systems, you’ll add them to SSM and then work with that.

As an example: Say you’ve grouped a set of systems as apache-dev. In the systems -> system groups menu, select use in ssm on the right side of the line with the group name in it. From there, you can select packages and follow the menu picks to upgrade the patches on all systems in the SSM.

You can also create completely temporary groupings in the SSM by accessing the Systems menu, checking the radio boxes as needed, then select Add selected to SSM.

In order to clear out the SSM, on the Systems menu, ensure all systems are unselected and the remove existing servers from the SSM radio button is checked, then select Add selected to SSM

Cloning repos:

If you’re starting a patching exercise, you need to ensure that the patches that get installed in the production envorinment are exactly the same ones that are getting installed in the development environment. If your repos are getting synced nightly, that may and probably will change during the patching cycle. This is bad.

The answer is to clone the repos somehow such that the main repos can continue to be patched but the cloned repos stay stable and are used as the source for patching your environment.

Spacewalk has a command called spacewalk-clone-by-date which will clone a parent and children repos by date, ensuring that all packages associated with any errata up to that point are copied.

Unfortunately, running centos in my home office environment, I don’t get errata - or, perhaps it hasn’t been running long enough for me to get any so I haven’t had a chance to try this yet. More on this as I get to it.

Backup and Restore:

Unlike most client facing systems/applications, your spacewalk server isn’t going to be very visible if it’s down for a bit. However, if it’s completely Tango Uniform, you’re going to want a way to bring it back to life without having to redo all of the steps above.

Backing up Spacewalk:

  • Stop spacewalk: spacewalk-service stop

  • Back up the following directories:

/var/satellite

14G

contains all the pushed packages

/var/lib/cobbler

592K

all the cobbler configuration

/etc

56M

Standard

/root/ssl-build

148K

with the package containing your SSL certificate

/home/nocpulse

?

Doesn’t exit

/var/lib/rhn/

37M

which contains all the kickstart files

  • Restart service.

  • Backup the database. This process assumes using the postgres DB. If you’re using a different one, use the apporpriate backup process for that database.

    yum -y install spacewalk-dobby
db-control backup ${root_writable_dir}  # cmd must be run as root

Restoring Spacewalk:

  • How much restoration to do will be determined by how much was damaged. If you have to replace the entire system, ensure you use the same IP address and hostname. SSL is very sensitive to those two.

  • Restore the following filesystms as needed:

/var/satellite

14G

contains all the pushed packages

/var/lib/cobbler

592K

all the cobbler configuration

/etc

56M

Standard

/root/ssl-build

148K

with the package containing your SSL certificate

/home/nocpulse

?

Doesn’t exit

/var/lib/rhn/

37M

which contains all the kickstart files

  • Restore the database using DB specific commands. For postgres:

    db-control restore ${dir}

Summary:

As stated way up there at the top, I was having problems initially figuring out how to use spacewalk after I had it running. There are lots of resources on how to install it; but, not so many telling how to use it. How do you find out the key fingerprint of a gpg key, how do you tie are repository to a software channel? Questions like that took a bit of time to find out. Hopefully, this doc will help someone shorten their learning curve.

Doug O’Leary dkoleary@olearycomputers.com