================================================= Spacewalk installation, configuration, and usage: ================================================= :Title: spacewalk: how-to :Author: Douglas O'Leary :Description: Spacewalk installation, configuration, and usage: :Date created: 03/28/15 :Date updated: 08/11/15 :Disclaimer: Standard: Use the information that follows at your own risk. If you screw up a system, don't blame it on me... .. contents:: Overview: ========= I've recently started in on a familiarization exercise in Spacewalk in order to support a patch management effort at my job. I found **lots** of good references on the net for installation, but fewer for initial configuration and pretty much none that covered patch management. This page will cover lessons learned first, followed by useful commands and URLs of note and, finally, the procedures. Lessons learned: ================ * Log rotation needed for /var/log/rhn/reposync * ``rm /etc/syscnfig/rhn/systemid`` makes a system believe it is no longer registered to a spacewalk server. System still has to be removed from the Spacewalk UI. * Bug in rhnChannel.py that prevents remote command line channel additions. Details of the bug are in the spacewalk archives_ and the actual patch is available on github as 1122438_ I don't know how to patch the file from github so I just edited the file and removed the extraneous comma at the end of line 1726. File location is: /usr/lib/python2.6/site-packages/spacewalk/server/rhnChannel.py Restart spacewalk for the change to take affect. * ``osad`` required to push patches to a system immediately. If it's not installed/running, the patches will eventually get pushed; but, it'll happen when the next system poll happens. * An alternative to osad is to execute ``rhn_check`` on the target system. * Logs located under /var/log/rhn. Repo sync logs located under /var/logs/rhn/reposync. * Only **one** instance of spacewalk-repo-sync can run at a time. This makes multiple cron jobs to sync repos a bit of a challenge, hence my global reposync script. * I need to patch to the latest point release available. Reason is Redhat's policy_ basically stating that they only support the latest point release. You can run previous versions, but if you come up with a bug and report it, Redhat will check to see if it's fixed in the current point release. If so, they'll tell you to upgrade. If not, they'll try to fix the bug but then only apply it **to the current point release** at which time, they'll tell you to upgrade. Makes sense, if you think about it. The only differenc between rhel6.5 and rhel6.6 is a kernel version and some of those kernel versions change due to security issues. * Required ports (inbound -> client): * 80: bi-directional * 443: bi-directional * 4545: outbound for monitoring * 5222: inbound (push actions to a client) * 5269: inbound, push actions to a spacewalk proxy * Ensure hosts resolve FQDN first and that ``hostname`` on the spacewalk server shows the fqdn. This will save you a boat-load of ssl problems later in the process. .. _archives: https://www.redhat.com/archives/spacewalk-list/2014-October/msg00148.html .. _1122438: https://github.com/spacewalkproject/spacewalk/commit/4bf063f97ae66d889236e11ea237dfd785dbce29 .. _policy: https://access.redhat.com/articles/64664 Useful commands: ================ rhn_check: Run on client to check the spacewalk server for any pending actions rhn-channel --list Displays channels the client can see. rhn-channel -a -c ${channel} -u ${user} -p ${pwd} Run on a target system, this command subscribes the system to the ${channel}. If ${usr} and ${pwd} aren't specified, the command will request them. spacewalk-common-channels Automates the process of creating base and child channels as well as creating the repo definitions spacewalk-repo-sync CLI to sync repos. Using the --parent-channel option, can be used to sync the repos that belong to all child repos as well. rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n' Displays all gpg keys used by rpm: :: # rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n' gpg-pubkey-4bd6ec30-4ff1e4fa --> gpg(Puppet Labs Release Key (Puppet Labs Release Key) ) gpg-pubkey-c105b9de-4e0fd3a3 --> gpg(CentOS-6 Key (CentOS 6 Official Signing Key) ) gpg-pubkey-863a853d-4f55f54d --> gpg(Spacewalk ) gpg-pubkey-0608b895-4bd22942 --> gpg(EPEL (6) ) gpg-pubkey-066e5810-53cfcd27 --> gpg(Spacewalk ) rpm -e gpg-pubkey-066e5810-53cfcd27 Removes rpm gpg key gpg --with-fingerprint ${key_file} Displays the key ID and fingerprint required for spacewalk software channels. In the example bleo, the key ID is 0608B895: :: # gpg --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 pub 4096R/0608B895 2010-04-23 EPEL (6) Key fingerprint = 8C3B E96A F230 9184 DA5C 0DAE 3B49 DF2A 0608 B895 db-control backup ${dir} Backs up the embedded postgres dabase db-control examine ${dir} Examines a backup of the postgres database. db-control verify ${dir} Verifies a postgres db backup. Interesting URLs: ================= https://fedorahosted.org/spacewalk/wiki/UserDocs Main wikidocs for spacewalk https://fedorahosted.org/spacewalk/wiki/SpacewalkWithLDAP Spacewalk installation w/users/groups in ldap http://yum.spacewalkproject.org/ Binary RPMS Yum repos: ========== * Spacewalk server: * http://yum.spacewalkproject.org/2.2/RHEL/6/x86_64/spacewalk-repo-2.2-1.el6.noarch.rpm * Spacewalk client: * rhel5: http://yum.spacewalkproject.org/2.2-client/RHEL/5/x86_64/spacewalk-client-repo-2.2-1.el5.noarch.rpm * rhel6: http://yum.spacewalkproject.org/2.2-client/RHEL/6/x86_64/spacewalk-client-repo-2.2-1.el6.noarch.rpm * rhel7: http://yum.spacewalkproject.org/2.2-client/RHEL/7/x86_64/spacewalk-client-repo-2.2-1.el7.noarch.rpm * Epel repo: * rhel5: http://dl.fedoraproject.org/pub/epel/5/`uname -i`/epel-release-5-4.noarch.rpm * rhel6: http://dl.fedoraproject.org/pub/epel/6/`uname -i`/epel-release-6-8.noarch.rpm * rhel7: http://dl.fedoraproject.org/pub/epel/beta/7/`uname -i`/epel-release-7-0.2.noarch.rpm Installation: ============= Install spacewalk server: ------------------------- * Ensure hosts resolve FQDN first and that ``hostname`` on the spacewalk server shows the fqdn. This will save you a boat-load of ssl problems later in the process. * Ensure OS on spacewalk server is up to date. * Set selinux permissve (need to get this working) on spacewalk server * Ensure appropriate space is available to the appropriate directories and/or they're linked to sufficiently sized filesystems. For my test environment, 20g postgres db filesystem and 50g repo storage area will results in ~50% allocation when centos6, epel, and rpmforge repos are fully synced. * Create 20g /var/lib/pgsql * Create 50g /var/satellite * Set firewal rules on spacealk server: :: # show input Chain INPUT (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 2 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 5 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited for p in 5222 443 80 do iptables -I INPUT 5 -m state --state NEW -p tcp \ -m tcp --dport ${p} -j ACCEPT done # show input Chain INPUT (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 2 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 5 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 6 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443 7 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5222 8 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited service iptables save * Configure requisite repos: * Download/install epel and spacewalk yum repo: :: yum -y install http://yum.spacewalkproject.org/2.2/RHEL/6/x86_64/spacewalk-repo-2.2-1.el6.noarch.rpm yum -y install http://dl.fedoraproject.org/pub/epel/6/`uname -i`/epel-release-6-8.noarch.rpm * Create jpackage repo: :: cat > /etc/yum.repos.d/jpackage-generic.repo << EOF [jpackage-generic] name=JPackage generic #baseurl=http://mirrors.dotsrc.org/pub/jpackage/5.0/generic/free/ mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0 enabled=1 gpgcheck=1 gpgkey=http://www.jpackage.org/jpackage.asc EOF * Install spacewalk: :: yum -y install spacewalk-postgresql \ spacewalk-setup-postgresql-2.2.2-1.el6.noarch \ spacewalk-utils-2.2.25-1.el6.noarch \ spacewalk-remote-utils-2.2.2-1.el6.noarch \ spacewalk-dobby # needed for db-control cmd * Set selinux contexts. Required even though we set the selinux policy to permissive above. restorecon -Rv /var/lib/pgsql restorecon -Rv /var/satellite * Configure spacewalk: ``spacewalk-setup --disconnected`` **USE FQDN** for spacewalk server * Configure the primary admin account by accessing the system's web server: :: https://${fqdn} * Create the default channels: :: spacewalk-common-channels -v -u ${user} -p '${pwd}' -a x86_64 \ -k unlimited 'centos6*' 'spacewalk-nightly-client*' * Set up epel, rpmforge and spacewalk client channels: * Use the repo files in /etc/yum.repos.d for the required information. $basearch can be determined by running ``uname -m`` * spacewalk client repo, at least for me, is required because I can't get the *spacewalk-nightly-client-centos6-x86_64* to work. Every time I try to load a package from it, it errors out saying its unsigned. * URLs to note: * epel gpg keys available at https://dl.fedoraproject.org/pub/epel/ * rpmforge gpg key available at http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt * Process: 1. Channels -> manage s/w channels -> manage repositories 2. create new repository. Fill in required menu items, then create repository. 3. Channels -> manage s/w channels, then create new channel. I don't remember where, but I did read a recommendation to use only one repo for each channel. That makes sense to me. Fill in the menu items: * channel name: human readable channel name * channel label: label for use w/spacewalk-repo-sync * Find a url for the repo gpg signing key. Paste it in ``gpg key url:``. Google, the repo url, and/or the yum repo file are all good places to look. * ID key id and fingerprinter. Paste information into appropriate boxes. See gpg command above. * create channel 4. On that same menu (Channels -> manage s/w channels), select the repositories tab. Check the radio box for the repository that you created in #2. Update repositories. 5. Plan on syncing the repo either through a global script or in this same menu. Note that only one spacewalk-repo-sync command can run at one time. On that same menu (Channels -> manage s/w channels), select the sync tab. Specify an appropriate time for automatic sync. Select schedule. Then, if so desired, select sync now. Monitor progress on the satellite server at /var/log/rhn/reposync/${channel}.log * Sync the repos: * Command: ``spacewalk-repo-sync --parent-channel centos6-x86_64`` * This will take a **very** long time. In excess of 24 hours... Ensure you run the command via tmux or an at job such that it can't be interrupted. Global reposync started at 07:44:34 on 03/12/15 +-------------+----------+----------+ | Started: | 07:44:37 | 03/12/15 | +-------------+----------+----------+ | Ended: | 15:49:11 | 03/13/15 | +-------------+----------+----------+ | Total time: | | 32:04:34 | +-------------+----------+----------+ * You can track progress by examining the appropirate log file under ${satellite}:/var/log/rhn/reposync * Create a activation key that clients will use for their initial regsitration. * Systems -> activation keys -> create new key * Enter appropriate information. Select universal default * In the child channels tab, select the appropriate channels. Ensure epel is selected as it'll be needed later. Also, ensure the spacewalk client we manually entered is selected and not the default *spacewalk client(n) for centos..* Install and register spacewalk clients: --------------------------------------- Execute the following process on all existing clients: * Install spacewalk client repo. See `Yum repos:`_ above for urls * Install epel client repo: See `Yum repos:`_ above for urls * Install spacewalk client packages: :: yum install osad rhn-client-tools rhn-check \ rhn-setup rhnsd m2crypto yum-rhn-plugin * Install spacewalk server's CA cert: :: yum -y install \ http://${fqdn}/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm * Backup and move old repos: :: cd /etc/yum.repos.d mkdir backup && mv *.repo backup * Register client: :: rhnreg_ks --serverUrl=https://${fqdn}/XMLRPC \ --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT \ --activationkey=1-centos6-x86_64 * ``yum clean all`` * Verify connectivity by executing ``yum repolist`` or``rhn-channel --list`` on each client. * Configure osad. Basically, identify the osa_ssl_cert. This perl one-liner is in my kickstart scripts: :: perl -i -ple s'/(^osa_ssl_cert).*/\1 = \/usr\/share\/rhn\/RHN-ORG-TRUSTED-SSL-CERT/g' \ /etc/sysconfig/rhn/osad.conf Patching: ========= Assuming you followed these directions, you now have a nice shiney new Spacewalk server with at least centos6, epel, and rpmforge related child repos. You should have at least one client registered as well. The spacewalk server itself can - and should - be registered as a client. Grouping systems: ----------------- Systems can be grouped together either permanently (more or less) or temporarily. The *permanent* option is under *Systems -> System groups*. In the right corner is a selectable link titled *create a new group*. Enter a name and description, then select *create group*. When fully rolled out, I will have groups like the following: :: ${application-X}-dev ${application-X}-QA ${application-X}-TRN ${application-X}-PRD #--------------------- ${application-Y}-dev ${application-Y}-QA ${application-Y}-TRN ${application-Y}-PRD This will bring up a new page. Along the top row of the main box you'll see *Details, Systems, Target System, Errata, Admins*, and *Probes*. Select *Target Systems* then check the radio box for the clients that should be part of this group. System Set Manager: ------------------- I may just be a little thick but it took me a bit to figure out how to work with groups of systems even temporarily. The system set manager (SSM) is the temporary grouping mechanism. If you're going to patch a set of systems, you'll add them to SSM and then work with that. As an example: Say you've grouped a set of systems as apache-dev. In the *systems -> system groups* menu, select *use in ssm* on the right side of the line with the group name in it. From there, you can select packages and follow the menu picks to upgrade the patches on all systems in the SSM. You can also create completely temporary groupings in the SSM by accessing the Systems menu, checking the radio boxes as needed, then select *Add selected to SSM*. In order to clear out the SSM, on the Systems menu, ensure all systems are **unselected** and the *remove existing servers from the SSM* radio button is checked, then select *Add selected to SSM* Cloning repos: -------------- If you're starting a patching exercise, you need to ensure that the patches that get installed in the production envorinment are exactly the same ones that are getting installed in the development environment. If your repos are getting synced nightly, that may and probably will change during the patching cycle. This is bad. The answer is to clone the repos somehow such that the main repos can continue to be patched but the cloned repos stay stable and are used as the source for patching your environment. Spacewalk has a command called spacewalk-clone-by-date which will clone a parent and children repos by date, ensuring that all packages associated with any errata up to that point are copied. Unfortunately, running centos in my home office environment, I don't get errata - or, perhaps it hasn't been running long enough for me to get any so I haven't had a chance to try this yet. More on this as I get to it. Backup and Restore: =================== Unlike most client facing systems/applications, your spacewalk server isn't going to be very visible if it's down for a bit. However, if it's completely Tango Uniform, you're going to want a way to bring it back to life without having to redo all of the steps above. Backing up Spacewalk: --------------------- * Stop spacewalk: ``spacewalk-service stop`` * Back up the following directories: +------------------+------+--------------------------------------------------+ | /var/satellite | 14G | contains all the pushed packages | +------------------+------+--------------------------------------------------+ | /var/lib/cobbler | 592K | all the cobbler configuration | +------------------+------+--------------------------------------------------+ | /etc | 56M | Standard | +------------------+------+--------------------------------------------------+ | /root/ssl-build | 148K | with the package containing your SSL certificate | +------------------+------+--------------------------------------------------+ | /home/nocpulse | ? | Doesn't exit | +------------------+------+--------------------------------------------------+ | /var/lib/rhn/ | 37M | which contains all the kickstart files | +------------------+------+--------------------------------------------------+ * Restart service. * Backup the database. This process assumes using the postgres DB. If you're using a different one, use the apporpriate backup process for that database. :: yum -y install spacewalk-dobby db-control backup ${root_writable_dir} # cmd must be run as root Restoring Spacewalk: -------------------- * How much restoration to do will be determined by how much was damaged. If you have to replace the entire system, ensure you use the same IP address and hostname. SSL is very sensitive to those two. * Restore the following filesystms as needed: +------------------+------+--------------------------------------------------+ | /var/satellite | 14G | contains all the pushed packages | +------------------+------+--------------------------------------------------+ | /var/lib/cobbler | 592K | all the cobbler configuration | +------------------+------+--------------------------------------------------+ | /etc | 56M | Standard | +------------------+------+--------------------------------------------------+ | /root/ssl-build | 148K | with the package containing your SSL certificate | +------------------+------+--------------------------------------------------+ | /home/nocpulse | ? | Doesn't exit | +------------------+------+--------------------------------------------------+ | /var/lib/rhn/ | 37M | which contains all the kickstart files | +------------------+------+--------------------------------------------------+ * Restore the database using DB specific commands. For postgres: :: db-control restore ${dir} Summary: ======== As stated **way** up there at the top, I was having problems initially figuring out how to use spacewalk after I had it running. There are lots of resources on how to install it; but, not so many telling how to use it. How do you find out the key fingerprint of a gpg key, how do you tie are repository to a software channel? Questions like that took a bit of time to find out. Hopefully, this doc will help someone shorten their learning curve. Doug O'Leary dkoleary@olearycomputers.com