Open Source Puppet v5 installation notes:

Author:

Doug O’Leary

Description:

OSPv2 installation notes, ll, issues, etc

Created:

07/05/17

Updated:

07/08/18

Overview:

This document lists the process I went through to get a fully funcitonal open source puppet (v5) environment running. This includes:

  • gitlab (community edition)

  • puppet server (obviously)

  • puppetdb

  • hiera and eyaml

  • r10k

  • external node classifier (ENC) tying into a mysql database.

  • Auto-signing hosts listed in the database.

It was a fun bit of work and seems to be pretty rock solid especially after setting up additional puppet servers for development and disaster recovery, though this document doesn’t cover that.

Issues and resolutions:

  • Puppet agent runs taking in excess of 500 seconds. Situation started when I inadvertently moved the postgres db datafiles without stopping postgres.

    • puppet agent debug runs point to slow connection to puppet server.

    • I noticed an excessive number of tim_wait commands.

    • Updating kernel didn’t resolve the issue.

      • echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

      • echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

      • echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

    • Just prior to rebuilding the server, I noticed that max-active-instances was commented in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf. Once that was uncomented and the server restarted, everything went back to normal.

  • Initial run of puppetserver generates an ssl cert for the server. That ssl cert should list both primary puppet server and alternates. Proper method is to ensure dns_alt_names is specified in normal puppet.conf prior to the first run. If that wasn’t done, the resolution is:

    • Update puppet.conf with dns_alt_names = walvdriconf2442.mycompany.com

    • service puppetserver stop

    • puppet cert clean ${puppset_server}.mycompany.com

    • puppet cert generate ${puppet_server}.mycompany.com \ --dns_alt_names ${alternate}.mycompany.com

    • service puppetserver start

    • Any agents configured prior to this will need to be resigned.

  • puppetdb fails to install/run:

    • First issue caused by incorrect version installed. Tried upgrading the module via:

      puppet module upgrade puppetlabs-puppetdb --version 6.0.1

    • Second issue: When running puppet agent, get an error that yum can’t find postgresql96 to install it. Going to start over from scratch.

    • The clean install worked much better. Need to do it correctly from the start.

  • r10k deploys from gitlab server were displaying ‘project not found’ errors. Turned out to be the same on the puppet server. Previously undiscovered due to ssh agent forwarding. Root cause is the fact that all but a very few modules were listed as private. Temporary(?) fix was to make all modules public. See to-dos for investigation task.

  • puppetdb not storing data? Turned out to be two issues:

    • puppet reports still being stored to the filesystem. To redirect reports to puppetdb, need to add reports = store,puppetdb to puppet.conf. Notes on puppetdb installation updated appropriately.

    • curl syntax to query puppetdb data is incredibly arcane. Installed puppetdbquery module from the forge which allows a much saner query syntax and ability to verify data’s getting stored. Report queries still go through curl:

      curl -G http://localhost:8080/pdb/query/v4/reports
[[boatload of text snipped]]

URLs:

https://docs.puppet.com/puppet/5.0/index.html:

Puppet 5.0 reference manual

https://docs.puppet.com/puppet/5.0/install_pre.html:

Pre-installation checks

https://yum.puppetlabs.com/puppet5/puppet5-release-el-[5-7].noarch.rpm:

puppet 5 yum repo file. Remember to set the el-[5-7] version for the OS being used.

https://docs.puppet.com/puppet/5.0/hiera_config_yaml_5.html:

Documentation for hiera5. Some significant differences between ver 5 and the one installed on the original OS puppet server.

https://andrewwippler.com/2016/10/21/deploying-puppet-open-source/:

All in one checklist for deplying puppet open source w/foreman.

https://docs.puppet.com/pe/latest/r10k.html:

Managing code w/r10k - assumes a PE installation. Still looking for the open source version of this.

https://docs.puppet.com/pe/latest/r10k_run.html:

r10k related commands. Written for PE but calls r10k directly so is probably still applicable.

https://puppet.com/blog/refactor-your-monolithic-code-repo-to-deploy-r10k

Discusses refactoring the monster git:puppet/environments repo into separate repos for use with r10k.

https://github.com/bentlema/puppet-tutorial-pe:

Labs 10-13 were particularly useful for tips on migrating puppet code into gitlab.

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/project/integrations/webhooks.md:

gitlab’s documentation on webhooks.

https://about.gitlab.com/products/:

gitlab’s production comparisons side by side.

Commands:

rpm -Uvh https://yum.puppetlabs.com/puppet5/puppet5-release-el-6.noarch.rpm:

Install the puppet 5 yum repo file on an oel6 box.

puppet config print [${parm}]:

Prints out configuration parameters. Better than searching through various text files, particularly for defaults which may not be specified.

puppet agent -t –noop –evaltrace –summarize:

Runs puppet agent in debug mode attempting to identify where agents are spending their time. Add a ‘-d’ to put agent run in debug mode.

Lessons learned:

  • Need to set dns_alt_names in server’s puppet.conf before initial run of puppetserver.

  • Need to install the correct version of puppetdb before attempting to configure it. Probably could have upgraded all relevant modules, but clean install is preferable.

  • hiera 5 uses three levels of hiera.yaml files. Example hiera5 yaml file is under /etc/puppetlabs/code/environments/production/hiera.yam by default.

    • ${confdir}/hiera.yaml: The normal one I’m used to - different format, though. Sets defaults that the others can override.

    • ${puppet_environment}/hiera.yaml: hiera parms for environments

    • ${module}/hiera.yaml: hiera parms for a specific module.

  • r10k gets intalled/configured before puppetdb.

  • r10k does not install dependencies from the forge. That’s intentional. Reason is that the app does not want to have the possibility of stomping other modules that we have in place. Bummer, but understandable.

  • puppet modules in gitlab must be marked public in order for r10k deployment to work. There is a possibility this can be locked down a bit which is a to-do entry.

  • This should go without saying, but - don’t move the /var/lib/pgsql directory without stopping postgress - not just puppetdb, but the damn db too.

  • If puppet agents across the board are taking 100s of seconds to complete, check the number of jruby instances. The parameter is max-active-instances in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf

To-dos:

Notes:

Prereqs:

  • Decide on architecture: monolithic install.

  • Which servers act as puppet server, puppet db.

  • OS version: OEL6.9 - should be supported, will confirm when running the install.

  • ntp: confirmed

Going with puppet ver 5:

Installation:

  1. Install gitlab and migrate puppet code to it.

    • Dependencies are standard on our systems:

      • openssh (both client/server)

      • curl

      • postfix

      • cronie

      • httpd

    • turn off and disable httpd:

      chkconfig httpd off
service httpd stop

    • Download the script to generate the gitlab repo file. Review the script to ensure gitlab’s not trying to do anything nefarious:

      curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh

    • yum install gitlab-ce

    • gitlab-ctl reconfigure Large amounts of chef configuration later, gitlab should be running on the host.

    • When you access the host, (http://${host}) it will automatically force you to change the account’s password.

    • Create a non-root local user by registering.

      • This account to be used for admin from here out.

      • Update privileges to Admin by accessing the admin area, edit user, set admin user type.

    • To enable ldap authentication, edit /etc/gitlab/gitlab.rb

      • Locate the ldap section by searching for LDAP.

      • Uncomment the main section and the trailing EOS

      • Update the parameters thusly:

        gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main: # 'main' is the GitLab 'provider ID' of this LDAP server
  label: 'LDAP'
  host: 'mpildap.mycompany.com'
  port: 636
  uid: 'uid'
  method: 'ssl' # "tls" or "ssl" or "plain"
  bind_dn: 'uid=linuxauth,ou=accounts,ou=AuthService,dc=corp,dc=viant,dc=net'
  password: 'M6lt1pl8N'
  active_directory: false
  allow_username_or_email_login: false
  block_auto_created_users: false
  base: 'ou=People,dc=corp,dc=viant,dc=net'
  user_filter: ''
  attributes:
    username: ['uid', 'userid', 'sAMAccountName']
    email:    ['mail', 'email', 'userPrincipalName']
    name:       'cn'
    first_name: 'givenName'
    last_name:  'sn'
    ## EE only
    group_base: ''
    admin_group: ''
    sync_ssh_keys: false

EOS

        NOTE: The formatting is very strict. Indentations and quotes are critical. The two gitlab_rails lines, main, and EOS are left justified - no spaces. Indent two spaces (not tabs) from there. Best case, copy the section from another gitlab.rb file. If the formatting is not correct, the resulting error is little better than saying “Something’s broke… fix it”:

        Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb

      • Re-run gitlab-ctl reconfigure

      • Verify ldap login.

      • Add ssh keys. Won’t be able to pull/push w/o them.

NOTE: There may very well be a puppet module based installation of gitlab.

  1. Create control repo:

    • Log in to web gui, create a new group called puppet

    • Under the puppet project, create a project called control

    • Create ssh shortcut:

      host gitlab
  hostname ${gitlab_server}
  user git

    • Clone the repo somewhere temporary:

      cd ~/working/puppet
git clone ssh://gitlab/puppet/control.git
cd control
vi README.md
git add README.md
git commit -am'initial commit'
git remote -v
origin  ssh://gitlab/puppet/control.git (fetch)
origin  ssh://gitlab/puppet/control.git (push)
git push -u origin master

    • Verify new file shows up in the webgui. If you re-clone the repo, you should see your README file.

    • Build the control repo:

      cd ~/working/puppet/control

mkdir manifests
touch manifests/site.pp
echo modules > .gitignore
vi Puppetfile
#################################################################
forge 'http://forge.puppetlabs.com'

#================================================================
# Puppet forge modules
#================================================================
mod 'puppetlabs-stdlib'
mod 'puppetlabs-inifile'
#----------------------------------------------------------------
# Following modules required to install puppetdb on puppetserver
#----------------------------------------------------------------
mod 'puppetlabs-puppetdb', '6.0.1'
mod 'puppetlabs-postgresql', '5.0.0'
mod 'puppetlabs-apt', '4.1.0'
mod 'puppetlabs-concat', '4.0.1'
mod 'puppetlabs-firewall', '1.9.0'
#----------------------------------------------------------------
# End puppet forge modules
#----------------------------------------------------------------
#################################################################

git status
# On branch production
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       Puppetfile
#       .gitignore
#       site/
git add -A
git commit -m'production branch initial commit'
[production 1dc723f] production branch initial commit
 4 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 .gitignore
 copy a => Puppetfile (100%)
 copy a => site/manifests/packages.pp (100%)
 rename a => site/manifests/roles/base.pp (100%)
git push -u origin production

  2. Configure gitlab:

    NOTE: Follow directions in the vbox labs to create control repo, deploy key and test it out. One note: per lab10, author creates another monolithic codebase repo in the control repo. We’re not doing that. Everything but that remains the same.

  • Delete the master branch. Required so r10k doesn’t create a master environment.

    • Access gitlab -> puppet/control project

    • Access project settings in the top menu bar.

    • Access Repository in the second menu bar.

    • Expand ‘protected branches’

    • Protect production, unprotect master

    • Access project settings in top menu bar, then General in second menu bar. Expand General project settings

    • Change the default branch to ‘production’

    • git branch -D master

    • git push origin --delete master

  1. Copy monolithic repos to puppet project on gitlab.

    • Create a public project in group puppet named after the module.

    • Trim each module so only it and its history remain:

      • cd ~/working/puppet

      • rm -fr ./environments

      • git clone git@nap1151:puppet/environments

      • cd environments

      • git filter-branch –subdirectory-filter modules/${module}

      • git tag 1.0.0

      • git remote add origin git@${gitlab_server}.mycompany.com:puppet/${module}.git

      • git push -u origin master

      • git commit -am…

      • git push -u production

    • Iterate the previous step until all modules have been migrated including roles and profiles.

    • Update the control repo

      • cd ~/working/puppet/control

      • vi Puppetfile. Add all the modules that you just migrated to gitlab. Syntax for the homegrown modules is:

        mod 'roles',
    :git => 'ssh://gitlab/puppet/roles.git',
    :tag => '1.0.0'

mod 'profiles',
    :git => 'ssh://gitlab/puppet/profiles.git',
    :tag => '1.0.1'

      • git commit -am'approprite message goes here'

      • git tag 1.#.# -m'appropriate message goes here

      • git push -u origin production

      • git push -u origin production --tags

    • Verify all the modules are in the production environment on the puppet server.

  2. Install puppet server v5 on the puppet server:

    1. Enable the puppet v5 package repo either through adding the puppet repo or through pulp

    2. Install puppet server ver 5 on the puppet server:

      # yum install puppetserver git
[[snip]]

      If installing on a system that’s configured for pulp:

      # yum install puppetserver  pulp-puppet-handlers
[[snip]]

### Rework #6 to have notes, then combine the various sections that identify changes.

  1. Configure puppet server. Some notes from initial install of puppet4 that still seem applicable:

    • Config files:

      • Primary config file at /etc/puppetlabs/puppetserver/conf.d

      • Historic config file at /etc/puppetlabs/puppet/puppet.conf

    • Any config options available in the conf.d files will be ignored in puppet.conf. Another warning: a setting left undefined in config files will revert to default value rather than using the values from puppet.conf. Long story short: don’t use puppet.conf for puppet server. puppet commands, though, will use this file so keep the parameters in sync. Options to keep in sync:

puppet.server

puppet.conf

Default

master-conf-dir

confdir

/etc/puppetlabs/puppet

master-code-dir

codedir

/etc/puppetlabs/code

master-var-dir

vardir

/opt/puppetlabs/server/data/puppetserver

master-run-dir

rundir

/var/run/puppetlabs/puppetserver

master-log-dir

logdir

/var/log/puppetlabs/puppetserver

  • Update puppet.conf to put volatile files under /var in puppet.conf and puppetserver.conf. Update ownership and perms:

    # /etc/puppetlabs/puppet/puppet.conf
[agent]  # req for puppet agent runs
vardir = /var/opt/puppetlabs/puppetserver

[user]
vardir = /var/opt/puppetlabs/puppetserver

[master]
vardir = /var/opt/puppetlabs/puppetserver

# /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf
# (optional) path to puppet conf dir; if not specified, will use
# /etc/puppetlabs/puppet
master-var-dir: /var/opt/puppetlabs/puppetserver

# mkdir -p -m 755 /var/opt/puppetlabs/puppetserver && \
chown puppet:puppet /var/opt/puppetlabs/puppetserver

  • Changes:

    • /etc/puppetlabs/puppet/puppet.conf:

      • mv vardir to /var/opt/puppetlabs/puppetserver

      • Update perms/ownership on /var/opt/puppetlabs/puppetserver

      • Add dns_alt_names to server puppet.conf

      • Add server = ${puppet_server}.mycompany.com to master and agent sections in puppet.conf

    • /etc/puppetlabs/puppetserver/conf.d/puppet.conf:

      • master-var-dir: /var/opt/puppetlabs/puppetserver

      • Update max-active-instances to 7 (one less than the # of cores)

    • Create 15gb lv and mount under /var/lib/pgsql

  1. Start the puppetserver: service puppetserver start Verify cert has both names. Update if not. Interestingly, puppetserver will not recreate /etc/puppetlabs/puppet/ssl if the directory is missing like the puppet client will.

puppet cert list --all

  1. Install/configure r10:

    • Install the r10k gem. I have absolutely no idea if I need to run both puppet{server,user}; however, both were required for hiera. Since r10k is used both by the server(?) and the user, it makes sense to install in a similar fasion

      • puppetserver:

        /opt/puppetlabs/bin/puppetserver gem install r10k

      • puppet user:

        /opt/puppetlabs/puppet/bin/gem install r10k

    • Link r10k script to /etc/puppetlabs/bin:

      ln -s /opt/puppetlabs/puppet/bin/r10k /opt/puppetlabs/bin/r10k

    • On the puppet server, create r10k.yaml file in /etc/puppetlabs/r10k:

      # mkdir -p -m 755 /var/cache/r10k
# mkdir -p -m 755 /etc/puppetlabs/r10k
# cat /etc/puppetlabs/r10k/r10k.yaml
---
cachedir: /var/cache/r10k
sources:
  puppet:
    basedir: /etc/puppetlabs/code/environments
    remote: gitlab:puppet/control

    • If needed/desired, create a backup of /opt/puppetlabs/code/environments

    • Update ssh config env:

      host gitlab
  hostname ${gitlab_server}
  user git
  identityfile /root/.ssh/r10k

    • Create /root/.ssh/r10k:

      # ssh-keygen -t rsa -b 2048 -P "" -f ./r10
[[snip]]

    • Add r10k.pub to gitlab’s deploy key:

      1. Access gitlab as root user.

      2. Access Admin area by selecting the wrench to the immediate right of the search box at the top of the screen.

      3. Expand the gear menu immediately below the user icon in the top right. Select deploy keys.

      4. Enter the key data as appropriate, then select ‘create’

      5. If needed, copy git’s authorized_keys file to /etc/sshkeys:

        # cp /var/opt/gitlab/.ssh/authorized_keys \
/etc/sshkeys/authorized_keys.git
# chmod 640 /etc/sshkeys/authorized_keys.git
# chgrp git /etc/sshkeys/authorized_keys.git

    • On the puppet server, execute r10k deploy environment -vp. If you get an error about host key verification failure, add gitlab’s host key to known hosts: ssh ${gitlab_server} hostname.

      • Assuming it works, this will remove anything under /opt/puppetlabs/code/environments replacing it with items from the control repo.

      • If you get an error like the one below, install git:

        ERROR    -> No Git providers are functional.

  1. Configure gitlab to push changes

    • Overview notes:

      • Issue is we want gitlab to execute an r10k run whenever someone pushes a control repo.

      • Right answer, according to the labs and other reading is a webook which, on the surface, doesn’t seem that trivial.

      • puppet/r10k is apparently the right way to manage r10k - and we’ll get there - but it also manages the webhook. Big issue with the module is that it has a host of dependencies some of which are already being maintained by our own repos (gcc).

      • So, for the time being, we will use the ssh key documented in the labs.

    • Create key:

      ssh ${gitlab_server}
su - git
cd .ssh
ssh-keygen -t rsa -b 2048 -N '' -f gl_r10k
vi config

    • Create /var/opt/gitlab/.ssh/config to look like:

      loglevel=error
host pm
  hostname ${puppet_server}.mycompany.com
  user root
  identityfile /var/opt/gitlab/.ssh/gl_r10k

    • Add key to ${puppet_server}:/root/.ssh/authorized_keys following normal process. If you’re managing your keys in puppet (like you should be), ensure the appropriate module gets updated.

    • Verify access:

      ssh -o stricthostkeychecking=no pm whoami
root

    • Create post-receive:

      cd /var/opt/gitlab/git-data/repositories/puppet/control.git
mkdir custom_hooks && cd custom_hooks
vi post-receive
#----------------------
#!/bin/bash

PATH=/bin:/usr/bin:/sbin/:/usr/sbin
export PATH
pms=(pm)   # in case we ever add more...

echo
echo "Running post-receive hook..."
for pm in ${pms[*]}
do
  ssh $pm \
  "echo \"[$pm] Updating...\" ; /opt/puppetlabs/bin/r10k deploy environment -p ; echo \"[$pm] Done.\""
done
echo
#----------------------
# chmod 755 ./post-receive

  2. Apply ENC. Note: this assumes you’re using a mysql db as a cmdb. If you’re not, adjust accordingly.

    • Install mysql

    • Copy /root/.my.cnf

    • Copy /opt/puppetlabs/server/data/puppetserver/.my.cnf. Verify ownership/permissions.

    • Copy /usr/local/bin/enc

    • Update master section of /etc/puppetlabs/puppet/puppet.conf:

      node_terminus  = exec
external_nodes = /usr/local/bin/enc

    • service puppetserver restart

  1. Configure hiera and eyaml:

    • We have our keys maintained in a gitolite git repo outside of gitlab so retrieving them is much easier. There are directions available on the net for generating new keys if needed. Clone keys:

      cd /etc/puppetlabs
mkdir -p -m 755 secure
cd secure
git clone git@${local_mgmt_server}:puppet/secrets ./keys
chown -R puppet:puppet /etc/puppetlabs/secure

    • Configure root to use keys:

      cd /root
mkdir -p -m 700 .eyaml

      create /root/.eyaml/config.yaml with:

      ---
pkcs7_private_key: /etc/puppetlabs/secure/keys/private_key.pkcs7.pem
pkcs7_public_key: /etc/puppetlabs/secure/keys/public_key.pkcs7.pem

    • Install hiera-eyaml gems installed:

      • Puppet server:

        /opt/puppetlabs/bin/puppetserver gem install hiera-eyaml

      • Puppet user:

        /opt/puppetlabs/puppet/bin/gem install hiera-eyaml

    • Link eyaml to a dir in the path:

      ln -s /opt/puppetlabs/puppet/bin/eyaml /opt/puppetlabs/bin/eyaml

    • Clone hiera data:

      cd /etc/puppetlabs/code
git clone git@nap1151:puppet/hiera ./hieradata

    • Create/copy /etc/puppetlabs/puppet/hiera.yaml

    • service puppetserver restart

  1. Install/configure puppetdb:

    • Overview notes:

      • puppetdb should be installed via module; however, on the first run through this process I was facing a distinctive chicken/egg scenario. I didn’t have gitlab, the control repo, or any of the modules migrated.

      • In order to install modules outside of being able to use the enc, I created a site.pp file.

      • Right way to do it is the the roles/profiles/ENC as that’ll be permanent. The wrong way is a short cut initially but caused problems when trying to manage more than just the puppet server.

    • Choose right way or wrong way to install puppetdb module:

      • Right way: update roles/profiles/ENC

        • Create/update mpipsrv profile:

          $ cd ../profiles/manifests
$ cat mpipsrv.pp
class profiles::mpipsrv  {

  class {'puppetdb': }
  class {'puppetdb::master::config':}
  class {'puppetdbquery':}
}

        • Create/upate mpipsrv role:

          $ cd ../../roles/manifests
$ cat mpipsrv.pp
class roles::mpipsrv  {

  include ::profiles::mpibase
  include ::profiles::mpipsrv
}

        • Update tags on both roles/profiles, then push.

        • Update control Puppetfile with new tags.

        • Update ENC to reflect new role:

          $ manage_enc --list --host ${puppet_server}
Host             Environment  Classes
=======================================================
${puppet_server}  dkoleary     roles::mpipsrv

      • Wrong way: Create ${control}/manifests/site.pp:

        node ${puppet_server} {
  class {'puppetdb': }
  class {'puppetdb::master::config':}
}

        NOTE: Over and above not being a permanent fix, the site.pp file will prevent puppet server from managing any node not listed in it. If used, it must be removed immediately afterwards.

    • Run puppet agent -t The agent run ran much better. See issues for the hassles experienced. I’m still getting the iptables errors, basically bitching about not having a config file. That recurrs on successive agent runs as well.

NOTE: major wrestling around with modules from different sources. Need a way to keep that reasonably straight.

  • Add the following to /etc/puppetlabs/puppet/puppet.conf to store reports in the db:

    # grep reports /etc/puppetlabs/puppet/puppet.conf
reports              = store,puppetdb
service puppetserver restart

  • Verifying that puppetdb is being populated is a little more problematic. Run an agent on any number of nodes a couple of times to ensure some data is in puppetd. Then, on the puppetdb server, assuming dalen-puppetdbquery is included, you can use puppetdb query:

    # puppet query facts --facts env 'hostname=walvprogit497'
walvprogit497.mycompany.com  {"env":"prd"}
# puppet query events 'hostname=walvprogit497'
walvprogit497.mycompany.com: 2017-07-17T12:58:19.868Z: Service[ossec-hids]/ensure (stopped -> running): ensure changed 'stopped' to 'running'
walvprogit497.mycompany.com: 2017-07-17T12:58:23.809Z: Service[xymon-client]/ensure (stopped -> running): ensure changed 'stopped' to 'running'
walvprogit497.mycompany.com: 2017-07-17T13:28:20.895Z: Service[ossec-hids]/ensure (stopped -> running): ensure changed 'stopped' to 'running'
walvprogit497.mycompany.com: 2017-07-17T13:28:24.899Z: Service[xymon-client]/ensure (stopped -> running): ensure changed 'stopped' to 'running'

    Veriyfing reports can’t be done through puppetdb query, unfortunately:

    curl -G http://localhost:8080/pdb/query/v4/reports
[[boatload of text snipped]]

  1. Configure policy based cert autosign:

    • Add autosign = /usr/local/bin/check_cert to [master] section of /etc/puppetlabs/puppet/puppet.conf

    • Create /usr/local/bin/check_cert with the following:

      # cat /usr/local/bin/check_cert
#!/bin/bash

export PATH=/bin:/usr/bin:/sbin:/usr/sbin
Req=$1
[[ ${#Req} -eq 0 ]] && exit 4
hostid=$(echo "select hostid from hosts
where hostname = '${Req%%\.*}'" | mysql -N)
[[ ${#hostid} -gt 0 ]] && exit 0 || exit 3

service puppetserver restart

  2. Still need to to mcollective; but skipping that for now.

Migration:

  1. Ensure puppet5 repos are available.

  2. On the client:

    1. Stop puppet.

    2. Erase puppet agent

    3. Install puppet5 agent.

    4. Remove /etc/puppetlabs/puppet/ssl

    5. Update /etc/puppetlabs/puppet/puppet.conf. If puppet is managing the puppet configuration, you can also:

      puppet agent -t --server ${puppet_server}.mycompany.com

## or::
# Add the following to /etc/puppetlabs/puppet/puppet.conf
[main]
  server = ${puppet_server}.mycompany.com

[agent]
  server = ${puppet_server}.mycompany.com

[user]
  server = ${puppet_server}.mycompany.com

    1. # puppet config print server ${puppet_server}.mycompany.com

puppet configuration repo:

Gitlab obviously requires accounts which users use to log in and modify repositories. Each account must have a unique email address which normally isn’t a problem.

Where this becomes an issue is when we need to push configuration repos from the puppet server using an ssh key (tied to an account) to gitlab.

The first choice is to use an account that belongs to a human. What happens if that person moves on either to a different role or to a different company? This is the same paradigm that prevents management scripts run via cron jobs to be run under someone’s personal account.

The second choice is to use the deploy key. That key has global read access to all repos but allowing write capability for specific repos is both difficult and ill-advised based on various google groups.

The third choice is to use psuedo accounts in gitlab. These accounts have an email configured in gitlab which, on its own is invalid, but is remapped to a valid one. This allows an ssh key to be assigned to it which can then be used to push the puppet configuration repo.

Details:

${puppet_server} (puppet server):

  • As root, create a null passphrased key. In this case, /root/.ssh/pconfig

  • Update /root/.ssh/config to create a host alias:

    host pconfig
  hostname ${gitlab_server}
  user git
  identityfile /root/.ssh/pconfig

  • Initialize a repo in /etc/puppetlabs:

    cd /etc/puppetlabs
git init

  • Configure repo to ignore the code directory:

    cd /etc/puppetlabs
echo 'code' > .gitignore
git config --global core.excludesfile .gitignore

  • Verify the ignore file by executing:

    git status | grep code

  • Add and commit the repo:

    git add -A
git commit -am'puppet configuration files including ssl certs'

  • Create the remote configuration:

    git remote add origin pconfig:puppet/puppet-configs.git

${gitlab_server} (gitlab):

  • Created a master psuedo account called pconfig with the email address pconfig@mycompany.com.

  • Apply ${puppet_server}:/root/.ssh/pconfig.pub as the ssh key for pconfig account on gitlab.

  • Create the puppet/puppet-configs.gi project/repo.

Back on ${puppet_server}:

  • Push the repo:

    cd /etc/puppetlabs
git push origin master

  • Create a script to be run via cron that will push the configurations when changes are detected:

    #!/bin/bash

hdirs=( /etc/puppetlabs/code/hieradata /etc/puppetlabs )
Dtg=$(date)
echo "Starting: ${Dtg}"

for d in ${hdirs[*]}
do
  echo "${d}" | grep -q ^/etc/puppetlabs$ && \
    msg="puppet config update - prob ssl" || \
    msg="Auto update as not everyone uses git like they should"
  echo "  checking: ${d}"
  cd ${d}
  ## pull any changes
  git pull
  ## how many updates
  n=$(git status --porcelain | wc -l)
  ## if any updates, add, commit, and push
  if [ ${n} -gt 0 ]
  then
    git add -A
    git commit -am"${msg}"
    git push
  fi
done

Dtg=$(date)
echo "Ending: ${Dtg}"