/===================== Puppet notes - take 2: ====================== Overview: ========= Going back over the pro puppet book. Now that we're closing in on installing puppet at work, it'd behoove me to be the best puppet engineer I can... and the best at MPI. To-do: ====== * Go over the url for bootstapping with puppet. * Review params subclass inheritance examples * Read puppet style guide: https://docs.puppetlabs.com/guides/style_guide.html * Examine vagrant (http://www.vagrantup.com) for interaction with kvm. Lessons learned: ================ * DSL = Domain Specific Language * puppet won't start without a sites.pp file. * puppet key's not working with the spacewalk. need to run the following command on all clients: :: rpm --import https://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs * Date needs to be in sync across the puppet environment. * Directories to back up: * /var/lib/puppet/ssl * Definition vs instantiation: * Definitions go in module files (vhosts.pp) * instantiations go in node files: :: node '$host' { include base include apache apache:vhost {'www.oci.com': port... docroot... ssl ... etc } apache:vhost {'www.nuther.com': ... } apache:vhost {'www.nuther1.com': ... } } * notify vs notic: * notify is a puppet resource: * ``notice {"Hello, world": }`` * Class is determined by location in the log and/or display msg * notice is a function: * ``notice("Hello, world") * Class is determined by output: :: # puppet apply -e 'notice("Hello, world")' Notice: Scope(Class[main]): Hello, world * Passing an array to a type (like notify) produces one resource of that type for every element in the array. :: # puppet apply -e 'notice {["one","two", "three"]:}' puppet apply -e 'notify {["one","two", "three"]:}' Notice: Compiled catalog for node1.olearycomputers.com in environment production in 0.03 seconds Notice: one Notice: /Stage[main]/Main/Notify[one]/message: defined 'message' as 'one' Notice: three Notice: /Stage[main]/Main/Notify[three]/message: defined 'message' as 'three' Notice: two Notice: /Stage[main]/Main/Notify[two]/message: defined 'message' as 'two' Notice: Finished catalog run in 0.07 seconds Valuable commands: ================== puppet help puppet config print ... Prints configuration settings. For instance: :: # puppet config print ssldir /var/lib/puppet/ssl puppet module generate ${name} Generates an involved puppet module after asking a bunch of questions puppet cert list [--all] lists outstanding certs. If *--all* is supplied, signed certs are prepended with a '+' puppet parser validate Validates the syntax of the module. :: # puppet parser validate manifests/init.pp # echo $? 0 erb -x -T '-' main.cf.erb | ruby -c Checks syntax of a template: :: # erb -x -T '-' main.cf.erb | ruby -c Syntax OK puppet apply -e 'if "Puppet" == "puppet" { notify {"true!?":}}' Demo of testing puppet syntax Interesting URLs: ================= http://docs.puppetlabs.com/references/stable/type.html: Full list of puppet manageable types http://projects.puppetlabs.com/projects/1/wiki/Bootstrapping_With_Puppet Bootstrapping with puppet http://docs.puppetlabs.com/guides/faq.html#Common+Misconceptions FAQ http://docs.puppetlabs.com/puppet/latest/reference/lang_node_definitions.html#inheritance Puppet documentation on inheritence and why you shouldn't... http://docs.puppetlabs.com/puppet/latest/reference/lang_summary.html#conditionals Puppet language basics: conditioanls specifically. http://docs.puppetlabs.com/guides/style_guide.html Puppet style guide https://github.com/openstack-infra/system-config/blob/master/manifests/site.pp openstack puppet infrastructure: example of using parameterized classes and a glue class. 03/31/15: ========= I have 2 kvm guests, one's ubuntu. I'm probably going to blast the centos one because it's already using puppet. I want to get this thing running via the puppet master this time. * Declarative vs imperative language. * Layers * Resource abstracton layer - facter * Transactional layer - the puppet engine * Creates graph showing resources, relationships. Used to calculate action order. * Mixed releases of puppet: * puppet master must be newest release of the batch * Older versions don't support everything. Stopping at the installation. It's getting late. 04/01/15: * Installation: * prereqs: * ruby * ruby-libs * ruby-shadow * epel repo * Master: * puppet * puppet-server * facter * Clients: * puppet * facter * W/my setup, all available packages are on the spacewalk server. * Ubuntu: * apt-get install puppet facter [puppetmaster] * Firewall config: 8140 needs to be open. * The /etc/puppet/ssl dir doesn't exist. Starting to notice a few things aren't right with this book and the ver 3.X ver: :: # puppet config print ssldir /var/lib/puppet/ssl # ll /var/lib/puppet/ssl total 36 drwxrwx--x. 8 puppet puppet 4096 Apr 1 20:58 ./ drwxr-x---. 11 puppet puppet 4096 Apr 1 20:58 ../ drwxr-xr-x. 5 puppet puppet 4096 Apr 1 20:58 ca/ drwxr-xr-x. 2 puppet puppet 4096 Apr 1 20:58 certificate_requests/ drwxr-xr-x. 2 puppet puppet 4096 Apr 1 20:58 certs/ -rw-r--r--. 1 puppet puppet 975 Apr 1 20:58 crl.pem drwxr-x---. 2 puppet puppet 4096 Apr 1 20:58 private/ drwxr-x---. 2 puppet puppet 4096 Apr 1 20:58 private_keys/ drwxr-xr-x. 2 puppet puppet 4096 Apr 1 20:58 public_keys/ 04/07/15: Need to do better on the studying... * Getting the hosts ready * First connection, as advertised: # puppet agent --server=pm.olearycomputers.com --no-daemonize --verbose Info: Creating a new SSL key for node1.olearycomputers.com Info: Caching certificate for ca Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml Info: Creating a new SSL certificate request for node1.olearycomputers.com Info: Certificate Request fingerprint (SHA256): CC:A2:AE:22:4A:32:54:4F:F2:D1:C9:0D:40:E9:A9:09:E0:47:F5:D2:17:38:4D:71:04:4D:F6:20:99:67:51:87 Info: Caching certificate for ca * puppet master: :: # h pm # puppet cert --list "node1.olearycomputers.com" (SHA256) CC:A2:AE:22:4A:32:54:4F:F2:D1:C9:0D:40:E9:A9:09:E0:47:F5:D2:17:38:4D:71:04:4D:F6:20:99:67:51:87 # puppet cert --sign node1.olearycomputers.com Notice: Signed certificate request for node1.olearycomputers.com Notice: Removing file Puppet::SSL::CertificateRequest node1.olearycomputers.com at '/var/lib/puppet/ssl/ca/requests/node1.olearycomputers.com.pem' * On target: :: [[after signing]] Info: Caching certificate for node1.olearycomputers.com Notice: Starting Puppet client version 3.7.4 Info: Caching certificate_revocation_list for ca Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for node1.olearycomputers.com Info: Applying configuration version '1428452868' Info: Creating state file /var/lib/puppet/state/state.yaml Notice: Finished catalog run in 0.04 seconds * Interesting, on puppetmaster, can import nodes via: :: import 'nodes/*' import 'classes/*' * Fuck me, that took longer than it should have. My first puppet module and I fucked it up... and took almost an hour to find the fucker. What's wrong with this picture? :: class sudo { package { sudo: ensure => present, } if $operatingsystem == 'Ubuntu' { package { "sudo-ldap": ensure => present, require => Package["sudo"], } } file { "/etc/sudoers": owner => 'root', group => 'root', mode => 0440, source => "puppet:///modules/sudo/etc/sudoers", require => Package["sudo"], } } When run on node1, I was getting: :: # puppet agent --server=pm.olearycomputers.com --no-daemonize --verbose --onetime Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for node1.olearycomputers.com Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false. (at /usr/lib/ruby/site_ruby/1.8/puppet/type/package.rb:430:in `default') Info: Applying configuration version '1428454886' Error: /Stage[main]/Sudo/File[/etc/sudoers]: Could not evaluate: Could not retrieve information from environment production source(s) puppet://pm.olearycomputers.com/modules/sudo/etc/sudoers WTF, over? Finally, as I'm typing up a question on server fault, I finally figured it out... :: source => "puppet:///modules/sudo/etc/sudoers", It's not in **etc** you slack-jawed numb nut! :: source => "puppet:///modules/sudo/sudoers", works just fine... fucktard! * Checked the book again. Noticed they had etc in the file resource; but, they actually made modules/sudo/files/etc and copied sudoers into it. Not a bad idea, actually. * Good place to stop; heading into chapter 2. 04/08/15: ========= Chapter 2: ---------- * node definitions: * node '${node1}', '${node2}', '${node3}' { ... } * node /^web\d+\.example\.com$/ {..} # perl regex * external node classifier * node inheritance: * Book gives example. From class, though: * Only use with a parameters subclass. * Reason to inherit vs include is to be able to override class parameters. Syntax is as follows: :: class apache ( $httpd_user = $apache::params::httpd_user, $httpd_group = $apache::params::httpd_group, $httpd_pkg = $apache::params::httpd_pkg, $httpd_svc = $apache::params::httpd_svc, $httpd_conf = $apache::params::httpd_conf, $httpd_confdir = $apache::params::httpd_confdir, $httpd_docroot = $apache::params::httpd_docroot, ) inherits apache::params { [[rest-o-class-def-snipped]] * Class 'inherits' reasonable defaults from $apache::params yet can override them using class resource declaratios as described in previous section. * Variable scoping. Bet that's going to bite me in the ass someday. For the most part, fairly comfortable with the concept. * Got up to the params class. all four systems are part of the puppet env now. :: # puppet cert list --all + "node1.olearycomputers.com" (SHA256) 88:9B:03:37:E9:05:D4:8F:63:4B:F2:0D:3F:60:5B:B3:9D:D5:3E:7B:2C:1F:0C:D4:2A:B6:97:DC:97:9A:45:07 + "pm.olearycomputers.com" (SHA256) F3:8B:D6:CF:8B:58:B7:3E:66:76:79:D3:99:F0:03:D3:45:C9:87:2F:30:57:FB:B1:39:C0:48:6C:9A:0D:F9:0D (alt names: "DNS:pm.olearycomputers.com", "DNS:puppet", "DNS:puppet.olearycomputers.com") + "ubuntu.olearycomputers.com" (SHA256) C9:4B:60:C6:A1:3E:8F:20:2A:51:45:50:17:B2:D3:05:60:AC:52:10:12:A7:86:AA:04:37:2C:25:CC:4B:33:F1 + "xymon.olearycomputers.com" (SHA256) DD:1E:66:48:11:81:7A:2C:B2:A4:08:62:80:BE:9A:E3:C9:05:91:09:10:05:AC:39:D0:B3:A1:8E:27:F1:42:DC Tired, time to go to bed. 04/12/15: ========= Been noticing many differences betwee the puppet I installed and the one the book was talking about. Looked on amazon and found an update to the book. Bit pricy, but I bought it. Starting over. Lessons learned, et al from above will remain. * Bit different first chapter. More heavily into the configuration. * Update puppet.conf to include ``server=puppet.oci.com`` in the main section. * Shows how to update iptables for puppet: :: iptables -A INPUT -p tcp -m state --state NEW --dport 8140 -j ACCEPT or, better yet, restrict to IPs that are in use only: :: iptables -A INPUT -p tcp -m state --state NEW \ --dport 8140 -s 192.168.0.0/24 -j ACCEPT * That ``server=pm.oci.com`` has to be everywhere... just tried running ``puppet agent --test`` on node1 and it choked with all sorts of ugly errors. Once server line was added to /etc/puppet/puppet.conf, everything worked well. * Book's for ver 3.2; I'm running 3.7. Seems there's been some changes since the book was written. They're talking about having to install puppet agent on the puppet master. I didn't do that on pm and it's already there. * DSL notes: * Parameterized classes: * Simple version: :: node /node1/ { include ::sudo { * Parameterized version: :: node /node2 { class { '::sudo': users => [ 'tom', 'jerry' ], } } * selector - puppet trinary, but more. Case insensitive: :: $packge_name = $::osfamily ? { 'RedHat' => "openssh-server", 'Debian' => "openssh-server", 'Solaris' => "openssh", } * Selector requirements: * All selectors and case statements must have a default value. Note the example above does not. * Defaults should call fail() if a default behavior is questionable. * defautl: {} is acceptable * Scope: * ``::${class}`` indiccates a top level class. Possibly similar in paradigm to adding {} around var names. * Facter vars are top scope. * Available scopes: * top - facter, or site.pp vars * node: vars in a node definition * parent: vars from an explicity inherited class * local: sub class vars * Misc: * ``==`` operator is case insensitive. To do case sensitive searches, need a perl syntax. ``$osfamily =~ /Debian$/`` * ``$::server`` is the hostname of the puppet master. * Seemngly backed away from the ``import nodes.pp`` in site.pp. * Node classification: * straight: ``node [...}`` * multiple nodes: :: node 'node1.oci.com', 'node2.oci.com', 'node3.oci.com' {...}`` * regex: ``node /^node[1-3]\.oci\.com$/ {...}`` * ENC: discussed in chapter 5 * Default node: particularly useful when auto-signing certs. Stopping at page 44 right before *Creating a module to manage ssh* 04/14/15: ========= * WTF with this syntax? it's not explained anywhere but passes the ``puppet parser validate`` Apparently, some type of short hand for ``include`` :: class ssh { class { '::ssh::package': } -> class { '::ssh::config': } -> class { '::ssh::service': } -> Class['ssh'] } * Notes on DSL stuff in the section above Short night; stopping right before params class on page 48. woohoo, 4 whole pages. 05/24/15: ========= Getting ready to hop back into puppet. Been gone for a bit getting the cmdb ready at work. That's mostly done so now I can concentrate on this. I briefly contemplated rebuilding everything, but decided against it. I haven't gone so far as to make that necessary. I ame taking a step back and starting chapter 2 completely over again. I'll start in on that later tonight or tomorrow. 05/25/15: ========= * Variable scoping: * Puppet DLS is a declarative language therefore order doesn't matter. As such, vars can't be reassigned once set as there would be no predictable value for the var. * Scopes: * top * Anything defined in facter, sites.pp or imported manifests * Explicity accessed by prepending '::' to var. * node * vars created within the brackets of a node definition * No explicit accessing. * local * Single class or defined type * No explicit accessing. * Has access to node scope vars unless overridden * parent * Class scope that is specifically inherited through the *inherits* keyword. * NOTE: This is **not** node inheritance. Class inheritance is possibly a good thing. * Class inheritance vs include statement: * Examples: * include (old style): :: class ssh::params { case $::osfamily { 'Debian': { $sshd_package = 'ssh' } 'RedHat': { $sshd_package = 'openssh-server' } default: {fail(" Login class does not work on osfamily: ${:: osfamily}")} } } class ssh { include ssh::params package { $::ssh::params::sshd_package: ensure = > installed, } } include ssh * parameterized: :: class ssh::params { case $::osfamily { 'Debian': { $ sshd_package = 'ssh' } 'RedHat': { $ sshd_package = 'openssh-server' } default: {fail(" Login class does not work on osfamily: ${::osfamily}")} } } class ssh inherits ssh::params { package { $::ssh::params::sshd_package: ensure = > installed, } } include ssh * On the surface, definitions don't seem that much different; however, paraemterized classes allow variables to be overridden. * Class definiont '->' WTF?? * Example: :: class ssh { class { '::ssh::package': } -> class { '::ssh::config': } -> class { '::ssh::service': } -> Class ['ssh'] } * Based on this line in the text, I **believe** this is an alternate form of the include function; however, I can't seem to find anything to confirm or refute that: :: In listing 2-13, we created a funcitnoal structure by dividing the components of the serve we're managing info functional domains: packages to be installed, files to be configured, and service to be executed. We created a class called ssh (which we need to ensure that the module is valid) and used the include function to all all the classes to the module. * Finally found something. It's puppet's chaining syntax. It's setting up an order of the classes in ::ssh, first package, then config, followed by service. WTF is that for??? * Templates: * <%= ... %> Ruby variable * Facter facts listed with '@' rather than '$' * Directories: * W/o a source parameter, a ``recurse => true`` statement can be used to force a user/group ownership on all files there in.... * Parameterized classes: * need to review the notes from class. * that being said, syntax very similar to a class definition inside of another class or node definition is another form of ``include`` * init.pp: :: class mysql ( $user = 'mysql', $group = 'mysql', $service_enabled = true, $service_running = true ) { class { 'mysql::install': user => $user, group => $group, } class { 'mysql::config': user => $user, group => $group, } class { 'mysql::service': ensure => $service_running, enabled => $service_enabled, } } * Node definitions using the same class with different parameters: :: node 'prod.example.com' { include base include ssh class { 'mysql': user => 'mysql', service_running => true, service_enabled => true, } } node 'qa.example.com' { include base include ssh class { 'mysql': user => 'qamysql', service_running => true, service_enabled => true, } } node 'dev.example.com' { include base include ssh class { 'mysql': user => 'devmysql', service_running => false, service_enabled => false, } } * Audit mode: * Can use puppet as a rich man's HIDS. :: file { $file: audit => [ owner, group, mode ], } Stopping at the apache stage. 05/30/15: ========= * Definitions: * Similar to classes but can be specified and evaluated multiple times. * Definitions go in module files. Using the example below, the definition would be in vhost.pp. * Instantiation goes in the nodes.pp file * Definition syntax: :: define apache::vhost ( $docroot, $port, $priority, $ssh=true, $serveralises = '', $template='apache/vhost.conf.erb', ) { include apapche file { "/etc/apache2/sites-enabled/${priority}-${name}": content => template($template), owner => 'root', group => 'group', mode => '0640', require => Class['apache::install'], notify => Class['apache::service'], } } * Instantiation syntax: :: apache::vhost { 'www.example.com': port => '80', docroot => '/var/www/www.example.com', ssl => false, priority => '10', serveraliase => 'home.example.com', } Chapter 3: Developing and deploying puppet: -------------------------------------------- 05/31/15: Started with some studying as documented below, then went on to rebuild the environment. My puppet git govno was starting to be quite different then the one in the book. * Some interesting tidbits about running pupppet apply, differences between notice and notify, and how arrays are handled. All ended up in the lessons learned section. * Typical layout for /etc/puppet on puppet master - starting at /etc/puppet: :: # tree environments/ environments/ ├── dev │   ├── manifests │   │   └── site.pp │   └── modules │   ├── apache │   ├── collectd │   ├── concat │   ├── nrpe │   ├── postfix │   ├── stdlib │   └── sudo ├── prod │   ├── manifests │   │   └── site.pp │   └── modules │   ├── apache │   ├── collectd │   ├── concat │   ├── nrpe │   ├── postfix │   ├── stdlib │   └── sudo └── staging ├── manifests │   └── site.pp └── modules ├── apache ├── collectd ├── concat ├── nrpe ├── postfix ├── stdlib └── sudo * Book goes into some depth on vagrant, a method of automating vm builds for vmware or virtualbox. might work with kvm as well. need to look into this some other time. * Environments: * module maintenance models: * Each module is its own repository * Single repository under modules * Tools for external modules: Covered in chapter 8 (this "we'll cover that later" thing is getting old) * puppet-librarian * r10k * Configuration: * Add support for environments in pm:/etc/puppet.conf: