RHCE Class (rh255) notes:

Things to do:

  • Verify my test on Friday.

  • Read/stufy ip command (man ip)

  • Review iscsi and potential for SAAS.

  • Experiment w/NIC bonding in vm lab.

Lessons Learned:

  • ethtool -p ${int} ${seconds} will blink the network card for the seconds identified. Useful for identifying nics.

  • Flush firewall rules before trying to recreate them. Otherwise, things just get added instead of replaced.

  • man -k _selinux: displays list of selinux related man pages.

  • Ace load balancers might have an issue w/4096 byte ssl keys. Something to be aware of for MPI.

  • Stop using chcon; restorecon is the right approach as it verifies the mapping.

  • Custom selinux contexts stored in /etc/selinux/targeted/contexts/files/file_contexts.local

  • To remove a custom context:

    semanage fcontext -d ${dir}

  • Can hit I during boot to select services interactively.

  • <shift> up arrow will cycle back through previous boot messages on kvm console

  • Network IS available in rescue mode.

Overview:

Day 1: 04/21/14

Get used to typing this function:

show()
{   chain=${1:-FORWARD}
    chain=$(echo ${chain} | <tr> '[a-z]' '[A-Z]')
    echo ${chain} | \
        grep -qi -e ^prerouting -e ^postrouting && args="-t nat" || args=""
    c=1; iptables ${args} -L ${chain} -n | while read line
    do
        echo "${line}" | grep -qi -e ^accept -e ^reject -e masq -e ^dnl
        if [ $? -eq 0 ]
        then
            printf "%02d %s\n" ${c} "${line}"
            c=$((c+1))
        else
            printf "%2s %s\n" "." "${line}"
        fi
    done
}

Redone to take advantage of iptables cli args:

show()
{   [[ $# -gt 1 ]] && v="-v" || v=""
    chain=${1:-forward}
    chain=$(echo ${chain} | tr 'a-z' 'A-Z')
    if [ "${chain}" == "ALL" ]
    then
        iptables ${v} --line-numbers -L -n
        echo "";
        echo "#----------------------------------------------------"
        echo "";
        iptables ${v} -t nat --line-numbers -L -n
    else
        echo ${chain} | grep -i -e pre -e post && args="-t nat" || args=""
        iptables ${args} ${v} --line-numbers -L ${chain} -n
    fi
}

Intro:

  • Fairly routine intro to people, redhat, and support.

  • xfs is going to be the default filesystem for rhel7

  • fedora 18/19 should be roughly equivalent to rhel7

  • soreport

  • Classroom network:

    • internal private network

    • server: 192.168.0.X; example.com; my server = 192.168.0.107

    • instructor: 192.168.0.254; gateway; instructor.example.com

    • remote: 192.168.1.X; remote.test domain

    • My desktop: desktop7: 192.168.0.7 (br0 is the nic)

    • root/redhat

  • Internationalization:

    • shows interesting method of changing text based languageenvironment.

    • table of LANG vars on xviii

Unit 1: review:

  • Virtualization:

    • virsh destroy ${host}: force power off.

    • add console=ttyS0 to end of kernel line of a virtual to redirect console to the serial console. Then, virsh console will work.

    • “root” in the kernel message is the /boot partition. Not /root.

  • ldap user authentication: Beginning, use system-config-authentication

  • ssh set up. nothing surprising.

  • ISET: steps a new service:

    • I)nstall

    • S)tart

    • E)nable (make it persistent)

    • T)est

Unit 2: enhance user security:

  • sudo: started out examining sudo rule syntax, etc.

  • kerberos access via authconfig. During the test, use system-config-authentication

  • Troubleshooting sssd:

    • How does one config sssd? authentication commands; (authconfig and gui)

Unit 3: bash scripting and tools:

  • Interesting tidbit: set shows local vars; env shows exported vars.

  • Very basic overview of scripting in bash

  • Ouch; looks like the rest of today is going to be very dull info on unix commands. df, sort, cut, really??

    • sort -u is equivalent to sort | uniq; however, we can do other things w/uniq (-d, -u, etc)

    • tr command doesn’t require []. Interesting.

  • Remember not to expand on any of the scripts asked to write. Output exactly what is requested, nothing more, nothing less.

  • head/cut. Some differences I have been looking for:

    $ nl /usr/share/dict/british-english | head -5
     1  A
     2  A's
     3  AA's
     4  AB's
     5  ABM's
$ nl /usr/share/dict/british-english | tail -5
 99152  épée's
 99153  épées
 99154  étude
 99155  étude's
 99156  études
$ nl /usr/share/dict/british-english | head -n -90000 | tail -2
  9155  Mable's
  9156  Mac
$ nl /usr/share/dict/british-english | tail -n +90000 | head -3
 90000  tenderness's
 90001  tenders
 90002  tending

    Need to start using the -n option. head -n -## removes the last ## lines. tail -n +## removes the top ## lines.

Unit 4: gpg:

No surprises

Unit 5: package management:

  • yum plugins:

    • yum-plugin-verify: similar to rpm -V. Useful.

    • yum-plugin-versionlock: prevents packages from being upgraded by subsequent updates. Potentially useful for puppet to ensure it doesn’t get upgraded by accident.

      yum-verify example:

      # yum verify-rpm openssh-server
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security,
              : verify, versionlock
==================== Installed Packages ====================
openssh-server.x86_64 : An open source SSH server daemon
    File: /etc/ssh/sshd_config
    Tags: configuration
        Problem:  checksum does not match
        Current:  sha256:66794402c4759165af5d281e252e59cb87d3b8495926d536b7dcd51e5c2a87f7
        Original: sha256:489b4194c7430d0277f6d4f6806a4fc0258e5cecb46d7edfd5e0792dfd1f09bc
                                   --------
        Problem:  size does not match
        Current:  2.3 k
        Original: 3.8 k
                                   --------
        Problem:  mtime does not match
        Current:  Fri Dec 20 21:14:31 2013 (301 days, 23:22:43 later)
        Original: Thu Feb 21 21:51:48 2013
verify-rpm done
# yum verify openssh-server
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security,
              : verify, versionlock
verify done

      version lock:

      # yum versionlock sudo
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security,
              : verify, versionlock
Adding versionlock on: 0:sudo-1.8.6p3-7.el6
versionlock added: 1

# yum versionlock delete 0:sudo*
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security,
              : verify, versionlock
Deleting versionlock for: 0:sudo-1.8.6p3-7.el6.*
versionlock deleted: 1

  • Additional commands: Damn; I thought I was reasonably familiar w/rpm!

    • rpm2cpio: extracts files from rpm packages:

      rpm2cpio ${pkg} | cpio -id

    • rpm -qd: lists docs

    • rpm -qc: lists config files

    • rpm -q –scripts: lists %pre, %prerun and %postrun scripts.

    • rpm -q ${pkg} –changelog: displays the changelog.

    • yum localinstall ${pkg}

  • Package specification:

    • rpm package parts:

      • intro/preamble

      • build

      • scriptlets

      • manifests

      • change log

    • Preamble directives: as expected from the book.

      • Group should be listed in /usr/share/doc/rpm-*/GROUPS

      • BuildArch: noarch

    • Required spec file sections:

      • %prep

      • %build

      • %install

      • %clean

      • %files

      • %changelog

    • VERY interesting; class doesn’t run the rpmdevtools rpm. Another way to get a spec file template is to run vimv ${file}.spec:Q

  • Fuck me! rpm build not a part of the test anymore! outstanding!

  • Creating remotely accessible repos:

    • Create directory - ${dir} - publically accessible.

    • Create ${dir}/Packages

    • Copy (don’t move) packages into ${dir}/Packages

    • Copy (don’t move) GPG key used to sign to ${dir}

    • createrepo -v ${dir}

Unit 6: tcpdump/wireshark:

No surprises:

Unit 7: Advanced networking:

  • IP aliases

    • Book says not to use ifconfig but rather ip addr command.

    • /etc/sysconfig/network-scripts/ifcfg-eth0:0 same entries except the ONBOOT should be ONPARENT

    • Old style alias:

      *   ``ip addr add ${ip}/${cidr} dev ${nic} label ${nic}:#``
*   ``ip addr delete ${ip}/${cidr} dev ${nic}``
*   Create /etc/sysconfig/network-scripts/ifcfg-${nic}:#

    • New style alias:

      *   ``ip addr add ${ip}/${cidr} dev ${nic}`` # no label
*   ``ip addr delete ${ip}/${cidr} dev ${nic}``
*   Edit /etc/sysconfig/network-scripts/ifcfg-${int}
*   Add *IPADDR#=${ip}* and *PREFIX#=${cidr}*
*   For multiple aliases, ensure '#' match

  • NIC Bonding

    • Create /etc/sysconfig/network-scripts/ifcfg-bond#

      DEVICE=bond0
IPADDR=${ip}
PREFIX=${cidr}
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="mode=[1-3] ..."

    • Update /etc/sysconfig/network-scripts/ifconfig-${nic}

      • Remove config info

      • Add MASTER=bond0 and SLAVE=yes

    • Add alias bond0 bonding to /etc/modprobe.d/bonding.conf

  • Static routes:

    • Enable kernel routing: sysctl -w net.ipv4.ip_forward = 1 and update /etc/sysctl.conf

    • ip route add ${network}/${cidr} via ${gateway}

    • Update /etc/sysconfig/network-scripts/route-${nic}:

      ADDRESS#=${network}
NETMASK#=${netmask}
GATEWAY#=${gateway}

Run through the lab at the end of unit 7 a time or two before the test.

Unit 8: Secure network traffic:

  • ssh port forwarding:

    • ssh -L ${local}:${remotehost}:${remote_port} ${sshhost}

      ssh -L 2025:mailhost:25 sshhost

    • Other args:

      • ${local} = the port on the ssh client system.

      • ${remotehost} = how ${sshhost} will resolve the name. Use localhost if you want ${remoteport} used on ${sshhost}.

      • ${sshhost} = the target system or gateway.

      • -N: Don’t execute remote commands

      • -f: run in background

    • I’ve used this for local traffic, where sshost is also remotehost. Could be useful. Need to remember this.

    • To set up a proxy web env:

      • ssh -D 8080 ${user}@${host}

      • Configure browser to use 127.0.0.1:8080 as the socks proxy. I can set up one of these for work so I can browse drudge and what not via my firewall.

  • iptables:

    • prerouting chain does destination nat’ing - ie traffic redirection.

    • postrouting chain does the normal nat.

    • Tables: a set of chains used for a specific purpose:

      • filter: the default: input, forward, output

      • nat: prerouting, postrouting, output

    • iptables -nL --line-numbers - displays line numbers. Don’t need the show function anymore… Update the show() function as follows:

      show()
{   [[ $# -gt 1 ]] && v="-v" || v=""
    chain=${1:-forward}
    chain=$(echo ${chain} | tr 'a-z' 'A-Z')
    if [ "${chain}" == "ALL" ]
    then
        iptables ${v} --line-numbers -L -n
        echo "";
        echo "#----------------------------------------------------"
        echo "";
        iptables ${v} -t nat --line-numbers -L -n
    else
        echo ${chain} | grep -i -e pre -e post && args="-t nat" || args=""
        iptables ${args} ${v} --line-numbers -L ${chain} -n
    fi
}

    • Use -v arg to display number of packets and bytes: (destination column snipped)

      # iptables -vL INPUT -n
Chain INPUT (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source
1      12M 1868M ACCEPT     all  --  eth0   *       192.168.12.0/24 [[snip]]
2     2352  643K ACCEPT     all  --  lo     *       0.0.0.0/0
3      255  191K ACCEPT     all  --  tun+   *       0.0.0.0/0
4     343K  113M REJECT     udp  --  eth1   *       96.120.26.109

    • ftp: why do I have to go through system-config-firewall-tui when adding ftp rather than CLI? because the ftp module is getting added to the /etc/sysconfig/iptables-config - not to /etc/sysconfig/iptables. The line in question is:

      IPTABLES_MODULES="nf_conntrack_ftp"

  • NAT:

    • Use DNAT to redirect web traffic, for instance. As in, I want my web traffic to go to mgmt. That’s done via the rerouting table:

      iptables -t nat -A PREROUTING -p tcp --dport 80 - j DNAT --to-destination ${ip}

    • I fucked up the Lab because I didn’t flush the nat chain. If multiple rewrites of the firewall rules, tends to make the chain fucked up.

Unit 9: NTP:

no surprises. Ensure the restrict line is accurate - mask does not equal netmask

Unit 10: system monitoring:

Looks like we’re going to go through AIDE

  • AIDE

    • AIDE tracks changes to system files and permissions only. No other HIDS functionality present. While good, there are better open source HIDS tools available.

    • Process:

      • aide --iniit:

        • Creates a database which should be stored offline somewhere.

        • Run the init when the system is in a known good state.

      • Store aide.conf and database in a secure location.

      • mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

      • aide --check: Runs through everything comparing against the stored values.

    • ossec is still much better.

  • tmpwatch and logrotate:

    • tmpwatch: /etc/cron.daily/tmpwatch

    • logrotate:

Unit 11: Centralized and secure storage:

  • ISCSI:

    • Terms:

      • Initiator: the end client

      • Target: the storage

      • Portal: the server holding the storage

      • Qualified name: unique name for the initiator and target. Similar to DNS.

    • Intialization process:

      • Install the s/w

      • Update /etc/iscsi/initiatorname.iscsi w/a unique name. This is the local name for the system.

      • Discover potential targets:

        iscsiadm -m discovery -t st -p ${ip}

      • Log in to the portal:

        iscsiadm -m node -T ${remote_name} -p ${ip} -l

      • ID the new device:

        • sda,

        • tail /var/log/messages, or

        • tail /proc/partitions

        • ll /dev/disk/by-ath

        • service iscsi status: seems like the most reliable.

        • Partition the new disk, as needed.

        • If drive is active, apparently, you have to reboot.

    • Use _netdev as the mount option to tell it not to mount until the system is up.

      UID=${UID} ${mp} ext4 _netdev,acl 1 2

    • To remove:

      • Unmount/update /etc/fstab

      • Log out of the portal:

        iscsiadm -m node -T ${id} -p ${portal} -u

      • Delete the local record:

        iscsiadm -m node -T ${id} -p ${portal} -delete

  • luks: no surprises.

Unit 12: SSL web service:

  • No real surprises. Method of adding specific CA to firefox is nice. basically, using genkey to do everything.

Unit 13: Web server; additional configuration:

  • Virtual host key words:

    • VirtualHost

    • NameVirtualHost

    • ServerName/ServerAlias

    • ServerAdmin

    • DocumentRoot

  • Config process (from scratch):

    • Uncomment NameVirtualhost *:80

    • Copy/Uncomment sample

    • Update information therein.

    • Update selinux info as needed.

      semanage fcontext -a -s system_u -t httpd_sys_content_t \
'${dir}(/.*)?'
restorecon -vFR ${dir}

      That semanage regex is important otherwise newly added info doesn’t get the updated context. Might be able to get around that by running semanage fcontext twice; once w/dir and once w/ ${dir}/*

      The regex is listed in the semanage man page.

    • If there’s one virtual, everything must be virtual. Multiple blocks means that the first one is the default.

  • CGI:

    • Create dirs, update selinux contexts.

    • Add scriptalias to virtual host: scriptalias /cgi-bin/ /www7/cgi-bin/

    • Apparently, a directory block for the cgi-bin isn’t required.

  • User Authentcation:

    • Flat file:

      • Create private dirs, ensuring selinux labels.

      • Add directory stanza to the appropriate virtual:

        <directory ${dir>
    authname "${name}"
    authtype basic
    authuserfile ${pwd_file_created_above}
    require valid-user
</directory>

    • ldap:

      • Same as above,

      • Updated directory structure:

        <directory ${dir>
    authname "${name}"
    authtype basic
    authbasicprovider ldap
    authldapurl "ldap://${url}/${search_dn}" TLS
    require valid-user
</directory>

      • Add LDAPTrustedGlobalCert line outside of the virtual host block:

        LDAPTrustedGlobalCert CA_BASE64 ${ldap_cert_loc}

  • Selinux issues:

    • Adding a non-standard port, use semanage:

      semanage port -a -t http_port_t -p tcp ${port}

    • Verify booleans (getsebool/setsebool). Don’t forget to make permanent w/-P option.

    • semanage boolean -l: displays status of booleans, their status, and their persistent status.

    • system-config-selinux

Unit 14: SMTP:

Test objectives have apparently changed. They simply say configure an MTA so the whole sendmail thing is completely overcome by events. Training manual doesn’t even go into it at all.

User authentication isn’t touched upon either. Nice.

  • Docs: BASIC_CONFIG* and STANDARD_CONFIG* are useful for identifying and configuring postfix in a standard method.

  • To completely disable local mail delivery is to update the local_transport parameter thusly:

    mydestination = ${default}
local_transport = error:local delivery disabled

  • Lab 14 answers:

    • Criteria:

      • Server:

        • Inbound mail server, accepting incoming mail for local delivery to addresses server7 and domain7.example.com

        • Relay only local interfaces

        • masquerading as domain7.example.com

        • Relay to desktop7

        • User elvis should be able to receive email and receive email sent to theking

      • Desktop:

        • Outbound mail relay accepting mail to realy from all hosts on 192.168.0.0/8 and localhost.

        • Should not deliver mail locally.

        • masquerading as domain7.example.com

    • Desktop:

      • myhostname

      • mydomain

      • myorigin = domain7.example.com

      • inet_interfaces = all

      • local_transport = error:local delivery refused.

      • mynetworks = 127.0.0.0/8, 192.168.0.0/8

    • server:

      • myhostname

      • mydomain

      • myorigin = domain7.example.com

      • inet_interfaces = al

      • mydestination = domain7.example.com, …

      • mynetworks = 127.0.0.0/8

      • relayhost = [desktop7.example.com]

      • Update aliases/ run newaliases.

      • Create user elvis

Don’t forget the inet_interfaces, dummy!

Unit 15: DNS:

No real surprises. If the name caching/forwarding isn’t working, turn off dnssec and reenable recursive queries.

Unit 16: NFS:

No surprises

Unit 17: CIFS:

  • Persistently mounting a cifs filesystem in /etc/fstab, use credentials arg:

    //${server}/${share}  ${mp}  cifs credentials=${abs_file} 0 0

    where ${abs_file} is absolute path to a file with the format:

    user=${user}
password=${clear_text_password}

  • Manually mount:

    mount -t cifs -o credentials=${abs_file} //${svr}/${share} /${mp}

  • samba-doc provides html docs of Samba by Example; install that if getting stuck.

  • Group collaboration directories:

    • Create dir:

      • mkdir

      • chgrp

      • chmod -R 2770

    • selinux: (see smbd_selinux man page, right at the top)

      • semanage fcontext … public_content(_rw)_t

      • setsebool -P allow_samba_anon_write 1 # needed if not using samba_share_t

      • restorecon -vFR

    • smb.conf stanza:

      [collaborators]
    comment     = ${name}
    path        = ${path}
    browseable  = no
    public      = no
    writable    = yes
    valid users = ${user_list} or +group
    hosts allow = ${list-o-hosts}

  • printers:

    • Turn off global via setting the load printers to no or comment the stanza

    • To share an individual printer:

      [printer_name]
    comment         = ${printer}
    path            = /var/spool/samba
    browseable      = no
    printable       = yes
    printer name    = ${cups_printer_name}

Unit 18: FTP:

Anonymous drop box:

  • Create dir, chmod 730 ${dir}

  • setsebool -P allow_ftpd_anon_write 1

  • vsftpd.conf:

    • local_enable = no

    • anon_upload_enable = yes

    • chown_uploads=yes; chown_username=daemon

  • Update firewall:

    • Update input rules

    • find /lib/modules -name \*_ftp\* ID the relevant modules:

      nf_conntrack_ftp
nf_nat_ftp

    • Update /etc/sysconfig/iptables-config with those modules and restart iptables

Unit 19: Troubleshooting the boot process:

  • Upstart crap under /etc/init now; all the stuff that used to be in the /etc/inittab.

  • Can hit I during boot to select services interactively.

  • Rescue mode:

    • Can add linux rescue to end of network boot kernel.

    • Go through questions until it tries to find the linux env.

    • If found, chroot /mnt/sysimage to see what your system looks.