========================= RHCE Class (rh255) notes: ========================= Things to do: ============= * Verify my test on Friday. * Read/stufy ip command (man ip) * Review iscsi and potential for SAAS. * Experiment w/NIC bonding in vm lab. Lessons Learned: ================ * ``ethtool -p ${int} ${seconds}`` will blink the network card for the seconds identified. Useful for identifying nics. * Flush firewall rules before trying to recreate them. Otherwise, things just get added instead of replaced. * ``man -k _selinux``: displays list of selinux related man pages. * Ace load balancers might have an issue w/4096 byte ssl keys. Something to be aware of for MPI. * Stop using chcon; restorecon is the right approach as it verifies the mapping. * Custom selinux contexts stored in /etc/selinux/targeted/contexts/files/file_contexts.local * To remove a custom context: :: semanage fcontext -d ${dir} * Can hit *I* during boot to select services interactively. * up arrow will cycle back through previous boot messages on kvm console * Network **IS** available in rescue mode. Overview: ========= Day 1: 04/21/14 * Instructor info * Dane Copeland * dcopelan@redhat.com Get used to typing this function: :: show() { chain=${1:-FORWARD} chain=$(echo ${chain} | '[a-z]' '[A-Z]') echo ${chain} | \ grep -qi -e ^prerouting -e ^postrouting && args="-t nat" || args="" c=1; iptables ${args} -L ${chain} -n | while read line do echo "${line}" | grep -qi -e ^accept -e ^reject -e masq -e ^dnl if [ $? -eq 0 ] then printf "%02d %s\n" ${c} "${line}" c=$((c+1)) else printf "%2s %s\n" "." "${line}" fi done } Redone to take advantage of iptables cli args: :: show() { [[ $# -gt 1 ]] && v="-v" || v="" chain=${1:-forward} chain=$(echo ${chain} | tr 'a-z' 'A-Z') if [ "${chain}" == "ALL" ] then iptables ${v} --line-numbers -L -n echo ""; echo "#----------------------------------------------------" echo ""; iptables ${v} -t nat --line-numbers -L -n else echo ${chain} | grep -i -e pre -e post && args="-t nat" || args="" iptables ${args} ${v} --line-numbers -L ${chain} -n fi } Intro: ====== * Fairly routine intro to people, redhat, and support. * xfs is going to be the default filesystem for rhel7 * fedora 18/19 should be roughly equivalent to rhel7 * soreport * Classroom network: * internal private network * server: 192.168.0.X; example.com; my server = 192.168.0.107 * instructor: 192.168.0.254; gateway; instructor.example.com * remote: 192.168.1.X; remote.test domain * My desktop: desktop7: 192.168.0.7 (br0 is the nic) * root/redhat * Internationalization: * shows interesting method of changing text based languageenvironment. * table of LANG vars on xviii Unit 1: review: =============== * Virtualization: * ``virsh destroy ${host}``: force power off. * add ``console=ttyS0`` to end of kernel line of a virtual to redirect console to the serial console. Then, ``virsh console`` will work. * "root" in the kernel message is the /boot partition. Not /root. * ldap user authentication: Beginning, use ``system-config-authentication`` * ldap info * ldap url: ldap://instructor.example.com; don't need to add ldap:// or ldaps:// * base: dc=example,dc=com * tls cert: ftp://instructor.example.com/pub/example-ca-crt * rhel5 will show ldap entries w/``getnent passwd`` info; rhel6 won't. apparently, by default. * Set up ldap and automounted ldap home dirs for the server environment. * ssh set up. nothing surprising. * ISET: steps a new service: * I)nstall * S)tart * E)nable (make it persistent) * T)est Unit 2: enhance user security: ============================== * sudo: started out examining sudo rule syntax, etc. * kerberos access via authconfig. During the test, use ``system-config-authentication`` * Troubleshooting sssd: * How does one config sssd? authentication commands; (authconfig and gui) Unit 3: bash scripting and tools: ================================= * Interesting tidbit: ``set`` shows local vars; ``env`` shows exported vars. * Very basic overview of scripting in bash * Ouch; looks like the rest of today is going to be very dull info on unix commands. df, sort, cut, really?? * ``sort -u`` is equivalent to ``sort | uniq``; however, we can do other things w/uniq (-d, -u, etc) * tr command doesn't require []. Interesting. * Remember not to expand on any of the scripts asked to write. Output exactly what is requested, nothing more, nothing less. * head/cut. Some differences I have been looking for: :: $ nl /usr/share/dict/british-english | head -5 1 A 2 A's 3 AA's 4 AB's 5 ABM's $ nl /usr/share/dict/british-english | tail -5 99152 épée's 99153 épées 99154 étude 99155 étude's 99156 études $ nl /usr/share/dict/british-english | head -n -90000 | tail -2 9155 Mable's 9156 Mac $ nl /usr/share/dict/british-english | tail -n +90000 | head -3 90000 tenderness's 90001 tenders 90002 tending Need to start using the *-n* option. ``head -n -##`` removes the last ## lines. tail -n +## removes the top ## lines. Unit 4: gpg: ============ No surprises Unit 5: package management: =========================== * yum plugins: * yum-plugin-verify: similar to rpm -V. Useful. * yum-plugin-versionlock: prevents packages from being upgraded by subsequent updates. Potentially useful for puppet to ensure it doesn't get upgraded by accident. yum-verify example: :: # yum verify-rpm openssh-server Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security, : verify, versionlock ==================== Installed Packages ==================== openssh-server.x86_64 : An open source SSH server daemon File: /etc/ssh/sshd_config Tags: configuration Problem: checksum does not match Current: sha256:66794402c4759165af5d281e252e59cb87d3b8495926d536b7dcd51e5c2a87f7 Original: sha256:489b4194c7430d0277f6d4f6806a4fc0258e5cecb46d7edfd5e0792dfd1f09bc -------- Problem: size does not match Current: 2.3 k Original: 3.8 k -------- Problem: mtime does not match Current: Fri Dec 20 21:14:31 2013 (301 days, 23:22:43 later) Original: Thu Feb 21 21:51:48 2013 verify-rpm done # yum verify openssh-server Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security, : verify, versionlock verify done version lock: :: # yum versionlock sudo Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security, : verify, versionlock Adding versionlock on: 0:sudo-1.8.6p3-7.el6 versionlock added: 1 # yum versionlock delete 0:sudo* Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security, : verify, versionlock Deleting versionlock for: 0:sudo-1.8.6p3-7.el6.* versionlock deleted: 1 * Additional commands: Damn; I thought I was reasonably familiar w/rpm! * rpm2cpio: extracts files from rpm packages: :: rpm2cpio ${pkg} | cpio -id * rpm -qd: lists docs * rpm -qc: lists config files * rpm -q --scripts: lists %pre, %prerun and %postrun scripts. * rpm -q ${pkg} --changelog: displays the changelog. * yum localinstall ${pkg} * Package specification: * rpm package parts: * intro/preamble * build * scriptlets * manifests * change log * Preamble directives: as expected from the book. * Group should be listed in /usr/share/doc/rpm-\*/GROUPS * BuildArch: noarch * Required spec file sections: * %prep * %build * %install * %clean * %files * %changelog * **VERY** interesting; class doesn't run the rpmdevtools rpm. Another way to get a spec file template is to run vimv ${file}.spec:Q * Fuck me! rpm build not a part of the test anymore! outstanding! * Creating remotely accessible repos: * Create directory - ${dir} - publically accessible. * Create ${dir}/Packages * Copy (don't move) packages into ${dir}/Packages * Copy (don't move) GPG key used to sign to ${dir} * ``createrepo -v ${dir}`` Unit 6: tcpdump/wireshark: ========================== No surprises: Unit 7: Advanced networking: ============================ * IP aliases * Book says not to use ifconfig but rather ``ip addr`` command. * /etc/sysconfig/network-scripts/ifcfg-eth0:0 same entries except the ONBOOT should be ONPARENT * Old style alias: :: * ``ip addr add ${ip}/${cidr} dev ${nic} label ${nic}:#`` * ``ip addr delete ${ip}/${cidr} dev ${nic}`` * Create /etc/sysconfig/network-scripts/ifcfg-${nic}:# * New style alias: :: * ``ip addr add ${ip}/${cidr} dev ${nic}`` # no label * ``ip addr delete ${ip}/${cidr} dev ${nic}`` * Edit /etc/sysconfig/network-scripts/ifcfg-${int} * Add *IPADDR#=${ip}* and *PREFIX#=${cidr}* * For multiple aliases, ensure '#' match * NIC Bonding * Create /etc/sysconfig/network-scripts/ifcfg-bond# :: DEVICE=bond0 IPADDR=${ip} PREFIX=${cidr} ONBOOT=yes BOOTPROTO=none USERCTL=no BONDING_OPTS="mode=[1-3] ..." * Update /etc/sysconfig/network-scripts/ifconfig-${nic} * Remove config info * Add ``MASTER=bond0`` and ``SLAVE=yes`` * Add ``alias bond0 bonding`` to /etc/modprobe.d/bonding.conf * Static routes: * Enable kernel routing: ``sysctl -w net.ipv4.ip_forward = 1`` and update /etc/sysctl.conf * ``ip route add ${network}/${cidr} via ${gateway}`` * Update /etc/sysconfig/network-scripts/route-${nic}: :: ADDRESS#=${network} NETMASK#=${netmask} GATEWAY#=${gateway} Run through the lab at the end of unit 7 a time or two before the test. Unit 8: Secure network traffic: =============================== * ssh port forwarding: * ssh -L ${local}:${remotehost}:${remote_port} ${sshhost} :: ssh -L 2025:mailhost:25 sshhost * Other args: * ${local} = the port on the ssh client system. * ${remotehost} = how ${sshhost} will resolve the name. Use localhost if you want ${remoteport} used on ${sshhost}. * ${sshhost} = the target system or gateway. * -N: Don't execute remote commands * -f: run in background * I've used this for local traffic, where sshost is also remotehost. Could be useful. Need to remember this. * To set up a proxy web env: * ``ssh -D 8080 ${user}@${host}`` * Configure browser to use 127.0.0.1:8080 as the socks proxy. I can set up one of these for work so I can browse drudge and what not via my firewall. * iptables: * prerouting chain does destination nat'ing - ie traffic redirection. * postrouting chain does the normal nat. * Tables: a set of chains used for a specific purpose: * filter: the default: input, forward, output * nat: prerouting, postrouting, output * * ``iptables -nL --line-numbers`` - displays line numbers. Don't need the show function anymore... Update the show() function as follows: :: show() { [[ $# -gt 1 ]] && v="-v" || v="" chain=${1:-forward} chain=$(echo ${chain} | tr 'a-z' 'A-Z') if [ "${chain}" == "ALL" ] then iptables ${v} --line-numbers -L -n echo ""; echo "#----------------------------------------------------" echo ""; iptables ${v} -t nat --line-numbers -L -n else echo ${chain} | grep -i -e pre -e post && args="-t nat" || args="" iptables ${args} ${v} --line-numbers -L ${chain} -n fi } * Use *-v* arg to display number of packets and bytes: (destination column snipped) :: # iptables -vL INPUT -n Chain INPUT (policy DROP 0 packets, 0 bytes) num pkts bytes target prot opt in out source 1 12M 1868M ACCEPT all -- eth0 * 192.168.12.0/24 [[snip]] 2 2352 643K ACCEPT all -- lo * 0.0.0.0/0 3 255 191K ACCEPT all -- tun+ * 0.0.0.0/0 4 343K 113M REJECT udp -- eth1 * 96.120.26.109 * ftp: why do I have to go through ``system-config-firewall-tui`` when adding ftp rather than CLI? because the ftp module is getting added to the /etc/sysconfig/iptables-config - not to /etc/sysconfig/iptables. The line in question is: :: IPTABLES_MODULES="nf_conntrack_ftp" * NAT: * Use DNAT to redirect web traffic, for instance. As in, I want my web traffic to go to mgmt. That's done via the rerouting table: :: iptables -t nat -A PREROUTING -p tcp --dport 80 - j DNAT --to-destination ${ip} * I fucked up the Lab because I didn't flush the nat chain. If multiple rewrites of the firewall rules, tends to make the chain fucked up. Unit 9: NTP: ============ no surprises. Ensure the restrict line is accurate - mask does not equal netmask Unit 10: system monitoring: =========================== Looks like we're going to go through AIDE * AIDE * AIDE tracks changes to system files and permissions only. No other HIDS functionality present. While good, there are better open source HIDS tools available. * Process: * ``aide --iniit``: * Creates a database which should be stored offline somewhere. * Run the init when the system is in a known good state. * Store aide.conf and database in a secure location. * mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz * ``aide --check``: Runs through everything comparing against the stored values. * ossec is still much better. * tmpwatch and logrotate: * tmpwatch: /etc/cron.daily/tmpwatch * logrotate: Unit 11: Centralized and secure storage: ======================================== * ISCSI: * Terms: * Initiator: the end client * Target: the storage * Portal: the server holding the storage * Qualified name: unique name for the initiator and target. Similar to DNS. * Intialization process: * Install the s/w * Update /etc/iscsi/initiatorname.iscsi w/a unique name. This is the local name for the system. * Discover potential targets: :: iscsiadm -m discovery -t st -p ${ip} * Log in to the portal: :: iscsiadm -m node -T ${remote_name} -p ${ip} -l * ID the new device: * ``sda``, * ``tail /var/log/messages``, or * ``tail /proc/partitions`` * ``ll /dev/disk/by-ath`` * ``service iscsi status``: seems like the most reliable. * Partition the new disk, as needed. * If drive is active, apparently, you have to reboot. * Use _netdev as the mount option to tell it not to mount until the system is up. :: UID=${UID} ${mp} ext4 _netdev,acl 1 2 * To remove: * Unmount/update /etc/fstab * Log out of the portal: :: iscsiadm -m node -T ${id} -p ${portal} -u * Delete the local record: :: iscsiadm -m node -T ${id} -p ${portal} -delete * luks: no surprises. Unit 12: SSL web service: ========================= * No real surprises. Method of adding specific CA to firefox is nice. basically, using genkey to do everything. Unit 13: Web server; additional configuration: ============================================== * Virtual host key words: * VirtualHost * NameVirtualHost * ServerName/ServerAlias * ServerAdmin * DocumentRoot * Config process (from scratch): * Uncomment NameVirtualhost \*:80 * Copy/Uncomment sample * Update information therein. * Update selinux info as needed. :: semanage fcontext -a -s system_u -t httpd_sys_content_t \ '${dir}(/.*)?' restorecon -vFR ${dir} That semanage regex is important otherwise newly added info doesn't get the updated context. Might be able to get around that by running semanage fcontext twice; once w/dir and once w/ ${dir}/* The regex is listed in the semanage man page. * If there's one virtual, everything must be virtual. Multiple blocks means that the first one is the default. * CGI: * Create dirs, update selinux contexts. * Add scriptalias to virtual host: ``scriptalias /cgi-bin/ /www7/cgi-bin/`` * Apparently, a directory block for the cgi-bin isn't required. * User Authentcation: * Flat file: * Create private dirs, ensuring selinux labels. * Add directory stanza to the appropriate virtual: :: authname "${name}" authtype basic authuserfile ${pwd_file_created_above} require valid-user * ldap: * Same as above, * Updated directory structure: :: authname "${name}" authtype basic authbasicprovider ldap authldapurl "ldap://${url}/${search_dn}" TLS require valid-user * Add LDAPTrustedGlobalCert line outside of the virtual host block: :: LDAPTrustedGlobalCert CA_BASE64 ${ldap_cert_loc} * Selinux issues: * Adding a non-standard port, use semanage: :: semanage port -a -t http_port_t -p tcp ${port} * Verify booleans (getsebool/setsebool). Don't forget to make permanent w/-P option. * ``semanage boolean -l``: displays status of booleans, their status, and their persistent status. * ``system-config-selinux`` Unit 14: SMTP: ============== Test objectives have apparently changed. They simply say *configure an MTA* so the whole sendmail thing is completely overcome by events. Training manual doesn't even go into it at all. User authentication isn't touched upon either. Nice. * Docs: BASIC_CONFIG* and STANDARD_CONFIG* are useful for identifying and configuring postfix in a standard method. * To completely disable local mail delivery is to update the local_transport parameter thusly: :: mydestination = ${default} local_transport = error:local delivery disabled * Lab 14 answers: * Criteria: * Server: * Inbound mail server, accepting incoming mail for local delivery to addresses server7 and domain7.example.com * Relay only local interfaces * masquerading as domain7.example.com * Relay to desktop7 * User elvis should be able to receive email and receive email sent to theking * Desktop: * Outbound mail relay accepting mail to realy from all hosts on 192.168.0.0/8 and localhost. * Should not deliver mail locally. * masquerading as domain7.example.com * Desktop: * myhostname * mydomain * myorigin = domain7.example.com * inet_interfaces = all * local_transport = error:local delivery refused. * mynetworks = 127.0.0.0/8, 192.168.0.0/8 * server: * myhostname * mydomain * myorigin = domain7.example.com * inet_interfaces = al * mydestination = domain7.example.com, ... * mynetworks = 127.0.0.0/8 * relayhost = [desktop7.example.com] * Update aliases/ run newaliases. * Create user elvis Don't forget the inet_interfaces, dummy! Unit 15: DNS: ============= No real surprises. If the name caching/forwarding isn't working, turn off dnssec and reenable recursive queries. Unit 16: NFS: ============= No surprises Unit 17: CIFS: ============== * Persistently mounting a cifs filesystem in /etc/fstab, use credentials arg: :: //${server}/${share} ${mp} cifs credentials=${abs_file} 0 0 where ${abs_file} is absolute path to a file with the format: :: user=${user} password=${clear_text_password} * Manually mount: :: mount -t cifs -o credentials=${abs_file} //${svr}/${share} /${mp} * samba-doc provides html docs of *Samba by Example*; install that if getting stuck. * Group collaboration directories: * Create dir: * mkdir * chgrp * chmod -R 2770 * selinux: (see smbd_selinux man page, right at the top) * semanage fcontext ... public_content(_rw)_t * setsebool -P allow_samba_anon_write 1 # needed if not using samba_share_t * restorecon -vFR * smb.conf stanza: :: [collaborators] comment = ${name} path = ${path} browseable = no public = no writable = yes valid users = ${user_list} or +group hosts allow = ${list-o-hosts} * printers: * Turn off global via setting the load printers to no or comment the stanza * To share an individual printer: :: [printer_name] comment = ${printer} path = /var/spool/samba browseable = no printable = yes printer name = ${cups_printer_name} Unit 18: FTP: ============= Anonymous drop box: * Create dir, chmod 730 ${dir} * ``setsebool -P allow_ftpd_anon_write 1`` * vsftpd.conf: * local_enable = no * anon_upload_enable = yes * chown_uploads=yes; chown_username=daemon * Update firewall: * Update input rules * ``find /lib/modules -name \*_ftp\*`` ID the relevant modules: :: nf_conntrack_ftp nf_nat_ftp * Update /etc/sysconfig/iptables-config with those modules and restart iptables Unit 19: Troubleshooting the boot process: ========================================== * Upstart crap under /etc/init now; all the stuff that used to be in the /etc/inittab. * Can hit *I* during boot to select services interactively. * Rescue mode: * Can add ``linux rescue`` to end of network boot kernel. * Go through questions until it tries to find the linux env. * If found, ``chroot /mnt/sysimage`` to see what your system looks.