centos – put things down

MongoDB 3.0: mongos CentOS 6 service scripts

Premise

When installing mongos (not mongod) on CentOS 6 using the RPMs provided in MongoDB’s repository, no startup/service/init.d scripts are created, or really anything for that matter, to aid in managing mongos as a service. All the RPM provides is the binary.

If you want to treat mongos as a service, there are quite a few steps to follow beyond just setting up a script in init.d:

mongos YAML configuration file
mongos sysconfig service overrides
SELinux port definition
a user to run mongos

Most of the time there are many mongos processes distributed within a MongoDB cluster. Configuring each of these can be a hassle, let alone installing and managing them, unless you use an automation tool.

Be sure to read “Install MongoDB on Red Hat Enterprise or CentOS Linux” first.

mongos Service Scripts

Below are the 3 files I use when preparing a mongos cluster:

The init.d service script for managing the mongos process
A base mongos configuration file in YAML
An automation script to prepare the CentOS 6 environment for mongos

/etc/init.d/mongos

The following init.d script has been retooled from the stock MongoDB 3.0 mongod for mongos. Watch out for the subtle differences.

#!/bin/bash

# mongos - Startup script for mongos

# chkconfig: 35 85 15
# description: Mongo is a scalable, document-oriented database.
# processname: mongos
# config: /etc/mongos.conf
# pidfile: /var/run/mongodb/mongos.pid

. /etc/rc.d/init.d/functions

# things from mongos.conf get there by mongos reading it


# NOTE: if you change any OPTIONS here, you get what you pay for:
# this script assumes all options are in the config file.
CONFIGFILE="/etc/mongos.conf"
OPTIONS=" -f $CONFIGFILE"
SYSCONFIG="/etc/sysconfig/mongos"
LOCKFILE="/var/lock/subsys/mongos"

PIDFILEPATH=`awk -F'[:=]' -v IGNORECASE=1 '/^[[:blank:]]*(processManagement\.)?pidfilepath[[:blank:]]*[:=][[:blank:]]*/{print $2}' "$CONFIGFILE" | tr -d "[:blank:]\"'"`

mongos=${MONGOS-/usr/bin/mongos}

MONGO_USER=mongod
MONGO_GROUP=mongod

if [ -f "$SYSCONFIG" ]; then
    . "$SYSCONFIG"
fi

PIDDIR=`dirname $PIDFILEPATH`

# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
    NUMACTL="numactl $NUMACTL_ARGS"
else
    NUMACTL=""
fi

start()
{
  # Make sure the default pidfile directory exists
  if [ ! -d $PIDDIR ]; then
    install -d -m 0755 -o $MONGO_USER -g $MONGO_GROUP $PIDDIR
  fi

  # Recommended ulimit values for mongod or mongos
  # See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
  #
  ulimit -f unlimited
  ulimit -t unlimited
  ulimit -v unlimited
  ulimit -n 64000
  ulimit -m unlimited
  ulimit -u 32000

  echo -n $"Starting mongos: "
  daemon --user "$MONGO_USER" --check $mongos "$NUMACTL $mongos $OPTIONS >/dev/null 2>&1"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch "$LOCKFILE"
}

stop()
{
  echo -n $"Stopping mongos: "
  mongo_killproc "$PIDFILEPATH" $mongos
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f "$LOCKFILE"
}

restart () {
	stop
	start
}

# Send TERM signal to process and wait up to 300 seconds for process to go away.
# If process is still alive after 300 seconds, send KILL signal.
# Built-in killproc() (found in /etc/init.d/functions) is on certain versions of Linux
# where it sleeps for the full $delay seconds if process does not respond fast enough to
# the initial TERM signal.
mongo_killproc()
{
  local pid_file=$1
  local procname=$2
  local -i delay=300
  local -i duration=10
  local pid=`pidofproc -p "${pid_file}" ${procname}`

  kill -TERM $pid >/dev/null 2>&1
  usleep 100000
  local -i x=0
  while [ $x -le $delay ] && checkpid $pid; do
    sleep $duration
    x=$(( $x + $duration))
  done

  kill -KILL $pid >/dev/null 2>&1
  usleep 100000

  rm -f "${pid_file}"

  checkpid $pid
  local RC=$?
  [ "$RC" -eq 0 ] && failure "${procname} shutdown" || success "${procname} shutdown"
  RC=$((! $RC))
  return $RC
}

RETVAL=0

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f "$LOCKFILE" ] && restart || :
    ;;
  status)
    status $mongos
    RETVAL=$?
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    RETVAL=1
esac

exit $RETVAL

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

#!/bin/bash

# mongos - Startup script for mongos

# chkconfig: 35 85 15

# description: Mongo is a scalable, document-oriented database.

# processname: mongos

# config: /etc/mongos.conf

# pidfile: /var/run/mongodb/mongos.pid

. /etc/rc.d/init.d/functions

# things from mongos.conf get there by mongos reading it

# NOTE: if you change any OPTIONS here, you get what you pay for:

# this script assumes all options are in the config file.

CONFIGFILE="/etc/mongos.conf"

OPTIONS=" -f $CONFIGFILE"

SYSCONFIG="/etc/sysconfig/mongos"

LOCKFILE="/var/lock/subsys/mongos"

PIDFILEPATH=`awk -F'[:=]' -v IGNORECASE=1 '/^[[:blank:]]*(processManagement\.)?pidfilepath[[:blank:]]*[:=][[:blank:]]*/{print $2}' "$CONFIGFILE" | tr -d "[:blank:]\"'"`

mongos=${MONGOS-/usr/bin/mongos}

MONGO_USER=mongod

MONGO_GROUP=mongod

if [ -f "$SYSCONFIG" ]; then

. "$SYSCONFIG"

PIDDIR=`dirname $PIDFILEPATH`

# Handle NUMA access to CPUs (SERVER-3574)

# This verifies the existence of numactl as well as testing that the command works

NUMACTL_ARGS="--interleave=all"

if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null

then

NUMACTL="numactl $NUMACTL_ARGS"

else

NUMACTL=""

start()

{

# Make sure the default pidfile directory exists

if [ ! -d $PIDDIR ]; then

install -d -m 0755 -o $MONGO_USER -g $MONGO_GROUP $PIDDIR

# Recommended ulimit values for mongod or mongos

# See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

ulimit -f unlimited

ulimit -t unlimited

ulimit -v unlimited

ulimit -n 64000

ulimit -m unlimited

ulimit -u 32000

echo -n $"Starting mongos: "

daemon --user "$MONGO_USER" --check $mongos "$NUMACTL $mongos $OPTIONS >/dev/null 2>&1"

RETVAL=$?

echo

[ $RETVAL -eq 0 ] && touch "$LOCKFILE"

}

stop()

{

echo -n $"Stopping mongos: "

mongo_killproc "$PIDFILEPATH" $mongos

RETVAL=$?

echo

[ $RETVAL -eq 0 ] && rm -f "$LOCKFILE"

}

restart () {

stop

start

}

# Send TERM signal to process and wait up to 300 seconds for process to go away.

# If process is still alive after 300 seconds, send KILL signal.

# Built-in killproc() (found in /etc/init.d/functions) is on certain versions of Linux

# where it sleeps for the full $delay seconds if process does not respond fast enough to

# the initial TERM signal.

mongo_killproc()

{

local pid_file=$1

local procname=$2

local -i delay=300

local -i duration=10

local pid=`pidofproc -p "${pid_file}" ${procname}`

kill -TERM $pid >/dev/null 2>&1

usleep 100000

local -i x=0

while [ $x -le $delay ] && checkpid $pid; do

sleep $duration

x=$(( $x + $duration))

done

kill -KILL $pid >/dev/null 2>&1

usleep 100000

rm -f "${pid_file}"

checkpid $pid

local RC=$?

[ "$RC" -eq 0 ] && failure "${procname} shutdown" || success "${procname} shutdown"

RC=$((! $RC))

return $RC

}

RETVAL=0

case "$1" in

start)

start

;;

stop)

stop

;;

restart|reload|force-reload)

restart

;;

condrestart)

[ -f "$LOCKFILE" ] && restart || :

;;

status)

status $mongos

RETVAL=$?

;;

echo "Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"

RETVAL=1

esac

exit $RETVAL

/etc/mongos.conf Base YAML Configuration

Be sure to set sharding.configDB appropriately.

# mongos base config in YAML
systemLog:
    destination: "file"
    path: "/var/log/mongodb/mongos.log"
    logAppend: true
processManagement:
    fork: true
    pidFilePath: "/var/run/mongodb/mongos.pid"
net:
    bindIp: "0.0.0.0"
    port: 27017
sharding:
    configDB: "localhost:27019"

# mongos base config in YAML

systemLog:

destination: "file"

path: "/var/log/mongodb/mongos.log"

logAppend: true

processManagement:

fork: true

pidFilePath: "/var/run/mongodb/mongos.pid"

net:

bindIp: "0.0.0.0"

port: 27017

sharding:

configDB: "localhost:27019"

Post-Install Environment Setup

Provided mongodb-org-mongos has already been installed, the following script will do most of the legwork to get the CentOS 6 environment up and running. Note: The script is pulling the configuration file and service file dependencies from a web server using wget; this should be tailored. It performs the following:

Add a mongod user (yes, with a d) with the same settings that would be supplied from mongodb-org-server.
Add a placeholder file for sysconfig.
Add the mongos.conf file with the base configuration noted in previous (wget).
Add the init.d service script (wget).
If SELinux is enabled, set the contexts for the files created in previous, and add the port definition via semanage. Note that it will automatically install the required tools to perform this task if required.
Add and enable the new mongos service

#!/bin/bash

append='[mongos-prep] ';

SYSCONFIG_FILE="/etc/sysconfig/mongos";
ETCCONFIG_FILE="/etc/mongos.conf";
INITD_FILE="/etc/init.d/mongos";
LOGDIR_PATH="/var/log/mongodb";
INITD_DOWNLOAD="http://host.tld/path/to/init.d/mongos";
ETCCONFIG_DOWNLOAD="http://host.tld/path/to/mongos.conf";

echo "${append}Adding mongod user";
useradd -l -M -r -U -s /bin/false -d /var/lib/mongo mongod;

echo "${append}Adding sysconfig file";
echo '# TODO: add relevant configuration stuff here.' >> "${SYSCONFIG_FILE}";
chmod 0644 "${SYSCONFIG_FILE}";

echo "${append}Downloading and installing init.d script...";
wget "${INITD_DOWNLOAD}" -O "${INITD_FILE}";
chmod 0755 "${INITD_FILE}";

echo "${append}Adding regular config file ${ETCCONFIG_FILE}";
wget "${ETCCONFIG_DOWNLOAD}" -O "${ETCCONFIG_FILE}";
chmod 0644 "${ETCCONFIG_FILE}";

echo "${append}Preparing log directory";
mkdir -p "${LOGDIR_PATH}";

if selinuxenabled; then
	echo "${append}SELinux enabled, checking checking required tools...";
	if ! hash restorecon 2>/dev/null; then
		echo "${append}restorecon missing, installing policycoreutils...";
		sleep 2;
		yum -y install policycoreutils;
	fi;
	if ! hash semanage 2>/dev/null; then
		echo "${append}semanage missing, installing policycoreutils-python...";
		sleep 2;
		yum -y install policycoreutils-python;
	fi;
	
	echo "${append}Restoring SELinux contexts";
	restorecon -F "${INITD_FILE}";
	restorecon -F "${SYSCONFIG_FILE}";
	restorecon -F "${ETCCONFIG_FILE}";
	restorecon -F "${LOGDIR_PATH}";
	
	echo "${append}Adding port definition mongod_port_t on tcp/27017 to SELinux port, this will take a while...";
	semanage port -a -t mongod_port_t -p tcp 27017;

	echo "${append}Adding file contexts...";
	semanage fcontext -a -t mongod_initrc_exec_t '/etc/rc\.d/init\.d/mongos';
	semanage fcontext -a -t mongod_exec_t '/usr/bin/mongos';
fi;

echo "${append}Adding and enabling mongos in the startup config";
chkconfig --add mongos;
chkconfig mongos on;

echo "${append}complete.";

#!/bin/bash

append='[mongos-prep] ';

SYSCONFIG_FILE="/etc/sysconfig/mongos";

ETCCONFIG_FILE="/etc/mongos.conf";

INITD_FILE="/etc/init.d/mongos";

LOGDIR_PATH="/var/log/mongodb";

INITD_DOWNLOAD="http://host.tld/path/to/init.d/mongos";

ETCCONFIG_DOWNLOAD="http://host.tld/path/to/mongos.conf";

echo "${append}Adding mongod user";

useradd -l -M -r -U -s /bin/false -d /var/lib/mongo mongod;

echo "${append}Adding sysconfig file";

echo '# TODO: add relevant configuration stuff here.' >> "${SYSCONFIG_FILE}";

chmod 0644 "${SYSCONFIG_FILE}";

echo "${append}Downloading and installing init.d script...";

wget "${INITD_DOWNLOAD}" -O "${INITD_FILE}";

chmod 0755 "${INITD_FILE}";

echo "${append}Adding regular config file ${ETCCONFIG_FILE}";

wget "${ETCCONFIG_DOWNLOAD}" -O "${ETCCONFIG_FILE}";

chmod 0644 "${ETCCONFIG_FILE}";

echo "${append}Preparing log directory";

mkdir -p "${LOGDIR_PATH}";

if selinuxenabled; then

echo "${append}SELinux enabled, checking checking required tools...";

if ! hash restorecon 2>/dev/null; then

echo "${append}restorecon missing, installing policycoreutils...";

sleep 2;

yum -y install policycoreutils;

fi;

if ! hash semanage 2>/dev/null; then

echo "${append}semanage missing, installing policycoreutils-python...";

sleep 2;

yum -y install policycoreutils-python;

fi;

echo "${append}Restoring SELinux contexts";

restorecon -F "${INITD_FILE}";

restorecon -F "${SYSCONFIG_FILE}";

restorecon -F "${ETCCONFIG_FILE}";

restorecon -F "${LOGDIR_PATH}";

echo "${append}Adding port definition mongod_port_t on tcp/27017 to SELinux port, this will take a while...";

semanage port -a -t mongod_port_t -p tcp 27017;

echo "${append}Adding file contexts...";

semanage fcontext -a -t mongod_initrc_exec_t '/etc/rc\.d/init\.d/mongos';

semanage fcontext -a -t mongod_exec_t '/usr/bin/mongos';

fi;

echo "${append}Adding and enabling mongos in the startup config";

chkconfig --add mongos;

chkconfig mongos on;

echo "${append}complete.";

SELinux and MongoDB

The automation script above does not add all of the required SELinux policies required for mongos. Install the mongodb-org package to initialize all of the policies; it can be removed later if need be while keeping these policies intact. Finding the script used by MongoDB (or making one up) is a task for another day. A policy dump from semanage is as follows:

semanage port -l | grep mongo
	mongod_port_t                  tcp      27017, 27017-27019, 28017-28019

semanage fcontext -l | grep mongo
	/etc/rc\.d/init\.d/mongod                          regular file       system_u:object_r:mongod_initrc_exec_t:s0
	/usr/bin/mongod                                    regular file       system_u:object_r:mongod_exec_t:s0
	/usr/share/aeolus-conductor/dbomatic/dbomatic      regular file       system_u:object_r:mongod_exec_t:s0
	/var/lib/mongodb(/.*)?                             all files          system_u:object_r:mongod_var_lib_t:s0
	/var/log/mongodb(/.*)?                             all files          system_u:object_r:mongod_log_t:s0
	/var/run/aeolus/dbomatic\.pid                      regular file       system_u:object_r:mongod_var_run_t:s0
	/var/run/mongodb(/.*)?                             all files          system_u:object_r:mongod_var_run_t:s0

semanage permissive -l | grep mongo
	mongo_t

semanage port -l | grep mongo

mongod_port_t tcp 27017, 27017-27019, 28017-28019

semanage fcontext -l | grep mongo

/etc/rc\.d/init\.d/mongod regular file system_u:object_r:mongod_initrc_exec_t:s0

/usr/bin/mongod regular file system_u:object_r:mongod_exec_t:s0

/usr/share/aeolus-conductor/dbomatic/dbomatic regular file system_u:object_r:mongod_exec_t:s0

/var/lib/mongodb(/.*)? all files system_u:object_r:mongod_var_lib_t:s0

/var/log/mongodb(/.*)? all files system_u:object_r:mongod_log_t:s0

/var/run/aeolus/dbomatic\.pid regular file system_u:object_r:mongod_var_run_t:s0

/var/run/mongodb(/.*)? all files system_u:object_r:mongod_var_run_t:s0

semanage permissive -l | grep mongo

mongo_t

Migrating Kickstart from CentOS 6 to CentOS 7 + ESXi VMware Tools

In another post I described how to install CentOS 6 via a kickstart file (be sure to check out the “Kickstart Sample Configuration” section). CentOS 7 was recently released, and with that I needed to also use a kickstart configuration. However, simply using the previous kickstart configuration was not as easy as copy-and-paste (besides updating the release version in the repository configuration).

Summary of Kickstart Changes

There were a few changes that needed to be made for a base/core installation of CentOS 7:

Include eula --agreed (read the documentation)
Include services --enabled=NetworkManager,sshd (read the documentation)
Update the install packages list ( %packages section)
CentOS 7 is also a bit more strict with the kickstart file, so I had to explicitly include %end where applicable
CentOS 7’s default file system is now xfs, in CentOS 6 it was ext4, so consider updating the automatic partitioning to use xfs
Package groups @scalable-file-systems, @server-platform, @server-policy, and @system-admin-tools no longer exist – I haven’t located suitable replacements yet
Things like ifconfig are no longer included by default (they are now deprecated), so if you need them be sure to include net-tools. You should be using ip by now anyway.

Kickstart Sample Configuration

And now, an updated kickstart config for CentOS 7, with consideration of the previously mentioned updates (compare it with the previous Kickstart Sample Configuration mentioned in the other post). I also chose to include some extra packages that don’t exist by default with a @core installation.

#version=RHEL7
install
nfs --server=repo.hostname.tld --dir=/repo/centos/7/os/x86_64
lang en_US.UTF-8
keyboard --vckeymap=us --xlayouts='us'
zerombr
eula --agreed
services --enabled=NetworkManager,sshd
reboot

# leaving network in here, but it'll be dhcp for now, make sure you have KVM access
#network --onboot no --device eth0 --bootproto static --ip 172.16.0.254 --netmask 255.255.0.0 --gateway 172.16.1.1 --noipv6 --nameserver 172.16.0.2 --hostname new.hostname.tld
rootpw --iscrypted DEFAULT_SALTED_ROOT_PASSWORD
firewall --service=ssh
auth --enableshadow --passalgo=sha512
selinux --enforcing
timezone America/Phoenix --isUtc --ntpservers=0.centos.pool.ntp.org,1.centos.pool.ntp.org,2.centos.pool.ntp.org,3.centos.pool.ntp.org

### LVM
bootloader --location=mbr --boot-drive=sda
clearpart --all --drives=sda
ignoredisk --only-use=sda

part /boot --fstype="xfs" --ondisk=sda --size=256
part pv.01 --fstype="lvmpv" --ondisk=sda --grow --size=1

volgroup vg_main pv.01
logvol / --fstype="xfs" --name=lv_root --vgname=vg_main --grow --size=8192
logvol swap --fstype="swap" --name=lv_swap --vgname=vg_main --grow --size=4096 --maxsize=4096

%packages
@core
net-tools
policycoreutils
screen
tree
vim
wget
%end

%post
wget http://your.hostn.tld/scripts/c7/post-install.sh -O post-install.sh
sh post-install.sh
rm -f post-install.sh
%end

#version=RHEL7

install

nfs --server=repo.hostname.tld --dir=/repo/centos/7/os/x86_64

lang en_US.UTF-8

keyboard --vckeymap=us --xlayouts='us'

zerombr

eula --agreed

services --enabled=NetworkManager,sshd

reboot

# leaving network in here, but it'll be dhcp for now, make sure you have KVM access

#network --onboot no --device eth0 --bootproto static --ip 172.16.0.254 --netmask 255.255.0.0 --gateway 172.16.1.1 --noipv6 --nameserver 172.16.0.2 --hostname new.hostname.tld

rootpw --iscrypted DEFAULT_SALTED_ROOT_PASSWORD

firewall --service=ssh

auth --enableshadow --passalgo=sha512

selinux --enforcing

timezone America/Phoenix --isUtc --ntpservers=0.centos.pool.ntp.org,1.centos.pool.ntp.org,2.centos.pool.ntp.org,3.centos.pool.ntp.org

### LVM

bootloader --location=mbr --boot-drive=sda

clearpart --all --drives=sda

ignoredisk --only-use=sda

part /boot --fstype="xfs" --ondisk=sda --size=256

part pv.01 --fstype="lvmpv" --ondisk=sda --grow --size=1

volgroup vg_main pv.01

logvol / --fstype="xfs" --name=lv_root --vgname=vg_main --grow --size=8192

logvol swap --fstype="swap" --name=lv_swap --vgname=vg_main --grow --size=4096 --maxsize=4096

%packages

@core

net-tools

policycoreutils

screen

tree

vim

wget

%end

%post

wget http://your.hostn.tld/scripts/c7/post-install.sh -O post-install.sh

sh post-install.sh

rm -f post-install.sh

%end

VMware Tools Change

If you’re like me and use ESXi, I’m currently on version 5.5 and 5.5u1, the installable tools for integrating with ESXi is a nice treat. However, the repository location has changed specifically for RHEL7, and so have the packages.

The new repository location is at http://packages.vmware.com/packages/rhel7/x86_64/.
The only package that needs to be installed is open-vm-tools.
Note that the tools are currently only available for the x86_64 architecture, but I’m still including $basearch in the URL.
The “packages” repo at packages.vmware.com does not include a GPG Key, but it is the same one from the older “tools” repo.

Add VMware Tools to YUM

Put the following repo configuration in /etc/yum.repos.d/vmware-tools.repo:

[vmwarec7]
name=VMware Tools - $basearch
baseurl=http://packages.vmware.com/packages/rhel7/$basearch/
gpgkey=http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub
gpgcheck=1
protect=1
enabled=1

[vmwarec7]

name=VMware Tools - $basearch

baseurl=http://packages.vmware.com/packages/rhel7/$basearch/

gpgkey=http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

gpgcheck=1

protect=1

enabled=1

Then update yum and install the tools:

# clean yum's cache
yum clean all
# reload yum.repos.d
yum check-update
# install the package
yum install open-vm-tools

# clean yum's cache

yum clean all

# reload yum.repos.d

yum check-update

# install the package

yum install open-vm-tools

Install CentOS 6 with Anaconda/Kickstart (plus ESXi VMware Tools)

Synopsis

I’ve been getting my feet wet with ESXi, CentOS 6 VMs, and YUM/RPM. Well the last two I have been using for years, but not like recently.

The goal is to be able to blindly install a controlled distribution of CentOS 6.x quickly and without error (maybe even install multiple at the same time). What I needed:

Anaconda Kickstart file (ks.cfg)
Local mirrored repository for CentOS 6.x (6.3 in my example)
Custom 3rd party repo
HTTP/NFS/RSYNC access to these
Variable disk/cpu/ram size – the partitions need to be dynamic

Without writing a book about all of this, I really want to just highlight some problems I ran into and how I solved them.

An example of my kickstart file is below for reference.

Automated Partition Schema

Since I’m in the world of virtualized hardware, it is important for the disk to scale easily without lost data. The prerequisite to this is of course Logical Volume Management (LVM). Now you may not agree with my LVM layout, and honestly, this isn’t my expertise (optimization of disk partitions), but at the least there must be “boot,” “root,” and “swap” partitions.

The goal here is to make the root partition grow to its maximum size without negating the swap. Also, the boot partition won’t be on the LVM, it will be fixed in the MBR. The kickstart section is as follows:

# delete everything on /dev/sda
clearpart --all --drives=sda

# only use the sda - this is totally fine in the case of a single physical disk (remember, we're virtual here, so this is probably for the best, considering you'll have SAN, DAS or NAS behind it)
ignoredisk --only-use=sda

# make the boot partition in the MBR (only 128mb)
part /boot --fstype=ext4 --size=128

# make the physical partition for the LVM to sit on "pv.01" is "partition volume number 01" alternatively something like "pv.02393" would be fine.
# this partition should "grow" to fill the entire disk, the initial size is 1mb (don't use 0 here)
part pv.01 --grow --size=1

# define the LVM volgroup called "vg_main" - this can be called whatever you want - and put it on the physical partition "pv.01" that was created in previous.
volgroup vg_main pv.01

# the root partition called "lv_root" sitting on the volume group "vg_main" with a minimum size of 8gb, it will grow to fill whatever is left of the disk
logvol / --fstype=ext4 --name=lv_root --vgname=vg_main --grow --size=8192

# the swap partition called "lv_swap" sitting on volgroup "vg_main" that is exactly 2 gb
logvol swap --name=lv_swap --vgname=vg_main --grow --size=2016 --maxsize=2016

# delete everything on /dev/sda

clearpart --all --drives=sda

# only use the sda - this is totally fine in the case of a single physical disk (remember, we're virtual here, so this is probably for the best, considering you'll have SAN, DAS or NAS behind it)

ignoredisk --only-use=sda

# make the boot partition in the MBR (only 128mb)

part /boot --fstype=ext4 --size=128

# make the physical partition for the LVM to sit on "pv.01" is "partition volume number 01" alternatively something like "pv.02393" would be fine.

# this partition should "grow" to fill the entire disk, the initial size is 1mb (don't use 0 here)

part pv.01 --grow --size=1

# define the LVM volgroup called "vg_main" - this can be called whatever you want - and put it on the physical partition "pv.01" that was created in previous.

volgroup vg_main pv.01

# the root partition called "lv_root" sitting on the volume group "vg_main" with a minimum size of 8gb, it will grow to fill whatever is left of the disk

logvol / --fstype=ext4 --name=lv_root --vgname=vg_main --grow --size=8192

# the swap partition called "lv_swap" sitting on volgroup "vg_main" that is exactly 2 gb

logvol swap --name=lv_swap --vgname=vg_main --grow --size=2016 --maxsize=2016

I do want to note that the logvol’s are interpreted in a random order, so it is perfectly fine for the swap logvol to be declared after the root (/) logical volume.

Bypassing “Storage Device Warning”

The only problem I had in regards to a prompt-less install was the “Storage Device Warning” asking if I was sure I wanted to write to the disk and lose all of my data. No matter what I put in the partition specification of kickstart, it would always prompt. The answer is to use zerombr yes. See the option “zerombr” as defined within the CentOS kickstart guide. This can be placed anywhere in the kickstarter file (well except in %packages, %post or similar); just put it up near the top.

Auto Reboot

After the installation is complete, automatically reboot the machine. This works perfectly in ESXi since it automatically unmounts the virtual cdrom after the first boot of the guest! Simply put reboot anywhere in your kickstart – near the top is probably best.

VMware Tools RPM

In order for the vSphere Client to monitor and execute certain tasks on the guest vm, VMware Tools is required. This will show you things like IP addresses, hostnames and guest state as well as integrated shutdown/reboot tools.

Add VMware Tools to YUM

Put the following repo configuration in /etc/yum.repos.d/vmware-tools.repo:

[vmware-tools]
name=VMware Tools
baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/x86_64
enabled=1
gpgcheck=1
gpgkey=http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

[vmware-tools]

name=VMware Tools

baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/x86_64

enabled=1

gpgcheck=1

gpgkey=http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

Then execute the following shell in %post of kickstart:

yum -y update
yum -y install vmware-tools-plugins-guestInfo

1 2	yum -y update yum -y install vmware-tools-plugins-guestInfo

The important part to mention here is that the package is called vmware-tools-plugins-guestInfo. All the dependencies will come with it, so no worries there.

Mirroring a Repository for NFS Kickstart Installation

Create the Repo Mirror

Remember, my goal is to be able to quickly add a CentOS VM. With that, I don’t want to wait 30 minutes to pull down packages from a mirror in Iowa, New York or Cali. I want to pull it down once, keep it up-to-date and have my local install pull from my local mirror. For simplicities sake, I’ll put the mirror in /repo/centos.

MIRROR=mirror.centos.com::centos/6.4
DEST=/repo/centos
mkdir -p /repo/centos
rsync -avSHP --delete --exclude "local*" --exclude "isos"  $MIRROR $DEST

MIRROR=mirror.centos.com::centos/6.4

DEST=/repo/centos

mkdir -p /repo/centos

rsync -avSHP --delete --exclude "local*" --exclude "isos" $MIRROR $DEST

I am choosing to exclude any local files/directories (“local*”) and also the huge DVD ISOs (“isos”). Also note that the mirror format is host::path and that the mirror host must support the rsync protocol.

Keep the Local Mirror Updated

To keep the local repo copy up-to-date, run this script via cron (by the way, I stole this from somewhere, I just don’t remember). Please don’t forget to swap out the mirror hostname and path with something that makes more geographical sense to you.

#!/bin/bash
LOCK=/var/lock/subsys/repo_sync;
CENTOS_REPO=mirrors.usc.edu::centos/6.4;
CENTOS_DEST=/repo/centos;

if [ -f $LOCK ]; then
        echo "Updates via rsync already running.";
        exit 1;
fi;

touch $LOCK;

if [ -d $DEST ] ; then
        rsync -avSHP --delete --exclude "local*" --exclude "isos" $CENTOS_REPO $CENTOS_DEST;
else
        echo "Target directory "$DEST" does not exist. Cannot update.";
fi;

/bin/rm -f $LOCK;

exit 0;

#!/bin/bash

LOCK=/var/lock/subsys/repo_sync;

CENTOS_REPO=mirrors.usc.edu::centos/6.4;

CENTOS_DEST=/repo/centos;

if [ -f $LOCK ]; then

echo "Updates via rsync already running.";

exit 1;

fi;

touch $LOCK;

if [ -d $DEST ] ; then

rsync -avSHP --delete --exclude "local*" --exclude "isos" $CENTOS_REPO $CENTOS_DEST;

else

echo "Target directory "$DEST" does not exist. Cannot update.";

fi;

/bin/rm -f $LOCK;

exit 0;

Configure NFS for Kickstart Network Installations

NFS server support is built into CentOS and running by default, so this is pretty easy. Add the following to /etc/exports:

/repo/centos 172.16.0.0/16(ro,sync,all_squash)

1	/repo/centos 172.16.0.0/16(ro,sync,all_squash)

This exports the directory “/repo/centos” for NFS. Only the subnet 172.16.0.0/16 is allowed access (no credentials required). It is mounted as read-only (ro), connection are synchronous as opposed to asynchronous (sync), and all connections are anonymous for security purposes (all_squash). Man exports(5) if you need more help.

Restart NFS via service nfs restart.

I feel like I’m missing something with NFS, but I don’t recall; this was too easy. In my memory there was a struggle with rpc!

Update iptables for NFS

Edit /etc/sysconfig/iptables and throw these rules in there before -A INPUT -j REJECT --reject-with icmp-host-prohibited.

-A INPUT -m state --state NEW -p tcp --dport 111 -j ACCEPT
-A INPUT -m state --state NEW -p udp --dport 111 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 662 -j ACCEPT
-A INPUT -m state --state NEW -p udp --dport 662 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 875 -j ACCEPT
-A INPUT -m state --state NEW -p udp --dport 875 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 892 -j ACCEPT
-A INPUT -m state --state NEW -p udp --dport 892 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 2049 -j ACCEPT
-A INPUT -m state --state NEW -p udp --dport 2049 -j ACCEPT
-A INPUT -m state --state NEW -p udp --dport 32769 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 32803 -j ACCEPT

-A INPUT -m state --state NEW -p tcp --dport 111 -j ACCEPT

-A INPUT -m state --state NEW -p udp --dport 111 -j ACCEPT

-A INPUT -m state --state NEW -p tcp --dport 662 -j ACCEPT

-A INPUT -m state --state NEW -p udp --dport 662 -j ACCEPT

-A INPUT -m state --state NEW -p tcp --dport 875 -j ACCEPT

-A INPUT -m state --state NEW -p udp --dport 875 -j ACCEPT

-A INPUT -m state --state NEW -p tcp --dport 892 -j ACCEPT

-A INPUT -m state --state NEW -p udp --dport 892 -j ACCEPT

-A INPUT -m state --state NEW -p tcp --dport 2049 -j ACCEPT

-A INPUT -m state --state NEW -p udp --dport 2049 -j ACCEPT

-A INPUT -m state --state NEW -p udp --dport 32769 -j ACCEPT

-A INPUT -m state --state NEW -p tcp --dport 32803 -j ACCEPT

And restart iptables via service iptables restart.

Configure Kickstart to Use Local Repo via NFS

This is an easy one-line if everything is set up correctly. Add the following after the “install” option within the kickstart configuration.

nfs --server=YOUR.HOSTNAME.TLD --dir=/repo/centos/6/os/x86_64

1	nfs --server=YOUR.HOSTNAME.TLD --dir=/repo/centos/6/os/x86_64

Use Local Repo Post Install

So you want to keep using your new local repo beyond the kickstart installation? No worries. Install apache, configure the vhost and update ks.cfg.

yum install httpd;
vim /etc/httpd/conf.d/vhosts.conf

1 2	yum install httpd; vim /etc/httpd/conf.d/vhosts.conf

Inside vhosts.conf:

<VirtualHost *:80>
        ServerAdmin webmaster@HOSTNAME.TLD
        DocumentRoot /repo
        ServerName YOUR.HOSTNAME.TLD
        ErrorLog logs/YOUR.HOSTNAME.TLD-error_log
        <Directory />
                Options Indexes MultiViews FollowSymLinks
                IndexOptions FancyIndexing FoldersFirst VersionSort XHTML
        </Directory>
</VirtualHost>

ServerAdmin webmaster@HOSTNAME.TLD

DocumentRoot /repo

ServerName YOUR.HOSTNAME.TLD

ErrorLog logs/YOUR.HOSTNAME.TLD-error_log

Options Indexes MultiViews FollowSymLinks

IndexOptions FancyIndexing FoldersFirst VersionSort XHTML

</Directory>

</VirtualHost>

Add the following rule to iptables:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

1	-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

Restart iptables via service iptables restart;.

Start httpd via service httpd start; chkconfig httpd on;.

Update the kickstart configuration:

repo --name=My Local Repo --baseurl=YOUR.HOSTNAME.TLD/centos/6/os/x86_64

1	repo --name=My Local Repo --baseurl=YOUR.HOSTNAME.TLD/centos/6/os/x86_64

Done!

Kickstart Sample Configuration

install
nfs --server=repo.hostname.tld --dir=/repo/centos/6/os/x86_64
lang en_US.UTF-8
keyboard us
zerombr yes
reboot

# leaving network in here, but it'll be dhcp for now, make sure you have KVM access
#network --onboot no --device eth0 --bootproto static --ip 172.16.0.254 --netmask 255.255.0.0 --gateway 172.16.0.1 --noipv6 --nameserver 172.16.0.2 --hostname new.hostname.tld
rootpw  --iscrypted DEFAULT_SALTED_ROOT_PASSWORD
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --enforcing
timezone --utc Europe/London
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

### LVM
clearpart --all --drives=sda
ignoredisk --only-use=sda

part /boot --fstype=ext4 --size=128
part pv.01 --grow --size=1

volgroup vg_main pv.01
logvol / --fstype=ext4 --name=lv_root --vgname=vg_main --grow --size=8192
logvol swap --name=lv_swap --vgname=vg_main --grow --size=2016 --maxsize=2016

%packages
@base
@core
@scalable-file-systems
@server-platform
@server-policy
@system-admin-tools
pax
sgpio
screen
crypto-utils

%post
wget http://your.hostn.tld/post-install-script.sh -O post-install-script.sh
sh post-install-script.sh
rm -f post-install-script.sh
yum -y upgrade
yum -y install vmware-tools-plugins-guestInfo

%end

install

nfs --server=repo.hostname.tld --dir=/repo/centos/6/os/x86_64

lang en_US.UTF-8

keyboard us

zerombr yes

reboot

# leaving network in here, but it'll be dhcp for now, make sure you have KVM access

#network --onboot no --device eth0 --bootproto static --ip 172.16.0.254 --netmask 255.255.0.0 --gateway 172.16.0.1 --noipv6 --nameserver 172.16.0.2 --hostname new.hostname.tld

rootpw --iscrypted DEFAULT_SALTED_ROOT_PASSWORD

firewall --service=ssh

authconfig --enableshadow --passalgo=sha512

selinux --enforcing

timezone --utc Europe/London

bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

### LVM

clearpart --all --drives=sda

ignoredisk --only-use=sda

part /boot --fstype=ext4 --size=128

part pv.01 --grow --size=1

volgroup vg_main pv.01

logvol / --fstype=ext4 --name=lv_root --vgname=vg_main --grow --size=8192

logvol swap --name=lv_swap --vgname=vg_main --grow --size=2016 --maxsize=2016

%packages

@base

@core

@scalable-file-systems

@server-platform

@server-policy

@system-admin-tools

pax

sgpio

screen

crypto-utils

%post

wget http://your.hostn.tld/post-install-script.sh -O post-install-script.sh

sh post-install-script.sh

rm -f post-install-script.sh

yum -y upgrade

yum -y install vmware-tools-plugins-guestInfo

%end

For the option “rootpw” use grub-crypt with the specified hash algorithm under authconfig –passalgo=X (to replace DEFAULT_SALTED_ROOT_PASSWORD). In the sample ks.cfg file, I have sha512, so:

$ grub-crypt --sha-512
Password: 12345
Retype password: 12345
$6$oavua3X2xzo08xkj$r1cYIK4ghmA7FEpIXbNsNQr1Xll13bGJbBX2CpoZGgKuLkDHkA71L/V8mfkiQh1pr4.JQwgW8hOWxhZW9W.y70

$ grub-crypt --sha-512

Password: 12345

Retype password: 12345

$6$oavua3X2xzo08xkj$r1cYIK4ghmA7FEpIXbNsNQr1Xll13bGJbBX2CpoZGgKuLkDHkA71L/V8mfkiQh1pr4.JQwgW8hOWxhZW9W.y70

Using the Kickstart Configuration

The idea is to create a custom ISO with the kickstart configuration embedded, but I haven’t done this yet. So for now, I’m hosting the file as ks.cfg on an intranet HTTP server and booting a centos 6.3 netinstall (~200mb). At the bootloader prompt, specify extra parameters vmlinux initrd=initrd.img ks=http://some.host.local/ks.cfg. This installs all the packages, updates as needed, partitions the disk, runs a custom script, and reboots the machine.

Brain dump complete.

Installing Jenkins CI on CentOS 6.x (Tomcat with an AJP Proxy)

This is on a fresh minimal install of CentOS 6.3 (but should work for 6.x and also to my knowledge works for the latest versions of 5.x). Judging by how easy this is and how aligned CentOS is with normal RedHat (RHEL), it should work on any RHEL-based system 5.x or 6.x but don’t take my word for it.

Don’t copy and paste this thing, it is a guideline, some of it won’t work if you copy and paste.

First get CentOS updated all the way in a safe session (I like to use screen, make sure it is installed though):

yum install screen;
screen;
yum upgrade;

yum install screen;

screen;

yum upgrade;

Install the Jenkins RPM via YUM as described on their RedHad Repository page for Jenkins; at the time of this writing the commands are:

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo;
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key;
yum install jenkins;

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo;

sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key;

yum install jenkins;

Don’t forget, you’ll need java as well (at least a JRE, but I can only seem to find the JDK, which will be overkill but sufficient):

yum install java;

1	yum install java;

Install httpd (Apache 2.2). It is bad practice to bind Tomcat (which Jenkins uses) to port 80. Tomcat is a service, not a web server. Apache will be used to proxy the requests to the Tomcat service and thus Jenkins through port 80 (the normal www port):

yum install httpd;

1	yum install httpd;

At this point Apache, nor Jenkins should be running. Update iptables and open TCP port 80 (no need to open port 8080 which Tomcat uses, all proxy comms will happen via the loopback):

vi /etc/sysconfig/iptables;

1	vi /etc/sysconfig/iptables;

And the iptables file is:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

# added the following line
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

# added the following line

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

Close vi (:wq), and restart iptables:

service iptables restart;

1	service iptables restart;

Now, configure Apache. Update the ServerName and any other necessary configurations. I’ll leave that up to you (the reader). What you DO need to know is the virtual host proxy configuration. I’ll be using AJP (Apache JServ Protocol). Some say it is slower, others say it is faster than a normal proxy configuration. I have seen AJP function superbly on an enterprise-level system and I have never had any problems with it.

First make sure the module is loaded:

grep ajp /etc/httpd/conf/httpd.conf

1	grep ajp /etc/httpd/conf/httpd.conf

Should yield the result: “LoadModule proxy_ajp_module modules/mod_proxy_ajp.so”

This is enabled and available by default within CentOS. Getting/building a proxy_ajp module is a PITA, and if it is not available to you, that is outside the scope of this doc (although I have done this before and have been successful – maybe I’ll write a guide later).

Now configure the vhost (this file didn’t exist, so vi will create it for me):

vi /etc/httpd/conf.d/vhosts.conf

1	vi /etc/httpd/conf.d/vhosts.conf

And add:

NameVirtualHost *:80
<VirtualHost *:80>
        ServerName jenkins.host.tld
        ProxyRequests Off
        ProxyPreserveHost On
        ProxyPass / ajp://127.0.0.1:8009/
        ProxyPassReverse / ajp://127.0.0.1:8009/
        ProxyPassReverseCookiePath / /
</VirtualHost>

NameVirtualHost *:80

ServerName jenkins.host.tld

ProxyRequests Off

ProxyPreserveHost On

ProxyPass / ajp://127.0.0.1:8009/

ProxyPassReverse / ajp://127.0.0.1:8009/

ProxyPassReverseCookiePath / /

</VirtualHost>

Save, quit vi (:wq), and start httpd for the first time. It should start without a problem; Tomcat (Jenkins) does not need to be running for this to work – the proxy will simply timeout and fail until the downstream service is online.

service httpd start;

1	service httpd start;

Go ahead and start Jenkins for the first time:

service jenkins start;

1	service jenkins start;

Navigate to the domain you install it under (in this example I used jenkins.host.tld). Happy building!

After Jenkins is installed consider the following plugins:

Confluence Publisher Plugin
Publish Over FTP (although I prefer LFTP)
Role-based Authorization Strategy
LDAP Plugin (installed by default)

Lastly, Jenkins support for SVN 1.7 is still up in the air. According to #JENKINS-11381, it is complete, but I haven’t had a chance to install and play with SVNKit 1.7 support.

Setting up BIND 9 on CentOS 6 and Securing a Private Nameserver on the Internet

Today I was setting up a brand new server over at LiquidWeb (I have been hosting with this Lansing, MI based company for years, although I’m stubborn and have never tried out their heroic support). I already had the IP addresses (2) and the box provisioned. It is a clean install of the latest CentOS 6 – that means no cPanel/WHM, Plesk or similar. The box will serve many purposes, but it also needs its own nameserver. For the sake of this tutorial, the example domain will be putthingsdown.com and the two IP addresses my host provided are 11.22.33.44 and 11.22.33.45.

Register Your Private Nameservers at Your Registrar

My one-and-only registrar is GoDaddy. They keep things simple and allow for flexibility as far as domain management goes. They are just my registrar: I do not host with them, use their mail servers, nor their nameservers.

This part is simple, when you register the domain name, navigate to the domain management tool and update the namservers to “ns1.putthingsdown.com” and “ns2.putthingsdown.com” – these do not have to exist yet and will be created below.

Lastly, register the nameservers and a utility host at GoDaddy by adding the following three “Host” entries (not subdomain entries, but host entries – there is a difference):

Hostname is ns1 and IP address is 11.22.33.44.
Hostname is ns2 and IP address is 11.22.33.45.
Hostname is host and IP address is 11.22.33.44.

Configuring named

First, get the named service installed: yum install bind-chroot
Notice that bind-chroot will install under /var/named in its own chrooted directory. This is for security purposes. There should be hardlinks to the chrooted “data” and “slave” directories (this was updated with EL6 – yay!)
Configure the rndc key: (Use the ampersand to send this process to the background, it will take 10-15 minutes to generate) rndc-confgen &
Secure the newly generated rndc.key file: chown named /etc/rndc.key; chmod 600 /etc/rndc.key;
Get the rndc key name which is encapsulated in double qutoes from the generated file (it should be rndc-key by default): grep key /etc/rndc.key
Note on CentOS 6 and bind 9.7.3, there will not be a /etc/rndc.conf
Create your first zone file in /var/named/data (example below): vi /var/named/data/putthingsdown.com.zone
Configure named (example below): vi /etc/named.conf

Example Master/Authoritative Zone File

$TTL 38400
putthingsdown.com.        86400   IN      SOA     ns1.putthingsdown.com. webmaster.putthingsdown.com. (
                                        2012062301      ;serial
                                        7200            ;refresh
                                        900             ;retry
                                        1209600         ;expire
                                        42300           ;minimum
)

putthingsdown.com.        86400   IN      NS      ns1.putthingsdown.com.
putthingsdown.com.        86400   IN      NS      ns2.putthingsdown.com.
ns1             86400   IN      A       11.22.33.44
ns2             86400   IN      A       11.22.33.45
putthingsdown.com.        14400   IN      A       11.22.33.44
host            14400   IN      A       11.22.33.44
www             14400   IN      CNAME   putthingsdown.com.

$TTL 38400

putthingsdown.com. 86400 IN SOA ns1.putthingsdown.com. webmaster.putthingsdown.com. (

2012062301 ;serial

7200 ;refresh

900 ;retry

1209600 ;expire

42300 ;minimum

)

putthingsdown.com. 86400 IN NS ns1.putthingsdown.com.

putthingsdown.com. 86400 IN NS ns2.putthingsdown.com.

ns1 86400 IN A 11.22.33.44

ns2 86400 IN A 11.22.33.45

putthingsdown.com. 14400 IN A 11.22.33.44

host 14400 IN A 11.22.33.44

www 14400 IN CNAME putthingsdown.com.

The “serial” should be updated every single time this zone file is modified and reloaded into named. The format is as follows: YYYMMDDnn (where nn is an incrementor for the same day, e.g. 01, 02, 03…. 11, 12, 13)
refresh, retry, expire, and minimum are measured in seconds, just a note that these aren’t always followed, especially by residential DNS mirrors/servers.
The SOA (Start of Authority) is the… well… start of the zone record. This section (including the parens) is what kickstarts the zone file and defines the meta data.
After the SOA the first two records are NS (NameServer): the TTL (time-to-live) is 86400 seconds, or 1 day, and point to the (non-existant, yet) nameservers ns1 and ns2.
The next 2 records are A (Address) records that register the ns1 and ns2 subdomains and bind them to IP addresses – now the two NS records have something to point to.
The third A record is the actual domain itself which is bound to the primary IP address. This is proof that you really don’t need the “www” in front of the domain name, although this is also dependent on the web server configuration
“host” will serve as a utility subdomain which also points to the primary IP. This is helpful in the future if there is a secondary util server used for SSH to manage all the servers within the private network.
Lastly, the www subdomain acts as a CNAME (Canonical Name), or alias to putthingsdown.com – essentially putthingsdown.com and www.putthingsdown.com will take you to the same place. Hint: try not to use CNAME records if you don’t have to, although they make your zone more flexible for future enhancements, since they require a secondary lookup.

Example named Configuration

The lines highlighted below are the ones that changed from the default named.conf provided by the installation script. I am not going to go over this in detail, but do what to highlight a few pieces. Note: this is the insecure version of the named configuration; continue reading for security enhancements.

include "/etc/rndc.key";

controls {
        inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };
};

acl "trusted" { 11.22.33.44; 11.22.33.45; };

options {
        listen-on port 53 { 127.0.0.1; 11.22.33.44; 11.22.33.45; };
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query     { any; };
        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

zone "putthingsdown.com" {
        type master;
        file "/var/named/data/putthingsdown.com.zone";
};

include "/etc/named.rfc1912.zones";

include "/etc/rndc.key";

controls {

inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };

};

acl "trusted" { 11.22.33.44; 11.22.33.45; };

options {

listen-on port 53 { 127.0.0.1; 11.22.33.44; 11.22.33.45; };

listen-on-v6 port 53 { ::1; };

directory "/var/named";

dump-file "/var/named/data/cache_dump.db";

statistics-file "/var/named/data/named_stats.txt";

memstatistics-file "/var/named/data/named_mem_stats.txt";

allow-query { any; };

recursion yes;

dnssec-enable yes;

dnssec-validation yes;

dnssec-lookaside auto;

/* Path to ISC DLV key */

bindkeys-file "/etc/named.iscdlv.key";

};

logging {

channel default_debug {

file "data/named.run";

severity dynamic;

};

zone "." IN {

type hint;

file "named.ca";

};

zone "putthingsdown.com" {

type master;

file "/var/named/data/putthingsdown.com.zone";

};

include "/etc/named.rfc1912.zones";

The first include is to the rndc.key file that was generated in step 3 above.
The “trusted” ACL has your two IP addresses in it.
Within the controls declaration, the key name found in step 5 above should be defined.
“listen-on” needs to have both IP address listed – named binds itself to port 53 on both these IP addresses
Setting “allow-query” to any will allow any upstream DNS server the ability to query yours.
The section at the bottom for the zone is the inclusion of the zone file created in previous.

Almost Done: Test Stuff

Before we take the step to secure the named server, let’s make sure it works first. Restart the service (hopefully it doesn’t throw any errors). Once it restarts successfully let’s start it on boot-up (with its default run levels); make sure the chkconfig took:

# service named restart
Stopping named:                                            [  OK  ]
Starting named:                                            [  OK  ]
# chkconfig named on
# chkconfig --list | grep named
named           0:off   1:off   2:on    3:on    4:on    5:on    6:off

# service named restart

Stopping named: [ OK ]

Starting named: [ OK ]

# chkconfig named on

# chkconfig --list | grep named

named 0:off 1:off 2:on 3:on 4:on 5:on 6:off

I’ll assume port 53 is open for TCP and UDP. Honestly, this isn’t me “not knowing” which protocol; DNS primarily uses UDP, but has been known to use TCP as a fail-over and will definitely be used in IPv6.

Lastly, use a public tool on the web to verify your DNS configuration. I like to use NsLookup by Network-Tools.com. Simply put “putthingsdown.com” in the domain field and hit GO. If everything is set up correctly, some records from the zone file will be listed, specifically the SOA and NS, as well as the primary A.

Securing named

After some super-fast searching, I found this nifty BSP (not sure what BSP stands for?) over at NIST.gov: How to Secure a Domain Name Server (DNS). Here’s the gist of §3.1.2 (most of these are snippets, they shouldn’t be interpreted verbatim):

override “version” number using options { version "dunno kthxbye"; };
restrict zone transfers: options { allow-transfer { localhost; trusted; }; };
restrict dynamic updates in each zone: zone "putthingsdown.com" { allow-update { localhost; trusted; }; };
protect against DNS spoofing: options { recursion no; };
restrict by default all queries: options { allow-query { localhost; trusted; }; };
allow individual zone queries: zone "putthingsdown.com" { allow-query { any; }; };
Verify with security tools: DNSWalk (online version) zone transfer should fail with “REFUSED”, ZoneCheck, and dlint. Happy hunting.

CentOS 5.8 + Apache 2.2 + PHP 5.3 + suPHP 0.7.1

So I’m a bit of a purist when it comes to CentOS administration. CentOS is built on the idea of stability and sustainability. Without the addition of extra 3rd-party repositories, it provides the bare necessities to run a reliable and secure server. Don’t get me wrong though, there are plenty of great packages out of the box (from OpenSSL, Apache, PHP to OpenLDAP, PostgreSQL and then some), but sometimes you need some heavy-duty next-gen power tools like ffmpeg, nginx, OpenVPN or suPHP. Most of these packages are not available from the “CentOS Certified” base, extras and updates repositories; in fact, you can’t get them via yum without adding a third-party repo like RPMForge.

With that said, I need suPHP for a PHP staging environment. I’m not going to talk about what suPHP is, you can read about it on your own time. Going back to me being a purist, I don’t use RPMForge repos or anything similar. I like to stick to base and extras only and since there isn’t a suPHP RPM available – I’ll have to build it myself. The proper way to do this is to build it as an RPM (Red Hat Package Manager) and install via yum from the locally built RPM, but for whatever reason I can never get myself to do it this way.

Reminder, suPHP can only use PHP CGI, not PHP CLI (so look for a php-cgi binary, not just a php one)

Download & Building suPHP from Source

Before we start, make sure you have dev tools:

yum groupinstall "Development Tools"

1	yum groupinstall "Development Tools"

We’ll also need development packages for httpd (Apache 2.2), php53 (PHP 5.3), and apr (Apache Runtime Libraries and Utilities):

yum install apr apr-devel apr-util apr-util-devel httpd-devel php53-devel

1	yum install apr apr-devel apr-util apr-util-devel httpd-devel php53-devel

Now create a working directory, download the suPHP src, configure it and build (make). Note that you need to figure out where the apr config is located, mine is at /usr/bin/apr-1-config

mkdir ~/mod_suphp
cd ~/mod_suphp
wget http://www.suphp.org/download/suphp-0.7.1.tar.gz
tar -xzf suphp-0.7.1.tar.gz
cd suphp-0.7.1

# RTFM
less doc/INSTALL
# quit less by typing "q"

./configure --with-apr=/usr/bin/apr-1-config
make
make install
# notice that suphp is in /usr/local/sbin/suphp
# also it should have auto-installed into the apache modules directory:
ls -al /usr/lib64/httpd/modules/ | grep suphp
-rwxr-xr-x 1 root root   60303 XXX XX XX:XX mod_suphp.so

mkdir ~/mod_suphp

cd ~/mod_suphp

wget http://www.suphp.org/download/suphp-0.7.1.tar.gz

tar -xzf suphp-0.7.1.tar.gz

cd suphp-0.7.1

# RTFM

less doc/INSTALL

# quit less by typing "q"

./configure --with-apr=/usr/bin/apr-1-config

make

make install

# notice that suphp is in /usr/local/sbin/suphp

# also it should have auto-installed into the apache modules directory:

ls -al /usr/lib64/httpd/modules/ | grep suphp

-rwxr-xr-x 1 root root 60303 XXX XX XX:XX mod_suphp.so

Configure Apache + PHP to use suPHP

I’ll admit, I relied heavily on the suPHP docs, but even then it was not 100% complete. That, and sites like this one didn’t provide any useful information – I’m mainly aggravated that they used RPMForge and did not use php53 packages. But, after some re-reading, reinterpreting and trial & error, I’m up and running… and this is how it went (starting to get tired of writing this post, this will be short and sweet):

Important Files

/usr/local/etc/suphp.conf (this is the core suPHP configuration)
/etc/httpd/conf.d/suphp.conf (this is the Apache mod_suphp configuration… needed to create this)
/etc/httpd/conf.d/php.conf (this is the php configuration that I had to disable)
/etc/httpd/conf/httpd.conf (for some of the primary virtual hosts… all my other vhosts are in separate files)

suPHP Core Configuration

/usr/local/etc/suphp.conf, I based it off of the suphp.conf-example file located in the source code’s doc directory. This is an ini-style configuration:

[global]
;Path to logfile
logfile=/var/log/suphp.log

;Loglevel
loglevel=info

;User Apache is running as
;in centos this is apache
;if you dont know, look it up via: grep Apache /etc/passwd
webserver_user=apache

;Path all scripts have to be in
;delimited by a colon (:)
;the default is docroot=/var/www:${HOME}/public_html
;but i added a few extras
docroot=/var/www:${HOME}/public_html:/var/custom/dir:/usr/local/www

;Path to chroot() to before executing script
;chroot=/do/not/bother/setting/this

; Security options
; set group writeable perms to true to allow for session writing and tmp storage
allow_file_group_writeable=true
allow_file_others_writeable=false
allow_directory_group_writeable=true
allow_directory_others_writeable=false

;Check whether script is within DOCUMENT_ROOT
;just turn this off, dont feel like explaining
check_vhost_docroot=false

;Send minor error messages to browser
errors_to_browser=false

;PATH environment variable
env_path=/bin:/usr/bin

;Umask to set, specify in octal notation
umask=0077

; Minimum UID
; allow for root and system-level groups
min_uid=0

; Minimum GID
; allow for root and system-level groups
min_gid=0

[handlers]
;Handler for php-scripts
;NOTE that this should match with the handler type
;in the mod_suphp config (suphp.conf)
application/x-httpd-php="php:/usr/bin/php-cgi"

;Handler for CGI-scripts
x-suphp-cgi="execute:!self"

[global]

;Path to logfile

logfile=/var/log/suphp.log

;Loglevel

loglevel=info

;User Apache is running as

;in centos this is apache

;if you dont know, look it up via: grep Apache /etc/passwd

webserver_user=apache

;Path all scripts have to be in

;delimited by a colon (:)

;the default is docroot=/var/www:${HOME}/public_html

;but i added a few extras

docroot=/var/www:${HOME}/public_html:/var/custom/dir:/usr/local/www

;Path to chroot() to before executing script

;chroot=/do/not/bother/setting/this

; Security options

; set group writeable perms to true to allow for session writing and tmp storage

allow_file_group_writeable=true

allow_file_others_writeable=false

allow_directory_group_writeable=true

allow_directory_others_writeable=false

;Check whether script is within DOCUMENT_ROOT

;just turn this off, dont feel like explaining

check_vhost_docroot=false

;Send minor error messages to browser

errors_to_browser=false

;PATH environment variable

env_path=/bin:/usr/bin

;Umask to set, specify in octal notation

umask=0077

; Minimum UID

; allow for root and system-level groups

min_uid=0

; Minimum GID

; allow for root and system-level groups

min_gid=0

[handlers]

;Handler for php-scripts

;NOTE that this should match with the handler type

;in the mod_suphp config (suphp.conf)

application/x-httpd-php="php:/usr/bin/php-cgi"

;Handler for CGI-scripts

x-suphp-cgi="execute:!self"

mod_suphp Configuration

/etc/httpd/conf.d/suphp.conf:

# load the core module that was installed previously
LoadModule suphp_module modules/mod_suphp.so

# register the x-httpd-php mime-type for the php extension
# we'll have to remove this in php.conf
# note that this mime-type aligns with the configuration in /usr/local/etc/suphp.conf
AddType application/x-httpd-php .php

# in case this isn't set anywhere else:
DirectoryIndex index.php

# turn suPHP on
suPHP_Engine on

# this is the path to the directory that contains php.ini
suPHP_ConfigPath /etc

# register the php mime-type with suPHP so it knows what to consider
suPHP_AddHandler application/x-httpd-php

# path to the PHP CGI executable
# this should not be the PHP CLI executable (/usr/bin/php)
suPHP_PHPPath /usr/bin/php-cgi

# the default user and group to execute php under
# this could be nobody:nobody, or apache:apache, or jack:jack
# this will be overridden based on the virtual host directives
suPHP_UserGroup apache apache

# load the core module that was installed previously

LoadModule suphp_module modules/mod_suphp.so

# register the x-httpd-php mime-type for the php extension

# we'll have to remove this in php.conf

# note that this mime-type aligns with the configuration in /usr/local/etc/suphp.conf

AddType application/x-httpd-php .php

# in case this isn't set anywhere else:

DirectoryIndex index.php

# turn suPHP on

suPHP_Engine on

# this is the path to the directory that contains php.ini

suPHP_ConfigPath /etc

# register the php mime-type with suPHP so it knows what to consider

suPHP_AddHandler application/x-httpd-php

# path to the PHP CGI executable

# this should not be the PHP CLI executable (/usr/bin/php)

suPHP_PHPPath /usr/bin/php-cgi

# the default user and group to execute php under

# this could be nobody:nobody, or apache:apache, or jack:jack

# this will be overridden based on the virtual host directives

suPHP_UserGroup apache apache

PHP Configuration

/etc/httpd/conf.d/php.conf, just comment everything out, you don’t need it

Apache Virtual Host (vhost) Configuration

This can be set in each individual vhost if you want to override. For example:

NameVirtualHost *:80

# all my vhosts are stored here...
Include conf.d/vhosts/*

# system-level virtual hosts
<VirtualHost *:80>
        ServerName stage.domain.tld
        DocumentRoot /var/www/html
        suPHP_UserGroup root root
</VirtualHost>
<VirtualHost *:80>
        ServerName util.domain.tld
        CustomLog logs/utilcommon_log common
        DocumentRoot /var/utilwww/www
        suPHP_UserGroup utilwww utilwww
        <Directory /var/util/www>
                Options FollowSymLinks Includes MultiViews
                AllowOverride All
        </Directory>
</VirtualHost>

NameVirtualHost *:80

# all my vhosts are stored here...

Include conf.d/vhosts/*

# system-level virtual hosts

ServerName stage.domain.tld

DocumentRoot /var/www/html

suPHP_UserGroup root root

</VirtualHost>

ServerName util.domain.tld

CustomLog logs/utilcommon_log common

DocumentRoot /var/utilwww/www

suPHP_UserGroup utilwww utilwww

Options FollowSymLinks Includes MultiViews

AllowOverride All

</Directory>

</VirtualHost>

Almost Done…

Now restart httpd:

service httpd restart

1	service httpd restart

Refresh a php page and check. If it didn’t work, re-read this post or email me (contact info in my resume) and I won’t help, but i’ll refine this post and provide more information.