November 2013 – put things down

Playing Around with PHP Namespaces, Classes, and Variable Instantiation

Just needed to do some sanity checks, figure out some syntactical sugar (i.e. tricks), and test certain scenarios. I figured I’d share my findings with the world. Explanations are very light (brain dump) and embedded in the code; output follows directly. Oh, and this is a single stand-alone file.

php://stdin

Well, this really isn’t php://stdin, but the headline looks cool.

<?php

// look! multiple namespaces in the same file, couldn't use `namespace ns;` in this case, needed to use block encapsulation.

namespace ns { // declare a namespace block
	interface INS { // declare a namespace

	}
	class NSMain implements INS { // declare a class that uses the interface

	}
}

namespace ns\sub { // declare a secondary namespace under the original one
	class NSSub { // and here's an empty class

	}
}

namespace other { // a third namespace (but a second top-level ns)
	class T { // the test class
		public static $s = array( // a static member full of class names
			'DateTime',
			'DOMDocument',
			'ns\\NSMain', // using double-escaped notation, albeit optional
			'ns\\sub\\NSSub'
		);
		public $a = array( // a instance member full of class names
			'DateTime',
			'DOMDocument',
			'ns\\NSMain',
			'ns\\sub\\NSSub'
		);
	}
}

namespace { // global namespace
	use other\T; // bring that test class in here from the third namespace "other", don't need to alias it with "as"
	$t = new T(); // we can instantiate it as if it were local to our namespace
	foreach ( $t->a as $className ) { // iterate through its class member "a"
		$temp = new $className(); // instantiate a class via a variable name (string)
		// dump some info about the class like if it exists, what its full namespaced path is and the object itself
		var_dump(class_exists($className,false),get_class($temp),$temp);
		echo '----';
	}

	$is = $t instanceof T; // check if an object (class instance) is of a specific class type
	var_dump($is);

	$c = new T::$s[2](); // instantiate a variabled class name via a static member with an array index
	var_dump($c);

	// check if a class implements an interface via key lookup
	$implements = class_implements('ns\NSMain'); // get all implementations
	var_dump(isset($implements['ns\INS'])); // check if implemented

}

?>

<?php

// look! multiple namespaces in the same file, couldn't use `namespace ns;` in this case, needed to use block encapsulation.

namespace ns { // declare a namespace block

interface INS { // declare a namespace

}

class NSMain implements INS { // declare a class that uses the interface

}

namespace ns\sub { // declare a secondary namespace under the original one

class NSSub { // and here's an empty class

}

namespace other { // a third namespace (but a second top-level ns)

class T { // the test class

public static $s = array( // a static member full of class names

'DateTime',

'DOMDocument',

'ns\\NSMain', // using double-escaped notation, albeit optional

'ns\\sub\\NSSub'

);

public $a = array( // a instance member full of class names

'DateTime',

'DOMDocument',

'ns\\NSMain',

'ns\\sub\\NSSub'

);

}

namespace { // global namespace

use other\T; // bring that test class in here from the third namespace "other", don't need to alias it with "as"

$t = new T(); // we can instantiate it as if it were local to our namespace

foreach ( $t->a as $className ) { // iterate through its class member "a"

$temp = new $className(); // instantiate a class via a variable name (string)

// dump some info about the class like if it exists, what its full namespaced path is and the object itself

var_dump(class_exists($className,false),get_class($temp),$temp);

echo '----';

}

$is = $t instanceof T; // check if an object (class instance) is of a specific class type

var_dump($is);

$c = new T::$s[2](); // instantiate a variabled class name via a static member with an array index

var_dump($c);

// check if a class implements an interface via key lookup

$implements = class_implements('ns\NSMain'); // get all implementations

var_dump(isset($implements['ns\INS'])); // check if implemented

}

php://stdout

Yes, this time it really is stdout:

boolean true
string 'DateTime' (length=8)
object(DateTime)[2]
  public 'date' => string '2013-11-16 08:44:57' (length=19)
  public 'timezone_type' => int 3
  public 'timezone' => string 'UTC' (length=3)
----
boolean true
string 'DOMDocument' (length=11)
object(DOMDocument)[3]
----
boolean true
string 'ns\NSMain' (length=9)
object(ns\NSMain)[2]
----
boolean true
string 'ns\sub\NSSub' (length=12)
object(ns\sub\NSSub)[3]
----
boolean true
object(ns\NSMain)[2]
boolean true

boolean true

string 'DateTime' (length=8)

object(DateTime)[2]

public 'date' => string '2013-11-16 08:44:57' (length=19)

public 'timezone_type' => int 3

public 'timezone' => string 'UTC' (length=3)

----

boolean true

string 'DOMDocument' (length=11)

object(DOMDocument)[3]

----

boolean true

string 'ns\NSMain' (length=9)

object(ns\NSMain)[2]

----

boolean true

string 'ns\sub\NSSub' (length=12)

object(ns\sub\NSSub)[3]

----

boolean true

object(ns\NSMain)[2]

boolean true

uberSVN: Cheap Backup Script Using hotcopy and rdiff-backup

SVN backups, and backing up in general, is critical for the purpose of disaster recovery. There are a few requirements to consider:

Backups should mirror (or come close to) the current state of system, i.e. a snapshot.
Backups should be incremental. In the event a snapshot is malformed, or the system during the snapshot is corrupt, one can roll back to previously working snapshot.
~~Take~~ Backup only what you need to survive. No sense in including temporary files, libs that are part of the distribution (easily downloadable), etc.

Lower-level note: I don’t have the resources for LVM snapshots, and the physical disks are RAID 6.

Considerations

Originally, I was backing up my uberSVN installation and all of the SVN repositories using a simple rdiff-backup command. This approach is shortsighted: a rsync of the internal repositories directory does not consider current commits, prop changes, hooks, etc. that are occurring while the rsync is happening which could cause a non-restorable backup; using svnadmin hotcopy addresses this concern. However, the issue with hotcopy is that it does not perform an incremental backup for you, so I needed to couple it with rdiff-backup. It is worth noting that performing this type of copy operation will include, props, hooks, commits and other information – it is more comprehensive than a normal svnadmin dump, but with the downside that it is only truly compatible with the version of SVN that generated it.

As if this wasn’t enough to think about, hotcopy does not preserve file system timestamps. This is problematic with rdiff-backup which relies on a timestamp + file size combination; even though it uses the same underlying algorithm as rsync, AFAIK it does not support the checksum feature for transfers. So after the svnadmin hotcopy is performed, file attributes should be synchronized as well (with slight risk I might add).

Lastly, uberSVN has a handful of configurations and metadata that must be backed up per installation. Its GUI has an easy backup feature, but there is no CLI access/equivalent that I could find. I’m sure I could dig through the source and figure out exactly what was included in the backup, but I decided to reverse-engineer the backup file (which uses ZIP compression by the way) and infer other details. uberSVN includes (or should include) the following directories from its install directory (default is /opt/ubersvn): /opt/ubersvn{conf,openssl,tomcat,ubersvn-db}. From this, a backup archive can be carefully constructed for later restoration within the GUI.

The Implementation (tl;dr)

This is a bash script that is configurable in the first two grouping of variables (lines are highlighted). It also uses a subsys lock mechanism (flock) so it cannot be run in parallel which is helpful when using it in a crontab. I haven’t extensively tested its reliability, but it does work… generally speaking. Here’s the hack:

#!/bin/bash

remotePath="/home/backup/ubersvn-repos";
remotePort=22;
remoteUser="backup";
remoteHost="example.com";
targetPrefix="/tmp/svnbackup"; # no trailing slash, where to put it temporarily
repoPrefix="/opt/ubersvn/repositories/"; # with trailing slash, contains all the repos
svnadmin="/opt/ubersvn/bin/svnadmin"; # path to ubersvn's svnadmin binary
rdb="/usr/bin/rdiff-backup"; # path to rdiff-backup binary
lockPath="/var/lock/subsys/backup-svn.sh.lock"; # path to store the lock file

extraPathsPrefix="/opt/ubersvn"; # no trailing slash, $extraPaths will be postfixed
read -r -d '' extraPaths <<EOT
conf
openssl
tomcat
ubersvn-db
EOT

# prepare repo hotcopy paths
timestamp=`date +"%s"`;
targetTemp="${targetPrefix}_${timestamp}";
repoList=`find "$repoPrefix" -maxdepth 1 -type d -not -wholename "$repoPrefix"`;

(
flock -x -w 10 200 || exit 1

# hotcopy all svn repos
for repoPath in $repoList; do
	repo=`echo "$repoPath" | rev | cut -d/ -f1 | rev`;
	tempRepoPath="${targetTemp}/repositories/${repo}";
	mkdir -p "$tempRepoPath";
	"$svnadmin" hotcopy --clean-logs "$repoPath" "$tempRepoPath";
done;

# sync file attributes (i.e. timestamps) to prepare incremental backup, this is kinda dirty
cd "${targetTemp}/repositories/";
find . -exec touch -r "$repoPrefix"\{\} \{\} \;
cd - >/dev/null 2>&1;

# copy extra paths (e.g. ubersvn configs)
for extra in $extraPaths; do
	/bin/cp -fRp "${extraPathsPrefix}/${extra}" "${targetTemp}/${extra}"; # preserve attributes
done;

"$rdb" --remote-schema "ssh -C -p $remotePort %s" "$targetTemp" "$remoteUser"@"$remoteHost"::"$remotePath"; # incremental backup

rm -fr "$targetTemp"; # clean up

) 200>"$lockPath";

#!/bin/bash

remotePath="/home/backup/ubersvn-repos";

remotePort=22;

remoteUser="backup";

remoteHost="example.com";

targetPrefix="/tmp/svnbackup"; # no trailing slash, where to put it temporarily

repoPrefix="/opt/ubersvn/repositories/"; # with trailing slash, contains all the repos

svnadmin="/opt/ubersvn/bin/svnadmin"; # path to ubersvn's svnadmin binary

rdb="/usr/bin/rdiff-backup"; # path to rdiff-backup binary

lockPath="/var/lock/subsys/backup-svn.sh.lock"; # path to store the lock file

extraPathsPrefix="/opt/ubersvn"; # no trailing slash, $extraPaths will be postfixed

read -r -d '' extraPaths <<EOT

conf

openssl

tomcat

ubersvn-db

EOT

# prepare repo hotcopy paths

timestamp=`date +"%s"`;

targetTemp="${targetPrefix}_${timestamp}";

repoList=`find "$repoPrefix" -maxdepth 1 -type d -not -wholename "$repoPrefix"`;

(

flock -x -w 10 200 || exit 1

# hotcopy all svn repos

for repoPath in $repoList; do

repo=`echo "$repoPath" | rev | cut -d/ -f1 | rev`;

tempRepoPath="${targetTemp}/repositories/${repo}";

mkdir -p "$tempRepoPath";

"$svnadmin" hotcopy --clean-logs "$repoPath" "$tempRepoPath";

done;

# sync file attributes (i.e. timestamps) to prepare incremental backup, this is kinda dirty

cd "${targetTemp}/repositories/";

find . -exec touch -r "$repoPrefix"\{\} \{\} \;

cd - >/dev/null 2>&1;

# copy extra paths (e.g. ubersvn configs)

for extra in $extraPaths; do

/bin/cp -fRp "${extraPathsPrefix}/${extra}" "${targetTemp}/${extra}"; # preserve attributes

done;

"$rdb" --remote-schema "ssh -C -p $remotePort %s" "$targetTemp" "$remoteUser"@"$remoteHost"::"$remotePath"; # incremental backup

rm -fr "$targetTemp"; # clean up

) 200>"$lockPath";

Scripting Parallel Bash Commands (Jobs)

Sometimes commands take a long time to process (in my case importing huge SQL dumps into a series of sandboxed MySQL databases), in which case it may be favorable to take advantage of multiple CPUs/cores. This can be handled in shell/bash with the background control operator & in combination with wait. Got a little insight from this StackOverflow answer.

The Technique

The break-down is commented inline.

#!/bin/bash

processes=4; # max number of parallel processes

#list of stuff
read -r -d '' theList <<EOT
big
long
list
of
args
or
commands
to
work
with
EOT

count=0;
pidList='';
for item in $theList; do
	count=$((count + 1));
	longOperationWith "$item"&
	lastPid=${!}; # get the PID of `longOperationWith "$item"`
	pidList=`echo "$pidList $lastPid" | sed 's/^ *//g'`; # concat and strip that initial beginning space
	if [ $count -ge $processes ]; then # when we've reached the max number of processes
		echo "Waiting for $pidList to finish...";
		wait $pidList; # wait for all PIDs in pidList to finish (don't quote)
		count=0; # then reset counter
		pidList=''; # and reset the list of PIDs
	fi;
done;

#!/bin/bash

processes=4; # max number of parallel processes

#list of stuff

read -r -d '' theList <<EOT

big

long

list

args

commands

work

with

EOT

count=0;

pidList='';

for item in $theList; do

count=$((count + 1));

longOperationWith "$item"&

lastPid=${!}; # get the PID of `longOperationWith "$item"`

pidList=`echo "$pidList $lastPid" | sed 's/^ *//g'`; # concat and strip that initial beginning space

if [ $count -ge $processes ]; then # when we've reached the max number of processes

echo "Waiting for $pidList to finish...";

wait $pidList; # wait for all PIDs in pidList to finish (don't quote)

count=0; # then reset counter

pidList=''; # and reset the list of PIDs

fi;

done;

Notes

This does not function like a pool (task queue) where once a “thread” is freed up it will be immediately eligible for the next task, although I don’t see why something like this could be implemented with a little work.
wait will pause script execution for all PID parameters it is provided to complete before moving on.
Once the if...fi control block is entered wait will cause the for...in loop to suspend.
$! (or ${!}) contains the PID of the most previously executed command; make sure it comes directly after the operation used with &. Throw it into a variable (like lastPid) for future use.
This is not multi-threading, although this simplified concept is similar. Each command spawns a separate process.
read is just creating a multiline list; in my specific case this was favorable over a bash array, but either will work.