Introduction
PmWiki's markup translation engine is handled by a set of rules; each rule searches for a specific pattern in the markup text and replaces it with some replacement text. Internally, this is accomplished by using PHP's "preg_replace" function.
Rules are added to the translation engine via PmWiki's Markup() function, which looks like
Markup($name, $when, $pattern, $replace);
where $name is a unique name (a string) given to the rule, $when says when the rule should be applied relative to other rules, $pattern is the pattern to be searched for in the markup text, and $replace is what the pattern should be replaced with.
For example, here's the code that creates the rule for ''emphasized text'' (in scripts/stdmarkup.php):
Markup("em", "inline", "/''(.*?)''/", "<em>$1</em>");
Basically this statement says to create a rule called "em" to be performed with the other "inline" markups, and the rule replaces any text inside two pairs of single quotes with the same text 
match not begin/end of strings but also begin/end of lines (i.e., right before/after a newline).
How do I get started writing recipes and creating my own custom markup?
(alternate) Introduction to custom markup for Beginners
This page may have a more recent version on pmwiki.org: PmWiki:CustomMarkup, and a talk page: PmWiki:CustomMarkup-Talk.