Aspyrin#21005
Total Leech = Total Physical Damage * (Leech%) * (Penalty) * (Drain Effectiveness)
The formula is the same for both life leech and mana leech. Penalty is 50% in Nightmare and 33% in Hell. Drain Effectiveness is a monster-specific stat that affects how much you can drain from them (think of it like leech resistance). For instance, Skeletons have a Drain Effectiveness of 0, which means they cannot be leeched from, no matter what.
As an example... if you're dealing 800 damage in Hell against a monster with no drain resistance, and you have 12% Life and 5% mana leech, you will leech:
800 * (.12) * (.33) * (1) = ~32 life and
800 * (.05) * (.33) * (1) = ~13 mana.