Bayes Rule

Bayes' Rule

	Product rule P(aÙb) = P(a | b) P(b) = P(b | a) P(a)
	Þ Bayes' rule: P(a | b) = P(b | a) P(a) / P(b)
	or in distribution form
	P(Y|X) = P(X|Y) P(Y) / P(X) = αP(X|Y) P(Y)
	Useful for assessing diagnostic probability from causal probability:
		P(Cause|Effect) = P(Effect|Cause) P(Cause) / P(Effect)
		E.g., let M be meningitis, S be stiff neck:
			P(m|s) = P(s|m) P(m) / P(s) = 0.5 × 0.0002 / 0.05 = 0.0002
		Note: posterior probability of meningitis still very small!

Bayes' Rule and conditional independence

	P(Cavity | toothache Ù catch)
		= α · P(toothache Ù catch | Cavity) P(Cavity)
		= α · P(toothache | Cavity) P(catch | Cavity) P(Cavity)
	This is an example of a naïve Bayes model:
		P(Cause,Effect1, … ,Effectn) = P(Cause) πiP(Effecti|Cause)
	Total number of parameters is linear in n

Naïve Bayes Classifier

	Calculate most probable function value
		Vmap = argmax P(vj| a1,a2, … , an)
		= argmax P(a1,a2, … , an| vj) P(vj)
		P(a1,a2, … , an)
		= argmax P(a1,a2, … , an| vj) P(vj)
		Naïve assumption: P(a1,a2, … , an) = P(a1)P(a2) … P(an)

Naïve Bayes Algorithm

	NaïveBayesLearn(examples) For each target value vj    P’(vj) ← estimate P(vj)    For each attribute value ai of each attribute a       P’(ai|vj) ← estimate P(ai|vj)
	ClassfyingNewInstance(x) vnb= argmax P’(vj) Π P’(ai|vj)

An Example

	(due to MIT’s open coursework slides)

An Example

	(due to MIT’s open coursework slides)