17 Mar 2004
CS 3243 - Learning
33
Information Content
nEntropy measures purity of sets of examples
¡Normally denoted H(x)
nOr as information content: the less you need to know (to determine class of new case), the more information you have
nWith two classes (P,N):
¡IC(S) = - (p/t) log2 (p/t)  - (n/t) log2 (n/t)
¡E.g., p=9, n=5;
IC([9,5]) = - (9/14) log2 (9/14) - (5/14) log2 (5/14)
¡ = 0.940
¡Also, IC([14,0])=0; IC([7,7])=1
T (total) = P + N