Huffman Codes and Greedy Algorithms

• Huffman Codes
  • An encoding for an alphabet
  • Reduces the length of the encoding for frequently used characters.
  • Input: alphabet, frequency distribution for the letters in the alphabet.
  • Output: an encoding
  • Algorithm:

    Huffman (C,f)
    (1)   n = |C|
    (2)   foreach c in C 
    (3)      L.PushBack(c)
    (4)   for i= 1 to n-1
    (5)      z = newnode
    (6)      z.leftchild =  ExtractMin(L)
    (7)      z.rightchild = ExtractMin(L)
    (8)      f(z) = f(z.leftchild) + f(z.rightchild)
    (9)      L.PushBack(z)
    	  

  • Analysis:

    The timing really depends on the cost of inserting
    into the list and extracting the min from the list
    (6,7,9)
    
    With a list these are  all O(n), and the outer loop
    makes this an O(n2) Algorithm.
    	  

  • Some comments on timing
    • maintaining an ordered list would help with line 6,7
    • Just a list (unordered, insert at front or back) would help with lines 9
    • Inserting into a binary search tree would help the average case
• Greedy Algorithms
  • Huffman's algorithm is an example of an optimization where a set of choices need to be made to arrive at an optimal (or best) solution
  • An optimization problems of this type can become exponential with a niave algorithm.
  • Think of trying to build all trees and see which is optimal
  • A Greedy Algorithm is an algorithm where we always make the "best" choice at each step and hope that it yields an optimal solution
  • In Huffman's algorithm, we always selected the least used sub tree to put lowest in the next generation of trees.
  • Or we tried to minimize bits * frequency.
  • A greedy technique does not always lead to an optimal solution
  • The proof that a greedy algorithm is correct (optimal) involves
    • Greedy Choice Property - Locally optimal choices lead to a gloobally optimal solution
    • Optimal Substructure - The optimal solution to a problem contains optimal substructures.
  • The proof for huffman's algorithm consists of two parts
    • Argue that the greedy choice propperty holds
      • This involves stating that if we have an optimal tree, not the one produced by the algorithm, the nodes in (6) and (7) can can be traded with the two lowest nodes in the tree to produce a tree of the same cost.
    • Argue that the sub problems are optimal
      • Show that replacing two lettters with a single letter with frequency sum of the two previous frequencies will lead to a optimal code for the new alphabet.
      • This involves a proof by contradiction by assuming the new tree is not optimal, and showing that this implys the old tree is not optimal either.
• A few notes
  • This is by no means a complete coverage of this topic
  • It is more of a brief introduction