The Mathematical Definitions of Big-O Big-Θ Big-Ω

Notes

• This is chapter 3 of the book
• We have been struggling to find a way to measure algorithm efficiency that is
  • Timeless
  • Does not depend on hardware
  • Scales as input size increases
• History
  • Wikipedia
  • The idea of minimizing operations in an "algorithm" apparently goes back to antiquity
    • In Discrete did you learn Euclid's Algorithm for finding the GCD of two positive integers.
    • This was described by Euclid in Elements about 300 BCE.

    • GCD(a,b)
      
       if a < b
          swap(a,b)
       while b != 0
          t = b
          b = a % b
          a = t
       return a

    • Turns out, that the above algorithm was not what Euclid described, this is an optimization from the 1800s
    • Wikipedia claims that "this marks the beginning of computational complexity theory" (9/2/2026, Euclidian_algorithm page)
  • The notation we use was proposed by mathematicians in 1844 and revised and extended
  • The systematic studies of computational complexity started in the 1960's
• We are primarily concerned about the worst-case running time for an algorithm.
  • This is given by O(f(n)) where n is the size of the input.
• Formally O(f(n)) is a set of functions
  • $O(g(n)) = \{ f(n) : \text{ there exists constants } c > 0, n_0 > 0 \text{ such that } 0 \le f(n) \le c \times g(n) \text{ for all } n \ge n_0 \}$
• Notation
  • We really should say that f(n) ∈ O(g(n))
  • But we frequently say f(n) = O(g(n))
  • Or even f(n) is O(g(n))
• But what does this mean?
  • Consider an algorithm that we decide has performance f(n) = 34n + 23
  • What is the 34 likely to be?
  • What is the 23 likely to be

  • setup                             //possibly 12 lines of code
    while some comparison based on n  //possibly 3 lines of code in loop, one out
       computation                    //possibly 31 lines of code including loop maintenance
    exit/cleanup/final computation    //possibly 10 lines of code 

  • Is f(n) ∈ O(n) , ie g(n) = n
    • Can we find a positive constant c, (how about 35) and a value of n₀ (how about 23)
    • is 35n ≥ 34n + 23 when n ≥ 23?
      • Subtract 34n from both sides
      • n ≥ 23
• That example is kinda yucky to compute graph, so let's try
  • Is f(n) = 3n + 2 ∈ O(n)?
  • First let's pick an easy constant for c, like 4
  • Then 4n > 3n + 2, subtract 3n from both sides, so n > 2
  • Let n₀ = 2
  • Yes, 3n + 2 ∈ O(n)
  • Graph this on desmos.com/calculator
• In general, c₁n + c₂ ∈ O(n)
  • c₁ is positive, or it doesn't make sense
  • So pick something bigger
  • Solve for n₀
• We really don't want to compute c and n₀ from the definition.
• What this definition really allows us to do is
  • Throw away everything but the dominant term in the timing equation
  • Throw away the constant coefficient of the dominant term.
• In the end, what does this really measure?
  • if something is O(n) what does the code do?
  • How about O(n²)?
  • An algorithm's performance is a× n² + b×n
    • What does this code look like?
• Common g(n) functions
  • log₂(n) or lg(n)
  • n
  • n lg(n)
  • n^c where c is a constant.
  • 2ⁿ
  • nⁿ
  • n!
  • Graph each of these at desmon.com/calculator
• A strange result of the definition
  • n ∈ O(n²)
  • n³ ∈ O(n!)
  • This is not a tight upper bound, it is just an upper bound.