Borel Sets and Kolmogorov Axioms

This is really foundational, but a foundation is a good place to start building. To define probabilities, we need something to define them on. Such structures are known by two names: sigma algebras and Borel sets. If S is a set, then a family of subsets of S is a Borel set if:

1. The empty set is included.
2. The family is closed under complements (which, combined with #1, means that the full set S is also included).
3. The family is closed under countable unions (which, with #2 and DeMorgan's Laws, implies it's also closed under countable intersections).

In the case where S is finite or countably infinite, the most obvious Borel set is the set of all subsets of S. When S is uncountable (for example, the real number line), things get a bit stickier. Fortunately, most probability questions are interested in ranges of the real line, so a Borel set that includes all intervals of the form [a, b], [a, b), (a, b], and (a, b) does nicely.

The point of defining the Borel set B for a sample space is that we can then define a measure P that maps the Borel set to [0,1]. To be a probability measure, P needs to meet the Kolmogorov Axioms:

1. P(A) ≥ 0 for all A ∈ B.
2. P(S) = 1.
3. if A₁, ..., A_n ∈ B are pairwise disjoint, then P(∪A_i) = ∑P(A_i)

Obviously, any finite measure P on B can be converted to a probability measure by simply norming it to 1. That is, if P is a measure on B and P(S) is finite, then  P'(A) = P(A) / P(S) is a probability measure on B. This fact turns out to be extremely useful in Bayesian statistics since the marginal likelihood is often difficult to compute but, as long as we know it's finite, we can often ignore it and then norm the resulting posterior after the fact.