Eikosstatistics & data science

Advanced & Applied Topics · Information Theory & Entropy

Information theory: Principles of modeling, model selection, and statistical learning

Public introduction · full guide in Eikos

Introduction

What this guide covers

Introduction Multivariate Analysis and Mathematical Theory

Overview

Topics covered

  • Introduction
  • Information Theory in Statistical Learning
  • Introduction to Information Theory in Statistical Modeling
  • 1.1 Structure of Academic Content
  • Shannon Entropy and Differential Entropy
  • 2.1 Shannon's Axiomatic Derivation

Related concepts

Main connections

  • Maximum likelihood estimation
  • Calculus & differentiation
  • Linear algebra & matrices
  • Bayes' theorem & priors
  • Normal (Gaussian) distribution
  • Regularization (Lasso / Ridge / Elastic Net)

Compatible notation

Mathematical sample in LaTeX

The expression is rendered accessibly, with its LaTeX source preserved below.

H(X)=𝔼[I(P(X))]=−∑x∈𝒳p(x)logp(x)
View LaTeX sourceH(X) = \mathbb{E}[I(P(X))] = -\sum_{x \in \mathcal{X}} p(x) \log p(x)