Machine Learning & Artificial Intelligence · Big Data & Distributed Computing

Big Data · In-memory distributed processing with Apache Spark — Architecture and optimization fundamentals

Public introduction · full guide in Eikos

Introduction

What this guide covers

Introduction Chart from the original guide Big Data · Distributed Systems Engineering

Overview

Topics covered

  • Introduction
  • In-Memory Processing with Apache Spark
  • Distributed computing paradigm and the evolution of Spark
  • 1.1 Analytical comparison of architectures
  • System architecture and execution topology
  • Driver Master

Related concepts

Main connections

  • Preprocessing pipelines & transform recipes
  • SQL & relational data modeling
  • Directed acyclic graphs (DAGs)
  • Mean & expected value
  • Calculus & differentiation
  • Principal component analysis (PCA)

Compatible notation

Mathematical sample in LaTeX

The expression is rendered accessibly, with its LaTeX source preserved below.

TMapReduce=O(I⋅(TI/O\_disk+TRed+TCPU))
View LaTeX sourceT_{\text{MapReduce}} = O\left(I \cdot (T_{\text{I/O\_disk}} + T_{\text{Red}} + T_{\text{CPU}})\right)