KOVIKI

Apache Spark's runtime architecture is designed to efficiently process and analyze large-scale distributed data across clusters. The architecture involves several key components that work together to execute Spark applications. Here's an overview of the Spark runtime architecture:

Driver Program

- The driver program is the entry point of a Spark application. It contains the application's main function and defines the high-level computation logic.
- The driver program communicates with the cluster manager to acquire resources and launch tasks on worker nodes.

Cluster Manager

- The cluster manager is responsible for managing the allocation of resources (CPU, memory) and scheduling tasks across the cluster.
- Common cluster managers include Apache Mesos, Hadoop YARN, and Kubernetes.

Spark Application Master (AM)(for YARN and Mesos)

- In YARN or Mesos deployments, the Spark AM is a separate process that coordinates the execution of Spark tasks on worker nodes.
- It negotiates resources from the cluster manager and monitors task execution.

Executor

- Executors are worker processes that run on cluster nodes and perform actual computation.
- Each executor manages its own memory and CPU resources.
- Executors are responsible for running tasks and storing cached data in memory.

Task

- A task represents a unit of work that can be executed on a single partition of data.
 - Tasks are created by the driver program and sent to executors for execution.
- Each task processes a subset of data and can perform transformations, aggregations, etc.

Resilient Distributed Dataset (RDD)

- RDDs are the core data abstraction in Spark, representing distributed collections of data.
- RDDs are partitioned and can be cached in memory to allow for efficient data processing.
- Transformations and actions are applied to RDDs to perform computations.

DAG Scheduler

 - The DAG (Directed Acyclic Graph) scheduler constructs a logical execution plan by analyzing the high-level transformations in the application code.
 - It breaks down the application into stages of tasks based on data dependencies.

Stage

   - A stage is a collection of tasks that can be executed in parallel without data shuffling (map tasks).
   - Stages are determined based on narrow transformations that do not require data movement.

Shuffle Manager

 - During wide transformations (transformations requiring data exchange between partitions), data shuffling occurs.
 - The shuffle manager handles the movement of data across partitions and nodes.

Block Manager

- The block manager is responsible for managing data blocks in memory, cached on disk, or shuffled across the cluster.
- It ensures data locality and efficient data access.

Catalyst Optimizer

- The Catalyst optimizer optimizes query plans, including DataFrame operations, to improve performance.
- It applies rule-based and cost-based optimizations to generate efficient execution plans.

Tungsten Execution Engine

- The Tungsten execution engine optimizes memory management and CPU usage to improve the efficiency of Spark applications.

These components work together to achieve fault tolerance, data locality, parallelism, and optimization in Spark applications. By distributing tasks across a cluster, Spark can process large volumes of data efficiently and provide high-level abstractions for various data processing tasks.