Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• Overview of the Big Data ecosystem and Spark's role in contemporary data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining appropriate use cases for each
• Creating and configuring SparkSession, along with understanding the fundamentals of application configuration

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data processing
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods in relation to Spark driver and executor processes, with a conceptual focus for cluster users rather than administrators
• Examining the execution of Spark jobs on Kubernetes: pod scheduling, resource requests and limits, and the mapping of session settings to these resources
• Distinguishing between parameters controlled via Spark session settings and those managed by the platform team through cluster provisioning, and why this distinction is crucial for tuning
• Practical Exercise: Examine a live Spark-on-Kubernetes job within the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Advanced Analysis

• Analysing executor count versus executor size: memory-versus-cores trade-offs and logical reasoning strategies
• Driver and executor memory allocation, memory overhead, and their translation into pod resources
• Dynamic allocation: behaviour on Kubernetes, scenarios where it conserves resources, and limitations
• Practical Exercise: Execute the same job with varying executor and core configurations to compare runtime and resource utilisation

Module 5: Scaling Dynamics & Cost Optimisation

• Understanding the impact of adding or removing nodes on job behaviour, completion time, and resource consumption
• Comparing numerous small executors against fewer large ones: performance and cost trade-offs
• Shuffle mechanics and partition tuning, including assessing the cost implications of configuration choices
• Practical Exercise: Scale the sandbox cluster from five to ten nodes to observe effects on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Investigating the small-files problem: how sources split into numerous 1–5 MB Parquet files degrade performance and disrupt partitioning
• Strategies for repartitioning and coalescing
• Controlling partition sizes during read and write operations
• Writing Parquet data efficiently to prevent downstream small-file issues
• Practical Exercise: Load a dataset consisting of many small Parquet partitions, apply various repartitioning strategies, and compare pre- and post-optimisation performance

Module 7: Pandas Memory Management & Failure Analysis

• Identifying root causes of out-of-memory errors in Pandas and recognising diagnostic symptoms
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when writing large datasets to CSV
• Utilising chunked processing and dtype optimisation in resource-constrained environments
• Practical Exercise: Replicate a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance characteristics, lazy evaluation, and memory behaviour
• Understanding the role of Polars alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Practical Exercise: Rewrite a Pandas-heavy transformation using Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory management principles across a realistic ETL pipeline
• Understanding optimisation considerations specific to machine learning workloads on the same cluster
• Implementing a practical tuning workflow to systematically diagnose cost and performance issues
• Practical Exercise: Complete an end-to-end mini-project combining data loading, transformation, and a simple model-training step, with participants independently tuning the Spark configuration