course-deatils-thumbnail

Developing with Spark for Big Data| Enterprise-Grade Spark Programming for the Hadoop & Big Data Ecosystem

Course Code: 1050
|

$2895

|

Overview

The Developing with Spark for Big Data course introduces participants to enterprise-grade Spark programming, covering intermediate-level and advanced-level concepts of Spark programming, and enabling participants to work with the key components of Apache Spark for developing data science solutions. The course equips participants with the skills and knowledge to work with Apache Spark in real-world enterprises and execute effective data-driven decisions. The course involves various hands-on exercises to ensure that participants have a thorough understanding of all the concepts covered in the course.

Schedule Classes

Looking for more sessions of this class?
Talk to us
Trivera

Course Delivery

This course is available in the following formats:

Live Classroom
Duration: 5 days

Live Virtual Classroom
Duration: 5 days

What You'll learn

  • Basics of Spark architecture and applications
  • Executing Spark programs
  • Creating and manipulating both RDDs (Resilient Distributed Datasets) and UDFs (Unified Data Frames)
  • Restoring data frames
  • Essential NoSQL access
  • Integrating machine learning into Spark applications
  • Sing Spark streaming and Kafka to create streaming applications

Outline

Module 1: Overview of Spark

  • Hadoop ecosystem
  • Hadoop YARN vs. Mesos
  • Spark vs. MapReduce
  • Spark with MapReduce – Lambda Architecture
  • Spark in the enterprise data science architecture

Module 2: Spark component overview

  • Spark Shell
  • RDDs: Resilient Distributed Datasets
  • Data frames
  • Spark 2 unified DataFrames
  • Spark sessions
  • Functional programming
  • Spark SQL
  • MLib
  • Structured streaming
  • Spark R
  • Spark and Python

Module 3: RDDs: Resilient Distributed Datasets

  • Coding with RDDs
  • Transformation
  • Actions
  • Lazy evaluation and optimization
  • RDDs n MapReduce

Module 4: DataFrames

  • RDDs vs. DataFrames
  • Unified DataFrames (UDF) in Spark 2.0
  • Partitioning

Module 5: Spark Applications

  • Spark sessions
  • Running applications
  • Logging

Module 6: DataFrame persistence

  • RDD persistence
  • DataFrame and Unified DataFrame persistence
  • Distributed Persistence

Module 7: Spark streaming

  • Streaming overview
  • Streams
  • Structured streaming
  • DStreams and Apache Kafka

Module 8: Accessing NoSQL Data

  • Ingesting data
  • Parquet files
  • Relational databases
  • Graph databases (Neo4J, GraphX)
  • Interacting with Hive
  • Accessing Cassandra data
  • Document databases (MongoDB, CouchDB)

Module 9: Enterprise Integration

  • MapReduce and Lambda integration
  • Camel integration
  • Drools and Spark

Module 10: Algorithms and Patterns

  • MLib and Mahout
  • Classification
  • Clustering
  • Decision trees
  • Decompositions
  • Pipelines
  • Spark packages

Module 11: Spark SQL

  • Spark SQL
  • SQL and DataFrames
  • Spark SQL and Hive
  • Spark SQL and JDBC

Module 12: GraphX

  • Graph APIs
  • GraphX
  • ETL in GraphX
  • Exploratory analysis
  • Graph computation
  • Pregel APi overview
  • GraphX algorithms
  • Neo4J as an alternative

Module 13: Alternate languages

  • Using web notebooks (Zeppelin, Jupyter)
  • R on Spark
  • Python on Spark
  • Scala on Spark

Module 14: Clustering Spark for Developers

  • Parallelizing Spark applications
  • Clustering concerns for developers

Module 15: Performance and Tuning

  • Monitoring Spark performance
  • Tuning memory
  • Tuning CPU
  • Tuning Data Locality
  • Troubleshooting
View More

Prerequisites

Participants need to have experience working in a development role and have an understanding of the Big Data and Hadoop ecosystem.

Who Should Attend

The course is highly recommended for –

  • Developers
  • Architects
  • Big Data professionals
  • Hadoop professionals

Interested in this course? Let’s connect!

Related Courses

Modeling Techniques for Business Analysts
Modeling Techniques for Business Analysts
Introduction to Data Analysis
Introduction to Data Analysis
Developing SQL data models
Developing SQL data models
Red Hat Boss Data Virtualization Development
Red Hat JBoss Data Virtualization Development

Popular Courses

ITIL 4 foundation
ITIL® 4 Foundation
ITIL® Foundation (v3, 2011)
Certified Information Systems Security Professional (CISSP)
Certified Information Systems Security Professional (CISSP)
Mastering Python Programming
Mastering Python Programming

Recent Posts

What is Docker and Kubernetes?

October 20, 2020

Getting started with learning DevOps

October 16, 2020

What is Automation Anywhere?

October 15, 2020

Building a career as an AWS Cloud Solutions Architect

September 14, 2020