Best Big Data Training in Chennai

Introduction to Apache Hadoop and the Hadoop Ecosystem

Apache Hadoop Overview

Data Ingestion and Storage

Data locality

Data Analysis and Exploration

Other Ecosystem Tools

Ubuntu 14.04 LTS Installation through VMware Player

Installing Hadoop 2.7.1 on Ubuntu 14.04 LTS (Single-Node Cluster)

Apache Spark Installation

Jdk -8 Installation

Scala Installation

SBT Installation

Why we need HDFS

Apache Hadoop Cluster Components

HDFS Architecture

Failures of HDFS 1.0

Reading and Writing Data in HDFS

Fault tolerance

Overview and Architecture of Map Reduce

Components of MapReduce

How MapReduce works

Flow and Difference of MapReduce Version

YARN Architecture

Hive Installation on Ubuntu 14.04 With MySQL Database Metastore

Hive Overview and Architecture

Hive command execution in shell and HUE

Hive Data Loading methods

Hive Partition and Bucketing

External and Managed tables in Hive

File formats in Hive

Hive Joins

Serde in Hive

Apache Sqoop Overview and Architecture

Apache Sqoop Import Examples

Apache Sqoop Export Examples

Sqoop Incremental load

Functional Programing Vs Object Orient Programing

Scala Overview

Configuring Apache Spark with Scala

Variable Declaration

Operations on variables

Conditional Expressions

Pattern Matching

Iteration

Scala Functions

Scala Oops Concept

Scala Abstract Class & Traits

Scala Access Modifier

Scala Array and String

Scala Exceptions

Scala Collections

Scala Tuples

Scala File handling

Scala Multithreading

Spark Ecosystem

Scala File handling

Introduction and Setting up of Scala

Setup Scala on Windows

Basic Programming Constructs

Functions

Object Oriented Concepts - Classes

Object Oriented Concepts - Objects

Object Oriented Concepts - Case Classes

Collections - Seq, Set and Map

Basic Map Reduce Operations

Setting up Data Sets for Basic I/O Operations

Basic I/O Operations and using Scala Collections APIs

Tuples

Development Cycle - Developing Source code

Development Cycle - Compile source code to jar using SBT

Development Cycle - Setup SBT on Windows

Development Cycle - Compile changes and run jar with arguments

Development Cycle - Setup IntelliJ with Scala

Development Cycle - Develop Scala application using SBT in IntelliJ

Setup Environment - Locally

Setup Environment - using Cloudera QuickStart VM

Using Windows - Putty and WinSCP

Using Windows - Cygwin

HDFS Quick Preview

YARN Quick Preview

Setup Data Sets

What is Apache Spark?

Starting the Spark Shell

Using the Spark Shell

Getting Started with Datasets and Data Frames

Data Frame Operations

Apache Spark Overview and Architecture

RDD Overview

RDD Data Sources

Creating and Saving RDDs

RDD Operations

Transformations and Actions

Converting Between RDDs and Data Frames

Key-Value Pair RDDs

Map-Reduce operations

Other Pair RDD Operations

Quick overview about Spark documentation

Initializing Spark job using spark-shell

Create Resilient Distributed Data Sets (RDD)

Previewing data from RDD

Reading different file formats - Brief overview using JSON

Transformations Overview

Manipulating Strings as part of transformations using Scala

Row level transformations using map

Row level transformations using flat Map

Filtering the data

Joining data sets - inner join

Joining data sets - outer join

Aggregations - Getting Started

Aggregations - using actions (reduce and countByKey)

Aggregations - understanding combiner

Aggregations using groupByKey - least preferred API for aggregations

Aggregations using reduceByKey

Aggregations using aggregateByKey

Sorting data using sortByKey

Global Ranking - using sortByKey with take and takeOrdered

By Key Ranking - Converting (K, V) pairs into (K, Iterable[V]) using groupByKey

Get topNPrices using Scala Collections API

Get topNPricedProducts using Scala Collections API

Get top n products by category using groupByKey, flatMap and Scala function

Set Operations - union, intersect, distinct as well as minus

Save data in Text Input Format

Save data in Text Input Format using Compression

Saving data in standard file formats - Overview

Revision of Problem Statement and Design the solution

Creating Data Frames from Data Sources

Saving Data Frames to Data Sources

Data Frame Schemas

Eager and Lazy Execution

Querying Data Frames Using Column Expressions

Grouping and Aggregation Queries

Joining Data Frames

Querying Tables, Files, Views in Spark Using SQL

Comparing Spark SQL and Apache Hive-on-Spark

Creating Datasets

Loading and Saving Datasets

Dataset Operations

Writing a Spark Application

Building and Running an Application

Application Deployment Mode

The Spark Application Web UI

Configuring Application Properties

RDD Partitions

Example: Partitioning in Queries

Stages and Tasks

Job Execution Planning

Example: Catalyst Execution Plan

Example: RDD Execution Plan

Data Frame and Dataset Persistence

Persistence Storage Levels

Viewing Persisted RDDs

Difference between RDD, Data frame and Dataset

Common Apache Spark Use Cases

Different interfaces to run Hive queries

Create Hive tables and load data in text file format

Create Hive tables and load data in ORC file format

Using spark-shell to run Hive queries or commands

Functions - Getting Started

Functions - Manipulating Strings

Functions - Manipulating Dates

Functions - Aggregations

Functions - CASE

Row level transformations

Joins

Aggregations

Sorting

Set Operations

Analytics Functions - Aggregations

Analytics Functions - Ranking

Windowing Functions

Create Data Frame and Register as Temp table

Writing Spark SQL Applications - process data

Writing Spark SQL Applications - Save data into Hive tables

Data Frame Operations

Introduction to Flume & features

Flume topology & core concepts

Flume Agents: Sources, Channels and Sinks

Property file parameters logic

Apache Kafka Installation

Apache Kafka Overview and Architecture

Consumer and Producer

Deploying Kafka in real world business scenarios

Integration with Spark for Spark Streaming

Introduction to zookeeper concepts

Overview and Architecture of Zookeeper

Zookeeper principles & usage in Hadoop framework

Use of Zookeeper in Hbase and Kafka

Oozie Fundamentals

Oozie workflow creations

Concepts of Coordinates and Bundles