Best Big Data Training in Chennai

Introduction to Apache Hadoop and the Hadoop Ecosystem

Apache Hadoop Overview

Data Ingestion and Storage

Data locality

Data Analysis and Exploration

Other Ecosystem Tools

Ubuntu 14.04 LTS Installation through VMware Player

Installing Hadoop 2.7.1 on Ubuntu 14.04 LTS (Single-Node Cluster)

Apache Spark Installation

Jdk -8 Installation

Scala Installation

SBT Installation

Why we need HDFS

Apache Hadoop Cluster Components

HDFS Architecture

Failures of HDFS 1.0

Reading and Writing Data in HDFS

Fault tolerance

Overview and Architecture of Map Reduce

Components of MapReduce

How MapReduce works

Flow and Difference of MapReduce Version

YARN Architecture

Hive Installation on Ubuntu 14.04 With MySQL Database Metastore

Hive Overview and Architecture

Hive command execution in shell and HUE

Hive Data Loading methods

Hive Partition and Bucketing

External and Managed tables in Hive

File formats in Hive

Hive Joins

Serde in Hive

Apache Sqoop Overview and Architecture

Apache Sqoop Import Examples

Apache Sqoop Export Examples

Sqoop Incremental load

Introduction and Setting up of Python

Basic Programming Constructs

Functions in Python

Python Collections

Map Reduce operations on Python Collections

Setting up Data Sets for Basic I/O Operations

Basic I/O operations and processing data using Collections

Get revenue for given order id - as application

Setup Environment - Locally

Setup Environment - using Cloudera QuickStart VM

Using Windows - Putty and WinSCP

Using Windows - Cygwin

HDFS Quick Preview

YARN Quick Preview

Setup Data Sets

Introduction

Introduction to Spark

Setup Spark on Windows

Quick overview about Spark documentation

Connecting to the environment

Initializing Spark job using pyspark

Create RDD from HDFS files

Create RDD from collection - using parallelize

ead data from different file formats - using sqlContext

Row level transformations - String Manipulation

Row level transformations using map

Row level transformations using flatMap

Filtering the data

Joining Data Sets - Introduction

Joining data sets - inner join

Joining data sets - outer join

Aggregations - Introduction

Aggregations - count and reduce - Get revenue for order id

Aggregations - reduce - Get order item with minimum subtotal for order id

Aggregations - countByKey - Get order count by status

Aggregations - understanding combiner

ggregations - groupByKey - Get revenue for each order id

groupByKey - Get order items sorted by order_item_subtotal for each order id

Aggregations - reduceByKey - Get revenue for each order id

Aggregations - aggregateByKey - Get revenue and count of items for each order id

Sorting - sortByKey - Sort data by product price

Sorting - sortByKey - Sort data by category id and then by price descending

Ranking - Introduction

Ranking - Global Ranking using sortByKey and take

Ranking - Global using takeOrdered or top

Ranking - By Key - Get top N products by price per category - Introduction

Ranking - By Key - Get top N products by price per category - Python collections

Ranking - By Key - Get top N products by price per category - using flatMap

Ranking - By Key - Get top N priced products - Introduction

Ranking - By Key - Get top N priced products - using Python collections API

Ranking - By Key - Get top N priced products - Create Function

Ranking - By Key - Get top N priced products - integrate with flatMap

Set Operations - Introduction

Set Operations - Prepare data

Set Operations - union and distinct

Set Operations - intersect and minus

Saving data into HDFS - text file format

Saving data into HDFS - text file format with compression

Saving data into HDFS using Data Frames - json

Different interfaces to run SQL - Hive, Spark SQL

Create database and tables of text file format - orders and order_items

Create database and tables of ORC file format - orders and order_items

Running SQL/Hive Commands using pyspark

Functions - Getting Started

Functions - String Manipulation

Functions - Date Manipulation

Functions - Aggregate Functions in brief

Functions - case and nvl

Row level transformations

Joining data between multiple tables

Group by and aggregations

Sorting the data

Set operations - union and union all

Analytics functions - aggregations

Analytics functions - ranking

Windowing functions

Creating Data Frames and register as temp tables

Write Spark Application - Processing Data using Spark SQL

Write Spark Application - Saving Data Frame to Hive tables

Data Frame Operations

Introduction

Data Frames - Overview

Create Data Frames from Text Files

Create Data Frames from Hive Tables

Create Data Frames using JDBC

Data Frame Operations - Overview

Spark SQL - Overview

Overview of Functions to manipulate data in Data Frame fields or columns

Define Problem Statement - Get Daily Product Revenue

Selection or Projection of Data in Data Frames

Filtering Data from Data Frames

Perform Aggregations using Data Frames

Sorting Data in Data Frames

Development Life Cycle using Data Frames

Run applications using Spark Submit

Data Frame Operations - Window Functions - Overview

Data Frames - Window Functions APIs - Overview

Define Problem Statement - Get Top N Daily Products

Data Frame Operations - Creating Window Spec

Data Frame Operations - Performing Aggregations using sum, avg etc

Data Frame Operations - Time Series Functions such as Lead, Lag etc

Data Frame Operations - Ranking Functions - rank, dense_rank, row_number, etc

Writing a Spark Application

Building and Running an Application

Application Deployment Mode

The Spark Application Web UI

Configuring Application Properties

Introduction to Flume & features

Flume topology & core concepts

Flume Agents: Sources, Channels and Sinks

Property file parameters logic

Apache Kafka Installation

Apache Kafka Overview and Architecture

Consumer and Producer

Deploying Kafka in real world business scenarios

Integration with Spark for Spark Streaming

Introduction to zookeeper concepts

Overview and Architecture of Zookeeper

Zookeeper principles & usage in Hadoop framework

Use of Zookeeper in Hbase and Kafka

Oozie Fundamentals

Oozie workflow creations

Concepts of Coordinates and Bundles