There are mainly 3 components in spark UI Jobs A spark application can have multiple jobs based on the number of actions (#jobs =#actions) in the application. These jobs can have a common rdd somewhere in the execution map or they can be separate al...
RDD(resilient distributed datasets) are the basic unit of storage in spark. you can think of an rdd as a collection distributed over multiple machines.Most of the time higher level structured APIs are used in spark applications which under the hood g...
In simple terms, Apache spark is an in-memory unified parallel compute engine. In Memory,Most of the operations in apache spark happen in memory and there is very less disk IO operation giving rise to faster data transformation and computation, unlik...
We cannot use an analytical storage system for transactional requirements and vice versa. But have you ever wondered why is that so? Transactional vs Analytical storage system Transactional storage (ex - MySQL, Postgres, etc.) is used to work with da...
Data ingestion is one of the crucial steps in the data lifecycle and when the source is a relational database, Sqoop can be a very easy and simple tool for this. What is Sqoop? Sqoop is a general tool for transferring bulk data from relational databa...
Although MapReduce is not much used in solving Big Data problems nowadays because of its poor performance compared to spark. But it's still a very good approach to understanding how distributed computing works in Big Data. What is Hadoop MapReduce? H...
