Sometimes in a spark application, we need to share small data across all the machines for processing. For example, if you want to filter some set of words from a large dataset residing in a datalake. Or if we simply just want to know how many blank l...
We cannot use an analytical storage system for transactional requirements and vice versa. But have you ever wondered why is that so? Transactional vs Analytical storage system Transactional storage (ex - MySQL, Postgres, etc.) is used to work with da...
Data ingestion is one of the crucial steps in the data lifecycle and when the source is a relational database, Sqoop can be a very easy and simple tool for this. What is Sqoop? Sqoop is a general tool for transferring bulk data from relational databa...
Although MapReduce is not much used in solving Big Data problems nowadays because of its poor performance compared to spark. But it's still a very good approach to understanding how distributed computing works in Big Data. What is Hadoop MapReduce? H...
Data nodes in HDFS are generally of commodity hardware which is low-priced but goes down very frequently. What happens to the data when a Data Node goes down?whenever a block is entered in HDFS, it is always replicated to multiple nodes. the number o...
What is HDFS? HDFS (Hadoop Distributed File System) is 1 of the 3 Hadoop core components. As its name suggests it's a distributed storage file system where large files are divided into multiple blocks of size 128mb (64mb in Hadoop 1) and stored acros...
