![](\"http:\/\/api.ning.com\/files\/8fzzcZRAVnbcZgd0FsJT7Fb0vjB0BOMsuLjDU**A3IuYjKjG3MGYoiUIVZOxjXl2tzuJ-Q5u7f7Ahrafke-6W0lCRGKP4Byt\/4.PNG\")
<\/a><\/p>\nStep 1<\/u> \u2013<\/strong> MapReduce workflow starts with the client program submitting job to the JobTracker*<\/strong><\/p>\n\n- *JobTracker<\/strong>– a job configuration which specifies the map and reduce functions, as well as input and output path of the data. It also schedules jobs and tracks the assign jobs to the TaskTracker*<\/strong><\/li>\n
- *TastTracker \u2013<\/strong> It tracks the tasks and reports status to JobTracker.<\/li>\n<\/ul>\n
Step 2<\/u><\/strong> \u2013 JobTracker will determine the number of splits from the input path of the data and it will select some TaskTrackers based on their network proximity to the data source. Then, the JobTracker send the task requests to those selected TaskTrackers.<\/p>\n<\/p>\nStep 3<\/u> \u2013<\/strong> Each Tasktracker will start the Map Phase processing<\/strong> by extracting input data from the splits. For each record parsed by the InputFormatter<\/strong>, it invokes the user provided ‘Map’ function, which stores several key-value pair in the circular memory buffer<\/strong>* (100MB default size)<\/p>\n\n- *Memory buffer \u2013<\/strong> it is found mainly in RAM and acts as an area where the CPU can store data temporarily.<\/li>\n<\/ul>\n
If the memory buffer fills up or if it reaches its maximum threshold (100 MB by default), mapper will block filling of data and spilling\/transferring of data takes place from memory buffer to the local disk until the buffer has space for incoming data.<\/p>\n<\/p>\n
Step 4<\/u> –<\/strong> Before spilling the data into the disk, the thread will divide the data into partition<\/strong> corresponding to the reducers which it will ultimately send to. \u00a0For each partition, the background thread will perform in-memory sort by Key. Each time memory buffer reaches the threshold of filling data, a new spill file is created. There could be several spill files after the map task has written its last output record.<\/p>\n<\/p>\nStep 5<\/u><\/strong> \u2013 Before a map task is finished, all spill files are merged<\/strong> into a single partition and stored into the output files in the disk.<\/p>\n<\/p>\nStep 6<\/u><\/strong> \u2013 When the map task completes (all splits are done), the TaskTracker will notify the JobTracker. When all the TaskTrackers are done, the JobTracker will notify the selected TaskTrackers for the Reduce Phase.<\/strong> Hence, each TaskTracker will read the output files remoted from the above step and sort the Key-Value pairs<\/strong>. For each key, it invokes the \u201creduce\u201d function, which collects the key- aggregatedValue<\/strong> and writes it into the output file (one per reducer node).<\/p>\nThe JobTracker keep tracks of the progress of each phase and periodically ping the TaskTracker for their health status. When any of the map phase TaskTracker crashes, the JobTracker will reassign the map task to a different TaskTracker node, which will rerun all the assigned splits. If the reduce phase TaskTracker crashes, the JobTracker will rerun the reduce at a different TaskTracker.<\/p>\n<\/p>\n
Step 7<\/u><\/strong> \u2013 After both phases are complete, the JobTracker will unblock the client program.<\/p>\n<\/p>\nWhat are the advantages of MapReduce?<\/strong><\/span><\/p>\nResilient to Failure<\/strong> \u2013 an application master watch mapper and reducer tasks on each partition.<\/p>\nData processing is easy to scale<\/strong> over multiple computing nodes.<\/p>\nParallel processing<\/strong> \u2013 In MapReduce, jobs are divided among multiple nodes and each node works with a part of the job simultaneously and hence helps to process the data using different machines. As the data is processed by different machines in parallel, the time taken to process the data gets reduced by a tremendous amount.<\/p>\nCost-effective solution –<\/strong> Hadoop\u2019s highly scalable structure also implies that it comes across as a very cost-effective solution for businesses that need to store ever growing data dictated by today\u2019s requirements. Hadoop\u2019s scale-out architecture with MapReduce, allows the storage and processing of data in a very affordable manner.<\/span><\/p>\nFast \u2013<\/strong> The tools of data processing; here MapReduce are often on the same servers where the data is located, resulting in faster data processing.<\/p>\n<\/p>\nIn the next blog, we shall discuss about YARN which is another key feature of Hadoop. Stay tuned.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Author: Aafrin Dabhoiwala Hadoop – MapReduce in an easy way In the previous blog, we discussed about HDFS, one of the main components of Hadoop. […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":1029,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[26],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/1028"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=1028"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/1028\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/1029"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=1028"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=1028"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=1028"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![](\"http:\/\/api.ning.com\/files\/bWsXXB2erox4J3xfWz3Cgocv*tj4vVQ0CnkkuroM8UJheNjcP5HkmigaG5no7dOaai43xG2n1jwoEzfAO9OyWaSugRqnhboj\/1.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/8fzzcZRAVnaaOqT*ZVy*GR*NOdJVYHsWMTRzp6*6oZ6PEnxZeh91mZb-eI8blrgdccXHoA1JBkyevlJIoRefVwvpKUGmNlfT\/2.PNG\")
<\/a><\/strong><\/span><\/p>\n![](\"http:\/\/api.ning.com\/files\/8fzzcZRAVnZofFxOxzIkLPxkBxWflTGSxfxwZw0dkOzanlC2p9ZBLt6bRtrU79CmVLCOZnxK9Ow*Etho7geQHU1Gs4uQzWj6\/3.PNG\")
<\/a><\/strong><\/span><\/p>\n<\/p>\n