Thursday, December 27, 2007

GFS - Google File System : Introduction

Google File System (GFS) is a proprietary distributed file system developed by Google for its own use.

GFS is optimized for Google's core data storage needs, web searching, which can generate enormous amounts of data that needs to be retained; Google File System grew out of an earlier Google effort, "BigFiles", developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford. The data is stored persistently, in very large, even multiple gigabyte-sized files which are only extremely rarely deleted, overwritten, or shrunk; files are usually appended to or read. It is also designed and optimized to run on Google's computing clusters, the nodes of which consist of cheap, "commodity" computers, which means precautions must be taken against the high failure rate of individual nodes and the subsequent data loss. Other design decisions select for high data throughputs, even when it comes at the cost of latency.

The nodes are divided into two types: Master nodes and Chunkservers. Chunkservers store the data files, with each individual file broken up into fixed size chunks (hence the name) of about 64 megabytes, similar to clusters or sectors in regular file systems. Each chunk is assigned a unique 64-bit label, and logical mappings of files to constituent chunks are maintained. Each chunk is replicated a fixed number of times throughout the network, the default being three, but even more for high demand files like executables.

The Best Experts Advice @

No comments:

Celebrities, Movies, Reviews, Photos & Trivia