A scalable cloud platform using matlab distributed computing server integrated with HDFS

Abstract

The Hadoop Distributed File System (HDFS) is a large data storage system which exhibits several features of a good distributed file system. In this paper we integrate Matlab Distributed Computing Server (MDCS) with HDFS to build a scalable, efficient platform for scientific computations. We use an FTP server on top of HDFS for data transfer from the Matlab system to HDFS. The motivation of using HDFS for storage with MDCS is to provide an efficient, fault-tolerant file system and also to utilize the resources efficiently by making each system serve as both data node for HDFS and worker for MDCS. We test the storage efficiency of HDFS and compare with normal file system for data transfer operations through MDCS. © 2012 IEEE.