Micro Data Cluster

Web Server Cluster

The main function of Web server cluster which consists of 7 Banana Pro is providing external web access though which users can have access to the resources though web service. 2 Banana Pro are used as the main nodes whose duty is to transfer users’ request to the 4 remaining web service node. Each single one can guarantee the normal function. That is to say even there are two main nodes, only one of them is the active node while the other one is the standby node which will turn into active if the earlier active node is shut down because of emergency. The remaining four Banana Pro are designed to be the nodes providing web services. The purpose of using a cluster to provide Web service is that we can provide fast-and-high concurrent access and 24/7 available services as a result of concentrating on fault tolerant and load balancing. The structure of the cluster is showed in figure 1-1.

Main nodes are designed to be reverse proxy server and load balancing server. Reverse proxy server receive the request from user in Internet and transfer the request to the service nodes. After web server sending back the data, the main node sends it to user. With the techniques of load balancing, we distribute the request to the idle service node so that we can make sure the cluster’s speed for response since none of the web server would be overloaded.

By using Keep lived, we improved the availability of the cluster. There are 2 main nodes and in normal circumstances only one of them is active while the other one is standby for emergency. The cluster supplies a floating IP through which users are capable of having an access though Web page. The active node’s IP is the default IP. However when the active node shuts down because of system-halted emergency, system error or other uncontrollable factors, the floating IP would switches to the standby node within 9 seconds. Whenever necessary, the standby node would switch to the active node responsible for services supply. After system administrator recovers the shut-down node, it becomes to the standby node automatically. The main node will also monitor the service node so that when a service node shuts down, it will avoid distributing the task to the shut-down one. The four web service nodes whose duty is responding user’s request are running with the same web page. So even when there’s a service node is halted, user won’t feel the break that they are transferred from one service node to the other one. Besides, the main node is capable of checking whether the Web service node function normally which we call proactive health check in backend. By logging in as administrator in the website, users can the detailed status of every web service node.

We choose LNMP which is short for Linux+Nginx+Mysql+PHP to be the environment. To minimize cost, we choose LNMP as operating system, reverse proxy server, database, and dynamic web develop environment because they are free to use. Despite the fact that LNMP is free and open-source, they have been used steadily in market as web server for several times. As Banana Pro is based on ARM structure, so we choose the operating system supporting ARM structure which is commonly known as Raspbian.

To sum up, in Web server cluster, we concentrate a lot on load balancing so that in concurrency test, the server can function normally while there are 10000 requests in one second. Also, we focus on fault tolerant. In the case that one or more node is halted, the server can still operating normally. Moreover, we can real-time monitor the status of every node in the back end.

Hadoop Computing Cluster

Hadoop computing cluster is made up of 8 Lemaker Banana Pro. It can provide the ability of large-scale data computing, such as massive data search, Web data mining and massive e-commerce transactions data processing, etc. In this proposal, we designed a movie recommendation system based on machine learning and through the web service, the recommendation will be send back to user. And we install the core package of Ubuntu14.04 whose memory-occupation in normal operation is less than 50 MB in each hadoop node which gives us more resources for computation and storage. Considering the inconvenience of shell command and database operating, we especially designed an interface to the outside through which we can monitor and control the cluster through socket. The detail of Hadoop structure is showed in figure 1-2.

HDFS is famous for the high fault tolerance and the high efficiency distributed file system. Hadoop provides massive data computation through HDFS and Map Reduce programming model. By distributing massive task, Hadoop enhances the efficiency compared with single computational element. Moreover, Hadoop is fault tolerant. It will consider the failure of computational element or the data storage, so that it maintains multiple copies of the work data, ensuring that the task distributed to the fail node will be redistributed. The administrator could notice if one node is halted by emergency. So we can monitor the cluster anytime efficiently.

All the data we use for movie recommendation comes from the project GroupLens, which is in school of computer science and technology Minnesota University. MovieLens founded by GroupLens is made of movie rating which relies on the statistic of 6040 users making 1 million ratings among 3900 movies. The recommendation algorithm will recommend the different movies to various users which they are most likely to be interested in based on the rating they give before.

In a word, Hadoop is fault tolerant by itself and what we do in the Hadoop cluster is trying to enhance its efficiency. We install the pure system with no redundant service and takes less system resources than other systems. In order to operate the cluster more convenient, we encapsulate Hadoop shell command with socket so that by logging in as the administrator, we can start, end or run the algorithm in any computer as long as connected to the Internet.

Data Storage Node

Storage server is built with 1 Banana Pro whose main function is to offer efficient storage. Storage server is equipped with a SSD. Compared with mechanical hard drive (HHD), SSD have the advantages in data transfer speed. In the Web server cluster, none of the nodes is equipped with SSD, so the data must be stored in the data storage server. The centralized data storage strategy can reduce the storage space, greatly reduce the data storage cost, and can effectively avoid data inconsistencies between different nodes. In our proposal, all of the pictures or the text about the movie which users might access are stored in the data storage node. We installed MySQL database so that every other node can have the access to the database through remote PHP. Data storage node is capable of transferring data with Web service cluster and Hadoop computing cluster, which is shown in figure1-3.

The Design of the Server Shell

The shell we design is called FavoSever, which is from the English word Favose, Whose meaning is honeycomb. The specific shape is as figure 1-3 shows.

Every FavoSevershape is standard honeycomb with hexagonal structure which is rugged and reliable in strength. No matter the number of FavoSever, they can be stacked as a compact overall structure, which utilizes the space extremely.

The exact size of a FavoSever is 173mm*150mm*150mm. It can be placed either portrait or landscape (as figure 1-3 P-1,P-2 shows). It is small in size and convenient in transportation. Moreover, there is a incorporated handle to facilitate manual handling(as figure 1-3 P-3 shows). Outside FavoSever, there are limited spaces recesses and projections designed to fix with others. One of the hexagonal faces integrated cable interface and power connector is used as the input and output ports of the entire system. Inside FavoSever, 6 Banana Pro/Pi can be arranged in the inner surface and there is a 2.5-inch drive bay(as figure 1-3 p4 shows). Each Banana Pro’s CPU is equipped with passive heat sink. The entire system can be cooled with wind through ventilation shaft in the portrait which guarantee both Banana Pro and the SSD are in the appropriate conditions. We advise FavoSever placed horizontally so that the ventilation shaft is clear and cable is easier to reach switch. The standard power adapter is 5V and 12A which supplies the necessary power to 6 Banana Pros.