Monday, 18 June 2012

Parallel Processing on the Pi (Bramble)

Parallel processing on the Raspberry Pi is possible, thanks to the ultra portable MPICH2 (Message Passing Interface). I was keen to try this out as soon as I managed to get hold of two of these brilliant little computers (yes I'm a lucky boy). Here I'm going to show how I managed to get it all working and will display the results :) (Bramble was a name an ingenious Raspberry Pi forum member made up, not myself!)

There are three ways which you can install MPICH2 (in case one doesn't seem to work for you), compiling and installing from source, my .deb package then following the rest of the tutorial, or the Python script file. Installing from source takes a while on the little Pi when not cross compiling.

Install - Choose Method 1, 2 or 3
1) Simply download the script;
wget https://github.com/downloads/philleonard/MPICH2-Armel-Raspberry-Pi/install.py


and then run it as root with the command;
sudo python install.py

Then follow the instructions on the screen entering all of the necessary info. The script will do a full install and configuration like the steps in the tutorial below. Do this for every node on your Bramble.

I have also written an RSA key script which sets up a secure public and private key between the Master and all of the Slave nodes so that you don't have to enter the password into SSH every time, but maintain a secure connection;
wget https://github.com/downloads/philleonard/MPICH2-Armel-Raspberry-Pi/rsa.py


python rsa.py
You can run as root, and enter the username "root" for the other Slave(s) also for when you want to SSH as root.



Finally you can join onto the Testing phase of the tutorial if you want to, otherwise you are all setup!
Please let me know of any problems with the scripts and I will fix it :) Thanks!

2) From .deb package
First, download the .deb package;
wget https://github.com/downloads/philleonard/MPICH2-Armel-Raspberry-Pi/mpich2_1.4.1p1-1_armel.deb

De-package and install;
sudo dpkg -i mpich2_1.4.1p1-1_armel.deb 

3) From source;
First, download the source tar ball:
wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.4.1p1/mpich2-1.4.1p1.tar.gz

Now extract the files;
tar zxfv mpich2-1.4.1p1.tar.gz

Go into the source folder;
cd mpich2-1.4.1p1

Now we must run ./configure to configure the files ready for compilation;
sudo ./configure --disable-f77 --disable-fc

If you have a FORTRAN compiler then you can run it without "--disable-f77" and "--disable-fc".
If you do not, and you don't want to run FORTRAN programs on your cluster then make sure you do configure with them disabled as the error occurs a good 20 mins in and you have to start all over again :).

Now that you have configured without error, compile then install the compiled program;
sudo make
sudo make install

Setup

Now you should have MPICH2 installed on both (all) of your Raspberry Pis. On your Bramble, you are going to need one Master node and the rest are Slave nodes.
First create this file in both the pi and root user home directories, 
touch ~/.rhosts
sudo touch /root/.rhosts

add the following lines to the normal user file (pi is the default username for both of my nodes);
Master pi
Slave pi

For the root user;
Master root
Slave root

Do this for every Pi/node on the Bramble.

Now edit the /etc/hosts file to contain the IP of each node and the master node like so;
192.168.0.14 Master
192.168.0.11 Slave

Edit /etc/hosts.allow to contain just "ALL+", or each IP address of the other nodes.

(To find out the IP on each node, type ifconfig)
Now edit /etc/securetty to contain the line to allow root access;
rsh, rlogin, rexec, pts/0, pts/1

Create the file /etc/pam.d/rsh if it is not there already and put the following in;
auth       sufficient   /lib/security/pam_nologin.so
auth       optional     /lib/security/pam_securetty.so
auth       sufficient   /lib/security/pam_env.so
auth       sufficient   /lib/security/pam_rhosts_auth.so
account    sufficient   /lib/security/pam_stack.so service=system-auth
session    sufficient   /lib/security/pam_stack.so service=system-auth

SSH Public Key Authentication - Optional

This section is optional but advised! The use of a public and private key for SSH will mean you won't have to type in the SSH password between the Master and the Slaves every time you want to run a program.

Use the Python script I have written;
wget https://github.com/downloads/philleonard/MPICH2-Armel-Raspberry-Pi/rsa.py


python rsa.py

Or if it does not work properly, follow the instructions at this address http://steve.dynedge.co.uk/2012/05/30/logging-into-a-rasberry-pi-using-publicprivate-keys/
All thanks go to Steve Smith who wrote the tutorial!

Testing

Now you should be ready to rock and roll! Lets test the cluster out.
If you installed from source, you will have this file to test in the ~/mpich2-1.4.1p1/examples directory (you will need to compile it). If you installed from the .deb package then download this test file to each of the nodes in the same directory of each node (or even better a centralised dir, perhaps with NFS);
wget https://github.com/downloads/philleonard/MPICH2-Armel-Raspberry-Pi/cpi_test.tar.gz

Extract the tar ball and make sure the permissions are set;
tar zxfv cp_test.tar.gz
sudo chmod -R 777 cpi_test

Enter the cpi_test directory. This cpi program is a little program written in C which outputs Pi to 16 decimal places and outputs run time. I think this program is quite apt given what is running this program :)
Now we have to setup a "machinefile". This tells MPICH2 where to delegate the processes to. Yes, more IP addresses to come! Create the machine file on the Master node;
touch machinefile

Now edit the file to contain the IP addresses of the node and a colon folowed by the number process to follow. For example my machine file looks like this;
192.168.0.14
192.168.0.11:2

This tells MPICH2 to pass the first process to my Master node at 192.168.0.14 and the next (second) process to my only slave node at 192.168.0.11. There is no process number after the first IP because empty is the same to the first. (i.e. 192.168.0.14 is the same as 192.168.0.14:1).

It's easier command wise if the machinefile and the program you are executing are in the same folder to reduce the length of the next command ;)

Now it's finally time to run our first program! Se sure that you are in the cpi_test folder and that the same folder is located in the same place in all the other nodes on the Bramble. Execute the following line;
mpiexec -f machinefile -n 100 ./cpi

-f shows mpiexec where the machinefile is. -n is the number of times to repeat the process. This is set at 100 to execute cpi 100 times so that we can see the difference in power easier
If it gives you an error about file not found, again make sure all the cpi files are in the same location on each node.

If the command executes successfully then you should get a record of the processes running and an output like;

pi is approximately 3.1415926544231274, Error is 0.0000000008333343
wall clock time = 0.728116

You can remove the Slave IPs from the machinefile to compare running this program on just the Master node. 

My Results

I ran the previous command 10 times, with both nodes enables and again with just the Master and took an average both times. The average I got from both nodes was; 0.731 Seconds – with a minimum of around 0.6 and a limit of 0.8. The average I got from one node enabled was; 1.418 Seconds –with a minimum of 0.85 and a limit of 5.0. 

Finish 

You should now have your Bramble up and running!

I just compiled and created a deb file, and found the best way to set the program up for the Pi, all thanks go to the team who created the awesome and portable MPICH2, and any sources I used to make the tutorial :)

This shows a lot of other useful applications of MPICH2 including compiling across your Bramble, would be nice to have a compiler farm made of Pis for big projects :)

Thanks for reading! Don't hesitate to ask for help below!

Phil


MPICH2 on the RPi

28 comments:

  1. I was just- thinking why bother with Raspberry Pi, thought of a cluster, googled "Raspberry Pi cluster" and found this via raspberrypi.org.

    Brilliant!

    };->

    RoyMathur.com

    ReplyDelete
  2. Hi Phil. Thank you so much for taking the time to share this. I'm a novice with Linux and the Raspberry Pi and would love to set up a bramble with students and explore parallel computing with it if possible. Could you tell me what I need, e.g. 2 of the Pi B version, _ power cord, ethernet cord, what SD card with what Linux version, what other supporting hardware or switch, etc. I lack the fundamentals. Thank you for any help you can give.

    ReplyDelete
  3. Hi Rob. That's right, 2 or more raspberry Pi model B's and the same number of Ethernet cables, connected to the same network. I used a 10/100 Ethernet switch like this one which is quite useful; http://tinyurl.com/8rnaku3 as you can see underneath the monitor in the picture above. I used this in conjunction with a router as the switch does not have IP address allocating capabilities alone.

    The power chords needed are Micro USB B cables which are common with many mobile phone chargers like HTC and Blackberry. You will also need a USB plug, as the Pi wont run of USB power from a motherboard or hub. I use a HTC charger like this; http://tinyurl.com/9ccfz5z (But I use a British plug).

    I recommend using any 4GB+ SDHC cards for installing the Raspberry Pi Debian distro onto, preferably from this list of tried and tested cards, as I know there have been a few SD cards which haven't worked for other users; http://elinux.org/RPi_VerifiedPeripherals#SD_cards

    The Debian distro can be found on the Raspberry Pi foundation's download page;
    http://www.raspberrypi.org/downloads
    I haven't yet tried with the Debian "wheezy" as I set this up on the "squeeze" version of Debian, but I'm pretty confident that It will work :)

    If you are wanting to use a distribution other than Debian then I recommend following the compiling part of the tutorial. If you are happy using the Debian distro then downloading and running the script will be fine for setup.

    No other hardware is required to make this work.

    Don't hesitate to ask any further questions, I am happy to help :)

    Kind Regards

    Phil

    ReplyDelete
    Replies
    1. Hi Phil. Thank you so much for your quick and detailed reply! I will look into the links right away and get moving. I'll let you know how it goes. Best, Rob

      Delete
  4. Hey Phil. Are you on Twitter? I want to keep an eye on your Bramble progress as I also am interested in making one (possibly with SLURM though I have much more experience with TORQUE). Of course, it would be much cooler with MPI too :)

    ReplyDelete
    Replies
    1. Hi Darkshine, I have an account but I'm afraid I rarely use it. If I do anymore on this project (which I probably will), I will post a detailed account on here. I hope that's OK.

      Delete
  5. Just wondered how good this is at Fault resilience (if a board failed) and scaling up or down in size?

    ReplyDelete
    Replies
    1. Sorry for the late reply. Scaling works well with this method, as evident in this project; http://www.southampton.ac.uk/mediacentre/features/raspberry_pi_supercomputer.shtml

      I need to test what happens when a board goes down. My guess is that processing on other nodes will continue unless there is a dependency on a result from the node which goes down.

      Delete
  6. Great project!

    I just want to inform that the easiest way to set up ssh for login without passwerd is to use ssh-copy-id(1) to copy your public key to remote machines.
    It will see to that the directories and files are created and proper permissions are set.

    But great project!

    ReplyDelete
  7. An easier, and more modern, way of finding out the address of a device is to use ip(1) command.
    That is usually much easier to use and have more options.

    Like "ip -4 addr list dev eth0" to get information about your IPv4 settings on eth0.

    ip -o addr show | gawk --re-interval -- '/([0-9]{1,3}(\.([0-9]{1,3})){3})\/([0-9]{1,2})/ { if ( $4 !~ /^127\./ ) { split($4, ip, "/"); netmask=$6; print "dev=" $2 "\tip=" ip[1] " / " ip[2] "\tnetmask=" $6; } }' -

    ReplyDelete
  8. Hi there, thanks for all the extensive writeup! My only wonder is how you're networking them - I know for two I guess you'd just put them in a router, but for a cluster of, say 16 RPis, what would you need?? A switch or something? Thanks, AngusP.

    ReplyDelete
    Replies
    1. Hi Angus. Yes I used a switch connected to a router, a switch alone does not have routing capabilities, so it would be redundant. But a router and then a switch connected so that there would be sufficient Ethernet ports is perfect. I used this connected to a Netgear router; http://tinyurl.com/8rnaku3
      So you could either opt for two of these or maybe a larger switch. Hope that helped! Regards, Phil.

      Delete
  9. have you ever tried to run hadoop on it?

    ReplyDelete
    Replies
    1. No I haven't. Might be good to test it out. My only fear is that the JVM runs poorly on the Pi, and if each node creates it's own instance of the VM then I can see it performing poorly as opposed to fewer nodes of a more powerful device or server. Please correct me if my assumption is wrong I have yet to use Hadoop! :) Regards Phil.

      Delete
    2. Usually I run C++ code for "heavier" calculations on hadoop so my only concern is that the datanode itself is too "heavy" and may have that effect. I'm still waiting for my raspberrys so I can't test it. =/

      Delete
  10. Do you know if it's possible to run the Java Minecraft server on a "Pi Bramble"?

    ReplyDelete
    Replies
    1. Upon a small bit of Googling, you can compile the Java binding; MPIJava to use with MPICH2. This is a good looking tutorial; https://help.ubuntu.com/community/Mpich2ClusterWithMpijava
      However this guy is compiling for x86 arch and not ARM so I am uncertain of the results but is worth a shot.

      The Open Source Minecraft server can be found here;
      http://opencraft.sourceforge.net/
      The Git repo here;
      http://gitorious.org/opencraft

      On inspection of the README, the following tools & libraries are required for compiling the server;

      "Tools Required:
      ant
      java

      Libraries Required:
      xstream
      mina-core
      slf4j
      jython"

      I'm uncertain of the performance, and there may be some coding required in order to pass threads to other machines.
      But as I say I am uncertain.

      Delete
  11. Hi Phil

    Thanks for your great tutorial.
    I have 2 different SD cards a 8GB and a 16GB, will this cause any problem on both Pi's ?

    ReplyDelete
    Replies
    1. Hi Alex,

      Glad you liked the tutorial. No that wont be a problem.
      Enjoy!

      Phil

      Delete
  12. Thanks Phil
    I followed exactly your tutorial and my Bramble is up and running now~~~
    Yeah.
    Thanks again for your effort.

    pi is approximately 3.1415926544231274, Error is 0.0000000008333343
    wall clock time = 0.658796



    ReplyDelete
  13. Hi

    First off I would like to thank you for posting your efforts it made putting my system together a lot simpler. I just have one problem. My times are reversed from yours a single machine reports ~.67 wall time and when I use both cards my times are about ~.998 wall time. This suggests to me a network bottleneck. I have the the later 512 card and am running the latest 'whezzy' which may explain the faster times for a single card. I have nothing to compare my results with except yours and other comments. My ping times are in .55-.78 ms range. Any insight would be helpful. Thanks again

    ReplyDelete
  14. I'm stuck at testing. I can't get past the firs command.

    ReplyDelete
  15. My 8 year old daughter is very excited to start this project with me ths summer. I currently have 2 pi boards I was thinking a modest 8. How much should I budget for this effort and can you recommend a experiment that will help explain parallel computing. O a 8 year old. Thanks for the great inspiration and work!

    ReplyDelete
  16. Hi, I have just finished building the cluster but it is not working yet. It shows the following message
    "[proxy:0:0@(none)] HYDU_sock_connect (/home/pi/mpich2/mpich2-1.4.1p1/src/pm/hydra/utils/sock/sock.c:203): unable to connect from "(none)" to "raspberrypi" (Connection refused)
    [proxy:0:0@(none)] main (/home/pi/mpich2/mpich2-1.4.1p1/src/pm/hydra/pm/pmiserv/pmip.c:209): unable to connect to server raspberrypi at port 39015 (check for firewalls!)
    "

    Have I missed something here?

    ReplyDelete
  17. hi is there a MPICH2 for armhf?

    ReplyDelete