## Monday, 18 June 2012

### Parallel Processing on the Pi (Bramble)

Parallel processing on the Raspberry Pi is possible, thanks to the portable MPICH2 (Message Passing Interface). I was keen to try this out as soon as I managed to get hold of two of these brilliant little computers. Here I'm going to show you how I managed to get it all working, and I will also show you the results I obtained. (Bramble was a name an ingenious Raspberry Pi forum member made up, not myself!)

There are three ways which you can install MPICH2 (in case one of them doesn't seem to work for you). Compiling and installing from source, using my .deb package then following the rest of the tutorial, or using the Python scripts I wrote. Installing from source takes a while on the little Pi when not cross compiling.

Install - Choose Method 1, 2 or 3

and then run it as root with the command;
sudo python install.py

Follow the instructions on the screen entering all of the necessary info. The script will do a full install and configuration like the steps in the tutorial below. Do this for every node on your Bramble.

I have also written an RSA key script which sets up a secure public and private key between the master and all of the worker nodes. This means that you don't have to enter the password into SSH every time, but still maintaining a secure connection;

python rsa.py

If you want to set it up for the root user, you can run as root, and enter the username "root" for the other worker(s).

Finally you can join onto the Testing phase of the tutorial if you want to, otherwise you are all set!

Please let me know of any problems with the scripts and I will fix it. Thanks!

2) From .deb package

De-package and install;
sudo dpkg -i mpich2_1.4.1p1-1_armel.deb

Now proceed to the setup phase of this tutorial.

3) From source;

Now extract the files;
tar zxfv mpich2-1.4.1p1.tar.gz

Go into the source folder;
cd mpich2-1.4.1p1

Now we must run ./configure to configure the files ready for compilation;
sudo ./configure --disable-f77 --disable-fc

If you have a FORTRAN compiler then you can run it without "--disable-f77" and "--disable-fc".
If you do not, and you don't want to run FORTRAN programs on your cluster then make sure you do configure with them disabled as the error occurs a good 20 mins in and you have to start all over again.

Now that you have configured without error, compile then install the compiled program;
sudo make
sudo make install

Now proceed to the setup phase of this tutorial below.

Setup

Now you should have MPICH2 installed on both (all) of your Raspberry Pis. On your Bramble, you are going to need one  node and the rest are worker nodes.
First create this file in both the pi and root user home directories,
touch ~/.rhosts
sudo touch /root/.rhosts

add the following lines to the normal user file (pi is the default username for both of my nodes);
Master pi
Slave pi

For the root user;
Master root
Slave root

Do this for every Pi/node on the Bramble.

Now edit the /etc/hosts file to contain the IP of each node and the master node like so;
192.168.0.14 Master
192.168.0.11 Slave

Edit /etc/hosts.allow to contain just "ALL+", or each IP address of the other nodes.

(To find out the IP on each node, type ifconfig)
Now edit /etc/securetty to contain the line to allow root access;

Create the file /etc/pam.d/rsh if it is not there already and put the following in;

auth       optional     /lib/security/pam_securetty.so auth       sufficient   /lib/security/pam_env.so auth       sufficient   /lib/security/pam_rhosts_auth.so account    sufficient   /lib/security/pam_stack.so service=system-auth session    sufficient   /lib/security/pam_stack.so service=system-auth

SSH Public Key Authentication - Optional

This section is optional but advised! The use of a public and private key for SSH will mean you won't have to type in the SSH password between the master and the worker every time you want to run a program.

Use the Python script I have written;

python rsa.py

Or if it does not work properly, follow the instructions at this link: http://steve.dynedge.co.uk/2012/05/30/logging-into-a-rasberry-pi-using-publicprivate-keys/
Thanks to Steve Smith who wrote the tutorial!

Testing

Now you should be ready to rock and roll! Lets test the cluster out.
If you installed from source, you will have this file to test in the ~/mpich2-1.4.1p1/examples directory (you will need to compile it). If you installed from the .deb package then download this test file to the same directory in each of the nodes (or even better a centralised dir, perhaps with NFS);

Extract the tar ball and make sure the permissions are set;
tar zxfv cp_test.tar.gz
sudo chmod -R 777 cpi_test

Enter the cpi_test directory. This cpi program is a little program written in C which outputs Pi to 16 decimal places and outputs run time. I think this program is quite apt given what is running it.
Now we have to set up a "machinefile". This tells MPICH2 where to delegate the processes to. Yes, more IP addresses to come! Create the machine file on the Master node;
touch machinefile

Now edit the file to contain the IP addresses of the node and a colon followed by the number process to follow. For example my machine file looks like this;
192.168.0.14
192.168.0.11:2

This tells MPICH2 to pass the first process to my master node at 192.168.0.14 and the next (second) process to my only worker node at 192.168.0.11. There is no process number after the first IP because empty is the same to the first. (i.e. 192.168.0.14 is the same as 192.168.0.14:1).

It's easier command wise if the machinefile and the program you are executing are in the same folder to reduce the length of the next command.

Now it's finally time to run our first program! Se sure that you are in the cpi_test folder and that the same folder is located in the same place in all the other nodes on the Bramble. Execute the following line;
mpiexec -f machinefile -n 100 ./cpi

-f shows mpiexec where the machinefile is. -n is the number of times to repeat the process. This is set at 100 to execute cpi 100 times so that we can see the difference in power easier.
If it gives you an error about file not found, again make sure all the cpi files are in the same location on each node.

If the command executes successfully then you should get a record of the processes running and an output like;

pi is approximately 3.1415926544231274, Error is 0.0000000008333343
wall clock time = 0.728116

You can remove the worker IPs from the machinefile to compare running this program on just the Master node.

My Results

I ran the previous command 10 times, with both nodes enables and again with just the Master and took an average both times. The average I got from both nodes was; 0.731 Seconds – with a minimum of around 0.6 and a limit of 0.8. The average I got from one node enabled was; 1.418 Seconds –with a minimum of 0.85 and a limit of 5.0.

Finish

You should now have your Bramble up and running!

I just compiled and created a deb file, and found the best way to set the program up for the Pi, all thanks go to the team who created the awesome and portable MPICH2, and any sources I used to make the tutorial.

This shows a lot of other useful applications of MPICH2 including compiling across your Bramble, it would be nice to have a compiler farm made of Pis.

Phil

 MPICH2 on the RPi

1. AWESOME!!!

2. I was just- thinking why bother with Raspberry Pi, thought of a cluster, googled "Raspberry Pi cluster" and found this via raspberrypi.org.

Brilliant!

};->

RoyMathur.com

3. Hi Phil. Thank you so much for taking the time to share this. I'm a novice with Linux and the Raspberry Pi and would love to set up a bramble with students and explore parallel computing with it if possible. Could you tell me what I need, e.g. 2 of the Pi B version, _ power cord, ethernet cord, what SD card with what Linux version, what other supporting hardware or switch, etc. I lack the fundamentals. Thank you for any help you can give.

4. Hi Rob. That's right, 2 or more raspberry Pi model B's and the same number of Ethernet cables, connected to the same network. I used a 10/100 Ethernet switch like this one which is quite useful; http://tinyurl.com/8rnaku3 as you can see underneath the monitor in the picture above. I used this in conjunction with a router as the switch does not have IP address allocating capabilities alone.

The power chords needed are Micro USB B cables which are common with many mobile phone chargers like HTC and Blackberry. You will also need a USB plug, as the Pi wont run of USB power from a motherboard or hub. I use a HTC charger like this; http://tinyurl.com/9ccfz5z (But I use a British plug).

I recommend using any 4GB+ SDHC cards for installing the Raspberry Pi Debian distro onto, preferably from this list of tried and tested cards, as I know there have been a few SD cards which haven't worked for other users; http://elinux.org/RPi_VerifiedPeripherals#SD_cards

The Debian distro can be found on the Raspberry Pi foundation's download page;
I haven't yet tried with the Debian "wheezy" as I set this up on the "squeeze" version of Debian, but I'm pretty confident that It will work :)

If you are wanting to use a distribution other than Debian then I recommend following the compiling part of the tutorial. If you are happy using the Debian distro then downloading and running the script will be fine for setup.

No other hardware is required to make this work.

Don't hesitate to ask any further questions, I am happy to help :)

Kind Regards

Phil

1. Hi Phil. Thank you so much for your quick and detailed reply! I will look into the links right away and get moving. I'll let you know how it goes. Best, Rob

5. Hey Phil. Are you on Twitter? I want to keep an eye on your Bramble progress as I also am interested in making one (possibly with SLURM though I have much more experience with TORQUE). Of course, it would be much cooler with MPI too :)

1. Hi Darkshine, I have an account but I'm afraid I rarely use it. If I do anymore on this project (which I probably will), I will post a detailed account on here. I hope that's OK.

6. Just wondered how good this is at Fault resilience (if a board failed) and scaling up or down in size?

1. Sorry for the late reply. Scaling works well with this method, as evident in this project; http://www.southampton.ac.uk/mediacentre/features/raspberry_pi_supercomputer.shtml

I need to test what happens when a board goes down. My guess is that processing on other nodes will continue unless there is a dependency on a result from the node which goes down.

7. Great project!

I just want to inform that the easiest way to set up ssh for login without passwerd is to use ssh-copy-id(1) to copy your public key to remote machines.
It will see to that the directories and files are created and proper permissions are set.

But great project!

8. An easier, and more modern, way of finding out the address of a device is to use ip(1) command.
That is usually much easier to use and have more options.

ip -o addr show | gawk --re-interval -- '/([0-9]{1,3}(\.([0-9]{1,3})){3})\/([0-9]{1,2})/ { if ( $4 !~ /^127\./ ) { split($4, ip, "/"); netmask=$6; print "dev="$2 "\tip=" ip[1] " / " ip[2] "\tnetmask=" \$6; } }' -

9. Hi there, thanks for all the extensive writeup! My only wonder is how you're networking them - I know for two I guess you'd just put them in a router, but for a cluster of, say 16 RPis, what would you need?? A switch or something? Thanks, AngusP.

1. Hi Angus. Yes I used a switch connected to a router, a switch alone does not have routing capabilities, so it would be redundant. But a router and then a switch connected so that there would be sufficient Ethernet ports is perfect. I used this connected to a Netgear router; http://tinyurl.com/8rnaku3
So you could either opt for two of these or maybe a larger switch. Hope that helped! Regards, Phil.

10. have you ever tried to run hadoop on it?

1. No I haven't. Might be good to test it out. My only fear is that the JVM runs poorly on the Pi, and if each node creates it's own instance of the VM then I can see it performing poorly as opposed to fewer nodes of a more powerful device or server. Please correct me if my assumption is wrong I have yet to use Hadoop! :) Regards Phil.

2. Usually I run C++ code for "heavier" calculations on hadoop so my only concern is that the datanode itself is too "heavy" and may have that effect. I'm still waiting for my raspberrys so I can't test it. =/

11. Do you know if it's possible to run the Java Minecraft server on a "Pi Bramble"?

1. Upon a small bit of Googling, you can compile the Java binding; MPIJava to use with MPICH2. This is a good looking tutorial; https://help.ubuntu.com/community/Mpich2ClusterWithMpijava
However this guy is compiling for x86 arch and not ARM so I am uncertain of the results but is worth a shot.

The Open Source Minecraft server can be found here;
http://opencraft.sourceforge.net/
The Git repo here;
http://gitorious.org/opencraft

On inspection of the README, the following tools & libraries are required for compiling the server;

"Tools Required:
ant
java

Libraries Required:
xstream
mina-core
slf4j
jython"

I'm uncertain of the performance, and there may be some coding required in order to pass threads to other machines.
But as I say I am uncertain.

12. thanks for sharing.

13. Hi Phil

I have 2 different SD cards a 8GB and a 16GB, will this cause any problem on both Pi's ?

1. Hi Alex,

Glad you liked the tutorial. No that wont be a problem.
Enjoy!

Phil

14. Thanks Phil
I followed exactly your tutorial and my Bramble is up and running now~~~
Yeah.

pi is approximately 3.1415926544231274, Error is 0.0000000008333343
wall clock time = 0.658796

15. Hi

First off I would like to thank you for posting your efforts it made putting my system together a lot simpler. I just have one problem. My times are reversed from yours a single machine reports ~.67 wall time and when I use both cards my times are about ~.998 wall time. This suggests to me a network bottleneck. I have the the later 512 card and am running the latest 'whezzy' which may explain the faster times for a single card. I have nothing to compare my results with except yours and other comments. My ping times are in .55-.78 ms range. Any insight would be helpful. Thanks again

16. I'm stuck at testing. I can't get past the firs command.

17. My 8 year old daughter is very excited to start this project with me ths summer. I currently have 2 pi boards I was thinking a modest 8. How much should I budget for this effort and can you recommend a experiment that will help explain parallel computing. O a 8 year old. Thanks for the great inspiration and work!

18. Hi, I have just finished building the cluster but it is not working yet. It shows the following message
"[proxy:0:0@(none)] HYDU_sock_connect (/home/pi/mpich2/mpich2-1.4.1p1/src/pm/hydra/utils/sock/sock.c:203): unable to connect from "(none)" to "raspberrypi" (Connection refused)
[proxy:0:0@(none)] main (/home/pi/mpich2/mpich2-1.4.1p1/src/pm/hydra/pm/pmiserv/pmip.c:209): unable to connect to server raspberrypi at port 39015 (check for firewalls!)
"

Have I missed something here?

19. hi is there a MPICH2 for armhf?

1. after running his script try running sudo apt-get install mpich2 -y

20. This comment has been removed by the author.

21. Hi Paul,

Firstly great tutorial. It helped me get MPICH set up with Arch Linux with my Raspberry Pi's. I've completed the tutorial all the way through and have been able to run the CPI example on both the master and the slave pi's my problem is I don't think it's working correctly. I don't see any improvement in the speed between 1 or 2 nodes and my output seems to suggest that it's just running the same program on both Pi's rather then sharing the processing load. Odds are I just don't understand what's going on completely and it is working correctly.

here is the command I execute - mpiexec --hostfile machinefile -n 100 ./cpi

and the returned output from 2 Pis-

Enter passphrase for key '/home/pi/.ssh/id_rsa':
Process 0 of 1 is on master-pi
pi is approximately 3.1415926544231341, Error is 0.0000000008333410
wall clock time = 0.001194
Process 0 of 1 is on slave-pi-1
pi is approximately 3.1415926544231341, Error is 0.0000000008333410
wall clock time = 0.001189

1. Hi Scott,

The pi calculation program is single threaded and therefore we can't share the load of a single thread. We can develop multi threaded programs for MPI fairly easily. Here is what seems to be a good tutorial for doing so. It has some simple examples on summing values in an array;

http://condor.cc.ku.edu/~grobe/docs/intro-MPI-C.shtml

So the cpi program we have is single threaded, which is why we add the parameter -n 100 which tells mpich2 to execute the program/process 100 times. Mpich2 then balances running the program 100 times across however many nodes you have setup. Thats why on the photo at the end of the original post shows the htop output on both pi's. You can just make out the repeated process running on both sessions.

Try doing the same and making sure that both pis are executing the cpi program. Your times seem odly high, especially considering (as seen in the original post) that I acheived a quicker time even on a single pi.

Let me know how you get on. Thanks,

Phil

22. This comment has been removed by the author.