/users/rlr/SwarmStuff/CSCS-Demos/Axelcult/run-axelcult-exper.txt ---------------------------------------------------------------- Before you try to run experiments with the Swarm implementation of the Axelrod Cultural Influence/Divserity model, please see /users/rlr/SwarmStuff/CSCS-Demos/Axelcult/demo-axelcult.txt and then (a) read at least Axelrod's paper, and other related papers. See http://cscs.umich.edu/education/CSCS-courses/cscs530/W04/WWW/outline.html on 14/16 January for more information. (a) create a version of the model in your own directory space (b) run it a bit in GUI mode to see how it behaves. ========================================================================== Running experiments ------------------- Suppose we want to run some experiments to replicate Axelrod's experiment in which he looked at the number of regions that result for various sized worlds. Looking at Figure 2 in his paper, it looks like he ran an experiment in which these were fixed: numFeatures: 5 numTraits: 15 and then he tried a series of worldSize values (approximately): worldSize: 2 4 6 8 10 12 14 16 18 20 25 30 50 100 Since the model has stochastic elements, it will give different results for different RNG (random number generator) seeds, so he averaged the results of several runs (40 for smaller world sizes, 10 for the larger ones). Call each combination of (numFeatures,numTraits,worldSize) a "case". It looks like he looked at 14 cases, and did 40 (or 10) runs for each. That is on the order of 500 runs of the model!! To do that "by hand" (i.e., start the model, set the parameter values, run the model, get the final count of numRegions from the report file) would be a lot of drudge work!! How can we make this a bit easier??? ============================================================================ Drone -- a tool to run experiments ----- Drone makes it easy to: * sweep parameters (1, 2, or more) * N runs for each case * organize data into output directories and files * automatically log experiments * easy to replicate a whole experiment, case or run * run on multiple machines at once From the CSCS Lab documents page, there are pointers to pages about drone, both the full manual and a quick note. Basically, you have to tell drone, through a "control file" - what program to run - how many runs to do - where to put the output - the basic (non-changing parameters) - any parameters to be varied across cases - comment for the log Let's make a directory for all Axelcut experiments, and copy a drone "control" file into it (this assumes you already make a Axelcult directory on your account): cd ~/Axelcult mkdir Experiments cd Experiments Then enter the "print working directory" command: pwd You should see something like /users/yourUniqName/Axelcult/Experiments where yourUniqName is replaced by your login. *** If you don't see that displayed, please ask an instructor *** or other student for help!! If you are in that directory, then copy an experiment control file into that directory: cp ~rlr/SwarmStuff/CSCS-Demos/Axelcult/Experiments/exp1.ctrl exp1.ctrl Now let's edit that control file, using the emacs editor: emacs exp1.ctrl & You should have a window pop up, with the contents of that exp1.ctrl file showing. The top should look like: # These parameters MUST be set up by the user param rootDir = /users/rlr/SwarmStuff/CSCS-Demos/Axelcult/ param programDir = /users/rlr/SwarmStuff/CSCS-Demos/Axelcult/ param programName = axelcult param seriesLogFile = axelcult.log param emailAddr = rlriolo@umich.edu param mailUser = 1 param numRuns = 10 The parts you may have to change include: rootDir -- all your simulation output goes in directories under this directory. So you should set it to: /users/yourlogin/Axelcult/Experiments where yourlogin is yourUniqName . programDir -- the directory that contains the program to run - change only if you use your own program! programName -- the simulation program to run - change only if you use your own program! emailAddr -- your email address numRuns -- 10 runs (or whatever you are doing) ** ** BE SURE to change the rootDir above to something like: ** param rootDir = /users/yourUniqName/Axelcult/Experiments ** using your login instead of yourUniqName ** AND BE SURE to change the email address! ** Next the file has a set of "base values" that act as defaults, some of which will be changed for specific cases: begin input file @begin reportFrequency = 1 stopT = 50000 numAgents = 100 worldSize = 10 numTraits = 10 numFeatures = 5 actMethod = axelrod @end end input file Then the sweep values are specified on a line a bit farther down in the file: sweep ws 2 4 6 8 10 12 14 16 18 20 25 30 ...or similar numbers... To do this as a collective experiment, we'll divide up the parameters to be swept, rather than have each person do them all. The assignment of what part of the sweep to do will be made at class time...see the list below as we fill it in. So...the changes to be made: 0. Change the rootDir emailAddr lines: param rootDir = /users/yourUniqName/Axelcult/Experiments param emailAddr = ***your email**** 1. Change the number of runs to do: param numRuns = 3 If you are assigned a worldSize <= 30, change the 3 to 40. If you are assigned a worldSize > 30, change it to 10 2. Change the reportFrequency if you are assigned worldSize > 20 Change from reportFrequency = 1 to reportFrequency = 100 3. Some of you will run it with Axelrod's original activation Method actMethod = axelrod Others will use actMethod = ktour See below for assignments. 4. Change the sweep line from sweep worldSize 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 40 50 to some subset of the values, as assigned below. The rest of the exp1.ctrl file you can ignore. ----------------------------- Emacs is a wyswyg editor. You can use menus at the top to do things like save the buffer (save changes to the file), undo changes, or exit. You use arrow and page-up/down keys to move around. So...use emacs to change the lines as indicated above. What I'd like to do is have different people at different machines run different parts of the overall experiment. Everyone should change, as appropriate, the rootDir, etc. Everyone will then do some of worldSize values for one of the activation methods (actMethod = axelrod or ktour) as assigned below (at class time): ActMethod=axelrod ActMethod=ktour Account worldSize Account worldSize ------- --------- ------- --------- After you have changed your exp1.ctrl file, **and saved it** (File->Save), then you can make a link from your Experiments directory to an area for your experimental output. (Be sure you are in your Experiments directory when you execute this.) mkdir -p /scratch/posare0/**YourUniqName**/Axelcult-expdata ln -s /scratch/posare0/**YourUniqName**/Axelcult-expdata expdata BE SURE: to replace all of **YourUniqName** with just your login/uniqName, without the **'s!!. If you then do the command: ll you should see something like this (all on one line): lrwxrwxrwx 1 yourUniqName users 39 Jul 25 16:26 expdata -> /scratch/posare0/yourUniqName/Axelcult-expdata in the list of files in your Axelcult/Experiments directory. (The technical details: That just means you have a "symbolic link" from the directory expdata/ to some space in another part of the CSCS disk space. However, you can treat that expdata/ directory just as if it was just another directory in your Axelcult/Experiments/ directory. For details on CSCS scratch space, see http://cscs.umich.edu/lab/scratch-disks.html . Now you can start the drone run as follows: /users/rlr/Drone/drone-linux exp1.ctrl You should see something like: Drone version 970829a Running on host battistini Started on Fri Jul 18 13:42:32 2003 Reading control file... Initializing... Running experiment experiment-1... Going into the background... If you now do the "list (files) long" command: ll you should see something like: total 16 drwxr-xr-x 2 yourUniqName users 4096 Jul 18 17:01 errorlogs -rw-r--r-- 1 yourUniqName users 4000 Jul 18 17:01 exp1.ctrl -rw-r--r-- 1 yourUniqName users 4035 Jul 18 16:59 exp1.ctrl~ lrwxrwxrwx 1 yourUniqName users 42 Jul 18 16:34 expdata -> /scratch/posare0/yourUniqName/Axelcult-expdata drwxr-xr-x 2 sum0300 users 4096 Jul 18 17:01 logs Note a few new directories in your Expdir/ directory: errorlogs/ -- where drone error messages go logs/ -- where experiment logs go expdata/ -- where experiment output goes To see what is inside the expdata/ directory, do ll expdata/ You should see experiment-1/ -- this first experiment you are running if you run more, they will be experiment-2/, etc. So as the experiment proceeds you can see how its doing by ll expdata/experiment-1 You should see something like this: total 12 -rw-r--r-- 1 yourUniqName users 225 Jul 26 11:18 experiment-1.in drwxr-xr-x 2 yourUniqName users 1024 Jul 26 11:46 ws=2/ drwxr-xr-x 2 yourUniqName users 1024 Jul 26 11:46 ws=4/ (Exactly which ws= directories you have depends on which worldSize parameter values you put on the sweep line in the exp1.ctrl file.) The above means so far it has finished the worldSize=2 case, and is working on the ws=4 case. If you do this command: ll expdata/experiment-1/ws=2 you may see something like: -rw-r--r-- 1 yourUniqName users 2750 Jul 26 11:46 case.log -rw-r--r-- 1 yourUniqName users 18192 Jul 26 11:45 report.00 -rw-r--r-- 1 yourUniqName users 18192 Jul 26 11:45 report.01 ...and more, with one report.** file for each run you asked it to do. ** NOTE: If you see any files named stderr.00 (or 01 ...), that means you have an error in your control file, and your experiment is not running correctly. *** Contact the instructor(s)!! So...eventually...depending on what machine you are on, you will have a full set of report files for each case. Drone will send an email to you telling you the job is done (if you changed the email line in the control file!). Now....how do we get the data we want out of all those files??? What do we want? At the least, we'd like to know the number of regions at the end of the runs. We could of course do "tail" commands on them and add them up by hand (or type them into some stat package and have it do them). However, being lazy I have written a little script (in perl) to do the averaging and print it all out. Here is how you run it for experiment-1: /users/rlr/Scripts/getLastLineAvgsFromDirs-2.pl \ -c4 -f"report.*" -d"expdata/experiment-1" That says: extract the values from column 4 (the numRegions column) of the last line of each report file, in each subdirectory in expdata/experiment-1, and average them over each case. You should see something like this: Last-step data Fri Jul 18 16:46:19 EDT 2003 From directory: expdata/experiment-1 FilePattern: report.* Column numbers in files: 4 Then print average (and StdDev) across files, and average, StdDev and Max last time-step. Overall: Avgs (stddev) LastStep LastStep Directory Avg (sd) Max Column 4 ------------------------------------------ -------------------- ----------------- ws=2 14 ( 9) 26 2.400 ( 0.966) ws=4 80 ( 25) 110 7.800 ( 1.874) ws=6 161 ( 97) 311 14.600 ( 3.534) You can see it prints for each case, averaged over all the runs in that case: * the average last time step of the run (and the StdDev) * the largest LastStep (i.e., the max number of steps it took for any of the runs to reach equilibrium) * the average value of column 4 at the end of the run (and the sd) In this case, column 4 is the number of regions. One can then take these values and import them into some graphing program, e.g., to get a plot of number of regions vs worldSize. On the CSCS machines, this could be done with xmgrace, mathematica, matlab, scilab, and so on. (There are pointers to xmgrace from the CSCS lab doc pages.) You could also move the data to a different computer (e.g. a wintel machine) and plot in Excel or whatever. **** Note: For the purposes of our collective experiment, please run the analysis program above, and then send your results to the CTNG email, so we can archive them all there. Given the data collected there, one (or more!) of us could produce graphs of the data. *** Also note: Feel free to run the same analysis program, but extract data from other columns, i.e., to see how other measures of segregation are affected by the parameters we swept. Look in any report* file produced by running the axelcult program for a description of the content of all the columns of data it produces. -------------------------------------------------------------------- And feel free to come up with other experiments. Please don't run them for too many time steps (eg 100,000!) or too many runs per case, or to many cases. For example, you could cp exp1.ctrl exp2.ctrl and use emacs to edit exp2.ctrl to replicate the experiment described in Table 2 of Axelrod's paper, in which he varied the numFeatures and numTraits for a worldSize = 10 world. To do this, you would use a double set of sweep lines: sweep numFeatures 5 10 15 sweep numTraits 5 10 15 It will do all those 9 combinations. ============================================================================= Drone is a tool that can be used with any program that meets a few constraints with respect to how the program is started, how parameters are sent to it, and so on. We have also be able to use Drone with programs that don't directly meet those constraints by writing "wrapper" scripts that do process parameters as Drone requires, by transforming those parameters into the form your program wants and then running it. For details, see the Drone documentation off the CSCS web pages: http://cscs.umich.edu/lab/documentation/SwarmStuff/drone-quicknote.html ============================================================================= In general, you will want to develop some kind of scripts or use other tools to automate the process of collecting and analyzing data "across cases" of an experiment. An alternative approach is to do the multiple runs for the varying cases all within one run of a program. I don't favor that approach because: - it makes the program with the model implementation more complex - it means you do one long run, and can't divide it up into runs on many machines - it means you have to re-implement tools (eg averagers, etc) within each model you implement =============================================================================