/users/rlr/Courses/ICPSR-02/Handouts/run-tipping-exper.txt ---------------------------------------------------------- Running experiments ------------------- Suppose we want to run some experiments to see how sensitive Schelling's tipping model is to the agents' threshold values. Suppose we also want to see what differences there are when the agents' use different methods to determine where to move to, i.e., the random, nearest-open and nearest-ok rules described in the writeup of the model. In particular, suppose we want to try the following thresholds: 0.01, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, ..., 0.75, 0.80, 0.85, 0.90. Let's call this one experiment. Each combination of threshold and movement rules we will call a "case." Thus there are 3 x 19 = 57 cases. Also, since the model has stochastic elements, we are pretty sure it will give different results for different RNG seeds, so we will want to average the results of several runs, say at least 10 runs for each case (30 would be better). That is a total of (at least) 570 runs of the model, yielding 570 "report" files from which we would want to read segregation measures (from the last lines of those files). For this exercise, suppose we will just look at the number of fully segregated red agents at the end of each run. One approach would be to make a directory for the experiment: mkdir experiment-1 cd experiment and then make a directory for each case, giving the directories names which reflect the values of the relevant parameters for that case: mkdir mm=random-tr=0.01 mkdir mm=random-tr=0.05 ... Then we could go into that first directory: cd mm=random-tr=0.01 and we could then do each of the 10 runs with different random number seeds, being sure to give them different report file suffixes: tipping -Dmm=random -Dtr=0.01 -Dtg=0.0 -DstopT=200 -DRN=00 --batch tipping -Dmm=random -Dtr=0.01 -Dtg=0.0 -DstopT=200 -DRN=01 --batch ... tipping -Dmm=random -Dtr=0.01 -Dtg=0.0 -DstopT=200 -DRN=09 --batch *** *** Try a couple of the tipping commands above. *** You should see a report file as listed below for each *** run you do (with a different RN= value). *** After these are all done, we will have 10 report files: report.00 report.01 .... report.09 each with the results from one run with a different RNG seed. (If you start tipping without specifying a RNG seed, it generates a new one by a combination of the clock time and the process number, which will always be a unique new seed.) Finally, we have to extract the number we want, the number of red agents that are fully segregated, from the last lines of those 10 report files, and average them to get a statistical summary of what happens for this particular case. We could do that by hand: tail -2 report.00 tail -2 report.01 ... and average the number of red agents that are fully segregated at the end of each run (i.e., the value in the 7th column). So....this is obviously a lot of drudge work!! How can we make this a bit easier??? ============================================================================ Drone -- a tool to run experiments ----- Drone makes it easy to: * sweep parameters (1, 2, or more) * N runs for each case * organize data into output directories and files * automatically log experiments * easy to replicate a whole experiment, case or run * run on multiple machines at once From the CSCS Lab documents page, there are pointers to pages about drone, both the full manual and a quick note. Basically, you have to tell drone, through a "control file" - what program to run - how many runs to do - where to put the output - the basic (non-changing parameters) - any parameters to be varied across cases - comment for the log Let's make a directory for all Tipping experiments, and copy a drone "control" file into it (this assumes you already make a Tipping directory on your account): cd ~/Tipping mkdir Experiments cd Experiments cp ~rlr/Courses/ICPSR-02/Experiments/Tipping2/exp1.ctrl exp1.ctrl Now let's edit that file control file, using the emacs editor: emacs exp1.ctrl & You should have a window pop up, with the contents of that exp1.ctrl file showing. The top should look like: # These parameters MUST be set up by the user param rootDir = /users/rlr/Courses/ICPSR-02/Experiments/Tipping2/ param programDir = /users/rlr/SwarmStuff/CSCS-Demos/Tipping2/ param programName = tipping param seriesLogFile = tipping.log param emailAddr = rlriolo@umich.edu param mailUser = 1 param numRuns = 10 The parts you have to change are rootDir -- all your simulation output goes in directories under this directory. So you should set it to: /users/yourlogin/Tipping/Experiments where yourlogin is sum02** . programDir -- the directory that contains the program to run - change only if you use your own program! programName -- the simulation program to run - change only if you use your own program! emailAddr -- your email address numRuns -- 10 runs (or whatever you are doing) ** ** BE SURE to change the rootDir above to something like: ** param rootDir = /users/sum0100/Tipping/Expdir ** using your login instead of sum0100 ** AND BE SURE to change the email address! ** Next are a set of "base values" that act as defaults, some of which will be changed for specific cases: begin input file @begin stopT = 200 reportFileName = report reportFrequency = 1 rDebug = 0 worldSize = 20 numAgents = 300 fractionRed = 0.5 thresholdRed = 0.35 thresholdGreen = 0.35 randomMoveProb = 0 moveMethod = random oneStepMode = 0 @end end input file Then the sweep values are specified: sweep moveMethod random nearest-open nearest-ok sweep (thresholdRed thresholdGreen) (0.01 0.01) (0.35 0.35) (0.50 0.50) (0.75 0.75) Now to do this as a collective experiment, we'll divide up the parameters to be swept, rather than have each person do them all. The assignment of what part of the sweep to do will be made at class time...see the list below as we fill it in. When you find out what part of the experiment you will be doing, you will need to: a) remove the two moveMethods you are NOT doing from the first line. b) put in only the threshold values you are doing on the second line. Note that for the second sweep line, we are varying two parameters together, i.e., the threshold for Red and Green agents. So be sure to include pairs of equal values just as shown above. Finally, you can specify comments to be included in the log file for the experiments: comment A simple test to be sure the tipping model comment is working ok with this control file setup. comment We are sweeping on Move Method, and threshold for comment Red and Green agents together. You can put whatever comment you want there, on lines that start with just the word "comment" as shown above. The rest of the exp1.ctrl file you can ignore. ----------------------------- Emacs is a wyswyg editor. You can use menus at the top to do things like save the buffer (save changes to the file), undo changes, or exit. You use arrow and page-up/down keys to move around. So...use emacs to change the lines as indicated above. What I'd like to do is have different people at different machines run different parts of the overall experiment. So....everyone should change, as appropriate, the rootDir, etc, and the comments. Everyone will then do some of the threshold values, as follows: Account sum02** move-method thresholds ------- ------------------------------------ random 0.01 0.05 0.10 0.15 random 0.20 0.25 0.30 0.35 random 0.40 0.45 0.50 0.55 random 0.60 0.65 0.70 0.75 random 0.80 0.85 0.90 0.95 nearest-open 0.01 0.05 0.10 0.15 nearest-open 0.20 0.25 0.30 0.35 nearest-open 0.40 0.45 0.50 0.55 nearest-open 0.60 0.65 0.70 0.75 nearest-open 0.80 0.85 0.90 0.95 nearest-ok 0.01 0.05 0.10 0.15 nearest-ok 0.20 0.25 0.30 0.35 nearest-ok 0.40 0.45 0.50 0.55 nearest-ok 0.60 0.65 0.70 0.75 nearest-ok 0.80 0.85 0.90 0.95 After you have changed your exp1.ctrl file, **and saved it**, **and maybe asked one of the instructors to look at it**, then you can make a link from your Experiments directory to an area for your experimental output. To spread out the load, please pick *one* of the following commands (flip a coin or whatever to pick randomly): *** Just do one of these: makeExperDir.pl -dTipping2 -spulcina2 OR makeExperDir.pl -dTipping2 -svolpe3 OR makeExperDir.pl -dTipping2 -slupo0 OR makeExperDir.pl -dTipping2 -spesce0 If you then do the command: ll you should see something like this (all on one line): lrwxrwxrwx 1 sum0100 users 39 Jul 25 16:26 expdata -> /scratch/pulcina2/sum0200/Tipping2-expdata -rw-r--r-- 1 sum0100 users 6058 Jul 25 15:38 report in the list of files in your Tipping/Experiments directory. (The technical details: That just means you have a "symbolic link" from the directory expdata/ to some space in another part of the CSCS disk space. However, you can treat that expdata/ directory just as if it was just another directory in your Tipping/Experiments/ directory.) Now you can start the drone run as follows: drone exp1.ctrl You should see something like: Drone version 970829a Running on host pulcina Started on Wed Jul 25 16:34:49 2001 Reading control file... Initializing... Running experiment experiment-1... Going into the background... The you should see a few new directories in your Expdir/ directory: errorlogs/ -- where drone error messages go logs/ -- where experiment logs go expdata/ -- where experiment output goes To see what is inside the expdata/ directory, do ll expdata/ You should see experiment-1/ -- this first experiment you are running if you run more, they will be experiment-2/, etc. So as the experiment proceeds you can see how its doing by ll expdata/experiment-1 You should see something like this: total 12 -rw-r--r-- 1 rlr car 225 Jul 26 11:18 experiment-1.in drwxr-xr-x 2 rlr car 1024 Jul 26 11:46 random-tr=0.01-tg=0.01 That means its so far working on the random move, threshold 0.01 and 0.05 cases. If you do this command: ll expdata/experiment-1/mm=random-tr=0.01-tg=0.01 you may see something like: -rw-r--r-- 1 rlr car 2750 Jul 26 11:46 case.log -rw-r--r-- 1 rlr car 18192 Jul 26 11:45 report.00 -rw-r--r-- 1 rlr car 18192 Jul 26 11:45 report.01 There are all the various reports files produced for that case (so far, if the experiment is still running). ** NOTE: If you see any files named stderr.00 (or 01 ...), that means you have an error in your control file, and your experiment is not running correctly. =================================================================== So...eventually...depending on what machine you are on, you will have a full set of 10 report files for each case (ie each combination of sweep values). Drone will send an email to you telling you the job is done. Now....how do we get averages out of all those??? We could of course do "tail" commands on them and add them up by hand (or type them into some stat package and have it do them). However, being lazy I have written a little script (in perl) to do the averaging and print it all out. Here is how you run it for experiment-1: getLastLineAvgsFromDirs.pl -c7 -f"report.*" -d"expdata/experiment-1" That says: extract the values from column 7 of the last line of each report file, in each subdirectory in expdata/experiment-1, and average them over each case. You should see something like this: Last-step data Wed Jul 24 14:56:56 EDT 2002 From directory: expdata/experiment-1 FilePattern: report.* Column numbers in files: 7 Then print average (and StdDev) across files, and average, StdDev and Max last time-step. Overall: Avgs (stddev) LastStep LastStep Directory Avg (sd) Max Column 7 ------------------------------ -------------------- ----------------- mm=random-tr=0.01-tg=0.01 201 ( 0) 201 12.500 ( 4.720) mm=random-tr=0.35-tg=0.35 201 ( 0) 201 69.400 ( 12.695) mm=random-tr=0.50-tg=0.50 201 ( 0) 201 89.300 ( 11.795) mm=random-tr=0.75-tg=0.75 201 ( 0) 201 145.300 ( 9.393) You can see it prints for each case, averaged over all the runs in that case: * the average last time step of the run (and the sd) * the average value of column 7 at the end of the run (and the sd) In this case, column 7 is the number of red agents who are fully segregated. (See the top of the report file from any tipping run to see the column headings...e.g., see the handout from last night.) One can then take these values and import them into some graphing program, e.g., to get a plot of number of fully segregated red agents vs threshold, one curve for each move method. On the CSCS machines, this could be done with xmgrace, mathematica, matlab, scilab, and so on. (There are pointers to xmgrace from the CSCS lab doc pages.) You could also move the data to a different computer (e.g. a wintel machine) and plot in Excell or whatever. **** Note: For the purposes of our collective experiment, please run the analysis program above, and then paste your results into the Swiki page off the course web page at www.pscs.umich.edu/~rlr/icpsr/ When prompted for a login and pw, enter: icpsr02 summer02 Click on the experiment 1 (or 2) page, and then click the Edit button. When the page with the editable area comes up, add an entry as described on the page, by using your mouse to copy/paste the output from the analysis program (eg as above) into the edit area. Then press the Save button. (If that doesn't work you could also email it to me at rlriolo@umich.edu) Given the data collected there, one (or more!) of us could produce graphs of the data. *** Also note: Feel free to run the same analysis program, but extract data from other columns, i.e., to see how other measures of segregation are affected by the parameters we swept. -------------------------------------------------------------------- Suppose we want to run an experiment to see how the randomMoveProb parameter affects the amount of segregation, say at a few threshold values and for each of the three move methods. To get a copy of a control file for an experiment to explore that: cp ~rlr/Courses/ICPSR-02/Experiments/Tipping2/exp2.ctrl exp2.ctrl Again, the lines at the top of the control file specifying the rootDir and the emailTo must be changed. Perhaps we could divide this experiment up as well: Account sum02** move-method randomMoveProb stopT ------- ------------------------------------------------------ ----- random 0.0005 0.001 0.002 0.005 0.01 0.02 0.05 500 random 0.0005 0.001 0.002 0.005 0.01 0.02 0.05 1000 nearest-open 0.0005 0.001 0.002 0.005 0.01 0.02 0.05 500 nearest-open 0.0005 0.001 0.002 0.005 0.01 0.02 0.05 1000 nearest-ok 0.0005 0.001 0.002 0.005 0.01 0.02 0.05 500 nearest-ok 0.0005 0.001 0.002 0.005 0.01 0.02 0.05 1000 Note we also have two stopT cases, to see if the system has settled down by those time periods. Again, after you edit the control file, and save it, you can run this with drone exp2.ctrl Looking in the expdata/ directory: ll expdata/ should reveal to directories, the original experiment-1/ and now the new experiment-2/ , which has the cases as specified in the sweep lines of exp2.ctrl file. ------------------------------------------------------------- Feel free to come up with other experiments. Please don't run them for too many time steps (eg 100,000!) or too many runs per case, or to many cases. ============================================================================= In general, you will want to develop some kind of scripts or use other tools to automate the process of collecting and analyzing data "across cases" of an experiment. An alternative approach is to do the multiple runs for the varying cases all within one run of a program. I don't favor that approach because: - it makes the program with the model implementation more complex - it means you do one long run, and can't divide it up into runs on many machines - it means you have to re-implement tools (eg averagers, etc) within each model you implement =============================================================================