$ontext =============================================================== LAST MODIFIED: 16 June, 2004 FILENAME: pprocess.gms PURPOSE: This tool performs all performance post-processing for a given set of trace files. For all paris of trace files, the routine runs the following performance tool routines: -solver square -resource time comparison -performance profiles REFERENCE: Performance World Website, Ptools section, online at http://www.gamsworld.org/performance/tools.htm. REQUIRED INPUTS: --trace1: GAMS trace file of solver 1 --trace2: GAMS trace file of solver 2 etc up to --trace8 --ntrace1: GENERIC data file 1 --ntrace2: GENERIC data file 2 up to --ntrace8 Users cannot input two tracefile numbers of different type with the same (n)trace number, i.e. --trace1=... --ntrace1 is not allowed, only --trace1=... --ntrace2=... OPTIONAL INPUTS: --outdir: output directory (default: results/) --outfile: output filename (default: results.htm) OPTIONAL PROFILE INPUTS: --factor factor for values of tau: 1, factor, factor^2, factor^3 (default 1.25) --numpts: number of data points for each solver for performance profile (default: 25) --delta actual relative tolerance for having exact best objective value. Because of rounding errors, this should generally not be exactly zero. (default: 1e-5) --resmin: minimum resource time threshold. If a solver reports a resource time below --resmin, then it is increased to resmin before being used in ratios. (default 0.05). OPTIONAL SQUARE INPUTS: --bnd: relative objective function difference. Prints "optimal" solution in boldface for HTML output for one solver if the relative objective function difference is greater than --bnd. (default 1e-5) --resmin: minimum resource time threshold. If both solvers report a resource time below --resmin, then no ratio is reported. In this case, both solver times are considered the same (default 0.05). OPTIONAL RESTIME INPUTS: --bnd: real. Bound for optimal outcome. Outcome is optimal if both solvers scaled solutions are within bnd of eachother, i.e. relative error (default: 1e-5) --resmin: minimum resource time threshold. If both solvers report a resource time below --resmin, then no ratio is reported. In this case, both solver times are considered the same (default 0.05). --tsame: bound in % for solver res. times to be considered same (default 10%) --tfaster: bound in % for solver res. times to be considered much faster (default 50%) REMARKS: User must input one tracefiles for each solver. Currently works for LP, RMIP, NLP, RMINLP, MIP, and MINLP type models. Each trace file can have model/solve information for only 1 solver. For example trace1.trc can have info for all LPs in PerformanceLib for solver A, and trace2.trc can have information for all LPs using solver B. Tracefiles containing multiple solvers will cause error messages upon compilation. $offtext ============================================================== *=== Set default pporcess.gms options $if not set delta $set delta 1e-5 $if not set outfile $set outfile results $if not set outdir $set outdir results *=== Set default square.gms options $if not set bnd $set bnd 1e-5 $if not set resmin $set resmin 0.05 *=== Set default restime.gms options $if not set tsame $set tsame 10 $if not set tfaster $set tfaster 50 * === Error checks for GNUPLOTXY $ set sep \ $if %system.filesys% == UNIX $set sep / $if %system.filesys% == UNIX $set no_unixgnuplot yes $if not %system.filesys% == UNIX $set no_unixgnuplot no $if not exist "%gams.sysdir%inclib\gnupltxy.gms" $set no_gnuplotxy yes $if exist "%gams.sysdir%inclib\gnupltxy.gms" $set no_gnuplotxy no *=== Indicator if PAVER run $if not set paver $set paver 0 $if not set numpts $set numpts 30 *=== Determine trace and non-GAMS data files that are input $echo > tracedata $echo > alltracedata $echo > ntracedata $if set trace1 $echo '"%trace1%"' >> tracedata $if set trace1 $echo '"%trace1%"' >> alltracedata $if set trace1 $set trace1def 1 $if not set trace1 $set trace1def 0 $if set ntrace1 $echo '"%ntrace1%"' >> ntracedata $if set ntrace1 $echo '"%ntrace1%"' >> alltracedata $if set ntrace1 $set ntrace1def 1 $if not set ntrace1 $set ntrace1def 0 $if set trace2 $echo '"%trace2%"' >> tracedata $if set trace2 $echo '"%trace2%"' >> alltracedata $if set trace2 $set trace2def 1 $if not set trace2 $set trace2def 0 $if set ntrace2 $echo '"%ntrace2%"' >> ntracedata $if set ntrace2 $echo '"%ntrace2%"' >> alltracedata $if set ntrace2 $set ntrace2def 1 $if not set ntrace2 $set ntrace2def 0 $if set trace3 $echo '"%trace3%"' >> tracedata $if set trace3 $echo '"%trace3%"' >> alltracedata $if set trace3 $set trace3def 1 $if not set trace3 $set trace3def 0 $if set ntrace3 $echo '"%ntrace3%"' >> ntracedata $if set ntrace3 $echo '"%ntrace3%"' >> alltracedata $if set ntrace3 $set ntrace3def 1 $if not set ntrace3 $set ntrace3def 0 $if set trace4 $echo '"%trace4%"' >> tracedata $if set trace4 $echo '"%trace4%"' >> alltracedata $if set trace4 $set trace4def 1 $if not set trace4 $set trace4def 0 $if set ntrace4 $echo '"%ntrace4%"' >> ntracedata $if set ntrace4 $echo '"%ntrace4%"' >> alltracedata $if set ntrace4 $set ntrace4def 1 $if not set ntrace4 $set ntrace4def 0 $if set trace5 $echo '"%trace5%"' >> tracedata $if set trace5 $echo '"%trace5%"' >> alltracedata $if set trace5 $set trace5def 1 $if not set trace5 $set trace5def 0 $if set ntrace5 $echo '"%ntrace5%"' >> ntracedata $if set ntrace5 $echo '"%ntrace5%"' >> alltracedata $if set ntrace5 $set ntrace5def 1 $if not set ntrace5 $set ntrace5def 0 $if set trace6 $echo '"%trace6%"' >> tracedata $if set trace6 $echo '"%trace6%"' >> alltracedata $if set trace6 $set trace6def 1 $if not set trace6 $set trace6def 0 $if set ntrace6 $echo '"%ntrace6%"' >> ntracedata $if set ntrace6 $echo '"%ntrace6%"' >> alltracedata $if set ntrace6 $set ntrace6def 1 $if not set ntrace6 $set ntrace6def 0 $if set trace7 $echo '"%trace7%"' >> tracedata $if set trace7 $echo '"%trace7%"' >> alltracedata $if set trace7 $set trace7def 1 $if not set trace7 $set trace7def 0 $if set ntrace7 $echo '"%ntrace7%"' >> ntracedata $if set ntrace7 $echo '"%ntrace7%"' >> alltracedata $if set ntrace7 $set ntrace7def 1 $if not set ntrace7 $set ntrace7def 0 $if set trace8 $echo '"%trace8%"' >> tracedata $if set trace8 $echo '"%trace8%"' >> alltracedata $if set trace8 $set trace8def 1 $if not set trace8 $set trace8def 0 $if set ntrace8 $echo '"%ntrace8%"' >> ntracedata $if set ntrace8 $echo '"%ntrace8%"' >> alltracedata $if set ntrace8 $set ntrace8def 1 $if not set ntrace8 $set ntrace8def 0 $echo 'empty' >> tracedata $echo 'empty' >> ntracedata *=== Create set of all input files (trace and non-GAMS data) $set alltracefile alltracedata Set allfiles "tracefiles and ntracefiles" / $onempty $include %alltracefile% $offempty /; Alias (allfilesA, allfilesB, allfiles); *=== Create set of trace files $set tracefile tracedata Set tfiles "tracefiles" / $onempty $include %tracefile% $offempty /; Alias (tfilesA, tfilesB, tfiles); *=== Create set of non-GAMS data files $set ntracefile ntracedata Set ntfiles "ntracefiles" / $onempty $include %ntracefile% $offempty /; Alias (ntfilesA, ntfilesB, ntfiles); display tfiles, ntfiles, allfiles; *=== Copy trace files to output directory execute 'mkdir %outdir%' file fcp /fcopy.tmp/ if(%trace1def%, put fcp '$call cp -f %trace1% %outdir%%sep%' /); if(%trace2def%, put fcp '$call cp -f %trace2% %outdir%%sep%' /); if(%trace3def%, put fcp '$call cp -f %trace3% %outdir%%sep%' /); if(%trace4def%, put fcp '$call cp -f %trace4% %outdir%%sep%' /); if(%trace5def%, put fcp '$call cp -f %trace5% %outdir%%sep%' /); if(%trace6def%, put fcp '$call cp -f %trace6% %outdir%%sep%' /); if(%trace7def%, put fcp '$call cp -f %trace7% %outdir%%sep%' /); if(%trace8def%, put fcp '$call cp -f %trace8% %outdir%%sep%' /); if(%ntrace1def%, put fcp '$call cp -f %ntrace1% %outdir%%sep%' /); if(%ntrace2def%, put fcp '$call cp -f %ntrace2% %outdir%%sep%' /); if(%ntrace3def%, put fcp '$call cp -f %ntrace3% %outdir%%sep%' /); if(%ntrace4def%, put fcp '$call cp -f %ntrace4% %outdir%%sep%' /); if(%ntrace5def%, put fcp '$call cp -f %ntrace5% %outdir%%sep%' /); if(%ntrace6def%, put fcp '$call cp -f %ntrace6% %outdir%%sep%' /); if(%ntrace7def%, put fcp '$call cp -f %ntrace7% %outdir%%sep%' /); if(%ntrace8def%, put fcp '$call cp -f %ntrace8% %outdir%%sep%' /); putclose fcp; execute 'gams fcopy.tmp'; execute 'rm fcopy.tmp'; *========================================================== *=== Run performance tools *========================================================== *=== Parmfile with all trace files file ftrace/ "%gams.scrdir%tfiles.txt" /; put ftrace; loop(allfiles, if(tfiles(allfiles), put ftrace /' --trace' ord(allfiles):0:0 '=' allfiles.tl:0; else put ftrace /' --ntrace' ord(allfiles):0:0 '=' allfiles.tl:0; ) ); putclose ftrace; file ftmp / pprocessbat.gms /; *=== Run solver square, resource time, and performance ==== *=== profile utilities ==================================== put ftmp / '*=== Batch runs for solver square utility' //; loop( (allfilesA, allfilesB)$(not sameas(allfilesA,allfilesB) and ord(allfilesB)>ord(allfilesA) ), put ftmp '$call =gams square.gms '; if(tfiles(allfilesA), put ' --trace1=' allfilesA.tl:0; else put ' --ntrace1=' allfilesA.tl:0; ); if(tfiles(allfilesB), put ' --trace2=' allfilesB.tl:0; else put ' --ntrace2=' allfilesB.tl:0; ); put ' --outfile=' allfilesA.tl:0 '_' allfilesB.tl:0 ' --bnd=%bnd% --resmin=%resmin%' /; put ftmp '$if errorlevel 1 $abort "probs with square"%' /; put ftmp '$call mv -f ' allfilesA.tl:0 '_'allfilesB.tl:0 '_sqr.htm %outdir%%sep%' /; ); put ftmp / '*=== Batch runs for resource time utility' //; loop( (allfilesA, allfilesB)$(not sameas(allfilesA,allfilesB) and ord(allfilesB)>ord(allfilesA) ), put ftmp '$call =gams restime.gms '; if(tfiles(allfilesA), put ' --trace1=' allfilesA.tl:0; else put ' --ntrace1=' allfilesA.tl:0; ); if(tfiles(allfilesB), put ' --trace2=' allfilesB.tl:0; else put ' --ntrace2=' allfilesB.tl:0; ); put ' --outfile=' allfilesA.tl:0 '_' allfilesB.tl:0 ' --bnd=%bnd% --resmin=%resmin% --tsame=%tsame%' ' --tfaster=%tfaster%' /; put ftmp '$call mv -f ' allfilesA.tl:0 '_' allfilesB.tl:0 '_res.htm %outdir%%sep%' /; put ftmp '$call mv -f ' allfilesA.tl:0 '_' allfilesB.tl:0 '_resA.htm %outdir%%sep%' /; ); put ftmp //; * set objwithin/0 "0","0.01" "1","0.02" "2","0.05" "5","0.10" "10","0.50" "50",Inf "Inf"/; set objwithin/0 "0",Inf "Inf"/; put ftmp / '*=== Batch runs for performance profile utility' //; if(%no_unixgnuplot% or %no_gnuplotxy%, loop(objwithin, put ftmp '$call =gams pprofile.gms parmfile="%gams.scrdir%tfiles.txt" '; put ftmp ' --outfile=profile_' objwithin.te(objwithin):0 '.txt '; if( sameas(objwithin,'0'), put ftmp ' --resmin=%resmin% ' ' --delta=%delta% ' /; else put ftmp ' --resmin=%resmin% ' ' --delta=' objwithin.tl:0 /; ); ); else loop(objwithin, put ftmp '$call gams plotprof.gms parmfile="%gams.scrdir%tfiles.txt" '; put ftmp ' --otype=png --figname=profile_' objwithin.te(objwithin):0 ; if( sameas(objwithin,'0'), put ' --delta= %delta% ' else put ' --delta=' objwithin.tl:0 ; ); put ' --numpts=%numpts% --outfile=profile_' objwithin.te(objwithin):0 '.txt ' * ' --factor=%factor% --resmin=%resmin%' /; /; put ftmp '$call mv -f profile_' objwithin.te(objwithin):0 '.png %outdir%\profile_' objwithin.te(objwithin):0 '.png' /; put ftmp '$call gams plotprof.gms parmfile="%gams.scrdir%tfiles.txt" '; put ftmp ' --otype=postscript --figname=profile_' objwithin.te(objwithin):0 ; if( sameas(objwithin,'0'), put ' --delta= %delta% ' else put ' --delta=' objwithin.tl:0 ; ); put ' --numpts=%numpts% --outfile=profile_' objwithin.te(objwithin):0 '.txt ' * ' --factor=%factor% --resmin=%resmin%' /; /; put ftmp '$call mv -f profile_' objwithin.te(objwithin):0 '.eps %outdir%\' /; ); ); loop(objwithin, put ftmp / '$call mv -f profile_' objwithin.te(objwithin):0 '.txt %outdir%%sep%' /; ); put ftmp '$call gams exceltrace.gms'; loop(allfiles, put ftmp ' --trace' ord(allfiles):0:0 '=' allfiles.tl:0; ); put /; put ftmp '$call mv -f exceltrace.csv %outdir%%sep%'; put ftmp / '$call gams timings.gms '; loop(allfiles, put ' --trace' ord(allfiles):0:0 '=' allfiles.tl:0; ); put ftmp / '$call mv -f timings.txt %outdir%%sep%'; put ftmp / '$call mv -f timings.htm %outdir%%sep%'; put /; putclose ftmp; execute '=gams pprocessbat.gms'; scalar rc; rc=errorlevel; abort$rc 'problems with pprocessbat.gms'; execute 'mv -f pprocessbat.gms %outdir%%sep%' *========================================================== *=== Create main performance summary HTM file *========================================================== file fhtm / "%outdir%/%outfile%.htm" /; put fhtm; put / ' Performance Results' / '' / '' / '

Performance Results

' /; if(%paver%, put / 'Results created using the PAVER Server:  ' / '' / 'http://www.gamsworld.org/performance/paver' / 'For a list of model and solve status codes see the ' / 'Status Codes Section.' / '

'; ); put / '' / '' / ' ' / ' ' / '' / '
Date / Time: %system.date% %system.time%
' / '
'; put / 'This page is a summary of the performance tests, using the' / 'solver square, resource time, and performance profile utilities.' / 'The Performance Tools are described in detail in the Performance' / 'World webpage at ' / 'http://www.gamsworld.org/performance.' put / '

' / '

Trace files used:

'; put / '' loop(allfiles, put /'' / ' ' / ' ' / ''; ); put / '
Tracefile ' ord(allfiles):0:0 ':' allfiles.tl:0 '

'/; *=== Performance profile summary ========================== put / '


' / '

Performance profile summary:

' / 'The profile utility compares solver outcomes of any number of ' / 'given solvers (up to 8) over a model test set. The tool is ' / 'described in detail at ' / '' / 'http://www.gamsworld.org/performance/pprofile.htm.' / '


' /; if(%no_unixgnuplot%, put / '

Warning:

' / 'Gnuplotxy, which automates plotting of results, does ' / 'not exist for UNIX, only for Windows. Performance profile' / 'result data is available only as a textfile (not as a' / 'plot). You will have to plot the results manually.' /; put / ''; elseif (%no_gnuplotxy% and not %no_unixgnuplot%), put / '

Warning:

' / 'Gnuplotxy, which automates plotting of results is not ' / 'installed on your machine. Please download GNUPLOTXY at' / '' / 'http://ageco.tamu.edu/faculty/mccarl/gnuplot/gnuplot.html' / 'if you wish to automate performance profile plotting.' / 'Performance profile result data is available only as a' / 'textfile (not as a plot). You will have to plot the results' / 'manually.' /; put / ''; else put / '' ); *=== Solver square summary ================================ put / '


' / '

Solver square summary:

' / 'The solver square utility compares solver outcomes of two given' / 'solvers over a model test set. The tool is described in detail' / 'at ' / 'http://www.gamsworld.org/performance/square.htm.' / '


' /; put / '' / '' / ' '; loop(allfilesB, put / ' '; ); put / ''; loop(allfilesA, put / '' / ' '; loop(allfilesB, if(ord(allfilesB)>ord(allfilesA), put / ' '; else put / ' '; ); ); put / ''; ); put / '
' allfilesB.tl:0 '
' allfilesA.tl:0 'results--
'; *=== Resource time summary ================================ put / '


' / '

Resource time summary:

'; put / 'The resource time utility compares solver resource times of two' / 'given solvers over a model test set. ' / '

The tool is described in' / 'detail at ' / 'http://www.gamsworld.org/performance/restime.htm.' / '


' /; put / '' / '' / ' '; loop(allfilesB, put / ' '; ); put / ''; loop(allfilesA, put / '' / ' '; loop(allfilesB, if(ord(allfilesB)>ord(allfilesA), put / ' '; else put / ' '; ); ); put / ''; ); put / '
' allfilesB.tl:0 '
' allfilesA.tl:0 'results' '--
'; put / '

' / 'For simple timing information, see the resource time benchmarking' / 'tables:' / '

' / '

'; putclose fhtm; *========================================================== *=== Create performance profile summary HTM file *=== profile_summary.htm *========================================================== file fhtm2 / "%outdir%/profile_summary.htm" /; put fhtm2; put / ' Performance Profile Summary' / '' / '' / '

Performance Profile Summary

' /; put / '' / '' / ' ' / ' ' / '' / '
Date / Time: %system.date% %system.time%
' / '

'; put / 'Performance profiles for different objective value tolerances. '; put / 'The profile utility compares solver outcomes of any number of given ' / 'solvers (up to 8) over a model test set. The tool is described in detail' / 'at ' / 'http://www.gamsworld.org/performance/pprofile.htm.'; put / '

' / 'A solver is considered feasible or optimal, if it has the proper model' / 'and solver return codes and the relative objective value error is within' / 'n % of the best possible solution. If the best found objective' / 'value is 0, then the absolute objective value error is used.'; put / '

' / 'The following relative objective value tolerances are used:' / ''; put / ''; loop(objwithin, put /''; ); put / '
' /' '; if( sameas(objwithin,'Inf'), put /' ' /' ' /' ' /' '; elseif( sameas(objwithin,'0')), put /' ' /' ' /' ' /' '; else put /' ' /' Data File: profile_' objwithin.te(objwithin):0 '.txt' /' ' /' '; ); put /'
Efficiency Only (within ' objwithin.te(objwithin):0 '% ' 'wrt to best value)
Data File: profile_' objwithin.te(objwithin):0 '.txt
Black and White EPS file - ' /' profile_' objwithin.te(objwithin):0 '.eps
Quality of Solution (within ' objwithin.te(objwithin):0 '% ' 'wrt to best value)
Data File: profile_' objwithin.te(objwithin):0 '.txt
Black and White EPS file: ' /' profile_' objwithin.te(objwithin):0 '.eps
(Within ' objwithin.te(objwithin):0 '% ' 'wrt to best value)
Black and White EPS file: ' /' profile_' objwithin.te(objwithin):0 '.eps
'; put /'
'; putclose fhtm2; *=========================================================== *=== Create HTM file with GAMS model and solver status codes *=========================================================== file fhtm3 / "%outdir%/status_codes.htm" /; put fhtm3; put / ' GAMS Model and Solver Status Codes' / '' / '' / '

GAMS Model and Solver Status Codes

' /; put / 'Below is the list of GAMS model and solver status codes:' / '

' put / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / '' / '' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '' / ' ' / ' ' / '' / '
MODEL STATUS CODEDESCRIPTION
1Optimal
2Locally Optimal
3Unbounded
4Infeasible
5Locally Infeasible
6Intermediate Infeasible
7Intermediate Nonoptimal
8Integer Solution
9Intermediate Non-Integer
10Integer Infeasible
11Licensing Problems - No Solution
12Error Unknown
13Error No Solution
14No Solution Returned
15Solved Unique
16Solved
17Solved Singular
18Unbounded - No Solution
19Infeasible - No Solution
SOLVER STATUS CODEDESCRIPTION
1Normal Completion
2Iteration Interrupt
3Resource Interrupt
4Terminated by Solver
5Evaluation Error Limit
6Capability Problems
7Licensing Problems
8User Interrupt
9Error Setup Failure
10Error Solver Failure
11Error Internal Solver Error
12Solve Processing Skipped
13Error System Failure
' / '

'; putclose fhtm3; *=== Output message(s) ==================================== file fstat /status.log/; put fstat '---' / '--- Result summary file is %outfile%.htm, located in' / '--- folder %outdir%/.' / '---' putclose fstat; execute 'head status.log' execute "rm tracedata ntracedata"