Performance World [ Performance World Home | Board | Tools | PerformanceLib | Links | GamsWorld group | Search | Contact ]

Comparing solver outcomes of two solvers

The Performance Tool solver outcome comparison utility, square.gms, is useful if users wish to compare the model/solver outcomes of two different solvers on a given model collection. Possible solver outcomes are optimal, infeasible, unbounded, interrupt (time or iteration) or fail. The results are displayed in a square, where each square element shows the number of models of that solver outcome pair. Output is in the form of an HTML file, where each square entry is a link to the associated models.

It is particularly useful for It is not useful for The resource time utility is more suited for this type of comparison. Below are descriptions on how to use the utility and its command line options.

Download:

Sample Output:




Using square.gms

The solver square comparison utility makes use of the --option to initialize environment variables from the command line. You must have a GAMS system 20.1 or later installed. See the GAMS 20.1 release notes for details on this feature.

The general command for square.gms is:

>>   gams square --trace1=(input 1 tracefile) --trace2=(input 2 tracefile) --outfile=(output filename) --modelfile=(optional file containing model names) --resmin=(lower bound on resource usage)

The options --trace1 and --trace2 are required inputs. All of the other --options are optional, that is if ommitted they are set to default options as described in the next section. A sample call is:

>>   gams square --trace1=trace1.trc --trace2=trace2.trc --outfile=out

This will create an output HTML file called out_sqr.htm showing the LP or NLP solver square results.

The tracefiles trace1.trc and trace2.trc can each have results for only a single solver on a given set of models. For example, trace1.trc can have results for CPLEX on the NETLIB models, and trace2.trc the results using XPRESS on NETLIB.

The utility also allows the ability to only consider a subset of the models listed in the tracefiles. Suppose the tracefiles trace1.trc and trace2.trc contain all LPs from LINLib and we are only interested in results for all of the medium sized models. If we have a file containing all of the medium sized models called medium_mod.txt, then we can include this using the --modelfile option. The sample call

>>   gams square --trace1=trace1.trc --trace2=trace2.trc --modelfile=medium_mod.txt --outfile=out

creates output comparing only the medium sized models in the two tracefiles. Users can download the following files:

and run square.gms with the command example given above with or without the --modelfile=medium_mod.txt option.

Top


List of square.gms --options and defaults


Required Inputs:

Option Input Type Description Default
--trace1 Text Trace file of solver 1 none (user must specify)
--trace2 Text Trace file of solver 2 none (user must specify)


Optional Inputs:

Option Input Type Description Default
--bnd Real Relative objective function difference. Prints "optimal" solution in boldface for HTML output for one solver if the relative objective function difference is greater than --bnd. 1e-5
--modelfile Text Filename containing models to be considered in comparison. For example, users may wish to only consider small models. Users can include a text file called medium_mod.txt containing all medium sized models. none
--outfile Text Root of filename output. Output files will be of the form (--outfile)_sol.htm and (--outfile)_sol.txt. out (creates files: out_sol.htm and out_sol.txt)
--resmin Real Lower bound on resource usage. If both solvers report resource times below --resmin, then neither solver is considered faster and the resource time ratio is listed as ---. 0.05

Top