Performance World [ Performance World Home | Board | Tools | PerformanceLib | Links | GamsWorld group | Search | Contact ]

Checking solver resource times of two solvers

The Performance Tool solver resource time comparison utility, restime.gms, is useful if users wish to compare solver resource times of two different solvers on a given model collection. Output is in the form of an HTML file, where each resource time table entry is a link to the associated models. It is particularly useful for

Below are descriptions on how to use the utility and its command line options.


Sample Outputs:

Using restime.gms

The solver resource time comparison utility makes use of the --option to initialize environment variables from the command line. You must have a GAMS system 20.1 or later installed. See the GAMS 20.1 release notes for details on this feature.

The general command for restime.gms is:

>>   gams restime --bnd=(bound for optimal solution error) --trace1=(input 1 tracefile) --trace2=(input 2 tracefile) --outfile=(output filename) --modelfile=(optional file containing model names) --tsame=(bound in % when solvers are considered same) --tfaster=(bound in % when solvers are considered faster) --resmin=(lower bound on resource usage)

The options --trace1 and --trace2 are required inputs. All of the other --options are optional, that is if ommitted they are set to default options as described in the next section. A sample call is:

>>   gams restime --trace1=trace1.trc --trace2=trace2.trc --outfile=out

This will create an output HTML file called trace_res.htm showing the solver resource time comparisons. The output file considers all models listed in the trace files, including those that were not solved optimally by both solvers.

The tracefiles trace1.trc and trace2.trc can each have results for only a single solver on a given set of models. For example, trace1.trc can have results for CPLEX on the NETLIB models, and trace2.trc the results using XPRESS on NETLIB.

The utility also allows the ability to only consider a subset of the models listed in the tracefiles. Suppose the tracefiles trace1.trc and trace2.trc contain all LPs from PerformanceLib and we are only interested in results for all of the medium sized models. If we have a file containing all of the medium sized models called medium_mod.txt, then we can include this using the --modelfile option. The sample call

>>   gams restime--trace1=trace1.trc --trace2=trace2.trc --modelfile=medium_mod.txt --outfile=out

creates output comparing only the medium sized models in the two tracefiles. Users can download the following files:

and run restime.gms with the command example given above with or without the --modelfile=medium_mod.txt option.


List of restime.gms --options and defaults

Required Inputs:

Option Input Type Description Default
--trace1 Text Trace file of solver 1 none (user must specify)
--trace2 Text Trace file of solver 2 none (user must specify)

Optional Inputs:

Option Input Type Description Default
--bnd Real Relative objective function difference. Prints "optimal" solution in boldface for HTML output for one solver if the relative objective function difference is greater than --bnd. 1e-5
--modelfile Text Filename containing models to be considered in comparison. For example, users may wish to only consider small models. Users can include a text file called medium_mod.txt containing all medium sized models. none
--outfile Text Root of filename output. Output files will be of the form (--outfile)_sol.htm and (--outfile)_sol.txt. out (creates files: out_sol.htm and out_sol.txt)
--resmin Real Lower bound on resource usage. If both solvers report resource times below --resmin, then neither solver is considered faster and the resource time ratio is listed as ---. 0.05
--tsame Real (in %) A solver is considered as fast as another, if resource times are within --tsame % of eachother. 10 (i.e. 10%)
--tfaster Real (in %) A solver is considered faster than another, if resource times are within --tfaster % of eachother, but not within --tsame % of eachother. A solver is considered much faster than another, if resource times are more than --tfaster % apart of one another. 50 (i.e. 50%)