Performance Profiles of Solvers
Performance profiles provide an effective means to compare solver
performance for several solvers at once, eliminating some of the bias
certain comparisons have. For some comparisons, a small number of problems
may unduly influence the results.
Initial performance comparisons were developed in
[1]
by Billups, Dirkse,
and Ferris.
In
[2]
Dolan and More expanded this approach to compare
solver performance using performanc profiles. In their
approach, they define the performance profile of a solver as a (cumulative)
distribution function for a performance metric. In particular, they
use the ratio of the solver resource time for a given solver versus
the best time of all the solvers.
The profile computation routine is
pprofile.gms. The user inputs different trace files, where
each trace file contains information for one solver only. Users
can input a maximum of 8 different trace files. The GAMS routine
creates a text file containing a table of the performance profiles,
which can be used to plot using a variety of software packages, such
as Gnuplot, Excel or others.
For Windows users, we also have a plotting routine available, which
automates the profile creation and plotting in one step. The routine
is called plotprof.gms and makes use
of the
GAMS/GNUPLOT interface,
for plotting from within GAMS.
Below are descriptions on how to use the utility and its command line options.
Download:
 pprofile.gms:
creates performance profile text file
 plotprof.gms:
plots performance profiles (currently only under Windows
using gnuplotxy)

gnuplotxy:
GAMS/Gnuplot interface for plotting under Windows
Sample Outputs:
Background on Performance Profiles
Much of the following information comes from
[1].
Definitions:
Suppose we solved a set of models SM with different solvers. Let
p denote a particular problem (model) and s a particular
solver. The idea is to compare the performance of solver s
on problem p with the best performance by any solver on this
particular problem. Dolan and More define the performance ratio
as
t_(p,s)
rho(p,s) = 
min{t_(p,s): 1<= s <=n_s}
where t_(p,s) refers to the time solver s spent on
problem p and n_s refers to the number of problems
in the model test set. For practical purposes, if a solver does not
solve a problem, we set it's performance ratio to infinity.
In order to obtain an overall assessment of a solver on the given model
test set, we define a cumulative distribution function
P_s (tau):
1
P_s(tau) =  size{ p in SM: rho(p,s) <= tau}.
n_p
P_s (tau) is the probability that a performance ratio
rho(p,s) is within a factor of tau of the best
possible ratio.
Interpretation:
In general, P_s (tau) for a particular solver s
gives information on the percentage of models that the solver will
solve if for each model, the solver can have a maximum resource
time of tau times the minimum time.
For tau=1 the probability P_s (1) of a particular solver is
the probability that the solver will win over all the others. Therefore, if
users are only interested the number of actual wins, users need only
consider the values of P_s (1) of all the solvers.
For large values of tau the probability function P_s (tau)
gives information if a solver actually solves a problem. Thus, if users
are interested only in the probability that a solver will be succesful,
users should consider P_s (tau) for all solvers as tau
becomes large.
Using pprofile.gms
The solver performance profile utility makes use of the
option to initialize environment variables from the command line.
You must have a GAMS system 20.1 or later installed.
See the
GAMS 20.1 release notes
for details on this feature.
The general command for pprofile.gms is:
>>
gams pprofile trace1=(input 1 tracefile) trace2=(input 2 tracefile)
... trace[n]=
factor=(factor for computing values of cutpoint tau)
outfile=(output filename)
modelfile=(optional file containing model names)
numpts=(number of data points for cutpoint tau)
resmin=(lower bound on resource usage)
objwithin=relative objc. function error

The options trace1 and trace2, ..., trace[n] are required inputs.
All of the other options are optional, that is if ommitted they are
set to default options as described in the next section.
A sample call is:
>>
gams pprofile trace1=A.trc trace2=B.trc trace3=C.trc
outfile=profile.txt numpts=10 factor=1.5

This will create a text file profile.txt
showing the performance profiles of the solvers listed in
the trace files. Users can use software such as Excel or GNUPLOT
to plot the resulting performance profiles.
The tracefiles trace1.trc and trace2.trc can each have results for only
a single solver on a given set of models. For example, trace1.trc can have
results for CPLEX on the NETLIB models, and trace2.trc the results using
XPRESS on NETLIB.
Users can download the following files:
Top
Using plotprof.gms
The routine plotprof.gms is a plotting routine which automates
the profile creation routine (using pprofile.gms) and the plotting
step in one task. Plotprof.gms works only on Windows systems, and users
must have
GAMS / GNUPLOT installed.
The GAMS call to plotprof.gms is exactly the same as for pprofile.gms.
A sample call is:
>>
gams plotprof trace1=A.trc trace2=B.trc trace3=C.trc
outfile=profile.txt numpts=10 factor=1.5 otype=png
figname=profileGnuplot

This creates the sample GNUPLOT plot
profileGnuplot.png. Users may also
use the profile data file created and plot using Excel or similar
plotting software.
Users can download the following files:
Top
Reference:
 [1] Billups, Stephen C., Dirkse, Steven P., and Ferris, Michael C.,
"A comparison of large scale mixed complementarity problem solvers",
Computational Optimizatin and Applications, 7, pp. 325, 1997.
 [2] Dolan, Elizabeth D. and More, Jorge J., "Benchmarking optimization
software with performance profiles", Mathematical Programming,
Volume 91 (2), pp. 201213, 2002.
Top
List of pprofile.gms and plotprof.gms options and defaults
The required and optional input parameters for pprofile.gms
and plotprof.gms are identical.
Required Inputs:
Option 
Input Type 
Description 
Default 
trace1 
Text 
Trace file of solver 1

none (user must specify) 
trace[n] 
Text 
Trace file of solver n. Users can specify up to 8 tracefiles

none (user must specify) 
Optional Inputs:
Option 
Input Type 
Description 
Default 
factor 
Real 
Factor for computing values the values of the cutpoint tau. Values
of tau are computed tau_(n+1)=tau_n*(factor), where tau_1=1.

1.25 
modelfile 
Text 
Filename containing models to be considered in comparison. For example,
users may wish to only consider small models. Users can include a text
file called medium_mod.txt containing all medium sized models.
 none 
objwithin 
Text 
Relative objective function error. If set then "success" is defined
only if solution within objwithin of best solution (using
relative error).
 Inf (i.e. no quality of solution metric) 
outfile 
Text 
Output filename.
 profile.txt 
numpts 
Integer 
Number of data points for cutpoint tau.
 25 
resmin 
Real (in %) 
Minimum resource time threshold. If a solver reports a
resource time below resmin, then it is increased to
resmin before being used in ratios.
 0.05 
Optional Inputs for plotprof.gms only:
Option 
Input Type 
Description 
Default 
figname 
Text 
Output filename of figure. 
fig1.* where * is the appropriate extension (.gif or .ps  none for windows output) 
otype 
Text 
Output type: windows, png, or postscript 
windows (i.e. to screen) 
Top