Performance World [ Performance World Home | Board | Tools | PerformanceLib | Links | GamsWorld group | Search | Contact ]

Performance Profiles of Solvers

Performance profiles provide an effective means to compare solver performance for several solvers at once, eliminating some of the bias certain comparisons have. For some comparisons, a small number of problems may unduly influence the results.

Initial performance comparisons were developed in [1] by Billups, Dirkse, and Ferris. In [2] Dolan and More expanded this approach to compare solver performance using  performanc profiles. In their approach, they define the performance profile of a solver as a (cumulative) distribution function for a performance metric. In particular, they use the ratio of the solver resource time for a given solver versus the best time of all the solvers.

The profile computation routine is pprofile.gms. The user inputs different trace files, where each trace file contains information for one solver only. Users can input a maximum of 8 different trace files. The GAMS routine creates a text file containing a table of the performance profiles, which can be used to plot using a variety of software packages, such as Gnuplot, Excel or others.

For Windows users, we also have a plotting routine available, which automates the profile creation and plotting in one step. The routine is called plotprof.gms and makes use of the GAMS/GNUPLOT interface, for plotting from within GAMS.

Below are descriptions on how to use the utility and its command line options.

Download:

Sample Outputs:




Background on Performance Profiles

Much of the following information comes from [1].

Definitions:

Suppose we solved a set of models SM with different solvers. Let p denote a particular problem (model) and s a particular solver. The idea is to compare the performance of solver s on problem p with the best performance by any solver on this particular problem. Dolan and More define the performance ratio as 
                      t_(p,s)
   rho(p,s) = ---------------------------
                min{t_(p,s): 1<= s <=n_s}
where t_(p,s) refers to the time solver s spent on problem p and n_s refers to the number of problems in the model test set. For practical purposes, if a solver does not solve a problem, we set it's performance ratio to infinity.

In order to obtain an overall assessment of a solver on the given model test set, we define a cumulative distribution function P_s (tau): 
                1
   P_s(tau) = ----- size{ p in SM: rho(p,s) <= tau}. 
               n_p
P_s (tau) is the probability that a performance ratio rho(p,s) is within a factor of tau of the best possible ratio.

Interpretation:

In general, P_s (tau) for a particular solver s gives information on the percentage of models that the solver will solve if for each model, the solver can have a maximum resource time of tau times the minimum time.

For tau=1 the probability P_s (1) of a particular solver is the probability that the solver will win over all the others. Therefore, if users are only interested the number of actual wins, users need only consider the values of P_s (1) of all the solvers.

For large values of tau the probability function P_s (tau) gives information if a solver actually solves a problem. Thus, if users are interested only in the probability that a solver will be succesful, users should consider P_s (tau) for all solvers as tau becomes large.


Using pprofile.gms

The solver performance profile utility makes use of the --option to initialize environment variables from the command line. You must have a GAMS system 20.1 or later installed. See the GAMS 20.1 release notes for details on this feature.

The general command for pprofile.gms is:

>>   gams pprofile --trace1=(input 1 tracefile) --trace2=(input 2 tracefile) ... trace[n]= --factor=(factor for computing values of cutpoint tau) --outfile=(output filename) --modelfile=(optional file containing model names) --numpts=(number of data points for cutpoint tau) --resmin=(lower bound on resource usage) --objwithin=relative objc. function error

The options --trace1 and --trace2, ..., --trace[n] are required inputs. All of the other --options are optional, that is if ommitted they are set to default options as described in the next section. A sample call is:

>>   gams pprofile --trace1=A.trc --trace2=B.trc --trace3=C.trc --outfile=profile.txt --numpts=10 --factor=1.5

This will create a text file profile.txt showing the performance profiles of the solvers listed in the trace files. Users can use software such as Excel or GNUPLOT to plot the resulting performance profiles.

The tracefiles trace1.trc and trace2.trc can each have results for only a single solver on a given set of models. For example, trace1.trc can have results for CPLEX on the NETLIB models, and trace2.trc the results using XPRESS on NETLIB.

Users can download the following files:

Top


Using plotprof.gms

The routine plotprof.gms is a plotting routine which automates the profile creation routine (using pprofile.gms) and the plotting step in one task. Plotprof.gms works only on Windows systems, and users must have GAMS / GNUPLOT installed.

The GAMS call to plotprof.gms is exactly the same as for pprofile.gms. A sample call is:

>>   gams plotprof --trace1=A.trc --trace2=B.trc --trace3=C.trc --outfile=profile.txt --numpts=10 --factor=1.5 --otype=png --figname=profileGnuplot

This creates the sample GNUPLOT plot profileGnuplot.png. Users may also use the profile data file created and plot using Excel or similar plotting software.

Users can download the following files:

Top


Reference:

Top


List of pprofile.gms and plotprof.gms --options and defaults

The required and optional input parameters for pprofile.gms and plotprof.gms are identical.

Required Inputs:

Option Input Type Description Default
--trace1 Text Trace file of solver 1 none (user must specify)
--trace[n] Text Trace file of solver n. Users can specify up to 8 tracefiles none (user must specify)


Optional Inputs:

Option Input Type Description Default
--factor Real Factor for computing values the values of the cutpoint tau. Values of tau are computed tau_(n+1)=tau_n*(--factor), where tau_1=1. 1.25
--modelfile Text Filename containing models to be considered in comparison. For example, users may wish to only consider small models. Users can include a text file called medium_mod.txt containing all medium sized models. none
--objwithin Text Relative objective function error. If set then "success" is defined only if solution within --objwithin of best solution (using relative error). Inf (i.e. no quality of solution metric)
--outfile Text Output filename. profile.txt
--numpts Integer Number of data points for cutpoint tau. 25
--resmin Real (in %) Minimum resource time threshold. If a solver reports a resource time below --resmin, then it is increased to resmin before being used in ratios. 0.05


Optional Inputs for plotprof.gms only:

Option Input Type Description Default
--figname Text Output filename of figure. fig1.* where * is the appropriate extension (.gif or .ps - none for windows output)
--otype Text Output type: windows, png, or postscript windows (i.e. to screen)

Top