Using the PAVER Server for non-GAMS Solvers

Prakhar Lodha
Department of Computer Science
Arizona State University
prakhar@asu.edu

Description:

This page describes how the output from any optimization engine can be used for performance analysis using the PAVER server. In particular, we describe a suite of parsing routines to convert AMPL solver log files to proper format for submission to the PAVER analysis server. The main goal of developing these parsers is to make the interaction with PAVER seamless for non-GAMS solvers or optimization engines.

See the PAVER server results using the AMPL solvers FILTER, IPOPT, KNITRO, and LOQO.

  • Overview
  • XML File Format
  • Available Solver Parsers
  • Installation and Instructions
  • How to and Summary of Results


    • Overview

    In the most general form, the benchmarking process for solvers works in the following terms: when solving an optimization model, the optimization engine creates a log output, which contains the solve data of interest. We wish to extract the important information and analyze it in some automatic fashion.

    Analysis of important data such as resource time, iterations and objective value can be done using the PAVER Server. But to extract this information from the solver log output, we need parsers.

    The process of converting solver log files for input to PAVER consists of two steps: converting the log file information to general XML format and conversion of the XML file to GAMS trace (traceopt=3) format. 

     
     --------------        --------------        ----------------
     | Solver Log | -----> | XML format | -----> | Trace record | ===> Input to PAVER
     --------------   |    --------------   |    ----------------
               Solver-specific           xml2trc
                   parser                 parser


    The parsing routines for the first step are solver specific. The parsing routine for conversion from XML to a trace records is uniform for all solvers and a single routine can be used. For the last step we have written a routine in JAVA to facilitate this step.

    We have written a suite of parsing routines for the following nonlinear programming solvers hooked up to AMPL:

  • KNITRO
  • LOQO
  • IPOPT
  • FILTER
  • MINLP


    • The routines are written in JAVA to take the log output from the optimization engine as input and output an XML format file.

    XML File Format

    The XML file consists of the following data (note that we have used sample data) 
             <?xml version="1.0" encoding="ISO8859-1"?>


         <!DOCTYPE DATA>
         <DATA>
         <REQUIRED>
             <NAME>transport</NAME>
             <SOLVER>knitro</SOLVER>
             <TYPE>NLP</TYPE>
             <SOLVESTAT>1</SOLVESTAT>
             <MODELSTAT>2</MODELSTAT>
             <OBJECTIVE_VALUE>1.00000465759343E+00</OBJECTIVE_VALUE>
             <RESOURCE_USAGE>0.00</RESOURCE_USAGE>
         </REQUIRED>
         <SOLVE_SUMMARY>
             <SOLVER_MSG>NORMAL COMPLETION</SOLVER_MSG>
             <MODEL_MSG>INFEASIBLE</MODEL_MSG>
             <ITERATIONS>5</ITERATIONS>
             <EVALUATION_ERRORS>0</EVALUATION_ERRORS>
         </SOLVE_SUMMARY>
         <MODEL_STATISTICS>
             <EQUATIONS>4</EQUATIONS>
             <VARIABLES>6</VARIABLES>
             <DISCRETE_VARIABLES>6</DISCRETE_VARIABLES>
             <NONZEROS>20</NONZEROS>
             <NONLINEAR_NONZEROS>0</NONLINEAR_NONZEROS>
         </MODEL_STATISTICS>
         </DATA>
    The XML output from the above parser is fed to another parser (xml2trc) which outputs a (traceopt=3) trace records. A set of trace records in a single file (referred to as a trace file) can be used as input to the PAVER server for analysis.

    Available Solver Parsers

    The parsers were developed in JAVA and are available for download. In order to use them, JAVA must be installed on the user machine, which can be downloaded, for example, at http://java.sun.com/.

    Currently, parsers are available for the following AMPL solvers:

  • KNITRO  (KNITRO 2.1 and KNITRO 3.0)
  • LOQO  (LOQO Version 6.03 and LOQO Version 6.02)
  • IPOPT  (IPOPT-2.0.1)
  • filterSQP  (20020316)
  • MINLP  (MINLP-B&B 20020418)

    • These parsers were tested on models obtained from Hans Mittelmann's collection of AMPL test problems. collection of nonlinear programming models.

    Installation and Instructions

    This section gives installation and step-by-step instructions how to make use of the parsing routines for obtaining benchmark analysis results.

    System Requirements:

    (Windows and UNIX Users)

  • JDK version 1.2 or higher (download for free at http://java.sun.com)
  • Install Java and set classpath in autoexec.bat for windows.
  • Download all parsers in one directory to run tests in and make it the current directory.
  • Download all parsing routines: parsers.zip
  • Compile parsers using the "javac" command: 
            >> javac filter2xml.java
        >> javac ipopt2xml.java
        >> javac knitro2xml.java
        >> javac loqo2xml.java
        >> javac xml2trc.java
    which creates .class files from the JAVA files.
    • You then may want to
  • Download the models files in the current directory.
  • Run parsers on individual model files or write a script run them in a batch.

    • Instructions to run parsers as standalone programs.

    For each solver there is a stand-alone parsing routine which takes as input the log output from the optimization engine and outputs the XML file.

    To execute the java programs (these are the .CLASS files created previously) follow these instructions:

  • Make sure all program files and input/output files are in the same directory and give the following command: 
     >> java (solvername)2xml [inputfile] [outputfile] [option]
    where
  • the available solvernames are are minlp, knitro, loqo, filter, and ipopt.
  • the available option is debug, which prints the output of output file on the command line.
  • the outputfile name is optional as default out filename is (modelname).xml
  • the option name is case-insensitive and is optional.
    • For example: 
      >> java knitro2trc trnsport.kni
    takes the log file trnsport.kni and creates and XML file called trnsport.xml.

  • If JAVA parsers and input log files are in different directories you will need to set the classpath using the -cp flag.

    • Instructions for generating trace records from XML output:

    There is just one parser for generating trace records from output of all XML parsers. It takes as input the XML file from the above parsing routine and outputs the trace record (traceopt=3).
  • From the command line type 
    >> java xml2trc [inputfile] [outputfile] [option]
    where
  • Available option is debug - it prints the output of output file on command line.
  • Optional output filename. Default filename is (modelname).trc
    • For example 
    >> java xml2trc trnsport.xml
    takes the XML file trnsport.xml and creates a trace record in a trace file called trnsport.trc. If the trace file does not exist, it is created, otherwise the trace record is appended.

    • Summary of Results

  • List of AMPL models used for testing: amplmods.txt
  • AMPL models for download: models.zip
  • Log files that result from running the various AMPL solvers
  • filterlogs.zip
  • knitrologs.zip
  • ipoptlogs.zip
  • loqologs.zip
  • Resulting XML files from running the log to xml parsers on log files. Note that while for each model a single XML file is created, but a single trace file is created for each solver, containing results for all models.
  • filterxml.zip
  • knitroxml.zip
  • ipoptxml.zip
  • loqoxml.zip
  • Collection of trace files that results from running the XML to trace parsers on XML files. The trace files can be used as input to the PAVER server
  • filter.trc
  • knitro.trc
  • ipopt.trc
  • loqo.trc
  • Performance analysis results using the PAVER server on the trace files generated above: results.htm