2.3. pyopus.evaluator.aggregate — Performance aggregation

Parformance aggregation module (PyOPUS subsystem name: AG)

Normalization is the process where a performance measure is scaled in such way that values not satifying the goal result in normalized values smaller than 0. If the performance measure value exceeds the goal the normalized value is greater than 0. Complex values are treated as failures and failure penalty is used for norm.

Shaping is the process where the normalized performance is shaped. Usually positive values (corresponding to performance not satifying the goal) are shaped differently than negative values (corresponding to performance satifying the goal). Shaping results in a shaped contribution for every corner.

Corner reduction is the process where shaped contributions of individual corners are incorporated into the aggregate function. There are several ways how to achieve this. For instance one could incorporate only the contribution of the corner in which worst peroformance is observed, or on the other hand, one could incorporate the mean contribution of all corners.

The main data structure is the aggregate function description which is a list of component descriptions. Every component description is a dictionary with the following members:

• measure - the name of the performance meeasure on which the aggregate function’s component is based.

• norm - an object performing the normalization of the performance measure.

• shape - an object performing the shaping of normalized performance measure. Defaults to Slinear2(1.0, 0.0).

• reduce - an object performing the corner reduction of the cotributions. Defaults to Rworst().

The ordering of parameters is a list of parameter names that defines the order in which parameter values apper in a parameter vector.

A parameter vector is a list or array of parameter values where the values are ordered according to a given ordering of input parameters.

class pyopus.evaluator.aggregate.Aggregator(perfEval, definition, inputOrder=None, useOnlyListedCorners=False, debug=0)

Aggregator class. Objects of this class are callable. The calling convention is object(paramVector) where paramvector is a list or an array of input parameter values. The ordering of input parameters is given at object construction. The return value is the value of the aggregate function.

perfEval is an object of the PerformanceEvaluator class which is used for evaluating the performance measures of the system. inputOrder is the ordering of system’s input parameters. definition is the aggregate function description. If useOnlyListedCorners is set to True the cost function will be constructed from those corners that are listed in measure definitions. If no corners are listed all evaluated corners are used for constructing the cost function. When set to False all evaluated corners are used.

If debug is set to a value greater than 0, debug messages are generated at the standard output.

Objects of this class store the details of the last evaluated aggregate function value in the results member which is a list (one member for every aggregate function component) of dictionaries with the following members:

• measure - the name of the performance measure on which this component is based

• worst - the worst value of corresponding performance mesure across corners where the performance measure was computed. This is the return value of the normalization object’s Nbase.worst() method when called with with total set to True. None if performance measure evaluation fails in at least one corner

• worst_vector - a vector with the worst values of the performance measure. If the performance measure is a scalar this is also a scalar. If it is an array of shape (m1, m2, …) then this is an array of the same shape. This is the return value of the normalization object’s Nbase.worst() method with total set to False. None if performance measure evaluation fails in at least one corner.

• worst_corner - the index of the corner in which the worst value of performance measure occurs. If the performance measure is an array of shape (m1, m2, ..) this is still a scalar which refers to the corner index of the worst performance measure across all components of the performance measure in all corners. This is the return value of the normalization object’s Nbase.worstCornerIndex() method with total set to True. If the performance evaluation fails in at least one corner this is the index of one of the corners where the failure occurred.

• worst_corner_vector - a vector of corner indices where the worst value of the performance measure is found. If the performance measure is a scalar this vector has only one component. If the performance measure is an array of shape (m1, m2, …) this is an array with the same shape. This is the return value of the normalization object’s Nbase.worstCornerIndex() method with total set to False. If the evaluation of a performance measure fails in at least one corner this vector holds the indices of corners in which the failure occured.

• contribution - the value of the contribution to the aggregate function. This is always a number, even if the evaluation of some performance measures fails (see failure argument to the constructor of normalization objects - e.g. Nbase).

• fulfilled - True if the corresponding performance measure is successfully evaluated in all of its corresponding corners and all resulting values satisfy the corresponding goal. False otherwise.

Corner indices refer to corners in the cornerOrder member which is a list of names of corners defined in perfEval (see PerformanceEvaluator). The cornerOrder is in fact the cornerOrder member of perfEval.

The measure2corner dictionary of corner lists from the perfEval is used for computing the global corner indices which correspond to corners in the cornerOrder member.

The paramVector member holds the input parameter values passed at the last call to this object.

allBelowOrAtZero()

Returns True if all components of the aggregate function computed with the last call to this object are not greater than zero. Assumes that the following holds:

The return value is True, if all performance measures corresponding to aggregate function components not using the Rexcluded corner reduction satisfy their goals; assuming that

• normalization produces positive values for satisfied goals and negative values for unsatisfied goals

• normalization returns a positive value in case of a failure to evaluate a performance measure (failed is greater than 0)

• aggregate function shaping is nondecreasing and is greater than zero for positive normalized performance measures

allFulfilled()

Returns True if the performance measures corresponding to all aggregate function components that were evaluated with the last call to this object were successfully evaluated and fulfill their corresponding goals. All components are taken into account, even those using the Rexcluded corner reduction.

formatParameters(x=None, nParamName=15, nNumber=15, nSig=6)

Formats a string corresponding to the parameters passed at the last call to this object. Generates one line for every parameter. If x is specified it is used instead of the stored parameter vector. nParamName and nNumber specify the width of the formatting for the parameter name and its value. nSig specifies the number of significant digits.

formatResults(nTargetSpec=29, nMeasureName=12, nNumber=12, nSig=3, nCornerName=6)

Formats a string representing the results obtained with the last call to this object. Only the worst performance across corners along with the corresponding aggregate function component value is reported. Generates one line for every aggregate function component.

nTargetSpec specifies the formatting width for the target specification (specified by the corresponding normalization object) of which nMeasureName is used for the name of the performance measure. nNumber and nSig specify the width of the formatting and the number of the significant digits for the aggregate function contribution. nCornerName specifies the width of the formatting for the worst corner name.

getAnnotator()

Returns an object of the CostAnnotator class which can be used as a plugin for iterative algorithms. The plugin takes care of aggregate function details (results member) propagation from the machine where the evaluation of the aggregate function takes place to the machine where the evaluation was requested (usually the master).

getCollector(chunkSize=10)

Returns an object of the CostCollector class which can be used as a plugin for iterative algorithms. The plugin gathers input parameter and aggregate function values across iterations of the algorithm.

chunkSize is the chunk size used when allocating space for stored values (10 means allocation takes place every 10 iterations).

getReporter(reportParameters=True)

Returns an object of the ReportCostCorners class which can be used as a plugin for iterative algorithms. Every time an iterative algorithm calls this Aggregator object the reporter is invoked and prints the details of the aggregate function components.

getStopWhenAllSatisfied()

Returns an object of the StopWhenAllSatisfied class which can be used as a plugin for iterative algorithms. The plugin signals the iterative algorithm to stop when all contributions obtained with the last call to this Aggregator object are smaller than zero (when the allBelowOrAtZero() method returns True).

resolveCornerIndex(ndx)

Returns the corner name for the given corner index.

class pyopus.evaluator.aggregate.Nabove(goal, norm=None, failure=10000.0)

Performance normalization class requiring the performance to to be above the given goal. See Nbase for more information.

report(name, nName=12, nGoal=12, nSigGoal=3)

Format the goal as a string. The output is a string of the form

name > goal

See Nbase.report() method for more information.

worst(values, total=False)

Find the worst value. See Nbase.worst() method for more information.

worstCornerIndex(values, corners, total=False)

Find the worst corner index. See Nbase.worstCornerIndex() method for more information.

class pyopus.evaluator.aggregate.Nbase(goal, norm=None, failure=10000.0)

Basic normalization class. Objects of this class are callable. The calling convention of the object is object(value) where value is a scalar or an array of performance measure values. When called with a scalar the return value is a scalar. When called with an array the return value is an array of the same shape where every component is treated as if it was a scalar.

The return value is greater than 0 if the passed value fails to satisfy the goal. It is less than 0 if the passed value exceeds the goal. Exceeding the goal by norm means that the return value is -1.0. Failing to satify the goal by norm results in a return value of 1.0. If the value passed at call is None, the return value is equal to failure.

If norm is not given the default normalization is used which is equal to goal or 1.0 if goal is equal to 0.0. If goal is a vector, norm must either be a vector of the same size or a scalar in which case it applies to all components of the goal.

report(name, nName=12, nGoal=12, nSigGoal=3)

Formats the goal as a string of the form

name normalization_symbol goal where name

is the name of the performance measure. The normalization_symbol depends on the type of normalization (derived class).

nName and nGoal specify the width of performance measure name and goal formatting. nSigGoal is the number of significant digits of the goal in the formatted string.

worst(values, total=False)

Returns the worst performance value across all corners (the one with the largest normalized value). The values across corners are given in the values array where first array index is the corner index.

If the array has more than 1 dimension the worst value is sought along the first dimension of the array. This means that if value is of shape (n, m1, m2, …) then the return value is of shape (m1, m2, …). The return value is an array of performance measure values.

If total is True the worst value is sought acros the whole array and the return value is a scalar worst performance value.

worstCornerIndex(values, corners, total=False)

Returns the index corresponding to the corner where the performance measure takes its worst value (the one with the largest normalized value). The values across corners are given in the values array where first array index is the corner index.

If the array has more than 1 dimension the worst value is sought along the first dimension of the array. This means that if value is of shape (n, m1, m2, …) then the return value is of shape (m1, m2, …). The return value is an array of corner indices.

The corner indices corresponding to the first dimension of values are given by corners.

If total is True the worst value is sought across the whole array and the return value is a scalar worst corner index.

class pyopus.evaluator.aggregate.Nbelow(goal, norm=None, failure=10000.0)

Performance normalization class requiring the performance to to be below the given goal. See Nbase for more information.

report(name, nName=12, nGoal=12, nSigGoal=3)

Format the goal as a string. The output is a string of the form

name < goal

See Nbase.report() method for more information.

worst(values, total=False)

Find the worst value. See Nbase.worst() method for more information.

worstCornerIndex(values, corners, total=False)

Find the worst corner index. See Nbase.worstCornerIndex() method for more information.

class pyopus.evaluator.aggregate.Nbetween(goal, goalHigh, norm=None, failure=10000.0)

Performance normalization class requiring the performance to to be above goal and below goalHigh. See Nbase for more information. This class is deprecated. Use two contributions instead (one with Nbelow and one with Nabove).

report(name, nName=12, nGoal=12, nSigGoal=3)

Format the goal as a string. The output is a string of the form

name < > (goal + goalHigh)/2

See Nbase.report() method for more information.

worst(values, total=False)

Find the worst value. See Nbase.worst() method for more information.

worstCornerIndex(values, corners, total=False)

Find the worst corner index. See Nbase.worstCornerIndex() method for more information.

class pyopus.evaluator.aggregate.NlogAbove(goal, base=10, theta=None, failure=10000.0)

Performance normalization class requiring the performance to to be above the given goal. Logarithmic scale is applied.

For positive goal the normalized value is computed as ln(goal/value)/ln(base). If theta is given this function is linearized below theta * goal so that the first derivative is continuous.

For negative goal the normalized value is ln(value/goal)/ln(base). If theta is given this function is linearized above theta * goal.

See Nbase for more information.

class pyopus.evaluator.aggregate.NlogBelow(goal, base=10, theta=None, failure=10000.0)

Performance normalization class requiring the performance to to be below the given goal. Logarithmic scale is applied.

For positive goal the normalized value is computed as ln(value/goal)/ln(base). If theta is given this function is linearized below theta * goal so that the first derivative is continuous.

For negative goal the normalized value is ln(goal/value)/ln(base). If theta is given this function is linearized above theta * goal.

See Nbase for more information.

class pyopus.evaluator.aggregate.NlogBetween(goal, goalHigh, base=10, failure=10000.0)

Performance normalization class requiring the performance to to be between goal and goalHigh. Logarithmic scale is applied. Both values, as well as, the value to be normalized must be positive.

The central value is the geometric mean of goal and goalHigh.

The normalized value is computed as ln(value/goalHigh)/ln(base) if value is above the central value. If not, the normalized value is ln(goal/value)/ln(base).

See Nbase for more information.

report(name, nName=12, nGoal=12, nSigGoal=3)

Format the goal as a string. The output is a string of the form

name < > (goal * goalHigh) ** 0.5

See Nbase.report() method for more information.

worst(values, total=False)

Find the worst value. See Nbase.worst() method for more information.

worstCornerIndex(values, corners, total=False)

Find the worst corner index. See Nbase.worstCornerIndex() method for more information.

class pyopus.evaluator.aggregate.Rbase

Basic corner reduction class. Objects of this class are callable. The calling convention of the object is object(shapedMeasure) where shapedMeasure is an array of shaped contributions. The return value is a scalar.

flagFailure()

Return a string that represents the flag for marking performance measures for which the process of evaluation failed (their value is None).

flagSuccess(fulfilled)

Return a string that represents a flag for marking performance measures that satify the goal (fulfilled is True) or fail to satisfy the goal (fulfilled is False).

class pyopus.evaluator.aggregate.Rexcluded

Corner reduction class for excluding the performance measure from the aggregate function. Objects of this class are callable and return 0. See Rbase for more information.

flagFailure()

Return a string that represents the flag for marking performance measures for which the process of evaluation failed (their value is None). Returns 'x'.

See Rbase.flagFailure() method for more information.

flagSuccess(fulfilled)

Return a string that represents a flag for marking performance measures. A successfully satisfied goal is marked by ' ' while a failure is marked by '.'.

See Rbase.flagSuccess() method for more information.

class pyopus.evaluator.aggregate.Rmean(component=None)

Corner reduction class for including only the mean contribution across all corners. Objects of this class are callable and return the mean of contributions passed at call.

See Rbase for more information.

flagFailure()

Return a string that represents the flag for marking performance measures for which the process of evaluation failed (their value is None). Returns 'X'.

See Rbase.flagFailure() method for more information.

flagSuccess(fulfilled)

Return a string that represents a flag for marking performance measures. A successfully satisfied goal is marked by ' ' while a failure is marked by 'o'.

See Rbase.flagSuccess() method for more information.

class pyopus.evaluator.aggregate.Rworst(component=None)

Corner reduction class for including only the worst performance measure across all corners. Objects of this class are callable and return the larget contribution.

component specifies the component of a vector to be included in the aggregate cost function. If set to None all components are considered.

See Rbase for more information.

flagFailure()

Return a string that represents the flag for marking performance measures for which the process of evaluation failed (their value is None). Returns 'X'.

See Rbase.flagFailure() method for more information.

flagSuccess(fulfilled)

Return a string that represents a flag for marking performance measures. A successfully satisfied goal is marked by ' ' while a failure is marked by 'o'.

See Rbase.flagSuccess() method for more information.

class pyopus.evaluator.aggregate.Slinear2(w=1.0, tw=0.0)

Two-segment linear shaping. Normalized performances above 0 (failing to satify the goal) are multiplied by w, while the ones below 0 (satisfying the goal) are multiplied by tw. If w differs from tw this shaping has a discontinuous first derivative at 0.

Objects of this class are callable. The calling comvention is object(value) where value is an array of normalized performance measures.

pyopus.evaluator.aggregate.formatParameters(param, inputOrder=None, nParamName=15, nNumber=24, nSig=16)

Returns string representation of a parameter dictionary param where the ordering of parameters is specified by inputOrder. The width of parameter name and parameter value formatting is given by nParamName and nNumber, while nSig specifies the number of significant digits.