d3m.contrib.primitives.compute_scores

class d3m.contrib.primitives.compute_scores.ComputeScoresPrimitive(*, hyperparams, random_seed=0, docker_containers=None, volumes=None, temporary_directory=None)

Bases: d3m.primitive_interfaces.transformer.TransformerPrimitiveBase[[d3m.container.pandas.DataFrame, d3m.container.pandas.DataFrame], d3m.contrib.primitives.compute_scores.Hyperparams]

A primitive that takes a DataFrame with predictions and a scoring Dataset (test split with target values present), and computes scores for given metrics and outputs them as a DataFrame.

It searches only the dataset entry point resource for target columns (which should be marked with https://metadata.datadrivendiscovery.org/types/TrueTarget semantic type) in the scoring Dataset.

Primitive does not align rows between truth DataFrame and predictions DataFrame, it is expected that metric code does that if necessary. Similarly, it does not align columns order either.

It uses metadata to construct the truth DataFrame and renames the index column to match the standard names d3mIndex. It encodes any float vectors as strings.

For predictions DataFrame it expects that it is already structured correctly with correct column names and it leaves to metric code to validate that truth DataFrame and predictions DataFrame match. It does not use or expect metadata on predictions DataFrame. Predictions DataFrame should already have float vectors encoded as strings.

fit_multi_produce(*, produce_methods, inputs, score_dataset, timeout=None, iterations=None)

A method calling fit and after that multiple produce methods at once.

Parameters

• produce_methods (Sequence[str]) – A list of names of produce methods to call.

• inputs (DataFrame) – The inputs given to all produce methods.

• timeout (Optional[float]) – A maximum time this primitive should take to both fit the primitive and produce outputs for all produce methods listed in produce_methods argument, in seconds.

• iterations (Optional[int]) – How many of internal iterations should the primitive do for both fitting and producing outputs of all produce methods.

Returns

A dict of values for each produce method wrapped inside MultiCallResult.

Return type

MultiCallResult

multi_produce(*, produce_methods, inputs, score_dataset, timeout=None, iterations=None)

A method calling multiple produce methods at once.

When a primitive has multiple produce methods it is common that they might compute the same internal results for same inputs but return different representations of those results. If caller is interested in multiple of those representations, calling multiple produce methods might lead to recomputing same internal results multiple times. To address this, this method allows primitive author to implement an optimized version which computes internal results only once for multiple calls of produce methods, but return those different representations.

If any additional method arguments are added to primitive’s produce method(s), they have to be added to this method as well. This method should accept an union of all arguments accepted by primitive’s produce method(s) and then use them accordingly when computing results. Despite accepting all arguments they can be passed as None by the caller when they are not needed by any of the produce methods in produce_methods.

The default implementation of this method just calls all produce methods listed in produce_methods in order and is potentially inefficient.

If primitive should have been fitted before calling this method, but it has not been, primitive should raise a PrimitiveNotFittedError exception.

Parameters

• produce_methods (Sequence[str]) – A list of names of produce methods to call.

• inputs (DataFrame) – The inputs given to all produce methods.

• timeout (Optional[float]) – A maximum time this primitive should take to produce outputs for all produce methods listed in produce_methods argument, in seconds.

• iterations (Optional[int]) – How many of internal iterations should the primitive do.

Returns

A dict of values for each produce method wrapped inside MultiCallResult.

Return type

MultiCallResult

produce(*, inputs, score_dataset, timeout=None, iterations=None)

Produce primitive’s best choice of the output for each of the inputs.

The output value should be wrapped inside CallResult object before returning.

In many cases producing an output is a quick operation in comparison with fit, but not all cases are like that. For example, a primitive can start a potentially long optimization process to compute outputs. timeout and iterations can serve as a way for a caller to guide the length of this process.

Ideally, a primitive should adapt its call to try to produce the best outputs possible inside the time allocated. If this is not possible and the primitive reaches the timeout before producing outputs, it should raise a TimeoutError exception to signal that the call was unsuccessful in the given time. The state of the primitive after the exception should be as the method call has never happened and primitive should continue to operate normally. The purpose of timeout is to give opportunity to a primitive to cleanly manage its state instead of interrupting execution from outside. Maintaining stable internal state should have precedence over respecting the timeout (caller can terminate the misbehaving primitive from outside anyway). If a longer timeout would produce different outputs, then CallResult’s has_finished should be set to False.

Some primitives have internal iterations (for example, optimization iterations). For those, caller can provide how many of primitive’s internal iterations should a primitive do before returning outputs. Primitives should make iterations as small as reasonable. If iterations is None, then there is no limit on how many iterations the primitive should do and primitive should choose the best amount of iterations on its own (potentially controlled through hyper-parameters). If iterations is a number, a primitive has to do those number of iterations, if possible. timeout should still be respected and potentially less iterations can be done because of that. Primitives with internal iterations should make CallResult contain correct values.

For primitives which do not have internal iterations, any value of iterations means that they should run fully, respecting only timeout.

If primitive should have been fitted before calling this method, but it has not been, primitive should raise a PrimitiveNotFittedError exception.

Parameters

• inputs (DataFrame) – The inputs of shape [num_inputs, …].

• timeout (Optional[float]) – A maximum time this primitive should take to produce outputs during this method call, in seconds.

• iterations (Optional[int]) – How many of internal iterations should the primitive do.

Returns

The outputs of shape [num_inputs, …] wrapped inside CallResult.

Return type

CallResult[DataFrame]

logger: typing.ClassVar[logging.Logger] = <Logger d3m.primitives.evaluation.compute_scores.Core (WARNING)>

Primitive’s logger. Available as a class attribute. This gets automatically set to primitive’s logger in metaclass.

metadata: ClassVar[d3m.metadata.base.PrimitiveMetadata] = <d3m.metadata.base.PrimitiveMetadata object>

Primitive’s metadata. Available as a class attribute. Primitive author should provide all fields which cannot be determined automatically inside the code. In this way metadata is close to the code and it is easier for consumers to make sure metadata they are using is really matching the code they are using. PrimitiveMetadata class updates itself with metadata about code and other things it can extract automatically.