d3m.metrics¶

class d3m.metrics.AccuracyMetric(all_labels=None)[source]¶

Bases: d3m.metrics._AllAsMultiLabelBase

Supports binary, multi-class, multi-label, and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]¶

Return type: float

class d3m.metrics.F1MacroMetric(all_labels=None)[source]¶

Bases: d3m.metrics._AllAsMultiLabelBase

Supports multi-class, multi-label, and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]¶

Return type: float

class d3m.metrics.F1Metric(pos_label)[source]¶

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]¶

Return type: float

class d3m.metrics.F1MicroMetric(all_labels=None)[source]¶

Bases: d3m.metrics._AllAsMultiLabelBase

Supports multi-class, multi-label, and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]¶

Return type: float

class d3m.metrics.HammingLossMetric(all_labels=None)[source]¶

Bases: d3m.metrics._AllAsMultiLabelBase

Hamming loss gives the percentage of wrong labels to the total number of labels. Lower the hamming loss, better is the performance of the method used.

Supports multi-label and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]¶

Return type: float

class d3m.metrics.HitsAtKMetric(k)[source]¶

Bases: d3m.metrics._RankMetricBase

The computes how many elements of a vector of ranks make it to the top ‘k’ positions. Consider the example:

learningData:
    d3mIndex    subject object      relationship (target)
         James   John        father
         John    Patricia    sister
         Robert  Thomas      brother
    ...
    ...

truth:
    d3mIndex    relationship
         father
         sister
         brother

predictions:
    d3mIndex    relationships   rank
         brother         1
         cousin          2
         mother          3
         father          4 *
         grandfather     5
         sister          1 *
         mother          2
         aunt            3
         father          1
         brother         2 *
         sister          3
         grandfather     4
         aunt            5

Note that ranks (of truth relationships in the predictions) = [4,1,2]

Hits@3 = 2/3 = 0.666666

Hits@1 = 1/3 = 0.3333333

Hits@5 = 3/3 = 1.0

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.JaccardSimilarityScoreMetric(pos_label)[source]¶

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]¶

Return type: float

class d3m.metrics.MeanAbsoluteErrorMetric[source]¶

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.MeanReciprocalRankMetric[source]¶

Bases: d3m.metrics._RankMetricBase

This computes the mean of the reciprocal of elements of a vector of rankings. This metric is used for linkPrediction problems. Consider the example:

learningData:
    d3mIndex    subject object      relationship (target)
         James   John        father
         John    Patricia    sister
         Robert  Thomas      brother
    ...
    ...

truth:
    d3mIndex    relationship
         father
         sister
         brother

predictions:
    d3mIndex    relationships   rank
         brother         1
         cousin          2
         mother          3
         father          4 *
         grandfather     5
         sister          1 *
         mother          2
         aunt            3
         father          1
         brother         2 *
         sister          3
         grandfather     4
         aunt            5

Note that ranks (of truth relationships in the predictions) = [4,1,2]

MRR = np.sum(1/ranks)/len(ranks)

MRR = 0.58333

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.MeanSquareErrorMetric[source]¶

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.Metric[source]¶

Bases: object

classmethod align(truth, predictions)[source]¶

Aligns columns and rows in predictions to match those in truth.

It requires that all index values in truth are present in predictions and only those. It requires that any column name in truth is also present in predictions. Any additional columns present in predictions are pushed to the right.

Parameters

truth (~Truth) – Truth DataFrame.
predictions (~Predictions) – Predictions DataFrame.

Returns

Predictions with aligned rows.

Return type

~Predictions

classmethod get_index_column(dataframe)[source]¶

Returns only index column present in dataframe.

Return type: DataFrame

classmethod get_rank_column(dataframe)[source]¶

Returns only rank column present in dataframe.

Return type: DataFrame

classmethod get_score_column(dataframe)[source]¶

Returns only score column present in dataframe.

Return type: DataFrame

classmethod get_target_columns(dataframe)[source]¶

Returns only target columns present in dataframe.

Return type: DataFrame

classmethod is_empty_value(v)[source]¶

Return type: bool

classmethod one_hot_encode_score(series, all_labels)[source]¶

Returns one-hot-encoded dataframe where the columns are the labels of the score column, which is provided as a series of tuples, where each tuple contains score for all labels of a given sample, ordered in order specified by labels.

Returned dataframe has instead of 0 or 1, a score value itself.

Return type: DataFrame

classmethod one_hot_encode_target(series, all_labels)[source]¶

Returns one-hot-encoded dataframe where the columns are the labels of the target column, which is provided as a series of tuples, where each tuple contains all labels of a given sample.

Return type: DataFrame

abstract score(truth, predictions)[source]¶

Return type: Any

classmethod vectorize_columns(dataframe)[source]¶

For every non-index column, convert all values in rows belonging to the same index to one row with value being a tuple of values. The order of values in a tuple follows the order of original rows. Missing/empty values are removed and not included in the tuple.

Return type: DataFrame

class d3m.metrics.NormalizeMutualInformationMetric[source]¶

Bases: d3m.metrics.Metric

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.ObjectDetectionAveragePrecisionMetric[source]¶

Bases: d3m.metrics.Metric

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.PrecisionAtTopKMetric(k)[source]¶

Bases: d3m.metrics.Metric

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.PrecisionMetric(pos_label)[source]¶

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]¶

Return type: float

class d3m.metrics.RSquaredMetric[source]¶

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.RecallMetric(pos_label)[source]¶

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]¶

Return type: float

class d3m.metrics.RocAucMacroMetric(all_labels=None)[source]¶

Bases: d3m.metrics._RocAucBase

Supports multi-class and multi-label predictions.

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.RocAucMetric(all_labels=None)[source]¶

Bases: d3m.metrics._RocAucBase

Supports binary predictions.

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.RocAucMicroMetric(all_labels=None)[source]¶

Bases: d3m.metrics._RocAucBase

Supports multi-class and multi-label predictions.

score(truth, predictions)[source]¶

Return type: float

class d3m.metrics.RootMeanSquareErrorMetric[source]¶

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]¶

Return type: float

d3m.metrics¶

Version

Table of Contents