d3m.metrics module

class d3m.metrics.AccuracyMetric(all_labels=None)[source]

Bases: d3m.metrics._AllAsMultiLabelBase

Supports binary, multi-class, multi-label, and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]
Return type

float

class d3m.metrics.F1MacroMetric(all_labels=None)[source]

Bases: d3m.metrics._AllAsMultiLabelBase

Supports multi-class, multi-label, and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]
Return type

float

class d3m.metrics.F1Metric(pos_label)[source]

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]
Return type

float

class d3m.metrics.F1MicroMetric(all_labels=None)[source]

Bases: d3m.metrics._AllAsMultiLabelBase

Supports multi-class, multi-label, and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]
Return type

float

class d3m.metrics.HammingLossMetric(all_labels=None)[source]

Bases: d3m.metrics._AllAsMultiLabelBase

Hamming loss gives the percentage of wrong labels to the total number of labels. Lower the hamming loss, better is the performance of the method used.

Supports multi-label and multi-task predictions.

score_one(truth_target_encoded, predictions_target_encoded, all_labels)[source]
Return type

float

class d3m.metrics.HitsAtKMetric(k)[source]

Bases: d3m.metrics._RankMetricBase

The computes how many elements of a vector of ranks make it to the top ‘k’ positions. Consider the example:

learningData:

d3mIndex subject object relationship (target) 0 James John father 1 John Patricia sister 2 Robert Thomas brother … …

truth:

d3mIndex relationship 0 father 1 sister 2 brother

predictions:

d3mIndex relationships rank 0 brother 1 0 cousin 2 0 mother 3 0 father 4 * 0 grandfather 5 1 sister 1 * 1 mother 2 1 aunt 3 2 father 1 2 brother 2 * 2 sister 3 2 grandfather 4 2 aunt 5

Note that ranks (of truth relationships in the predictions) = [4,1,2] Hits@3 = 2/3 = 0.666666 Hits@1 = 1/3 = 0.3333333 Hits@5 = 3/3 = 1.0

score(truth, predictions)[source]
Return type

float

class d3m.metrics.JaccardSimilarityScoreMetric(pos_label)[source]

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]
Return type

float

class d3m.metrics.MeanAbsoluteErrorMetric[source]

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]
Return type

float

class d3m.metrics.MeanReciprocalRankMetric[source]

Bases: d3m.metrics._RankMetricBase

This computes the mean of the reciprocal of elements of a vector of rankings. This metric is used for linkPrediction problems. Consider the example:

learningData:

d3mIndex subject object relationship (target) 0 James John father 1 John Patricia sister 2 Robert Thomas brother … …

truth:

d3mIndex relationship 0 father 1 sister 2 brother

predictions:

d3mIndex relationships rank 0 brother 1 0 cousin 2 0 mother 3 0 father 4 * 0 grandfather 5 1 sister 1 * 1 mother 2 1 aunt 3 2 father 1 2 brother 2 * 2 sister 3 2 grandfather 4 2 aunt 5

Note that ranks (of truth relationships in the predictions) = [4,1,2] MRR = np.sum(1/ranks)/len(ranks) MRR = 0.58333

score(truth, predictions)[source]
Return type

float

class d3m.metrics.MeanSquareErrorMetric[source]

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]
Return type

float

class d3m.metrics.Metric[source]

Bases: object

classmethod align(truth, predictions)[source]

Aligns columns and rows in predictions to match those in truth.

It requires that all index values in truth are present in predictions and only those. It requires that any column name in truth is also present in predictions. Any additional columns present in predictions are pushed to the right.

Parameters
  • truth (~Truth) – Truth DataFrame.

  • predictions (~Predictions) – Predictions DataFrame.

Returns

Return type

Predictions with aligned rows.

classmethod get_confidence_column(dataframe)[source]

Returns only confidence column present in dataframe.

Return type

DataFrame

classmethod get_index_column(dataframe)[source]

Returns only index column present in dataframe.

Return type

DataFrame

classmethod get_rank_column(dataframe)[source]

Returns only rank column present in dataframe.

Return type

DataFrame

classmethod get_target_columns(dataframe)[source]

Returns only target columns present in dataframe.

Return type

DataFrame

classmethod is_empty_value(v)[source]
Return type

bool

classmethod one_hot_encode_confidence(series, all_labels)[source]

Returns one-hot-encoded dataframe where the columns are the labels of the confidence column, which is provided as a series of tuples, where each tuple contains confidence for all labels of a given sample, ordered in order specified by labels.

Returned dataframe has instead of 0 or 1, a confidence value itself.

Return type

DataFrame

classmethod one_hot_encode_target(series, all_labels)[source]

Returns one-hot-encoded dataframe where the columns are the labels of the target column, which is provided as a series of tuples, where each tuple contains all labels of a given sample.

Return type

DataFrame

abstract score(truth, predictions)[source]
Return type

Any

classmethod vectorize_columns(dataframe)[source]

For every non-index column, convert all values in rows belonging to the same index to one row with value being a tuple of values. The order of values in a tuple follows the order of original rows. Missing/empty values are removed and not included in the tuple.

Return type

DataFrame

class d3m.metrics.NormalizeMutualInformationMetric[source]

Bases: d3m.metrics.Metric

score(truth, predictions)[source]
Return type

float

class d3m.metrics.ObjectDetectionAveragePrecisionMetric[source]

Bases: d3m.metrics.Metric

score(truth, predictions)[source]
Return type

float

class d3m.metrics.PrecisionAtTopKMetric(k)[source]

Bases: d3m.metrics.Metric

score(truth, predictions)[source]
Return type

float

class d3m.metrics.PrecisionMetric(pos_label)[source]

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]
Return type

float

class d3m.metrics.RSquaredMetric[source]

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]
Return type

float

class d3m.metrics.RecallMetric(pos_label)[source]

Bases: d3m.metrics._MultiTaskBase

Supports binary and multi-task predictions.

score_one(truth_target, predictions_target)[source]
Return type

float

class d3m.metrics.RocAucMacroMetric(all_labels=None)[source]

Bases: d3m.metrics._RocAucBase

Supports multi-class and multi-label predictions.

score(truth, predictions)[source]
Return type

float

class d3m.metrics.RocAucMetric(all_labels=None)[source]

Bases: d3m.metrics._RocAucBase

Supports binary predictions.

score(truth, predictions)[source]
Return type

float

class d3m.metrics.RocAucMicroMetric(all_labels=None)[source]

Bases: d3m.metrics._RocAucBase

Supports multi-class and multi-label predictions.

score(truth, predictions)[source]
Return type

float

class d3m.metrics.RootMeanSquareErrorMetric[source]

Bases: d3m.metrics.Metric

Supports univariate and multivariate.

score(truth, predictions)[source]
Return type

float