d3m.metadata.base¶
-
class
d3m.metadata.base.ArgumentType(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.Context(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.DataMetadata(metadata=None, for_value=None, *, generate_metadata=True, check=True, source=None, timestamp=None)[source]¶ Bases:
d3m.metadata.base.MetadataA class for metadata for data values.
It checks all updates against container and data schemas. Note that as such empty (just created) metadata object does not validate against schemas. Consider setting required fields manually or use generate method as a helper to do so.
It has additional helper methods for operating on metadata of tabular data.
- Parameters
metadata (
Optional[Dict[str,Any]]) – Optional initial metadata for the top-level of the value.for_value (
Optional[Any]) – Optional value to automatically generate metadata for. DEPRECATED: use explicit generate method call instead.generate_metadata (bool) – Automatically generate metadata from
for_valueand update the metadata accordingly. DEPRECATED: use explicit generate method call instead.check (
bool) – DEPRECATED: argument ignored.timestamp (
Optional[datetime]) – DEPRECATED: argument ignored.
-
append_columns(right, *, use_right_metadata=False)[source]¶ Appends metadata for all columns from
rightto the right of this metadata.Top-level metadata of
rightis ignored, not merged, except ifuse_right_metadatais set, in which case top-level metadata of this metadata is ignored and one fromrightis used instead.- Return type
~D
-
check(value)[source]¶ Checks that all metadata has a corresponding data in
valueand that every metadata value is valid according to schema. If not it raises an exception.- Parameters
value (
Any) – Value to check against.- Return type
None
-
classmethod
check_selector(cls, selector, for_value=None)[source]¶ Checks that a given
selectoris a valid selector. Ifselectoris invalid it raises an exception.It checks that it is a tuple or a list and currently we require that all segments of a selector are strings, integers, or a special value
ALL_ELEMENTS.
-
clear(metadata=None, *, for_value=None, generate_metadata=True, check=True, source=None, timestamp=None)[source]¶ DEPRECATED: create a DataMetadata instance explicitly instead.
Creates and returns a new (clear) metadata object.
- Parameters
metadata (
Optional[Dict[str,Any]]) – Optional new initial metadata for the top-level of the value.for_value (
Optional[Any]) – Optional value to automatically generate metadata for.generate_metadata (bool) – Automatically generate metadata from
for_valueand update the metadata accordingly.check (
bool) – DEPRECATED: argument ignored.timestamp (
Optional[datetime]) – DEPRECATED: argument ignored.
- Returns
New metadata object.
- Return type
~D
-
generate(value=None, *, compact=False)[source]¶ Metadata about structure of data (dimensions) and structural types is generated for the
value, and existing metadata is updated accordingly.- Parameters
- Returns
Metadata object updated with automatically generated metadata.
- Return type
~D
-
get_index_columns(*, at=())[source]¶ Returns column indices of the primary index columns.
It makes sure
d3mIndexis always first listed.
-
horizontal_concat(right, *, use_index=True, remove_second_index=True, use_right_metadata=False)[source]¶ Similar to
append_columns, but it respects primary index columns, by default.It is required that both inputs have the same number of samples.
- Return type
~D
-
insert_columns(columns, at_column_index)[source]¶ Inserts metadata for all columns from
columnsbeforeat_column_indexcolumn in this metadata, pushing all existing columns to the right.E.g.,
at_column_index == 0means insertingcolumnsat the beginning of this metadata.Top-level metadata of
columnsis ignored.- Return type
~D
-
list_columns_with_semantic_types(semantic_types, *, at=())[source]¶ This is similar to
get_columns_with_semantic_type, but it returns all column indices for a dimension instead ofALL_ELEMENTSelement.Moreover, it operates on a list of semantic types, where a column is returned if it matches any semantic type on the list.
-
list_columns_with_structural_types(structural_types, *, at=())[source]¶ Returns a list of columns matching any of the structural types listed in
structural_types. Matching allows subclasses of those types.structural_typescan also be a function to call to check a structural type.
-
query_column(column_index, *, at=(), ignore_all_elements=False)[source]¶ Returns column metadata.
This assumes that column metadata is stored under
(ALL_ELEMENTS, column_index), at optionallyatselector, which might not necessary hold if metadata is not compacted. Consider using query_column_field.- Parameters
column_index (
int) – Column index to use.at (
Union[List[Union[int,str,ALL_ELEMENTS_TYPE]],Tuple[Union[int,str,ALL_ELEMENTS_TYPE], …]]) – Selector at which to assume tabular metadata.ignore_all_elements (
bool) – By default, metadata from ALL_ELEMENTS is merged with metadata for an element itself. By setting this argument toTrue, this is disabled and just metadata from an element is returned.
- Returns
Metadata of a given column.
- Return type
FrozenOrderedDict
-
query_column_field(column_index, field, *, at=(), strict_all_elements=True)[source]¶ Returns
fieldvalue of column metadata. Raises KeyError exception if metadata or field is not set.fieldrepresents only top-level fields in metadata.- Parameters
column_index (
int) – Column index to use.field (
str) – A field name to query.at (
Union[List[Union[int,str,ALL_ELEMENTS_TYPE]],Tuple[Union[int,str,ALL_ELEMENTS_TYPE], …]]) – Selector at which to assume tabular metadata.strict_all_elements (
bool) – If set, the method does not just returnfieldvalue of column metadata, but checks that the value really holds for all rows matching theselector, without exception. This is helpful also if metadata is not compacted andfieldvalue is the same across all rows, butALL_ELEMENTSmetadata does not contain that field.
- Returns
A value of
fieldof a given column.- Return type
-
remove(selector, *, recursive=False, strict_all_elements=False, for_value=None, check=True, source=None, timestamp=None)[source]¶ Removes all metadata at
selector.- Parameters
selector (
Union[List[Union[int,str,ALL_ELEMENTS_TYPE]],Tuple[Union[int,str,ALL_ELEMENTS_TYPE], …]]) – A selector to remove metadata at.recursive (
bool) – Should remove also all metadata under theselector?strict_all_elements (
bool) – IfTrue, then when removingALL_ELEMENTSentry, do not remove also metadata for all elements it matches.check (
bool) – DEPRECATED: argument ignored.timestamp (
Optional[datetime]) – DEPRECATED: argument ignored.
- Returns
Updated metadata.
- Return type
~D
-
remove_column(column_index, *, at=(), recursive=False, strict_all_elements=False, for_value=None, source=None, timestamp=None)[source]¶ Removes all column metadata for column
column_index.This removes column metadata under
(ALL_ELEMENTS, column_index), at optionallyatselector. It does not move to the left metadata for columns after the removed column. If you want that, useremove_columns.- Parameters
column_index (
int) – Column index to remove.at (
Union[List[Union[int,str,ALL_ELEMENTS_TYPE]],Tuple[Union[int,str,ALL_ELEMENTS_TYPE], …]]) – Selector at which to assume tabular metadata.recursive (
bool) – Should remove also all metadata under theselector?strict_all_elements (
bool) – IfTrue, then when removingALL_ELEMENTSentry, do not remove also metadata for all elements it matches.timestamp (
Optional[datetime]) – DEPRECATED: argument ignored.
- Returns
Updated metadata.
- Return type
~D
-
remove_columns(column_indices)[source]¶ Removes columns from metadata.
It moves to the left metadata for columns after removed columns. If you do not want that, use
remove_column.It throws an exception if no columns would be left after removing columns.
- Return type
~D
-
remove_semantic_type(selector, semantic_type, *, source=None, timestamp=None)[source]¶ - Return type
~D
-
replace_columns(columns, column_indices)[source]¶ Replaces columns listed in
column_indiceswithcolumns, in order, in this metadata.column_indicesandcolumnsdo not have to match in number of columns. Columns are first replaced in order for matching indices and columns. If then there are morecolumn_indicesthancolumns, additionalcolumn_indicescolumns are removed. If there are morecolumnsthancolumn_indicescolumns, then additionalcolumnsare inserted after the last replaced column.If
column_indicesis empty, then the behavior is equivalent to callingappend_columns.Top-level metadata of
columnsis ignored.- Return type
~D
-
select_columns(columns, *, allow_empty_columns=False)[source]¶ Returns a new metadata object with metadata only for given
columns. Moreover, columns are renumbered based on the position incolumnslist. Top-level metadata stays unchanged, except for updating the length of the columns dimension to the number of columns.So if the
columnsis[3, 6, 5]then output metadata will have three columns,[0, 1, 2], mapping metadata for columns3to0,6to1and5to2.This allows also duplication of columns.
- Return type
~D
-
set_for_value(for_value=None, *, generate_metadata=True, check=True, source=None, timestamp=None)[source]¶ DEPRECATED: use
generatemethod instead.If
generate_metadatais set, generate metadata fromfor_valueand update the metadata accordingly.- Parameters
- Returns
Metadata object updated with automatically generated metadata.
- Return type
~D
-
update(selector, metadata, *, for_value=None, check=True, source=None, timestamp=None)[source]¶ Updates metadata with new
metadatafor data pointed to withselector.If value of any field is
NO_VALUE, that field is deleted.It returns a copy of this metadata object with new metadata applied.
- Parameters
selector (
Union[List[Union[int,str,ALL_ELEMENTS_TYPE]],Tuple[Union[int,str,ALL_ELEMENTS_TYPE], …]]) – A selector pointing to data.metadata (
Dict[str,Any]) – A map of fields and values with metadata.check (
bool) – DEPRECATED: argument ignored.timestamp (
Optional[datetime]) – DEPRECATED: argument ignored.
- Returns
Updated metadata.
- Return type
~D
-
update_column(column_index, metadata, *, at=(), source=None, timestamp=None)[source]¶ Updates column metadata with new
metadatafor column identified bycolumn_index.This stores column metadata under
(ALL_ELEMENTS, column_index), at optionallyatselector.- Parameters
column_index (
int) – Column index to update.metadata (
Dict[str,Any]) – A map of fields and values with metadata.at (
Union[List[Union[int,str,ALL_ELEMENTS_TYPE]],Tuple[Union[int,str,ALL_ELEMENTS_TYPE], …]]) – Selector at which to assume tabular metadata.timestamp (
Optional[datetime]) – DEPRECATED: argument ignored.
- Returns
Updated column metadata.
- Return type
~D
-
class
d3m.metadata.base.ForeignKeyType(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PipelineRunPhase(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PipelineStepType(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PrimitiveAlgorithmType(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
EVOLUTIONARY_ACQUISITION_OF_NEURAL_TOPOLOGIES= 'EVOLUTIONARY_ACQUISITION_OF_NEURAL_TOPOLOGIES'[source]¶
-
INFORMATION_THEORETIC_METAFEATURE_EXTRACTION= 'INFORMATION_THEORETIC_METAFEATURE_EXTRACTION'[source]¶
-
PREFRONTAL_CORTEX_BASAL_GANGLIA_WORKING_MEMORY= 'PREFRONTAL_CORTEX_BASAL_GANGLIA_WORKING_MEMORY'[source]¶
-
STOCHASTIC_CHAINS_WITH_MEMORY_OF_VARIABLE_LENGTH= 'STOCHASTIC_CHAINS_WITH_MEMORY_OF_VARIABLE_LENGTH'[source]¶
-
STRICT_PARTITIONING_CLUSTERING_WITH_OUTLIERS= 'STRICT_PARTITIONING_CLUSTERING_WITH_OUTLIERS'[source]¶
-
-
class
d3m.metadata.base.PrimitiveArgumentKind(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PrimitiveEffect(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PrimitiveFamily(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PrimitiveInstallationType(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PrimitiveMetadata(metadata=None)[source]¶ Bases:
d3m.metadata.base.MetadataA class for metadata for primitives.
It checks all updates against primitive schema. Note that as such empty (just created) metadata object does not validate against the schema. If an instance is set on a primitive class, primitive’s metaclass logic will automatically link metadata object with the primitive class and generate required metadata.
-
class
d3m.metadata.base.PrimitiveMethodKind(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.
-
class
d3m.metadata.base.PrimitivePrecondition(value)[source]¶ Bases:
d3m.utils.EnumAn enumeration.