d3m.metadata.base¶
-
class
d3m.metadata.base.
ArgumentType
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
Context
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
DataMetadata
(metadata=None, for_value=None, *, generate_metadata=True, check=True, source=None, timestamp=None)[source]¶ Bases:
d3m.metadata.base.Metadata
A class for metadata for data values.
It checks all updates against container and data schemas. Note that as such empty (just created) metadata object does not validate against schemas. Consider setting required fields manually or use generate method as a helper to do so.
It has additional helper methods for operating on metadata of tabular data.
- Parameters
metadata (
Optional
[Dict
[str
,Any
]]) – Optional initial metadata for the top-level of the value.for_value (
Optional
[Any
]) – Optional value to automatically generate metadata for. DEPRECATED: use explicit generate method call instead.generate_metadata (bool) – Automatically generate metadata from
for_value
and update the metadata accordingly. DEPRECATED: use explicit generate method call instead.check (
bool
) – DEPRECATED: argument ignored.timestamp (
Optional
[datetime
]) – DEPRECATED: argument ignored.
-
append_columns
(right, *, use_right_metadata=False)[source]¶ Appends metadata for all columns from
right
to the right of this metadata.Top-level metadata of
right
is ignored, not merged, except ifuse_right_metadata
is set, in which case top-level metadata of this metadata is ignored and one fromright
is used instead.- Return type
~D
-
check
(value)[source]¶ Checks that all metadata has a corresponding data in
value
and that every metadata value is valid according to schema. If not it raises an exception.- Parameters
value (
Any
) – Value to check against.- Return type
None
-
classmethod
check_selector
(cls, selector, for_value=None)[source]¶ Checks that a given
selector
is a valid selector. Ifselector
is invalid it raises an exception.It checks that it is a tuple or a list and currently we require that all segments of a selector are strings, integers, or a special value
ALL_ELEMENTS
.
-
clear
(metadata=None, *, for_value=None, generate_metadata=True, check=True, source=None, timestamp=None)[source]¶ DEPRECATED: create a DataMetadata instance explicitly instead.
Creates and returns a new (clear) metadata object.
- Parameters
metadata (
Optional
[Dict
[str
,Any
]]) – Optional new initial metadata for the top-level of the value.for_value (
Optional
[Any
]) – Optional value to automatically generate metadata for.generate_metadata (bool) – Automatically generate metadata from
for_value
and update the metadata accordingly.check (
bool
) – DEPRECATED: argument ignored.timestamp (
Optional
[datetime
]) – DEPRECATED: argument ignored.
- Returns
New metadata object.
- Return type
~D
-
generate
(value=None, *, compact=False)[source]¶ Metadata about structure of data (dimensions) and structural types is generated for the
value
, and existing metadata is updated accordingly.- Parameters
- Returns
Metadata object updated with automatically generated metadata.
- Return type
~D
-
get_index_columns
(*, at=())[source]¶ Returns column indices of the primary index columns.
It makes sure
d3mIndex
is always first listed.
-
horizontal_concat
(right, *, use_index=True, remove_second_index=True, use_right_metadata=False)[source]¶ Similar to
append_columns
, but it respects primary index columns, by default.It is required that both inputs have the same number of samples.
- Return type
~D
-
insert_columns
(columns, at_column_index)[source]¶ Inserts metadata for all columns from
columns
beforeat_column_index
column in this metadata, pushing all existing columns to the right.E.g.,
at_column_index == 0
means insertingcolumns
at the beginning of this metadata.Top-level metadata of
columns
is ignored.- Return type
~D
-
list_columns_with_semantic_types
(semantic_types, *, at=())[source]¶ This is similar to
get_columns_with_semantic_type
, but it returns all column indices for a dimension instead ofALL_ELEMENTS
element.Moreover, it operates on a list of semantic types, where a column is returned if it matches any semantic type on the list.
-
list_columns_with_structural_types
(structural_types, *, at=())[source]¶ Returns a list of columns matching any of the structural types listed in
structural_types
. Matching allows subclasses of those types.structural_types
can also be a function to call to check a structural type.
-
query_column
(column_index, *, at=(), ignore_all_elements=False)[source]¶ Returns column metadata.
This assumes that column metadata is stored under
(ALL_ELEMENTS, column_index)
, at optionallyat
selector, which might not necessary hold if metadata is not compacted. Consider using query_column_field.- Parameters
column_index (
int
) – Column index to use.at (
Union
[List
[Union
[int
,str
,ALL_ELEMENTS_TYPE
]],Tuple
[Union
[int
,str
,ALL_ELEMENTS_TYPE
], …]]) – Selector at which to assume tabular metadata.ignore_all_elements (
bool
) – By default, metadata from ALL_ELEMENTS is merged with metadata for an element itself. By setting this argument toTrue
, this is disabled and just metadata from an element is returned.
- Returns
Metadata of a given column.
- Return type
FrozenOrderedDict
-
query_column_field
(column_index, field, *, at=(), strict_all_elements=True)[source]¶ Returns
field
value of column metadata. Raises KeyError exception if metadata or field is not set.field
represents only top-level fields in metadata.- Parameters
column_index (
int
) – Column index to use.field (
str
) – A field name to query.at (
Union
[List
[Union
[int
,str
,ALL_ELEMENTS_TYPE
]],Tuple
[Union
[int
,str
,ALL_ELEMENTS_TYPE
], …]]) – Selector at which to assume tabular metadata.strict_all_elements (
bool
) – If set, the method does not just returnfield
value of column metadata, but checks that the value really holds for all rows matching theselector
, without exception. This is helpful also if metadata is not compacted andfield
value is the same across all rows, butALL_ELEMENTS
metadata does not contain that field.
- Returns
A value of
field
of a given column.- Return type
-
remove
(selector, *, recursive=False, strict_all_elements=False, for_value=None, check=True, source=None, timestamp=None)[source]¶ Removes all metadata at
selector
.- Parameters
selector (
Union
[List
[Union
[int
,str
,ALL_ELEMENTS_TYPE
]],Tuple
[Union
[int
,str
,ALL_ELEMENTS_TYPE
], …]]) – A selector to remove metadata at.recursive (
bool
) – Should remove also all metadata under theselector
?strict_all_elements (
bool
) – IfTrue
, then when removingALL_ELEMENTS
entry, do not remove also metadata for all elements it matches.check (
bool
) – DEPRECATED: argument ignored.timestamp (
Optional
[datetime
]) – DEPRECATED: argument ignored.
- Returns
Updated metadata.
- Return type
~D
-
remove_column
(column_index, *, at=(), recursive=False, strict_all_elements=False, for_value=None, source=None, timestamp=None)[source]¶ Removes all column metadata for column
column_index
.This removes column metadata under
(ALL_ELEMENTS, column_index)
, at optionallyat
selector. It does not move to the left metadata for columns after the removed column. If you want that, useremove_columns
.- Parameters
column_index (
int
) – Column index to remove.at (
Union
[List
[Union
[int
,str
,ALL_ELEMENTS_TYPE
]],Tuple
[Union
[int
,str
,ALL_ELEMENTS_TYPE
], …]]) – Selector at which to assume tabular metadata.recursive (
bool
) – Should remove also all metadata under theselector
?strict_all_elements (
bool
) – IfTrue
, then when removingALL_ELEMENTS
entry, do not remove also metadata for all elements it matches.timestamp (
Optional
[datetime
]) – DEPRECATED: argument ignored.
- Returns
Updated metadata.
- Return type
~D
-
remove_columns
(column_indices)[source]¶ Removes columns from metadata.
It moves to the left metadata for columns after removed columns. If you do not want that, use
remove_column
.It throws an exception if no columns would be left after removing columns.
- Return type
~D
-
remove_semantic_type
(selector, semantic_type, *, source=None, timestamp=None)[source]¶ - Return type
~D
-
replace_columns
(columns, column_indices)[source]¶ Replaces columns listed in
column_indices
withcolumns
, in order, in this metadata.column_indices
andcolumns
do not have to match in number of columns. Columns are first replaced in order for matching indices and columns. If then there are morecolumn_indices
thancolumns
, additionalcolumn_indices
columns are removed. If there are morecolumns
thancolumn_indices
columns, then additionalcolumns
are inserted after the last replaced column.If
column_indices
is empty, then the behavior is equivalent to callingappend_columns
.Top-level metadata of
columns
is ignored.- Return type
~D
-
select_columns
(columns, *, allow_empty_columns=False)[source]¶ Returns a new metadata object with metadata only for given
columns
. Moreover, columns are renumbered based on the position incolumns
list. Top-level metadata stays unchanged, except for updating the length of the columns dimension to the number of columns.So if the
columns
is[3, 6, 5]
then output metadata will have three columns,[0, 1, 2]
, mapping metadata for columns3
to0
,6
to1
and5
to2
.This allows also duplication of columns.
- Return type
~D
-
set_for_value
(for_value=None, *, generate_metadata=True, check=True, source=None, timestamp=None)[source]¶ DEPRECATED: use
generate
method instead.If
generate_metadata
is set, generate metadata fromfor_value
and update the metadata accordingly.- Parameters
- Returns
Metadata object updated with automatically generated metadata.
- Return type
~D
-
update
(selector, metadata, *, for_value=None, check=True, source=None, timestamp=None)[source]¶ Updates metadata with new
metadata
for data pointed to withselector
.If value of any field is
NO_VALUE
, that field is deleted.It returns a copy of this metadata object with new metadata applied.
- Parameters
selector (
Union
[List
[Union
[int
,str
,ALL_ELEMENTS_TYPE
]],Tuple
[Union
[int
,str
,ALL_ELEMENTS_TYPE
], …]]) – A selector pointing to data.metadata (
Dict
[str
,Any
]) – A map of fields and values with metadata.check (
bool
) – DEPRECATED: argument ignored.timestamp (
Optional
[datetime
]) – DEPRECATED: argument ignored.
- Returns
Updated metadata.
- Return type
~D
-
update_column
(column_index, metadata, *, at=(), source=None, timestamp=None)[source]¶ Updates column metadata with new
metadata
for column identified bycolumn_index
.This stores column metadata under
(ALL_ELEMENTS, column_index)
, at optionallyat
selector.- Parameters
column_index (
int
) – Column index to update.metadata (
Dict
[str
,Any
]) – A map of fields and values with metadata.at (
Union
[List
[Union
[int
,str
,ALL_ELEMENTS_TYPE
]],Tuple
[Union
[int
,str
,ALL_ELEMENTS_TYPE
], …]]) – Selector at which to assume tabular metadata.timestamp (
Optional
[datetime
]) – DEPRECATED: argument ignored.
- Returns
Updated column metadata.
- Return type
~D
-
class
d3m.metadata.base.
ForeignKeyType
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PipelineRunPhase
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PipelineStepType
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PrimitiveAlgorithmType
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
EVOLUTIONARY_ACQUISITION_OF_NEURAL_TOPOLOGIES
= 'EVOLUTIONARY_ACQUISITION_OF_NEURAL_TOPOLOGIES'[source]¶
-
INFORMATION_THEORETIC_METAFEATURE_EXTRACTION
= 'INFORMATION_THEORETIC_METAFEATURE_EXTRACTION'[source]¶
-
PREFRONTAL_CORTEX_BASAL_GANGLIA_WORKING_MEMORY
= 'PREFRONTAL_CORTEX_BASAL_GANGLIA_WORKING_MEMORY'[source]¶
-
STOCHASTIC_CHAINS_WITH_MEMORY_OF_VARIABLE_LENGTH
= 'STOCHASTIC_CHAINS_WITH_MEMORY_OF_VARIABLE_LENGTH'[source]¶
-
STRICT_PARTITIONING_CLUSTERING_WITH_OUTLIERS
= 'STRICT_PARTITIONING_CLUSTERING_WITH_OUTLIERS'[source]¶
-
-
class
d3m.metadata.base.
PrimitiveArgumentKind
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PrimitiveEffect
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PrimitiveFamily
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PrimitiveInstallationType
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PrimitiveMetadata
(metadata=None)[source]¶ Bases:
d3m.metadata.base.Metadata
A class for metadata for primitives.
It checks all updates against primitive schema. Note that as such empty (just created) metadata object does not validate against the schema. If an instance is set on a primitive class, primitive’s metaclass logic will automatically link metadata object with the primitive class and generate required metadata.
-
class
d3m.metadata.base.
PrimitiveMethodKind
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.
-
class
d3m.metadata.base.
PrimitivePrecondition
(value)[source]¶ Bases:
d3m.utils.Enum
An enumeration.