Skip to content

Croissant

The croissant kind stores an ML Croissant dataset defined by a metadata.json file and its referenced local files. Use this kind to load the dataset through the mlcroissant library, which is available through the digitalhub[mlcroissant] extra in the SDK installation guide.

When logging a Croissant dataitem, the metadata file must be named metadata.json; its referenced local files are uploaded with it. An explicit destination path must be an S3 partition path ending with /. See supported paths.

Create

Log

log_croissant

Create and upload a Croissant dataitem entity.

Parameters:

Name Type Description Default
project str

Project name.

required
name str

Entity name.

required
source str

Croissant metadata JSON file path.

required
drop_existing bool

Whether to drop existing entity with the same name.

False
path str

Destination path. If omitted, it is generated.

None
description str

Human-readable entity description.

None
labels list[str]

Entity labels.

None
version str

Entity version.

None
**kwargs dict

Additional dataitem specification parameters.

{}

Returns:

Type Description
DataitemCroissant

Created Croissant dataitem entity with uploaded files.

Examples:

Log a Croissant dataset from its metadata file:

import digitalhub as dh

dataitem = dh.log_croissant(
    project="my-project",
    name="catalog",
    source="./data/croissant/metadata.json",
    description="Product catalog described with Croissant metadata.",
)

Register

register_croissant

Register a Croissant dataitem entity for an existing source.

Parameters:

Name Type Description Default
project str

Project name.

required
source SourcesOrListOfSources

Path or URI of the existing Croissant dataitem.

required
name str

Entity name. If omitted, it is inferred from source.

None
uuid str

Entity identifier.

None
version str

Entity version.

None
description str

Human-readable entity description.

None
labels list[str]

Entity labels.

None
embedded bool

Whether to embed the entity specification in the project specification.

False
extensions list[dict]

Entity extensions.

None
**kwargs dict

Additional dataitem specification parameters.

{}

Returns:

Type Description
DataitemCroissant

Registered Croissant dataitem entity.

Examples:

Register a Croissant dataset that already exists in remote storage:

import digitalhub as dh

dataitem = dh.register_croissant(
    project="my-project",
    name="registered-catalog",
    source="s3://my-bucket/data/croissant/metadata.json",
)

New

new_dataitem

Create a new dataitem entity in the backend.

Parameters:

Name Type Description Default
project str

Project name.

required
name str

Entity name.

required
kind str

Entity kind.

required
uuid str

Entity identifier.

None
version str

Entity version.

None
description str

Human-readable entity description.

None
labels list[str]

Entity labels.

None
embedded bool

Whether to embed the entity specification in the project specification.

False
path str

Entity path on the local file system or remote storage. It is also the destination path for uploads.

None
extensions list[dict]

Entity extensions.

None
**kwargs dict

Additional entity specification parameters.

{}

Returns:

Type Description
Dataitem

Created dataitem entity.

import digitalhub as dh

dataitem = dh.new_dataitem(
    project="my-project",
    name="catalog",
    kind="croissant",
    path="s3://my-bucket/data/croissant/catalog/",
)

Methods

The croissant kind adds dataset loading to the common Dataitem methods.

as_dataset

Get the Croissant Dataset object from the Dataitem.

Parameters:

Name Type Description Default
overwrite bool

Flag to indicate overwrite of local files.

False

Returns:

Type Description
Dataset

Croissant Dataset object.