Skip to content

Table

The table kind represents tabular data that can be read and written as a dataframe. The default dataframe engine is pandas; polars is also supported. Install the optional SDK extra for the engine you use, as described in the SDK installation guide.

The path identifies the table data and can use a supported local or remote scheme. log_table() generates a destination path when one is not supplied. An optional schema follows the Frictionless Table Schema format; log_table() infers it from the data when omitted.

Create

Log

log_table

Log a dataitem table to the project.

Parameters:

Name Type Description Default
project str

Project name.

required
name str

Entity name.

None
source SourcesOrListOfSources

Dataitem location on local path. Alternative to data or sql.

None
data Dataframe

Dataframe to log. Alternative to source or sql.

None
sql str

SQL query to log. Alternative to source or data.

None
drop_existing bool

Whether to drop existing entity with the same name.

False
path str

Destination path. If omitted, it is generated.

None
description str

Human-readable entity description.

None
labels list[str]

Entity labels.

None
file_format str

Extension of the file source (parquet, csv, json, xlsx, txt).

None
read_df_params dict

Parameters to pass to the dataframe reader.

None
engine str

Dataframe engine (pandas, polars, etc.).

"pandas"
version str

Entity version.

None
schema dict

Table schema. If omitted, it is inferred from the data.

None
**kwargs dict

Additional dataitem specification parameters.

{}

Returns:

Type Description
Dataitem

Created table dataitem entity with uploaded files.

Examples:

Create a table from a local file or dataframe:

import digitalhub as dh
import pandas as pd

table_from_source = dh.log_table(
    project="my-project",
    name="sales-from-file",
    source="./data/sales.csv",
    file_format="csv",
)

table_from_data = dh.log_table(
    project="my-project",
    name="sales-from-dataframe",
    data=pd.DataFrame(
        {
            "customer_id": [101, 102],
            "amount": [12.50, 8.75],
        }
    ),
)

Register

register_table

Register a table dataitem entity for an existing source.

Parameters:

Name Type Description Default
project str

Project name.

required
source SourcesOrListOfSources

Path or URI of the existing table dataitem.

required
name str

Entity name. If omitted, it is inferred from source.

None
uuid str

Entity identifier.

None
version str

Entity version.

None
description str

Human-readable entity description.

None
labels list[str]

Entity labels.

None
embedded bool

Whether to embed the entity specification in the project specification.

False
extensions list[dict]

Entity extensions.

None
schema dict

Table schema.

None
**kwargs dict

Additional dataitem specification parameters.

{}

Returns:

Type Description
Dataitem

Registered table dataitem entity.

Examples:

Register a table that already exists in remote storage:

import digitalhub as dh

dataitem = dh.register_table(
    project="my-project",
    name="registered-sales",
    source="s3://my-bucket/data/sales.parquet",
    schema={
        "fields": [
            {"name": "customer_id", "type": "integer"},
            {"name": "amount", "type": "float"},
        ]
    },
)

New

new_dataitem

Create a new dataitem entity in the backend.

Parameters:

Name Type Description Default
project str

Project name.

required
name str

Entity name.

required
kind str

Entity kind.

required
uuid str

Entity identifier.

None
version str

Entity version.

None
description str

Human-readable entity description.

None
labels list[str]

Entity labels.

None
embedded bool

Whether to embed the entity specification in the project specification.

False
path str

Entity path on the local file system or remote storage. It is also the destination path for uploads.

None
extensions list[dict]

Entity extensions.

None
**kwargs dict

Additional entity specification parameters.

{}

Returns:

Type Description
Dataitem

Created dataitem entity.

import digitalhub as dh

dataitem = dh.new_dataitem(
    project="my-project",
    name="sales",
    kind="table",
    path="s3://my-bucket/data/sales.parquet",
)

Methods

The table kind adds dataframe operations to the common Dataitem methods.

as_df

Read dataitem file (csv or parquet) as a DataFrame from spec.path. It's possible to pass additional arguments to the this function. These keyword arguments will be passed to the DataFrame reader function such as pandas's read_csv or read_parquet.

Parameters:

Name Type Description Default
file_format str

Format of the file to read. By default, it will be inferred from the extension of the file.

None
engine str

Dataframe framework, by default pandas.

'pandas'
**kwargs dict

Keyword arguments passed to the read_df function.

{}

Returns:

Type Description
Any

DataFrame.

write_df

Write DataFrame as parquet/csv/table into dataitem spec.path. keyword arguments will be passed to the DataFrame reader function such as pandas's to_csv or to_parquet. Note that by default the index is dropped when writing the dataframe. To keep the index, you can pass index=True as a keyword argument. If the dataitem path is a SQL scheme, the dataframe will be written to the table specified in the path (sql://(/)/).

Parameters:

Name Type Description Default
df Any

DataFrame to write.

required
extension str

Extension of the file (supported parquet and csv).

None
**kwargs dict

Keyword arguments passed to the write_df function.

{}

Returns:

Type Description
str

Path to the written dataframe.

Examples:

>>> import digitalhub as dh
>>> import pandas as pd
>>>
>>> p = dh.get_project("my_project")
>>> df = pd.read_df("data/my_data.csv")
>>> di = p.new_dataitem(
...     name="my_dataitem",
...     kind="table",
...     path="s3://my-bucket/my-data.parquet",
... )
>>> di.write_df(
...     df,
...     extension="parquet",
...     index=True,
... )
's3://my-bucket/my-data.parquet'