Table
The table kind represents tabular data that can be read and written as a dataframe. The default dataframe engine is pandas; polars is also supported. Install the optional SDK extra for the engine you use, as described in the SDK installation guide.
The path identifies the table data and can use a supported local or remote scheme. log_table() generates a destination path when one is not supplied. An optional schema follows the Frictionless Table Schema format; log_table() infers it from the data when omitted.
Create
Log
log_table
Log a dataitem table to the project.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project
|
str
|
Project name. |
required |
name
|
str
|
Entity name. |
None
|
source
|
SourcesOrListOfSources
|
Dataitem location on local path. Alternative to data or sql. |
None
|
data
|
Dataframe
|
Dataframe to log. Alternative to source or sql. |
None
|
sql
|
str
|
SQL query to log. Alternative to source or data. |
None
|
drop_existing
|
bool
|
Whether to drop existing entity with the same name. |
False
|
path
|
str
|
Destination path. If omitted, it is generated. |
None
|
description
|
str
|
Human-readable entity description. |
None
|
labels
|
list[str]
|
Entity labels. |
None
|
file_format
|
str
|
Extension of the file source (parquet, csv, json, xlsx, txt). |
None
|
read_df_params
|
dict
|
Parameters to pass to the dataframe reader. |
None
|
engine
|
str
|
Dataframe engine (pandas, polars, etc.). |
"pandas"
|
version
|
str
|
Entity version. |
None
|
schema
|
dict
|
Table schema. If omitted, it is inferred from the data. |
None
|
**kwargs
|
dict
|
Additional dataitem specification parameters. |
{}
|
Returns:
| Type | Description |
|---|---|
Dataitem
|
Created table dataitem entity with uploaded files. |
Examples:
Create a table from a local file or dataframe:
import digitalhub as dh
import pandas as pd
table_from_source = dh.log_table(
project="my-project",
name="sales-from-file",
source="./data/sales.csv",
file_format="csv",
)
table_from_data = dh.log_table(
project="my-project",
name="sales-from-dataframe",
data=pd.DataFrame(
{
"customer_id": [101, 102],
"amount": [12.50, 8.75],
}
),
)
Register
register_table
Register a table dataitem entity for an existing source.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project
|
str
|
Project name. |
required |
source
|
SourcesOrListOfSources
|
Path or URI of the existing table dataitem. |
required |
name
|
str
|
Entity name. If omitted, it is inferred from |
None
|
uuid
|
str
|
Entity identifier. |
None
|
version
|
str
|
Entity version. |
None
|
description
|
str
|
Human-readable entity description. |
None
|
labels
|
list[str]
|
Entity labels. |
None
|
embedded
|
bool
|
Whether to embed the entity specification in the project specification. |
False
|
extensions
|
list[dict]
|
Entity extensions. |
None
|
schema
|
dict
|
Table schema. |
None
|
**kwargs
|
dict
|
Additional dataitem specification parameters. |
{}
|
Returns:
| Type | Description |
|---|---|
Dataitem
|
Registered table dataitem entity. |
Examples:
Register a table that already exists in remote storage:
New
new_dataitem
Create a new dataitem entity in the backend.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project
|
str
|
Project name. |
required |
name
|
str
|
Entity name. |
required |
kind
|
str
|
Entity kind. |
required |
uuid
|
str
|
Entity identifier. |
None
|
version
|
str
|
Entity version. |
None
|
description
|
str
|
Human-readable entity description. |
None
|
labels
|
list[str]
|
Entity labels. |
None
|
embedded
|
bool
|
Whether to embed the entity specification in the project specification. |
False
|
path
|
str
|
Entity path on the local file system or remote storage. It is also the destination path for uploads. |
None
|
extensions
|
list[dict]
|
Entity extensions. |
None
|
**kwargs
|
dict
|
Additional entity specification parameters. |
{}
|
Returns:
| Type | Description |
|---|---|
Dataitem
|
Created dataitem entity. |
Methods
The table kind adds dataframe operations to the common Dataitem methods.
as_df
Read dataitem file (csv or parquet) as a DataFrame from spec.path. It's possible to pass additional arguments to the this function. These keyword arguments will be passed to the DataFrame reader function such as pandas's read_csv or read_parquet.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_format
|
str
|
Format of the file to read. By default, it will be inferred from the extension of the file. |
None
|
engine
|
str
|
Dataframe framework, by default pandas. |
'pandas'
|
**kwargs
|
dict
|
Keyword arguments passed to the read_df function. |
{}
|
Returns:
| Type | Description |
|---|---|
Any
|
DataFrame. |
write_df
Write DataFrame as parquet/csv/table into dataitem spec.path.
keyword arguments will be passed to the DataFrame reader function such as
pandas's to_csv or to_parquet.
Note that by default the index is dropped when writing the dataframe. To
keep the index, you can pass index=True as a keyword argument.
If the dataitem path is a SQL scheme, the dataframe will be written to the
table specified in the path (sql://
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
Any
|
DataFrame to write. |
required |
extension
|
str
|
Extension of the file (supported parquet and csv). |
None
|
**kwargs
|
dict
|
Keyword arguments passed to the write_df function. |
{}
|
Returns:
| Type | Description |
|---|---|
str
|
Path to the written dataframe. |
Examples:
>>> import digitalhub as dh
>>> import pandas as pd
>>>
>>> p = dh.get_project("my_project")
>>> df = pd.read_df("data/my_data.csv")
>>> di = p.new_dataitem(
... name="my_dataitem",
... kind="table",
... path="s3://my-bucket/my-data.parquet",
... )
>>> di.write_df(
... df,
... extension="parquet",
... index=True,
... )
's3://my-bucket/my-data.parquet'