Teradata reference
TeradataResourceoperations- Data transfer
- VantageCloud Lake Compute Cluster management
- Teradata operators documentation
TeradataResource operations
Execute a query (execute_query)
This operation executes a SQL query within Teradata Vantage.
Args:
sql(str) – The query to be executed.fetch_results(bool, optional) – If True, fetch the query results. Defaults to False.single_result_row(bool, optional) – If True, return only the first row of the result set. Effective only iffetch_resultsis True. Defaults to False.
Execute multiple queries (execute_queries)
This operation executes a series of SQL queries within Teradata Vantage.
Args:
sql_queries(Sequence[str]) – List of queries to be executed in series.fetch_results(bool, optional) – If True, fetch the query results. Defaults to False.single_result_row(bool, optional) – If True, return only the first row of the result set. Effective only iffetch_resultsis True. Defaults to False.
Drop a database (drop_database)
This operation drops one or more databases from Teradata Vantage.
Args:
databases(Union[str, Sequence[str]]) – Database name or list of database names to drop.
Drop a table (drop_table)
This operation drops one or more tables from Teradata Vantage.
Args:
tables(Union[str, Sequence[str]]) – Table name or list of table names to drop.
Data transfer
Transfer data from AWS S3 to Teradata Vantage using dagster-teradata
import os
from dagster import job, op, Definitions, EnvVar, DagsterError
from dagster_aws.s3 import S3Resource, s3_resource
from dagster_teradata import TeradataResource, teradata_resource
s3_resource = S3Resource(
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
aws_session_token=os.getenv("AWS_SESSION_TOKEN"),
)
td_resource = TeradataResource(
host=os.getenv("TERADATA_HOST"),
user=os.getenv("TERADATA_USER"),
password=os.getenv("TERADATA_PASSWORD"),
database=os.getenv("TERADATA_DATABASE"),
)
@op(required_resource_keys={"teradata"})
def drop_existing_table(context):
context.resources.teradata.drop_table("people")
return "Tables Dropped"
@op(required_resource_keys={"teradata", "s3"})
def ingest_s3_to_teradata(context, status):
if status == "Tables Dropped":
context.resources.teradata.s3_to_teradata(s3_resource, os.getenv("AWS_S3_LOCATION"), "people")
else:
raise DagsterError("Tables not dropped")
@job(resource_defs={"teradata": td_resource, "s3": s3_resource})
def example_job():
ingest_s3_to_teradata(drop_existing_table())
defs = Definitions(
jobs=[example_job]
)
The s3_to_teradata method is used to load data from an S3 bucket into a Teradata table. It leverages Teradata Vantage Native Object Store (NOS), which allows direct querying and loading of external object store data (like AWS S3) into Teradata tables.
Arguments supported by s3_blob_to_teradata
-
s3 (S3Resource): The
S3Resourceobject used to interact with the S3 bucket. -
s3_source_key (str): The URI specifying the location of the S3 bucket. The URI format is:
/s3/YOUR-BUCKET.s3.amazonaws.com/YOUR-BUCKET-NAMEFor more details, see: Teradata Documentation - Native Object Store -
teradata_table (str):
The name of the Teradata table to which the data will be loaded. -
public_bucket (bool):
Indicates whether the provided S3 bucket is public. IfTrue, the objects within the bucket can be accessed via a URL without authentication. IfFalse, the bucket is considered private, and authentication must be provided.
Defaults toFalse. -
teradata_authorization_name (str):
The name of the Teradata Authorization Database Object, which controls access to the S3 object store.
For more details, see: Teradata Vantage Native Object Store - Setting Up Access
Transfer data from Azure Blob to Teradata Vantage using dagster-teradata
import os
from dagster import job, op, Definitions, EnvVar, DagsterError
from dagster_azure.adls2 import ADLS2Resource, ADLS2SASToken
from dagster_teradata import TeradataResource, teradata_resource
azure_resource = ADLS2Resource(
storage_account="",
credential=ADLS2SASToken(token=""),
)
td_resource = TeradataResource(
host=os.getenv("TERADATA_HOST"),
user=os.getenv("TERADATA_USER"),
password=os.getenv("TERADATA_PASSWORD"),
database=os.getenv("TERADATA_DATABASE"),
)
@op(required_resource_keys={"teradata"})
def drop_existing_table(context):
context.resources.teradata.drop_table("people")
return "Tables Dropped"
@op(required_resource_keys={"teradata", "azure"})
def ingest_azure_to_teradata(context, status):
if status == "Tables Dropped":
context.resources.teradata.azure_blob_to_teradata(azure_resource, "/az/akiaxox5jikeotfww4ul.blob.core.windows.net/td-usgs/CSVDATA/09380000/2018/06/", "people", True)
else:
raise DagsterError("Tables not dropped")
@job(resource_defs={"teradata": td_resource, "azure": azure_resource})
def example_job():
ingest_azure_to_teradata(drop_existing_table())
defs = Definitions(
jobs=[example_job]
)
The azure_blob_to_teradata method is used to load data from Azure Data Lake Storage (ADLS) into a Teradata table. This method leverages Teradata Vantage Native Object Store (NOS) to directly query and load external object store data (such as Azure Blob Storage) into Teradata.
Arguments supported by azure_blob_to_teradata
-
azure (ADLS2Resource):
TheADLS2Resourceobject used to interact with the Azure Blob Storage. -
blob_source_key (str):
The URI specifying the location of the Azure Blob object. The format is:/az/YOUR-STORAGE-ACCOUNT.blob.core.windows.net/YOUR-CONTAINER/YOUR-BLOB-LOCATION
For more details, see the Teradata documentation:
Teradata Documentation - Native Object Store -
teradata_table (str):
The name of the Teradata table where the data will be loaded. -
public_bucket (bool, optional):
Indicates whether the Azure Blob container is public. IfTrue, the objects in the container can be accessed without authentication.
Defaults toFalse. -
teradata_authorization_name (str, optional):
The name of the Teradata Authorization Database Object used to control access to the Azure Blob object store. This is required for secure access to private containers.
Defaults to an empty string.
For more details, see the documentation:
Teradata Vantage Native Object Store - Setting Up Access