Job
job consist of:
definition Definition of a job as it is described in Job documentation. Job Definitiontrigger_schedule_expression (Deprecated) Trigger schedule expression. Create Triggertriggers Triggers definition, that contains trigger_schedule_expression and request_payload.
trigger_schedule_expression and triggers are optional. Only one of trigger_schedule_expression or triggers can be used.
If job not specified, it is required to use Put Data operation
definition
Job definition as it is described in the Create Job
There are few recommendations to consider:
Save Data
As of now, Job definition should contain a step that saves data to a pipeline
Save data example
{
"SaveData": {
"End": true,
"Method": "POST",
"Path": "console-pipeline/pipelines/{pipeline_name}/data",
"PathParameters": {
"pipeline_name": "$.outputs.Start.pipeline_name"
},
"RequestPayload": "$.outputs.Health.response_payload.metrics",
"Type": "InvokeProduct"
}
}
Get Latest from a pipeline
it is also recommended to start job definition from getting latest data from a pipeline
Get Latest data example
{
"LatestData": {
"ExceptNext": "Health",
"Method": "GET",
"Next": "Health",
"Path": "console-pipeline/pipelines/{pipeline_name}/data/latest?sort_column=created_at",
"PathParameters": {
"pipeline_name": "$.outputs.Start.pipeline_name"
},
"Projection": {
"created_at": "$jmespath.created_at | date_format(@, '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d')"
},
"Type": "InvokeProduct"
}
}
Use pipeline_name from request_payload
Dashboard Pipeline will Execute Job with request_payload, where pipeline_name equals Dashboard Pipeline name.
To be able to run
PathParameters:
pipeline_name: $.request_payload.pipeline_name
trigger_schedule_expression
A rate expression starts when you create the pipeline, and then runs on its defined schedule.
Rate expressions have two required fields. Fields are separated by white space.
Syntax
rate(value unit)
Where:
value a positive numberunit the unit of time.
Different units are required for values of 1, such as minute, and values over 1, such as minutes.
Valid values: minute | minutes | hour | hours | day | days
Example of trigger_schedule_expression definition
{
"trigger_schedule_expression": "rate(1 day)"
}
triggers
Allow to specify one or more executions for the pipeline. Can be used to run pipeline many times with different request_payload.
Consist of:
trigger_schedule_expression. For more information please see trigger_schedule_expression documentationrequest_payload (Optional)
triggers can be used to collection data from different Staircase accounts by specifying host and api_key in the request_payload.
Maximum number of triggers is 100.
Example of triggers definition
{
"triggers": [
{
"trigger_schedule_expression": "rate(1 day)",
"request_payload": {
"Host": "documentation.staicaseapi.com",
"ApiKey": "API_KEY"
}
}
]
}
Another example a trigger with custom payload, containing some data and some Staircase environment informations.
{
"triggers": [
{
"trigger_schedule_expression": "rate(1 day)",
"request_payload": {
"my_custom_data": {
"foo": "bar"
},
"Env_A": {
"my_host": "A.staicaseapi.com",
"my_api_key": "API_KEY"
},
"Env_B": {
"my_host": "B.staicaseapi.com",
"my_api_key": "API_KEY"
}
}
}
]
}
Datasets
Allow to define more than one dataset in a Pipeline.
Datasets definition contains Dataset Name and Dataset Configuration in a form of JSON object key and value
Example of a Pipeline that have 2 datasets
{
"datasets": {
"sales": {
"columns": [
{
"Name": "start_date",
"Type": "timestamp"
},
{
"Name": "transaction_id",
"Type": "string"
}
]
},
"communications": {
"columns": [
{
"Name": "sent_date",
"Type": "timestamp"
},
{
"Name": "transaction_id",
"Type": "string"
}
]
}
}
}
Dataset
dataset consist of:
columnscalculated_columns (Optional)
columns
Is an array of column definitions. Columns from this array will be available in the QuickSight dataset.
At least one column is required in columns array.
Column definition consist of:
NameType one of the following types: int, float, string, timestamp
The following example creates pipeline with job and dataset.
It will run pipeline every 5 mins. Will generate some random data for "metric_number", "metric_status" and "date".
{
"name": "pipe1",
"verbose_name": "Pipe 1",
"job": {
"definition": {
"StartAt": "StartFrom",
"States": {
"StartFrom": {
"Type": "InvokeProduct",
"Method": "POST",
"Path": "console-pipeline/generate_data_example",
"Next":"SaveRandomData"
},
"SaveRandomData": {
"Type": "InvokeProduct",
"Method": "POST",
"Path": "console-pipeline/pipelines/{pipeline_name}/data",
"PathParameters": {
"pipeline_name": "pipe1"
},
"RequestPayload": "$.outputs.StartFrom.response_payload",
"End": true
}
}
},
"trigger_schedule_expression": "rate(5 minutes)"
},
"dataset": {
"columns": [
{
"Name": "metric_number",
"Type": "int"
},
{
"Name": "metric_status",
"Type": "string"
},
{
"Name": "date",
"Type": "timestamp"
}
]
}
}
calculated_columns
It is an array of QuickSight Calculated Columns.
Calculated column definition consists of:
NameExpression - Calculated Columns expression to calculate a value
Expression transform columns by using one or more of the following:
- Operators
- Functions
- Aggregate functions (you can only add these to an analysis)
- Fields that contain data
- Other calculated fields
{
"calculated_columns": [
{
"Name":"status_len",
"Expression": "strlen({status})"
},
{
"Name":"now",
"Expression": "now"
}
]
}
