Job

State: States are the steps used to define Jobs. A different operation is performed in each State and the next step is specified.

Quick links

Available States are:

• InvokeProduct
• AthenaStartQuery
• AthenaGetResults
• AthenaGetResultsAsUrl
• AthenaGetResultsAsReference
• Choice
• Wait
• Mockdata
• Data
• Outputaggregate
• WaitForAction
• WaitForCallback

Available InvokeProduct settings:

• Path The staircase path where you will call the API (Required)
• PathParameters (Optional). Can be specified if path should be dynamically changed during API calls.
• QueryParameters (Optional). Can be specified if query string should be dynamically changed during API calls.
• Method Type of api call you will make
• RequestPayload (Optional). Request body of a http call.
• EnvironmentSettings (Optional). Environment settings allow invoking Staircase product on the other Staircase environment.
• Headers (Optional). Environment settings allow invoking Staircase product on the other Staircase environment.
• CallbackSettings (Optional). Allow invoking Staircase product with callback_url. If WaitForCallback present CallbackSettings going to be added by default.
• WaitForCallback (Optional). If present, State will wait for callback from a product. WaitForCallback can be used to specify success and/or failed values.
• IterationSettings (Optional). Job supports iteration settings for dynamic parallelism.
• Projection (Optional). Specifies the field(s) to return. To return all fields, omit this parameter.
• TimeoutSeconds (Optional). If the task runs longer than the specified seconds, this state fails.
• ExceptNext (Optional). Define your next state if any error occurs.
• Retry (Optional). Retry logic if any error in InvokeProduct happens it does retry.
• After (Optional). After action to execute after Product was invoked.

Available InvokeJob settings:

• JobName Job Name to invoke
• TransactionId Persistence Transaction ID
• RequestPayload. Request payload for Job invocation.
• Projection (Optional). Specifies the field(s) to return. To return all fields, omit this parameter.
• IterationSettings (Optional). Job supports iteration settings for dynamic parallelism.
• TimeoutSeconds (Optional). If the task runs longer than the specified seconds, this state fails.
• ExceptNext (Optional). Define your next state if any error occurs.
• Retry (Optional). Retry logic if any error in InvokeJob happens it does retry.

Available Intrinsic Functions are:

• ulid
• array_get
• array_chunks
• add
• concat
• split
• replace
• merge
• date_format
• date_end_day
• date_start_day
• date_now
• date_next_day
• date_previous_day
• date_to_iso
• date_to_iso_tz
• date_from_iso

Job Language

There are few ways to work with States definition.

As a JSON-path, string should start from $.

 "PathParameters": {
 "transaction_id": "$.transaction_id"
 } 

It is possible to escape a string with JSON-path like with \$., for example

RequestPayload:
 query:
 path:
 path: \$.people[*].has_taxpayer_identifier_value.has_value
 format: jsonpath
 operation: eq
 value: $.request_payload.data.has_taxpayer_identifier_value.has_value 

As a JMESPath, where string should start from $jmespath.

Example selecting 2 fields from metrics array of objects

 "Projection": {
 "metrics": "$jmespath.metrics[*].{created_at: date_format(created_at, '%Y-%m-%dT%H:%M:%S.%f','%Y-%m-%d %H:%M:%S'), transaction_id: transaction_id}",
 "next": "$.next_token"
 } 

JMESPath pipe example

 "Projection": {
 "created_at": "$jmespath.created_at | date_format(@, '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d')"
 } 

JMESPath if-exists else example

 "PathParameters": {
 "end_date": "$jmespath.outputs.LatestData.response_payload.created_at || outputs.Start.date",
 "product_name": "$.outputs.Start.product_name",
 "start_date": "$jmespath.outputs.LatestData.response_payload.created_at || outputs.Start.date"
 } 

Follow the link bellow to review other JMESPath Examples

JMESPath Intrinsic Functions

According to JMESPath specification there is a list of Builtin functions. However, Job Language also provides a small number of "Intrinsic Functions", constructs which look like functions in programming languages and can be used to help process the data going to and from States. Date formats should be defined with Python datetime formatting When a function fail it returns null.

ulid

Allow to generate ULID

ulid

 {
 "name": "ulid-test",
 "definition": {
 "StartAt": "Start",
 "States": {
 "Start": {
 "Type": "Data",
 "Data": {
 "ulid": "$jmespath.ulid"
 },
 "End": true
 }
 
 }
 }
 } 

array_get

Allow to get element from an array using element index as jmespath

array_get(array, index)

 {
 "PathParameters": {
 "flow_name": "$jmespath.array_get(outputs.Start.flow_names,outputs.Loop.index)",
 "start_date": "$jmespath.(outputs.LatestData.response_payload.start_date || outputs.Start.date)"
 }
 } 

array_chunks

Allow to create array of arrays with the fixed size.

array_chunks(array, chunk_size)

 {
 "name": "job_name",
 "definition": {
 "StartAt": "Start",
 "States": {
 "Start": {
 "Type": "MockData",
 "Data": {
 "array": [
 "1","2","3","4","5","6","7","8","9"
 ]
 },
 "Next": "Data"
 },
 "Data": {
 "Type": "Data",
 "Data": {
 "chunks": "$jmespath.array_chunks(outputs.Start.array,`2`)"
 },
 "End": true
 }
 }
 }
 } 

Should provide following output

{
 "response_payload": {
 "chunks": [
 [
 "1",
 "2"
 ],
 [
 "3",
 "4"
 ],
 [
 "5",
 "6"
 ],
 [
 "7",
 "8"
 ],
 [
 "9"
 ]
 ]
 } 
}

add

Allow to add a number to number. Jmespath require that every Number should be inside "`" quotes

add(number1, number2)

{
 "Start": {
 "Data": {
 "index": 2
 },
 "Next": "Math",
 "Type": "MockData"
 },
 "Math": {
 "Data": {
 "index": "$jmespath.add(outputs.Start.index,`-1`)"
 },
 "End": true,
 "Type": "Data"
 }
} 

concat

Allow to concat 2 strings. Jmespath require that every String should be inside "'" quotes

concat(string1, string2)

{
 "Start": {
 "Data": {
 "name":"foo"
 },
 "Next": "Math",
 "Type": "MockData"
 },
 "Math": {
 "Data": {
 "name": "$jmespath.concat('bar ',outputs.Start.name)"
 },
 "End": true,
 "Type": "Data"
 }
} 

split

Split a string using separator

split(string_1, separator)

 {
 "name": "split-test",
 "definition": {
 "StartAt": "Start",
 "States": {
 "Start": {
 "Type": "Data",
 "Data": {
 "people": [
 {
 "@id": "id_replace",
 "@type": "person",
 "first_name": "person_name",
 "email_address": "email@email.com"
 }
 ]
 },
 "Next": "Split"
 },
 "Split": {
 "Type": "Data",
 "Data": {
 "domain": "$jmespath.outputs.Start.people[0].email_address | split(@,'@')[1]"
 },
 "End": true
 } 
 }
 }
 }

Will return,

 {
 "name": "split-test",
 "status": "SUCCEEDED",
 "current_step": "Split",
 "response_payload": {
 "domain": "email.com"
 }
 } 

replace

Allow to replace sub_string in a string. Jmespath require that every String should be inside "'" quotes

replace(where, to_replace, replace_with)

{
 "States": {
 "Start": {
 "Type": "Data",
 "Data": {
 "data": "people[?@personal_identifier == PERSONAL_ID]"
 },
 "Next": "Replace"
 },
 "Replace": {
 "Type": "Data",
 "Data": {
 "replaced": "$jmespath.replace(outputs.Start.data,'PERSONAL_ID','123')"
 },
 "End": true
 }
 }
} 

merge

Allow to concat 2 objects. Will merge two objects into one. All keys from object2 will be present in result. For identical keys object2 values will override object1 values. This function does NOT do recursive merge.

merge(object1, object2)

 {
 "name": "test",
 "definition": {
 "StartAt": "A",
 "States": {
 "A": {
 "Type": "MockData",
 "Data": {
 "f1": "A",
 "f2": "A"
 },
 "Next": "B"
 },
 "B": {
 "Type": "MockData",
 "Data": {
 "f2": "B",
 "f3": "B"
 },
 "Next": "C"
 },
 "C": {
 "Type": "Data",
 "Data": {
 "result": "$jmespath.merge(outputs.A,outputs.B)"
 },
 "End": true
 }
 }
 }
 } 

date_format

Allow to format a date field formatting.

date_format(date, from_format, to_format)

 "Projection": {
 "created_at": "$jmespath.created_at | date_format(@, '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d')"
 } 

date_end_day

Takes the date and make it end of the day.

date_end_day(date, format)

 "Projection": {
 "created_at_end":"$jmespath.created_at | date_end_day(@, '%Y-%m-%dT%H:%M:%S.%f%z')"
 } 

date_start_day

Takes the date and make it start of the day.

date_start_day(date, format)

 "Projection": {
 "created_at_start":"$jmespath.created_at | date_start_day(@, '%Y-%m-%dT%H:%M:%S.%f%z')"
 } 

date_now

Current datetime.

date_now(format)

 "Projection": {
 "now": "$jmespath.date_now('%Y-%m-%d %H:%M:%S.%f')"
 } 

date_next_day

Takes the date and make it the next day. Equvalent of +1 day operation

date_next_day(date, format)

 "Projection": {
 "created_at_next": "$jmespath.created_at | date_next_day(@,'%Y-%m-%dT%H:%M:%S.%f%z') | date_format(@, '%Y-%m-%dT%H:%M:%S.%f%z','%Y-%m-%d %H:%M:%S')"
 } 

date_previous_day

Takes the date and make it previoud day. Equvalent of -1 day operation

date_next_day(date, format)

 "Projection": {
 "created_at_prev": "$jmespath.created_at | date_previous_day(@,'%Y-%m-%dT%H:%M:%S.%f%z') | date_format(@, '%Y-%m-%dT%H:%M:%S.%f%z','%Y-%m-%d %H:%M:%S')"
 } 

Example using Intrinsic Functions

 {
 "name": "test",
 "description": "Create Transaction",
 "definition": {
 "StartAt": "CreateTransaction",
 "States": {
 "CreateTransaction": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions",
 "Method": "POST",
 "RequestPayload": {},
 "Projection": {
 "transaction_id": "$jmespath.transaction_id",
 "created_at": "$jmespath.created_at",
 "created_at_next": "$jmespath.created_at | date_next_day(@,'%Y-%m-%dT%H:%M:%S.%f%z') | date_format(@, '%Y-%m-%dT%H:%M:%S.%f%z','%Y-%m-%d %H:%M:%S')",
 "created_at_prev": "$jmespath.created_at | date_previous_day(@,'%Y-%m-%dT%H:%M:%S.%f%z') | date_format(@, '%Y-%m-%dT%H:%M:%S.%f%z','%Y-%m-%d %H:%M:%S')", 
 "created_at_end":"$jmespath.created_at | date_end_day(@, '%Y-%m-%dT%H:%M:%S.%f%z')",
 "created_at_start":"$jmespath.created_at | date_start_day(@, '%Y-%m-%dT%H:%M:%S.%f%z')",
 "created_at_formatted": "$jmespath.created_at | date_format(@, '%Y-%m-%dT%H:%M:%S.%f%z','%Y-%m-%d %H:%M:%S')",
 "now": "$jmespath.date_now('%Y-%m-%d %H:%M:%S.%f')"
 },
 "End": true
 }
 }
 }
 } 

Expected result

 {

 "response_payload": {
 "transaction_id": "01GB858918AEZVTQMPAV5T1PF0",
 "created_at": "2022-08-24T10:49:11.465048-04:00",
 "created_at_next": "2022-08-25 10:49:11",
 "created_at_prev": "2022-08-23 10:49:11",
 "created_at_end": "2022-08-24T23:59:59.999999-0400",
 "created_at_start": "2022-08-24T00:00:00.000000-0400",
 "created_at_formatted": "2022-08-24 10:49:11",
 "now": "2022-08-24 14:49:11.494322"
 }

 } 

date_to_iso

Takes the date and converts it to ISO format

date_to_iso(date, from_format)

date_to_iso_tz

Takes the date and converts it to ISO format by specifying TZ name

date_to_iso_tz(date, from_format, tz_name)

date_from_iso

Takes the date in ISO and converts it to to_format

date_from_iso(date, to_format)

date_to_iso_tz and date_from_iso example

name: test-20221119-15
definition:
 StartAt: A
 States:
 A:
 Type: Data
 Data:
 date_EST: "2022-11-17 10:00:22.466"
 date_EDT: "2022-06-17 10:00:22.466"
 iso_EST: "2022-11-17T10:00:22.466000-05:00"
 iso_EDT: "2022-06-17T10:00:22.466000-04:00"
 from_iso: "2022-06-17T10:00:22.466000-04:00"
 foo: bar
 Next: B
 B:
 Type: Data
 Data:
 result: >-
 $jmespath.outputs.A.{
 foo: foo, 
 date_EST_iso: date_to_iso(date_EST, '%Y-%m-%d %H:%M:%S.%f'), 
 date_EST: date_to_iso_tz(date_EST, '%Y-%m-%d %H:%M:%S.%f','America/New_York'), 
 date_EDT: date_to_iso_tz(date_EDT, '%Y-%m-%d %H:%M:%S.%f','America/New_York'), 
 iso_EST: date_from_iso(iso_EST,'%Y-%m-%d %H:%M:%S.%f TZ %z'), 
 iso_EDT: date_from_iso(iso_EDT,'%Y-%m-%d %H:%M:%S.%f TZ %z')
 }
 result_2: >-
 $jmespath.outputs.A.{
 date_EST: date_from_iso(
 date_to_iso_tz(date_EST, 
 '%Y-%m-%d %H:%M:%S.%f','America/New_York'),'%Y-%m-%d %H:%M:%S.%f')
 }
 from_iso: >-
 $jmespath.outputs.A.{from_iso: date_from_iso(from_iso, '%Y-%m-%d %H:%M:%S.%f TZ %z')}
 End: true

{
 "response_payload": {
 "result": {
 "date_EST_iso": "2022-11-17T10:00:22.466000",
 "date_EST": "2022-11-17T10:00:22.466000-05:00",
 "date_EDT": "2022-06-17T10:00:22.466000-04:00",
 "foo": "bar",
 "iso_EST": "2022-11-17 10:00:22.466000 TZ -05:00",
 "iso_EDT": "2022-06-17 10:00:22.466000 TZ -04:00"
 },
 "result_2": {
 "date_EST": "2022-11-17 10:00:22.466000"
 },
 "from_iso": {
 "from_iso": "2022-06-17 10:00:22.466000 TZ -04:00"
 }
 }
 }

States

InvokeProduct

Invoke product state is main state type in the job. An API call is made to one of the Staircase products in the InvokeProduct state. You must define the resource definition for the action you will take.

The following example demonstrates how to invoke Persistence Product to create a transaction

"CreateTransaction": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions",
 "Method": "POST",
 "RequestPayload": {}
 "Next": "CreateCollection"
 }

Path

Full path to a Staircase product api. Should NOT start from /. Can contain {path_name} path parameters and/or query parameters

 {
 "Path": "code-health-checker/performance/metrics/products/{product_name}?start_date={start_date}&end_date={end_date}&status=failed&limit=10"
 }

PathParameters

 {
 "Path": "console-pipeline/pipelines/{pipeline_name}/data/latest?sort_column=created_at",
 "PathParameters": {
 "pipeline_name": "$.outputs.Start.pipeline_name"
 } 
 }

EnvironmentSettings

Environment settings allow to invoke Staircase product on the other Staircase environment.

EnvironmentSettings consists of:

• Host
• ApiKey

It is NOT allowed to set static string values.

Both Host and ApiKey can contain on of following:

• JSON-path
• JMESPath
• $$.CurrentValue

In case Host and/or ApiKey use JSON-path or JMESPath, Host and ApiKey can ONLY refer to $.request_payload. Host and ApiKey can be defined while Executing a Job in request_payload.

Example of a job to create Transaction on other Staircase eenvironment.

 {
 "definition": {
 "StartAt": "CreateTransaction",
 "States": {
 "CreateTransaction": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions",
 "EnvironmentSettings": {
 "Host": "$.request_payload.Host",
 "ApiKey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
 },
 "Method": "POST",
 "RequestPayload": {
 "label":"$.request_payload.label"
 },
 "End": true
 }
 }
 }
 }

In case Host and/or ApiKey use $$.CurrentValue,

{
 "GetServiceKey": {
 "Type": "InvokeProduct",
 "Method": "GET",
 "Path": "environment-manager/service-key",
 "EnvironmentSettings": {
 "Host": "$$.CurrentValue"
 },
 "IterationSettings": {
 "IterateOverPath": "$.outputs.Subscribers.response_payload.subscribed_domains",
 "MaxConcurrency": 10
 },
 "Next": "SomeData"
 } 
}

Headers

Headers allow to add custom headers to the request. For example, Authorization header. It is possible to use JSON-path or JMESPath inside Headers.

Example of a job with EnvironmentSettings and Headers.

 {
 "name": "test",
 "definition": {
 "StartAt": "Start",
 "States": {
 "Start": {
 "Type": "MockData",
 "Data": {
 "token": "bar"
 },
 "Next": "Product"
 },
 "Product": {
 "Type": "InvokeProduct",
 "Path": "administrator/costs",
 "Method": "GET",
 "EnvironmentSettings": {
 "Host": "$.request_payload.Host",
 "ApiKey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
 },
 "Headers": {
 "Authorization": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
 },
 "End": true
 }
 }
 }
 }

QueryParameters

Query Paramerts allow to specify Query string of a HTTP request. Only non-null values will be added to a query string.

 {
 "Type": "InvokeProduct",
 "Method": "GET",
 "Path": "code-health-checker/performance/metrics/products/{product_name}?status=failed",
 "QueryParameters": {
 "start_date": "$jmespath.(outputs.LatestData.response_payload.created_at || outputs.Start.date) | date_next_day(@,'%Y-%m-%d')",
 "end_date": "$jmespath.date_now('%Y-%m-%d') | date_previous_day(@,'%Y-%m-%d')",
 "limit": 10,
 "next_token": "$jmespath.outputs.Current_Step.response_payload.next_token || outputs.Start.next_token"
 },
 "PathParameters": {
 "product_name": "$.outputs.Start.product_name"
 }
 
 }

Method

Any valid HTTP method

RequestPayload

Can be an object,

{
 "RequestPayload": {
 "collection_without_soft_credit": "$.outputs.DeleteSoftCreditInformation.response_payload.collection_id",
 "hard_credit_collection": "$.outputs.PatchHardCreditCollectionSsn.response_payload.collection_id",
 "loan_collection": "$.outputs.CreateChosenLoanCollection.response_payload.collection_id",
 "transaction_id": "$.transaction_id"
 } 
} 

Can be a string,

 {
 RequestPayload": "$.outputs.Health.response_payload.metrics"
 }

Can be an array,

{
 "Product": {
 "Type": "InvokeProduct",
 "Path": "path/",
 "Method": "POST",
 "EnvironmentSettings": {
 "Host": "$.request_payload.Host"
 },
 "QueryParameters": {
 "foo": "$.outputs.Start.foo"
 },
 "RequestPayload": [
 "$jmespath.outputs.Start"
 ],
 "End": true
 } 
}

CallbackSettings

For callback settings, the expected path must be specified in the incoming callback message.

It can optionally be specified in the value corresponding to this path.

"Build": {
 "Type": "InvokeProduct",
 "Path": "infra-builder/builds",
 "Method": "POST",
 "RequestPayload": {
 "source_url": "$.outputs.RetrieveAssessResponse.response_payload.source_url"
 },
 "CallbackSettings": {
 "Path": "$.callback_url",
 },
 "Next": "NextState"
}

Path defines path to callback_url in request payload to Product. You can see default value above.

WaitForCallback

It can optionally be specified in the value corresponding to this path.

WaitForCallback can be true. In this case all default values will be applied.

The following example invoke Staircase Build product

"Build": {
 "Type": "InvokeProduct",
 "Path": "infra-builder/builds",
 "Method": "POST",
 "RequestPayload": {
 "source_url": "$.outputs.RetrieveAssessResponse.response_payload.source_url"
 },
 "WaitForCallback": {
 "ExpectedPath": "$.status",
 "ExpectedValues": ["SUCCEEDED"],
 "FailedValues": ["ERROR", "FAILED"]
 },
 "Next": "NextState"
}

WaitForCallback default values are

"ExpectedPath": "status",
"ExpectedValues": [
 "SUCCEEDED",
 "Succeeded",
 "COMPLETED",
 "Completed"
],
"FailedValues": [
 "CANCELLED",
 "Cancelled",
 "FAILED",
 "Failed",
 "ERROR",
 "Error",
 "TIME_OUT",
 "Time_out"
],

TimeoutSeconds

TimeoutSeconds (Optional). If the task runs longer than the specified seconds, this state fails with a Timeout. Must be a positive, non-zero integer. If not provided, the default value is 3600.

Retry

Retry (Optional). Retry logic if any error in InvokeProduct happens it does retry. Example of State with Retry usage:

"CreateCollection": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions/{transaction_id}/collections",
 "Method": "POST",
 "PathParameters": {
 "transaction_id": "$.transaction_id"
 },
 "RequestPayload": {
 "metadata": {
 "version": 2,
 "validation": true
 },
 "data": "$.request_payload.collection_data"
 },
 "Retry": {
 "IntervalSeconds": 10,
 "MaxAttempts": 10,
 },
 "Next": "MockData"
},

Both values are required and accepts int values.

After

After (Optional). Specifies type of action to execute after response from a product was received. Usually this is used when Product returns data instead of response_collection_id.

Supported after actions:

• CreateCollection will create a collection using Product response as a payload. Accepts product_response_data_path to prodive json path to response data
• CreateReference will create a reference to a dataset. Can be used later f.e. IterationSettings -> Reference

CreateReference

Accepts:

• reference_name (Optional). Name of the reference to be save. Going to available later in current Job Execution. If not specifed, Step name will be used.
• product_response_data_path (Optional). Path in the Product response to a data to be saved as a reference.
• product_response_data_url (Optional). Path in the Product response to a URL where data for a Reference is located. Job product will download the data and place and create new Reference.

It is possible to Get All saved references back from the Job Execution.

Example of After action that saves InvokeProduct response as a reference.

 {
 "GetTelemetry": {
 "Type": "InvokeProduct",
 "Method": "POST",
 "Path": "job/stats",
 "Projection": "$.data",
 "After": [
 {
 "CreateReference": {
 "reference_name": "job_stats",
 "product_response_data_path": "$"
 }
 }
 ],
 "Next": "SaveTelemetry"
 }
 }

Example of After action that saves Connector response with URL as a reference.

 {
 "ToJSON": {
 "Type": "InvokeProduct",
 "Path": "connector-jobs/transformations/csv-to-json/async",
 "Method": "PUT",
 "RequestPayload": {
 "transaction_id": "$.transaction_id",
 "urls": {
 "download_from_url": "$.outputs.RunSQL.response_payload.url"
 },
 "configuration": {
 "delimiter": ","
 }
 },
 "WaitForCallback": {
 "ExpectedPath": "$.status"
 },
 "After": [
 {
 "CreateReference": {
 "reference_name": "sql_results",
 "product_response_data_url": "$.result.temporary_url"
 }
 }
 ],
 "Next": "MoveToMortgage"
 }
 }

CreateCollection For example to be able to Translate payload and save to collection

Instead of

 {
 "TranslateAssets": {
 "Method": "POST",
 "Next": "CreateCollectionV2Assets",
 "Path": "translator/translate",
 "RequestPayload": {
 "data": "$.outputs.CheckStatusAssets.response_payload.data",
 "lang_from": "staircase",
 "lang_to": "staircase",
 "metadata": {
 "transaction_id": "$.transaction_id"
 },
 "sc_version_from": 0,
 "sc_version_to": 2
 },
 "Type": "InvokeProduct"
 },
 "CreateCollectionV2Assets": {
 "Method": "POST",
 "Next": "InvokeSoftCredit",
 "Path": "persistence/transactions/{transaction_id}/collections",
 "PathParameters": {
 "transaction_id": "$.transaction_id"
 },
 "RequestPayload": {
 "data": "$.outputs.TranslateAssets.response_payload",
 "metadata": {
 "version": 2
 }
 },
 "Retry": {
 "IntervalSeconds": 3,
 "MaxAttempts": 2
 },
 "Type": "InvokeProduct"
 }
 } 

it is recommended to use After action

 "TranslateAssets": {
 "Method": "POST",
 "Next": "CreateCollectionV2Assets",
 "Path": "translator/translate",
 "RequestPayload": {
 "data": "$.outputs.CheckStatusAssets.response_payload.data",
 "lang_from": "staircase",
 "lang_to": "staircase",
 "metadata": {
 "transaction_id": "$.transaction_id"
 },
 "sc_version_from": 0,
 "sc_version_to": 2
 },
 "Type": "InvokeProduct",
 "After": [
 {
 "CreateCollection": {
 "product_response_data_path": "$.response_payload",
 "validation": false
 }
 }
 ]
 } 

Projection

Projection (Optional). Specifies the field(s) to return. To return all fields, omit this parameter. Can be a String or an Object.

In case InvokeProduct has CallbackSetting Projection is going to be applied to callback response payload $, otherwise Projection is going to be applied to $.response_payload.

In case InvokeProduct has IterationSettings it is possible to use $$.CurrentValue inside projection.

Example of Projection as a String "Projection":"$.collection_id". It will put only "$.collection_id" value in your response_payload object.

{
 "name": "demo-job-01",
 "description": "Demo demo-job-01",
 "definition": {
 "StartAt": "CreateCollection",
 "States": {
 "CreateCollection": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions/{transaction_id}/collections",
 "Method": "POST",
 "PathParameters": {
 "transaction_id": "$.transaction_id"
 },
 "RequestPayload": {
 "metadata": {
 "version": 2,
 "validation": true
 },
 "data": "$.request_payload.collection_data"
 
 },
 "Projection":"$.collection_id",
 "Next": "MockData"
 },
 "MockData": {
 "Type": "MockData",
 "Data": {
 "foo": "bar"
 },
 "Next": "Aggregation"
 },
 "Aggregation": {
 "Type": "OutputAggregate",
 "Aggregation": {
 "collection_id.$": "$.outputs.CreateCollection.response_payload",
 "mock_foo.$": "$.outputs.MockData.foo"
 },
 "End": true
 }
 }
 }
} 

Example of Projection as a Object

"Projection": {"collection_id": "$.collection_id"}

It will put my_collection_id in your response_payload with a value of "$.collection_id".

{
 "name": "demo-job-01",
 "description": "Demo demo-job-01",
 "definition": {
 "StartAt": "CreateCollection",
 "States": {
 "CreateCollection": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions/{transaction_id}/collections",
 "Method": "POST",
 "PathParameters": {
 "transaction_id": "$.transaction_id"
 },
 "RequestPayload": {
 "metadata": {
 "version": 2,
 "validation": true
 },
 "data": "$.request_payload.collection_data"
 
 },
 "Projection": {"my_collection_id": "$.collection_id"},
 "Next": "MockData"
 },
 "MockData": {
 "Type": "MockData",
 "Data": {
 "foo": "bar"
 },
 "Next": "Aggregation"
 },
 "Aggregation": {
 "Type": "OutputAggregate",
 "Aggregation": {
 "collection_id.$": "$.outputs.CreateCollection.response_payload.my_collection_id",
 "mock_foo.$": "$.outputs.MockData.foo"
 },
 "End": true
 }
 }
 }
} 

Example using Projection with $$.CurrentValue

{
 "GetServiceKey": {
 "Type": "InvokeProduct",
 "Method": "GET",
 "Path": "environment-manager/service-key",
 "Projection": {
 "domain_name": "$$.CurrentValue",
 "service_key": "$.service_key"
 },
 "IterationSettings": {
 "IterateOverPath": "$.outputs.Subscribers.response_payload.subscribed_domains",
 "MaxConcurrency": 10
 },
 "Next": "SomeData"
 } 
}

Projection Examples

Example of Projection for Invoke Product

 "Projection": {
 "invocation_status": "$.invocation_status",
 "request_collection_id": "$.request_collection_id",
 "response_collection_id": "$.response_collection_id"
 }

Example of Projection for Execute Job

 "Projection": {
 "response_payload": "$.response_payload",
 "execution_id": "$.execution_id"
 }

Iteration Settings

Job supports iteration settings for dynamic parallelism.

IterationSettings can be used to run a set of steps for each element of an input array or reference.

IterationSettings for an array If you add IterationSettings to you state, InvokeProduct state will execute the same steps for multiple entries of an array in the state input.

Accepts:

• IterateOverPath The IterateOverPath field's value is a reference path identifying where in the effective input the array field is found (Required)
• MaxConcurrency (Optional) The MaxConcurrency field's value is an integer that provides an upper bound on how many invocations of the Iterator may run in parallel. Default value is 4.

For instance, a MaxConcurrency value of 10 will limit your Map state to 10 concurrent iterations running at one time. The value of 0, will places no limit on concurency. Step Functions invokes iterations as concurrently as possible.

There are two additional items available in the context object when processing a state with IterationSetting

The $$.CurrentIndex contains the index number for the arraIterationSettingsitem that is being processed in the current iteration The $$.CurrentValue contains the value for the array item that is being processed in the current iteration

"CreateCollection": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions/{transaction_id}/collections",
 "PathParameters": {
 "transaction_id": "$.outputs.CreateTransaction.response_payload.transaction_id"
 },
 "Method": "POST",
 "RequestPayload": {"data": "$$.CurrentValue"},
 "IterationSettings": {
 "IterateOverPath": "$.request_payload.items",
 "MaxConcurrency": 5,
 },
 "Next": "CreateTransaction2",
}

IterationSettings using Reference

Will use one the Saved References as an input for iteration. Iterating over Reference allow to expand limits of Job product, such as events history size, state machine size etc. It is important to make sure reference saved points to an array.

Accepts:

• Reference Name of the saved reference.
• ReferenceType Type of the saved reference JSON or CSV
• BatchSize (Optional) Batch size for single Iteration. Please make sure InvokeProduct enpoint can accepts array instead of object.
• ToleratedFailureCount (Optional) Error handling configuration where you can set count of errors to tolerate.
• ToleratedFailurePercentage (Optional) Error handling configuration where you can set percentage of errors to tolerate.
• MaxConcurrency (Optional) If you set it to zero, Job doesn't limit concurreny and runs 10,000 parallel child workflow executions. Default value is 4.

Example of IterationSettings using Reference

 {
 "SaveTelemetry": {
 "Method": "POST",
 "Path": "console-pipeline/pipelines/{pipeline_name}/datasets/main/data",
 "PathParameters": {
 "pipeline_name": "$.request_payload.pipeline_name"
 },
 "QueryParameters": {
 "account_name": "$.request_payload.domain_name",
 "transaction_id": "$.transaction_id",
 "created_at": "$jmespath.start_time | date_format(@, '%Y-%m-%dT%H:%M:%S.%f%z', '%Y-%m-%d %H:%M:%S')"
 },
 "RequestPayload": "$$.CurrentValue",
 "Type": "InvokeProduct",
 "IterationSettings": {
 "Reference": "job_stats",
 "BatchSize": 20,
 "ToleratedFailureCount": 100,
 "ToleratedFailurePercentage": 0
 },
 "Retry": {
 "IntervalSeconds": 10,
 "MaxAttempts": 5
 },
 "End": true
 }
 }

InvokeJob

InvokeJob initiates a direct synchronous call to another Job. Being inherently synchronous, InvokeJob will invariably wait for the other job to finish before proceeding.

The following example demonstrates how to invoke other Job

 "FIPS": {
 "Type": "InvokeJob",
 "JobName": "get-data-fips",
 "RequestPayload": {}
 "Next": "SQL"
 }

JobName

Job Name to invoke. Job should be created before using Create Job

RequestPayload

Request Payload to be passed to Job execution. See Request Payload in Execute Job Should always be an object,

 {
 "Create": {
 "Type": "InvokeJob",
 "JobName": "sql-execute-and-wait",
 "TransactionId": "$.transaction_id",
 "RequestPayload": {
 "sql": "$.outputs.Start.create_sql",
 "parameters": [
 "$.outputs.FIPS.fips_counties_prefixes_start"
 ]
 },
 "Next": "Insert"
 }
 }

TransactionId

TransactionID to be passed to Job execution. See Transaction ID in Execute Job If TransactionID was not provided, Job product will generate new Transaction ID.

Projection

Specifies the field(s) to return. To return all fields, omit this parameter. Should always be an object. Projection is going to be applied to $.response_payload of a Job. Projection does not support JMESPATH.

Example of Projection

 {
 "Projection": {
 "fips_counties_prefixes_start": "$.fips_counties_prefixes_start",
 "fips_counties_prefixes_continue": "$.fips_counties_prefixes_continue"
 }
 }

 {
 "FIPS": {
 "Type": "InvokeJob",
 "JobName": "get-data-fips",
 "TransactionId": "$.transaction_id",
 "Projection": {
 "fips_counties_prefixes_start": "$.fips_counties_prefixes_start",
 "fips_counties_prefixes_continue": "$.fips_counties_prefixes_continue"
 },
 "Next": "SQL"
 }
 }

AthenaStartQuery

This type allows invoking AWS Athena directly, without API invocations or additional job definitions. AthenaStartQuery is a synchronous operation, which means its state will be 'RUNNING' as soon as the SQL request starts running.

Accepts:

• Query. SQL query to execute. Should be a string
• Parameters (Optional). Arrays of SQL Query parameters to use
• WorkGroup (Optional). AWS Athena workgroup to use. Default value is: primary

Outputs:

• QueryExecutionId. ID of the query execution

The following example executes SQL and get results as URL or CSV Reference

 {
 "name": "athena-test",
 "definition": {
 "StartAt": "Data",
 "States": {
 "Data": {
 "Type": "Data",
 "Next": "Start",
 "Data": {
 "params": [
 "NY"
 ]
 }
 },
 "Start": {
 "Type": "AthenaStartQuery",
 "Query": "SELECT * FROM \"db_name\".\"addresses\" WHERE state = ? limit 10",
 "Parameters": "$.outputs.Data.params",
 "Next": "Get"
 },
 "Get": {
 "Type": "AthenaGetResults",
 "QueryExecutionId": "$.outputs.Start.QueryExecutionId",
 "Next": "GetUrl"
 },
 "GetUrl": {
 "Type": "AthenaGetResultsAsUrl",
 "QueryExecutionId": "$.outputs.Start.QueryExecutionId",
 "Next": "GetReference"
 },
 "GetReference": {
 "Type": "AthenaGetResultsAsReference",
 "QueryExecutionId": "$.outputs.Start.QueryExecutionId",
 "ReferenceName": "reference_name_foo",
 "End": true
 }
 }
 }
 }

AthenaGetResults

Gets SQL Results as data inside the Job state machine

Accepts:

• QueryExecutionId. SQL query to execute. Should be a string

Outputs:

• Data. JSON array that represents rows from SQL query results

AthenaGetResultsAsUrl

Accepts:

• QueryExecutionId. SQL query to execute. Should be a string

Outputs:

• URL. Pre-signed url to download results. File is in the format of CSV

AthenaGetResultsAsReference

Accepts:

• QueryExecutionId. SQL query to execute. Should be a string
• ReferenceName. Reference name to be created in current Job execution. Similar to CreateReference

Outputs:

• Reference name that was created

The following example executes SQL, creates Reference and executes IterationSettings using Reference

 {
 "name": "athena-test",
 "definition": {
 "StartAt": "Start",
 "States": {
 "Start": {
 "Type": "AthenaStartQuery",
 "Query": "SELECT * FROM \"batch-data-load\".\"sc-first-american-listings-unique\" limit 10000;",
 "Next": "GetReference"
 },
 "GetReference": {
 "Type": "AthenaGetResultsAsReference",
 "QueryExecutionId": "$.outputs.Start.QueryExecutionId",
 "ReferenceName": "listings",
 "Next": "InvokeJobs"
 },
 "InvokeJobs": {
 "Type": "InvokeProduct",
 "Path": "job/jobs/{job_name}/executions",
 "Method": "POST",
 "IterationSettings": {
 "IterateOverPath": "$",
 "Reference": "listings",
 "ReferenceType": "CSV",
 "BatchSize": 2,
 "ToleratedFailurePercentage": 5,
 "MaxConcurrency": 10
 },
 "PathParameters": {
 "job_name": "athena-test-empty"
 },
 "RequestPayload": {
 "transaction_id": "$.transaction_id",
 "request_payload": {
 "data": "$$.CurrentValue"
 }
 },
 "WaitForCallback": {
 "ExpectedPath": "$.status"
 },
 "End": true
 }
 }
 }
 }

Choice

A Choice state adds branching logic to a state machine.

Choices (Required): An array of Choice Rules that determines which state the state machine transitions to next

Default (Optional, Recommended): The name of the state to transition to if none of the transitions in Choices is taken

Choice Rules

A Choice state must have a Choices field whose value is a non-empty array, and whose every element is an object called a Choice Rule. A Choice Rule contains the following:

• A comparison - Two fields that specify an input variable to compare, the type of comparison, and the value to compare the variable to. Choice Rules support comparison between two variables. Within a Choice Rule, the value of Variable can be compared with another value from the state input by appending Path to name of supported comparison operators.
• A Next field - The value of this field must match a state name in the state machine.

The following example checks whether the numerical value is equal to 1.

{
 "Variable": "$.foo",
 "NumericEquals": 1,
 "Next": "FirstMatchState"
}

The following example checks whether the string is equal to MyString.

{
 "Variable": "$.foo",
 "StringEquals": "MyString",
 "Next": "FirstMatchState"
}

Supported Operations

The following comparison operators are supported:

• And
• BooleanEquals,BooleanEqualsPath
• IsBoolean
• IsNull
• IsNumeric
• IsPresent
• IsString
• IsTimestamp
• Not
• NumericEquals,NumericEqualsPath
• NumericGreaterThan,NumericGreaterThanPath
• NumericGreaterThanEquals,NumericGreaterThanEqualsPath
• NumericLessThan,NumericLessThanPath
• NumericLessThanEquals,NumericLessThanEqualsPath
• Or
• StringEquals,StringEqualsPath
• StringGreaterThan,StringGreaterThanPath
• StringGreaterThanEquals,StringGreaterThanEqualsPath
• StringLessThan,StringLessThanPath
• StringLessThanEquals,StringLessThanEqualsPath
• StringMatches
• TimestampEquals,TimestampEqualsPath
• TimestampGreaterThan,TimestampGreaterThanPath
• TimestampGreaterThanEquals,TimestampGreaterThanEqualsPath
• TimestampLessThan,TimestampLessThanPath
• TimestampLessThanEquals,TimestampLessThanEqualsPath

Wait

A Wait state delays the state machine from continuing for a specified time.

The following Wait state introduces a 10-second delay into a state machine

"wait_ten_seconds": {
 "Type": "Wait",
 "Seconds": 10,
 "Next": "NextState"
}

MockData

MockData could be used to generate and place some static data. Common usecase would be a usage of this step instead of InvokeProduct if it is not implemented.

"AddingSomeDataInsideJob": {
 "Type": "MockData",
 "Data": {
 "foo": "bar",
 },
 "Next": "NextState"
 }

Will produce following result

{
 "outputs": {
 "AddingSomeDataInsideJob": {
 "foo": "bar"
 }
 }
}

Data

Data could be used to generate new data from the $.outputs.*.

 {
 "Start": {
 "Data": {
 "array": [
 "a",
 "b",
 "c",
 "d",
 "e"
 ]
 },
 "Next": "Data",
 "Type": "MockData"
 }, 
 "Data": {
 "Data": {
 "size.$": "States.ArrayLength($.outputs.Start.array)"
 },
 "End": true,
 "Type": "Data"
 },

 }

Will produce following result

 {
 "Data": {
 "size": 5
 },
 "Start": {
 "array": [
 "a",
 "b",
 "c",
 "d",
 "e"
 ]
 }
 }

OutputAggregate

OutputAggregate is used to clear all state results from outputs and create new step data.

"AddingSomeDataInsideJob": {
 "Type": "OutputAggregate",
 "Aggregation": {
 "foo": "bar",
 "foo.$": "$.outputs.PreviousStep.foo" 
 },
 "Next": "NextState"
 }

Aggregation as an object allow to construct Step output with custom fields.

Example of MockData used together with OutputAggregate

 {
 "name": "JobRandom",
 "description": "",
 "definition": {
 "StartAt": "State",
 "States": {
 "State": {
 "Type": "MockData",
 "Result": { "foo": "bar" },
 "Next": "State1"
 },
 "State1": {
 "Type": "MockData",
 "Result": { "bar": "bar" },
 "Next": "State2"
 },
 "State2": {
 "Type": "OutputAggregate",
 "Aggregation": { "foonew.$": "$.outputs.State.foo" },
 "End": true
 }
 }
 }
 }

Aggregation as a String allow to set object from from different step.

Example of OutputAggregate override state outputs with results from "State"

 {
 "name": "JobRandom",
 "description": "",
 "definition": {
 "StartAt": "State",
 "States": {
 "State": {
 "Type": "MockData",
 "Result": { "foo": "bar" },
 "Next": "State1"
 },
 "State1": {
 "Type": "MockData",
 "Result": { "bar": "bar" },
 "Next": "State2"
 },
 "State2": {
 "Type": "OutputAggregate",
 "Aggregation": "$.outputs.State",
 "End": true
 }
 }
 }
 }

Wait for action

Wait for action allows to pause job execution and wait for resume action.

The following Wait for action will pause job execution

"wait_for_action": {
 "Type": "WaitForAction",
 "Next": "NextState"
}

Wait for callback

State waits for callback from a product. WaitForCallback can be used to specify success and/or failed values. This step behaviour is the same as waitforcallback in the InvokeProduct

StepName should point to a Step defined before with CallbackSettings

It is also possible to specify Projection in the WaitForCallback. In that case it will be applied to the to callback response payload $. See more details in the InvokeProduct section Projection

The following Wait for action will pause job execution

"WaitForCallback": {
 "Type": "WaitForCallback",
 "StepName": "step_name",
 "ExpectedPath": "$.status",
 "ExpectedValues": ["SUCCEEDED"],
 "FailedValues": ["ERROR", "FAILED"],
 "Next": "NextState"
}

Examples

Create Transaction Job Example

{
 "name": "CreateTransaction",
 "description": "Create Transaction",
 "definition": {
 "StartAt": "CreateTransaction",
 "States": {
 "CreateTransaction": {
 "Type": "InvokeProduct",
 "Path": "persistence/transactions",
 "Method": "POST",
 "RequestPayload": {}
 "End": true
 }
 }
 }
}

DevOps Pipeline Job

{
 "name": "pipeline",
 "description": "Devops Pipeline",
 "definition": {
 "StartAt": "Clone",
 "States": {
 "Clone": {
 "Type": "InvokeProduct",
 "Path": "code/clone",
 "Method": "POST",
 "RequestPayload": {
 "project_name": "$.request_payload.product_name",
 "branch": "$.request_payload.branch_name",
 "github_account": "GITHUB_ACCOUNT",
 "github_token": "GITHUB_TOKEN"
 },
 "Next": "WaitCloneStatus20Secs"
 },
 "WaitCloneStatus20Secs": {
 "Type": "Wait",
 "Seconds": 20,
 "Next": "RetrieveCloneStatus"
 },
 "RetrieveCloneStatus": {
 "Type": "InvokeProduct",
 "Path": "code/clone/{bundle_id}",
 "Method": "GET",
 "PathParameters": {
 "bundle_id": "$.outputs.Clone.response_payload.bundle_id"
 },
 "Next": "CheckCloneStatus"
 },
 "CheckCloneStatus": {
 "Type": "Choice",
 "Choices": [
 {
 "Variable": "$.outputs.RetrieveCloneStatus.response_payload.clone_status",
 "StringEquals": "SUCCEEDED",
 "Next": "Assess"
 },
 {
 "Variable": "$.outputs.RetrieveCloneStatus.response_payload.clone_status",
 "StringEquals": "FAILED",
 "Next": "FailState"
 },
 {
 "Variable": "$.outputs.RetrieveCloneStatus.response_payload.clone_status",
 "StringEquals": "IN_PROGRESS",
 "Next": "WaitCloneStatus20Secs"
 }
 ]
 },
 "Assess": {
 "Type": "InvokeProduct",
 "Path": "code-assessor/assessments",
 "Method": "POST",
 "RequestPayload": {
 "source_url": "$.outputs.RetrieveCloneStatus.response_payload.source_url"
 },
 "Next": "WaitAssessStatus30Secs"
 },
 "WaitAssessStatus30Secs": {
 "Type": "Wait",
 "Seconds": 30,
 "Next": "RetrieveAssessStatus"
 },
 "RetrieveAssessStatus": {
 "Type": "InvokeProduct",
 "Path": "code-assessor/assessments/{assessment_id}",
 "Method": "GET",
 "PathParameters": {
 "assessment_id": "$.outputs.Assess.response_payload.assessment_id"
 },
 "Next": "CheckAssessStatus"
 },
 "CheckAssessStatus": {
 "Type": "Choice",
 "Choices": [
 {
 "Variable": "$.outputs.RetrieveAssessStatus.response_payload.status",
 "StringEquals": "SUCCEEDED",
 "Next": "Build"
 },
 {
 "Variable": "$.outputs.RetrieveAssessStatus.response_payload.status",
 "StringEquals": "FAILED",
 "Next": "FailState"
 },
 {
 "Variable": "$.outputs.RetrieveAssessStatus.response_payload.status",
 "StringEquals": "IN_PROGRESS",
 "Next": "WaitAssessStatus30Secs"
 }
 ]
 },
 "Build": {
 "Type": "InvokeProduct",
 "Path": "infra-builder/builds",
 "Method": "POST",
 "RequestPayload": {
 "source_url": "$.outputs.RetrieveAssessStatus.response_payload.source_url"
 },
 "CallbackSettings": {
 "ExpectedPath": "$.status",
 "ExpectedValue": "SUCCEEDED"
 },
 "Next": "Wait10SecGetBuildResults"
 },
 "Wait10SecGetBuildResults": {
 "Type": "Wait",
 "Seconds": 10,
 "Next": "GetBuildResults"
 },
 "GetBuildResults": {
 "Type": "InvokeProduct",
 "Path": "infra-builder/builds/{build_id}",
 "Method": "GET",
 "PathParameters": {
 "build_id": "$.outputs.Build.response_payload.bundle_id"
 },
 "Next": "Deploy"
 },
 "Deploy": {
 "Type": "InvokeProduct",
 "Path": "infra-deployer/deploy-by-token",
 "Method": "POST",
 "RequestPayload": {
 "artifacts_url": "$.outputs.GetBuildResults.response_payload.artifacts_url",
 "environment_token": "$.request_payload.environment_token"
 },
 "CallbackSettings": {
 "ExpectedPath": "$.status",
 "ExpectedValue": "SUCCEEDED"
 },
 "Next": "Wait10SecGetDeployResults"
 },
 "Wait10SecGetDeployResults": {
 "Type": "Wait",
 "Seconds": 10,
 "Next": "GetDeployResults"
 },
 "GetDeployResults": {
 "Type": "InvokeProduct",
 "Path": "infra-deployer/deploy/{bundle_id}",
 "Method": "GET",
 "PathParameters": {
 "bundle_id": "$.outputs.Deploy.response_payload.bundle_id"
 },
 "End": true
 },
 "FailState": {
 "Type": "Fail",
 "Cause": "Failed.",
 "Error": "Failed"
 }
 }
 }
}

{
 "request_payload": {
 "form_data": {
 "buyer_name": "Buyer Name",
 "buyer_2_name": "Co Buyer Name",
 "agent_name": "Agent-Name",
 "agent_email": "agent@staircase.co",
 "brokerage_name": "remax",
 "brokerage_phone": "34342342",
 "loan_officer_email": "lo@staircase.co",
 "loan_officer_name": "LO",
 "property_street_address": "209 N Highway 45",
 "property_city": "Bonanza",
 "property_state": "AR",
 "property_zip_coShow the rest

de": "72916", "down_payment_amount": 45000, "down_payment_percent": 5, "offer_amount": 120000, "loan_id": "87376716-9351-4abb-93d6-ddd08364f897", "preapproval_loan_amount": 114000, "loan_officer_phone": "", "lender_organization_name": "envoymortgage" }, "transaction_id": "01GG54QMC4GCQBQ5YY91VVDCVP", } }

Example of using runtime variables

 {
 "definition": {
 "StartAt": "A",
 "States": {
 "A": {
 "Data": {
 "execution_id": "$.execution_id",
 "start_time": "$.start_time",
 "transaction_id": "$.transaction_id"
 },
 "End": true,
 "Type": "Data"
 }
 }
 },
 "name": "test"
 } 

{
"type": "SCHEDULE",
"schedule_expression": "rate(5 minutes)"
}

You can filter the messages and invoke your jobs according to your business needs

{
"type": "EVENT",
"filter_expression": {
"source_name": ["LOS_NAME"],
"event_type": ["LOAN_CREATED","LOAN_MODIFIED"]
}
}

Filter Expression:

You can only add filter expressions for the string elements in the message or the partner name

• Exact Matching

Exact matching occurs when a filter expression value matches one or more message values.

Consider the following policy attribute:

"football_team": ["liverpool", "chelsea"]

It matches the following message:

{
"football_team": "liverpool",
"players": [
"Salah",
"Mane",
"Firmino"
]
}

However, it doesn't match the following message:

{
"football_team": "man_united",
"players": [
"Cavani",
"Rashford",
"Ronaldo"
]
}

• Prefix matching

When a filter expression includes the keyword prefix, it matches any message value that begins with the specified characters.

Consider the following policy attribute:

"sport": [{"prefix": "bas"}]

It matches either of the following messages:

{
"sport": "baseball"
}

{
"sport": "basketball"
}

However, it doesn't match the following message:

{
"sport": "rugby"
}

• Anything-but matching

When a filter expression includes the keyword anything-but, it matches any message that doesn't include any of the filter expression values.

Consider the following policy attribute:

"sport": [{"anything-but": ["rugby", "tennis"]}]

It matches either of the following message attributes:

{
"sport": "baseball"
}

{
"sport": "football"
}

However, it doesn't match the following message:

{
"sport": "rugby"
}