Transformation syntax
Blacksmith exposes a Python / Django-like syntax at two different layers in the data lifecycle.
Firstly, when working inside the transformation
objects of the gateway. This
adds a convenient way to Transform data from triggers to integrations by
simply using a key / value mapping. When Loading data to a blob store, NoSQL
database, topic, and HTTP API, the JSON returned by the transformation
is the
one actually Loaded into the integration:
sources:
- name: "my-source"
# ...
triggers:
- name: "my-trigger"
mode: "subscription"
# ...
integrations:
- name: "my-nosql-store"
transformation:
users: "{% query 'body.friends.#(last=Murphy)#' %}"
Secondly, when working within SQL files: Loads, operations, and selects.
This does not apply for migrations because they must be predictable in order to
be reliable. When Loading data to a SQL database, the JSON returned by the
transformation
adds a convenient way to generate SQL dynamically with contextual
data and returning the compiled SQL file:
{% if order %}
INSERT INTO orders (amount, user_id) VALUES
({{ order.amount }}, '{{ user.id }}');
{% endif %}
The Transformation syntax involves four constructs: variables, tags, filters, and comments.
Variables
Variables are surrounded by {{
and }}
like this:
INSERT INTO orders (amount, user_id) VALUES
({{ order.amount }}, '{{ user.id }}');
When working in the transformation
object, these variables are the ones returned
by the trigger when Extracting data from the gateway.
When working in SQL files, these variables are the ones defined previously in the
transformation
object. When running operations from the CLI, variables are the
ones passed from a CSV or
from a JSON file.
Tags & Queries
Tags provide arbitrary logic in the rendering process. Tags are surrounded by
{%
and %}
like this:
{% if order %}
INSERT INTO orders (amount, user_id) VALUES
({{ order.amount }}, '{{ user.id }}');
{% endif %}
Queries (defined by the tag query
) are a particular kind of tags. They allow to
retrieve a subset of data within a single line of configuration. They can be used
for filtering, mapping, grouping, etc. In some cases query
tags may overlap
features offered by others. While the others are mostly suited for dynamically
templating SQL files, query
tags are best suited inside transformation
objects between triggers and integrations:
sources:
- name: "my-source"
# ...
triggers:
- name: "my-trigger"
mode: "subscription"
# ...
integrations:
- name: "my-nosql-store"
transformation:
users: "{% query 'body.friends.#(last=Murphy)#' %}"
Then the document put in the NoSQL store would have a single key users
with the
result of the query body.friends.#(last='Murphy')#
as value, which given the
example in the appropriate section, is:
{
"users": [
{
"first": "Dale",
"last": "Murphy",
"age": 44,
"nets": ["ig", "fb", "tw"]
},
{
"first": "Jane",
"last": "Murphy",
"age": 47,
"nets": ["ig", "tw"]
}
]
}
Modifiers
Modifiers transform the values of variables with tag arguments. They look like this:
INSERT INTO orders (amount, user_id) VALUES
({{ order.amount|floatformat:2 }}, '{{ user.id }}');
Comments
As the name suggests, comments lets you comment your template and are not compiled in the result. They look like this:
{# This won't be compiled #}
If you notice something we've missed or could be improved on, please follow this link and submit a pull request to the repository. Once we merge it, the changes will be reflected on the website the next time it is deployed.
Thank you for your contributions!