Documentation Data Pipeline

Basic Data Pipeline

If you joined after January 6, 2020, you might be using our new Visual SQL interface, and this content may not look familiar to you. Check out our Visual SQL page for more relevant content.

Our Data Pipeline allows you to perform a number of transformations on your query results in a series of steps. These steps include a variety of operations such as column sorting, pivoting data, and adding calculated columns. The flexibility of adding any steps in any order allows you to get your data exactly how you want it.

Data Pipeline

Adding a Step

Steps can be added to the Pipeline under any Dataset to edit the results of that specific dataset’s query. Steps can also be added after merging Datasets to manipulate the merged data.

Click the + Add Transformation button in the Pipeline to add a new transformation step to your query results. You can apply as many steps as you’d like, in any order. You can also use each step multiple times. The available steps and examples are outlined on the Data Pipeline Steps page.

Some steps allow flexibility to select the type of formulas used to transform your data as well as add custom transformations using SQLite syntax. The various formula options are outlined on the Formulas page by type of operation.

Note: Data Pipeline Steps are applied in order from first to last, so in some cases it is possible for a later step to override an earlier one.

Adding a step in the Data Pipeline

Preview data in the Pipeline

To preview how your Data Pipeline Step transforms your result set before actually saving it, click Apply when you’re editing a Pipeline Step. The top result set is your data before the transformation, and the bottom result set is your data afterward. It can be incredibly useful to compare how your data looks before and after a transformation step is applied.

You can also toggle these previews by clicking Hide Input & Output or Show Input & Output at the bottom-left corner of the Pipeline Step.

Preview Data

Reorder a Step

To reorder any Pipeline Steps, simply switch over to Advanced Mode, disconnect the step in question and re-add it where you’d like. Make sure to also reconnect the nodes before and after the removed Pipeline Step. You can switch back to Basic Mode after completing this as long as you haven’t added any additional Merge steps.