Run and track a Hamilton dataflow in less than 5 minutes.
Here’s a brief overview of capabilities you get by using the Hosted Hamilton UI on DAGWorks:
For more detailed information see the Capabilities page. Otherwise, let’s get started!
Navigate to app.dagworks.io and login with your credentials. If you do not have an account, click “sign up” and you can get started with the 14-day trial of the “team” plan for free. After that, you can choose to continue with the team plan, select another, or automatically downgrade to the community plan for free.
You should be directed to the projects page (four squares). Click the + New Project
button, and add a name, a description, and set visibility.
You can add your team (if you have one) to the “Visible by” and “Modifiable by” sections, as well as anyone you wish to share the project with.
Once the project is created, select it and find the ID — you’ll need it for later.
You can always find the name/ID of your project by going back to the projects page — its the first when you log in.
Navigate to the “API Keys” page (key icon) and click + Create new key
. Copy it by clickin on the copy icon and write it down.
Be careful with your API keys! These are like passwords — store them in a safe place and never check them in. You can always generate a new one if you forget, or delete the ones you have.
This assumes you have pip
installed/know how to manage python environments. If not, it is worth a read here. To install the library/CLI, run:
If you’re already using Hamilton in your project you can skip this and go directly to the DAGWorks Tracking Adapter page.
Now that you’ve installed the CLI, you can run the following command to generate from a template:
We have a few sample templates to choose from:
hello_world
The basic “hello world app”. Try it out at tryhamilton.dev.
machine_learning
A project that shows training and evaluating a machine learning model with Hamilton + DAGWorks. Meant for MLEs and Data Scientists who create models on a daily basis.
time_series_feature_engineering
A project that demonstrates how to use Hamilton for building out a time series feature engineering pipeline with Hamilton + DAGWorks. Time-series-forecasting was Hamilton’s original purpose. Meant for data scientists who solve forecasting problems.
data_processing
A project that demonstrates how to use Hamilton to build out an ETL. This is geared largely towards data engineers.
Now you’re ready for the fun part! Navigate into the directory, and run it!
Go back to the UI, and click Select
on the project you just created (or click the DAG icon in the table row). You should see the DAG!
Hamilton is the framework that helps you write and organize python functions. It is entirely open-source, meaning that any code that you write to work with DAGWorks can be used outside of the platform as well!
The basics of it are simple — you write a collection of functions, each of which have a specific shape. The literal parameter names refer to the upstream dependencies (either nodes in the DAG or external inputs), and the return value is the output of the function (which can then be referred to as well).
For example, the function:
Will be a transformation node in a DAG. This transformation will depend on two “upstream” transformations:
upstream_data
— a dataframeupstream_param
— an integerBy using the “driver”, you specify the inputs you want, and hamilton will execute it.
Organizing your code in Hamilton gives you a lot of power. Since it is also very lightweight, you can therefore:
Hamilton gives you, the data/ML pipeline creator, a lot of power. Read more about it here, and try it out (in the browser!) here: tryhamilton.dev.
Congrats, you’ve done it! Feel free to stop reading and start building. If you want to learn more, check out the following resources:
Learn to build highly expressive and powerful dataflows with Hamilton.
Learn how to use the DAGWorks client to store and track your Hamilton dataflows.
Learn to build highly expressive and powerful dataflows with Hamilton.
Book some time with our founder to get personalized onboarding and help!