DAGWorks aims to be your one-stop-shop for tracking, visualizing, and debugging Hamilton dataflows. By seamlessly integrating with Hamilton, DAGWorks allows you to:
Share project with your team/anyone who needs access, enabling you to quickly onboard new developers and
collaborate on Hamilton dataflows. You can share a project by first navigating to the projects page
and then clicking Edit
on your project and modifying who can see and/or write to your project.
Notes:
Every Hamilton dataflow executed with the Hosted Hamilton UI on DAGWorks is tracked and stored, along with a basic set of data quality/results. Results are streamed in, and this allows you to:
You can do this all (and more) by navigating to the “runs” page for you project.
The Hosted Hamilton UI on DAGWorks allows you to keep track of code and artifacts generated. This enables you to:
You can explore this under the “Structure” tab in your project.
DAGWorks allows for diffing of DAG versions and runs, enabling you to answer questions like:
You can answer these questions in one of the following ways:
compare
.compare
.DAGWorks is centered around “Projects”. Projects are a collection of DAGs that share a common business goal. All DAGs in a project share the same codebase.
Every time you create a DAG by using the Hamilton Driver + DAGWorksTracker, it has the potential to create a new “project version”. Project versions are uniquely determined by the following attributes:
You can always reset the DAG Name for a certain driver instance by calling set_name(new_name)
on the driver. This will force a new
version to be logged. This is useful when calling DAG-manipulation functions, such as materialize
.
Note that if you archive a version (in the DAGWorks UI in the browse/versions page), it will no longer be visible in the DAGWorks UI. That said, if you then save to the same version (E.G. with the same DAG Name/code version), it will un-archive that version, and you will be able to see it again.
Every time you execute a run attaching the DAGWorksTracker to Hamilton, it uploads a run. These are associated with a project and a project version, i.e. an instantiation of a DAG. This consists of metadata about the run, task-level summaries of data, and any information about errors you encountered.
When you run a DAG with the DAGWorksTracker, the following happens:
run
appears.Chrome is the only supported browser at this time. We are working on supporting other browsers, but for now, please use Chrome. Reach out to support@dagworks.io for questions/help.
DAGWorks aims to be your one-stop-shop for tracking, visualizing, and debugging Hamilton dataflows. By seamlessly integrating with Hamilton, DAGWorks allows you to:
Share project with your team/anyone who needs access, enabling you to quickly onboard new developers and
collaborate on Hamilton dataflows. You can share a project by first navigating to the projects page
and then clicking Edit
on your project and modifying who can see and/or write to your project.
Notes:
Every Hamilton dataflow executed with the Hosted Hamilton UI on DAGWorks is tracked and stored, along with a basic set of data quality/results. Results are streamed in, and this allows you to:
You can do this all (and more) by navigating to the “runs” page for you project.
The Hosted Hamilton UI on DAGWorks allows you to keep track of code and artifacts generated. This enables you to:
You can explore this under the “Structure” tab in your project.
DAGWorks allows for diffing of DAG versions and runs, enabling you to answer questions like:
You can answer these questions in one of the following ways:
compare
.compare
.DAGWorks is centered around “Projects”. Projects are a collection of DAGs that share a common business goal. All DAGs in a project share the same codebase.
Every time you create a DAG by using the Hamilton Driver + DAGWorksTracker, it has the potential to create a new “project version”. Project versions are uniquely determined by the following attributes:
You can always reset the DAG Name for a certain driver instance by calling set_name(new_name)
on the driver. This will force a new
version to be logged. This is useful when calling DAG-manipulation functions, such as materialize
.
Note that if you archive a version (in the DAGWorks UI in the browse/versions page), it will no longer be visible in the DAGWorks UI. That said, if you then save to the same version (E.G. with the same DAG Name/code version), it will un-archive that version, and you will be able to see it again.
Every time you execute a run attaching the DAGWorksTracker to Hamilton, it uploads a run. These are associated with a project and a project version, i.e. an instantiation of a DAG. This consists of metadata about the run, task-level summaries of data, and any information about errors you encountered.
When you run a DAG with the DAGWorksTracker, the following happens:
run
appears.Chrome is the only supported browser at this time. We are working on supporting other browsers, but for now, please use Chrome. Reach out to support@dagworks.io for questions/help.