August 18, 2022 • 3 minute read
Our Pricing Philosophy for Dagster Cloud
Dagster started off as an open-source project and our team at Elementl is committed to the open-source philosophy. Our goal is for Dagster to become a ubiquitous open standard for designing, running, and managing data pipelines and assets.
As we mentioned in explaining the reasoning behind our open-core model, we believe that there are three types of complexity being addressed by the Dagster project: application complexity, operational complexity, and enterprise complexity.
With the introduction of our hosted orchestration service, Dagster Cloud, we are accelerating adoption of Dagster. Stakeholders can now solely focus on managing application complexity such as structuring their code and writing their business logic using the Dagster framework.
Other concerns, like the operational complexity of running and maintaining Dagster’s central control plane, as well as the enterprise complexity of feature requirements to manage Dagster with many stakeholders are now offloaded. Utilities to address these complexities come out of the box in the hosted service.
Since December 2021, our commercial offering has been providing value to a wide variety of early access customers, ranging from small startups to Fortune 500 companies. We now feel confident in making the next move: opening up Dagster Cloud to general availability.
Of course, this raised new questions. For these general availability customers, how would we price Dagster Cloud in a standardized way? What should our customers be paying for and how can we align the customer and Elementl team incentives to sustainably scale usage?
In this post, we summarize our process in developing a pricing model for Dagster Cloud. In addition, we explain the different feature packages and hosted options that are available once you select a plan.
Pricing a hosted orchestration service
We started by pinning down what was important to us and what was important to members of the Dagster community who were looking for such a service.
Again, we wanted to offer a solution that would scale, so it would be accessible to all: from a solo practitioner looking to run a one-off proof of concept, all the way to the most demanding enterprise looking to free its data team of orchestration infrastructure concerns so they can spend more time focused on building and maintaining their data pipelines and assets.
We know that pricing has to be straightforward and predictable. It should align both the platform and the customer with the correct incentives.
Naturally, we looked at how other orchestration providers have priced similar services. What we saw and what we heard from the community gave us pause.
For example, most commercial orchestration solutions charge per task, but this encourages users to consolidate their data pipelines into a few large tasks to minimize their bill within budget. Likewise, billing per seat would prevent teams from including all practitioners and stakeholders of the data platform.
We wanted practitioners to work with the framework, not against it. We never want to see a data team have to compromise its engineering principles, inhibit its productivity, or move away from best practices.
A consumption based model
The most popular pricing model for contemporary managed cloud services is consumption based. With this pricing model, you pay for what you use, and nothing more. When the system runs, the usage meter runs. When the system stops, the usage meter stops.
In our consumption based model, we price on the duration of computations that you run on Dagster Cloud. Computation time is measured in millseconds of run-time for each individual step within an job. We aggregate the total run-time across all of those steps and round down to the nearest whole minute. Any infrastructure spin up time does not get counted.
We have found this to encourage teams to optimize computations. By building efficient orchestration jobs with Dagster’s observation and control capabilities, you run fewer, faster computations and your bill decreases. The more you optimize your orchestration layer, the less it costs to run.
You can also adopt Dagster Cloud without committing to an upfront cost, allowing you to incrementally realize the value of a managed data platform.
A simple single self-serve option
With a pricing model in place, we also looked at the best package plans and deployment options.
First, we offer a simple, low-friction, self-serve standard plan. You can get started straight from the Dagster website and get set up in a matter of minutes.
This standard plan includes everything that you need to run a single production deployment, which is plenty for most teams given the amazing capability of Dagster Cloud’s branch deployments.
There are no other usage limits this plan: you can have unlimited users and run any amount of computation to meet your data platform’s needs.
Two flexible hosted options: Serverless and Hybrid
In addition to choosing a plan, you can also choose how Dagster Cloud is hosted on your behalf.
If you are looking to have us host and manage the orchestration control plane and data plane for Dagster, we have you covered with our Serverless offering. We provide a straightforward usage-based fee per minute of computation usage, billed to the second. The price-per-minute comes down as you hit certain usage levels.
If you want full customization of the operational complexity of your data plane, we have that option with our Hybrid offering. We host and manage only the control plane for Dagster while you select and manage the data plane in your infrastructure. Again, the billing is simple. You pay a lower usage-based fee since you manage your own compute resources.
Additional orchestration tools for enterprise
If you are a larger enterprise, there will be some additional things you will need.
In our enterprise plan, we include unlimited production deployments. We also provide built-in security along with single-sign-on capabilities as well as more granular user roles. Furthermore, enterprise plans have robust service level agreements for those mission-critical implementations. You receive higher levels of support and a dedicated account manager for your team.
Get started today
We're always happy to hear your feedback, so please reach out to us! If you have any questions, ask them in the Dagster community Slack (join here!) or start a Github discussion. If you run into any bugs, let us know with a Github issue. And if you're interested in working with us, check out our open roles!