The world of business is migrating to the Cloud. This migration started in the early 2000s with COVID-19 accelerating the migration to support remote workers and enable digital business to grow. The benefits of the cloud include reduced costs, time, and encumbrance of maintaining hardware, thus allowing IT professionals to spend less time on their infrastructure and more time on their IT initiatives. While these benefits are real, usage-based cloud computing costs can build up based on the workload's size, duration, and complexity.
According to O’Reilly’s Cloud adoption report, more than 90% of organizations use the cloud, and cloud-based workloads account for 75% of workloads in 1 out of 5 organizations (Source: Fortinet 2021). Along with the benefits of cloud computing, a dark side effect has surfaced…unexpected spikes in usage-based costs. The causes of rising costs vary, but most can be attributed to a lack of understanding and experience with the complex world of compute, storage, and consumption. Project requirement “scope creep” can also contribute to ”cost creep” and higher-than-expected usage costs. According to the Flexera 2020 State of the Cloud Report, almost 59% of enterprises expected higher cloud usage than what they had actually planned. No one wants to pay for compute and storage they don’t need; this is a big appeal of cloud base resources. But it is very hard to forecast and budget the precise need for development projects. While some may be able to estimate usage and storage based on past projects and what’s been done historically, for most, anticipating needs requires a different mindset from behaviors that favor having too much rather than too little.
Kythera built Wayfinder healthcare data science platform on a Databricks foundation in part because of the features that help minimize the most common causes of “spend anxiety” and maximize the efficiency of cloud computing. We have been users of healthcare big data for years and know that big data users need a platform to meet their data needs, yet face new challenges that did not exist in their data centers and tech stacks. We embedded our knowledge of managing costs in AWS andDatabricks into Wayfinder, so our clients aren’t surprised by unexpected costs because no one wants to learn the hard way.
There are lots of statistics out there regarding the recognized problem of cost overruns. A survey of 350 IT and cloud decision-makers by Virtana Inc. found that 82% said they had incurred unnecessary cloud costs, 56% lack tools to manage their spending programmatically, and 86% can’t easily get a global view of all their costs when they need it.
While these stats may be disputed, the problem is real, and financial/economic cloud experts posit that the bulk of the problem comes from provisioning, storage and being able to monitor and assess computing costs accurately. It is more imperative than ever that organizations identify their project objectives, understand what data best fits those objectives, and maximize their cloud resources.
Databricks is packed with thousands of optimizations to provide its users with the best performance, like scaling clusters, spot pricing, Photon, Serverless, and spark optimized runtimes. Databricks SQL serverless removes the need to manage, configure or scale cloud infrastructure on the Lakehouse, and Databricks SQL warehouses provide instant, elastic SQL compute that will automatically scale. Wayfinder, built on Databricks, multiplies these cost savings for users of healthcare big data with features focused on efficiency, cost, and quality.
Let's take a look at some of the problem areas in cloud computing and how we help our clients avoid these costly mistakes.
In computing, provisioning simply means equipping your cloud with what it needs to run your workloads… things like which cloud features to leverage, capacity, operating system, and server types. The problem comes from overprovisioning or buying more than you actually need for storage, capacity or bandwidth. Why do companies do this? It comes from a historical mindset when IT departments had to provision for peak demand and a “just in case” mentality…you’ll have the ability to handle any unforeseen additional demand that may not even materialize, resulting in wasted money. The flip side to this, which may be even more costly, is not allocating enough, causing project delays while infrastructure is being provisioned.
With Big Data projects, resource needs spike and lull depending on the flow of volume, complex logic and enhancements. Databricks offers scaling clusters and spot pricing to help address this problem. Although these features are available, some users may not think to use them or automate them or find they don't have the expertise or time to optimize every component of infrastructure and processing. Wayfinder keeps teams in a structured environment until they “get their feet wet” and develop the understanding and expertise of working in the platform and with healthcare big data. We understand that for many users, some experience is needed to fully take advantage of both the platform features and healthcare data without getting hit with high costs during the learning curve. We often start by giving clients 1 cluster appropriately sized for their general needs, which caps the spend to 24x7 full-scaled cost (which they never, ever hit).
Another cause for cost overruns is forgotten instances. Think of it like forgetting to turn off the lights when you leave the room….forgotten instances occur when you forget to turn off your cloud instances/clusters when you no longer need them. This can rack up high costs if no one notices that a cluster is spinning. Databricks takes this one as a core feature, and Wayfinder helps mitigate forgotten instances by ensuring users have the correct settings and helping to control cluster permissions until they feel ready to take on full admin rights.
Another problem area related to costs is data storage. We find many organizations simply underestimate the amount of data they have. They may migrate all their data over to the cloud and then find much of it unusable. Some studies show that up to 46 % of data is unusable.* Another big problem is underestimating the time it takes to get the data ready for your projects. This is an area where Kythera shines. It is commonly cited that data preparation takes up to 80% of data analysts' and data scientists' time. We not only eliminate all the claims data prep, like cleaning and standardizing, we also enhance the data to produce instantly actionable data. We estimate that our data engineering and preparation save clients hundreds of thousands of dollars. Additionally, leveraging Databricks Unity Catalog capabilities allows the data to be distributed to users without ETL costs and duplicate storage. Users in different workspaces can share access to the same data, depending on privileges granted centrally in Unity Catalog.
Another contributor to rising costs is the inability to understand what contributes to those costs truly. 49% of cloud-based businesses struggle to control cloud costs (Source: Anodot), and in 54% of cases, cloud waste stems from a lack of visibility into cloud costs.
We help counter this by developing dashboards and tools that provide visibility into platform usage for our clients. As users of Databricks, we can easily see our own production costs by tagging each job and tracking each step of the process with that job. We can look at a job and determine usage by cluster, break down costs like compute versus storage, usage by date, and track our billable versus non-billable expenses. We have visibility into our costs over time, by job, so we can have a clearer view of what our costs will be prospectively.
Cost optimization will no doubt continue to be a high priority for cloud computing. This should not be in conflict with using big data to improve healthcare and develop new therapies. Wayfinder’s managed pipelines and ready-to-analyze data provide the tools so Life Science and Healthcare organizations can meet their business challenges and not worry about the intricacies of the Cloud.
Want to learn more about containing your cloud costs while getting more value from healthcare data? Get in touch at seek@kytheralabs.com or connect with me on LinkedIn: Matt Ryan, Co-Founder and Head of Engineering at Kythera.
