Innovations in technology platforms, like Databricks and Kythera Labs data technology platform Wayfinder, are making finding answers faster and more cost-effective in Real-World Data (RWD). Features like serverless compute make it possible to manage time and resources with minimal management and less cost. Let’s look more closely at the benefits of SQL Serverless Compute and how Kythera Labs leverages this feature in Wayfinder.
Serverless compute is a cloud-based, on-demand architecture without the burdens of managing machine resources and servers. This means users can avoid the cost and tedious time spent managing hardware, maintaining servers, and waiting for clusters to spin up. Instead, analysts can easily select the serverless option when logging into Wayfinder and spend more time writing queries, building pipelines, and getting work done. Wayfinder, built on Databricks, uses serverless compute to automatically scale and optimize your workload. If more compute power is needed, additional servers are added to your cluster. This efficiently leverages resources and saves time and money because you don’t have to worry about cloud waste like capacity planning and overprovisioning.
Cloud waste results in paying for more resources than you need or using resources that are not optimized. A 2022 survey of over 750 executives found that just over 30% of cloud spending (or roughly $175 billion of total cloud spending) was in excess of their actual business needs.
Another benefit of serverless is usage billing. Serverless warehouses are generally less costly compared to traditional clusters because cloud resources are utilized less frequently, for example. Users only pay for the capacity used once they start running queries or reports, without paying for start-up, fade-out or idle time. According to Matt, “Traditionally, when a cluster starts the cloud compute resources have to be found, initialized, and clustered before the workloads can run. With serverless, which is essentially a pool of servers on ‘warm standby,’ they are like a blank slate and are ready to go when you are ready to begin executing code. In essence, they are always on standby. The servers are already provisioned, performance-tuned, scaled, optimized, and ready for your workload. Using the serverless option in Wayfinder, users can run queries, write code, and build novel data sets within seconds, essentially outsourcing all administration so users can get to work quickly. We focus on the backend…the data, the processing power, security, and users focus on getting their work done quickly and cost-effectively.”
As an analogy, think about television streaming services. There is a new movie everyone is talking about, and you’d like to watch it. You already pay for four streaming services, and of course, this particular movie is on a streaming service you do not subscribe to. You could subscribe to yet another streaming service and pay for more than what you actually need because, after all, it's just one movie you are interested in seeing. Now, imagine being able to purchase only that movie and nothing more. Once you are done with the movie, you don’t have to continue paying for all the shows you are not interested in seeing. In essence, like serverless, you only paid for what you needed, in this case, one movie. And like serverless, if there were two movies you wanted to see, you could automatically only pay for those two movies and nothing more.
“The popularity of a serverless strategy is growing because it provides the opportunity for faster time to market by dynamically and automatically allocating compute and memory based on user requests. It also provides cost savings through hands-off infrastructure management, which enables organizations to redirect IT budget and development resources from operations to innovation.” Determining the Total Cost of Ownership: Comparing Serverless and Server-based Technologies
Serverless also provides a secure environment and has enhancements for working with regulated HIPAA-protected data. Workloads are executed with multiple layers of security and with encrypted storage that cannot be reassigned after use. They are securely wiped as soon as the warehouse shuts down.
Serverless computing benefits not only analysts but also developers. Developers can maximize their time developing technologies and building pipelines rather than spending time on maintenance. “I use serverless every day to research, test, and validate. Our healthcare claims data is over 44 billion rows with sophisticated joins already complete to make the data more useful for our clients. It’s great for working with big data because it has an easy-to-use interface and is fast and totally integrated with our Data Lake and big data pipelines.” (Matt Ryan)
Business analysts can get right to work writing SQL queries and reports rather than waiting for clusters to spin up. In fact, it’s a great way to get fast answers to questions hidden in big data, and once you get your answer, you can simply turn serverless off. And you can decide how many minutes you’d like the servers to run before it automatically turns off.
According to Markwood, he uses serverless compute daily. “Serverless has really transformed my job by making me so much more productive. Using traditional clusters, I may wait 10-15 minutes for them to spin up. During that time, I may get distracted from my work while I wait. It's those little gaps of time when I may run to get a cup of coffee, get into a conversation or find another job to do. With serverless, I can get into the flow state in a matter of seconds….you just get going. And it's highly efficient. I use Wayfinder for writing queries for our clients and to support sales. For example, I may be building a cohort of patients who are prescribed a particular NDC (National Drug Code) for a particular diagnosis. Many of my queries require going through a massive amount of data; with serverless, I just write my query and get to work. Queries that take an hour or more to run on traditional clusters can be completed in a fraction of the time to complete. For me, speed is the biggest selling point, but another great benefit is how simple it is to hook your BI tool into your Data Lake.”
As more Life Sciences and Healthcare organizations process more data to answer complicated business questions, the benefits of Serverless are front and center. Hear Matt Ryan discuss how Kythera innovates with data and architecture during the Databricks Next-Generation Products Built on Databricks Lakehouse webinar here. If you’d like to talk about how we benefit from Serverless or how Wayfinder makes RWD more accessible, efficient and cost-effective, get in touch.
