Ahana Cloud for Presto assessment: Quick SQL queries in opposition to knowledge lakes

Hope springs everlasting within the database enterprise. Whereas we’re nonetheless listening to about knowledge warehouses (quick evaluation databases, sometimes that includes in-memory columnar storage) and instruments that enhance the ETL step (extract, rework, and cargo), we’re additionally listening to about enhancements in knowledge lakes (which retailer knowledge in its native format) and knowledge federation (on-demand knowledge integration of heterogeneous knowledge shops).

Presto retains developing as a quick technique to carry out SQL queries on huge knowledge that resides in knowledge lake information. Presto is an open supply distributed SQL question engine for operating interactive analytic queries in opposition to knowledge sources of all sizes. Presto permits querying knowledge the place it lives, together with Hive, Cassandra, relational databases, and proprietary knowledge shops. A single Presto question can mix knowledge from a number of sources. Fb makes use of Presto for interactive queries in opposition to a number of inside knowledge shops, together with their 300PB knowledge warehouse.

The Presto Basis is the group that oversees the event of the Presto open supply undertaking. Fb, Uber, Twitter, and Alibaba based the Presto Basis. Extra members now embody Alluxio, Ahana, Upsolver, and Intel.

Ahana Cloud for Presto, the topic of this assessment, is a managed service that simplifies Presto for the cloud. As we’ll see, Ahana Cloud for Presto runs on Amazon, has a reasonably easy consumer interface, and has end-to-end cluster lifecycle administration. It runs in Kubernetes and is very scalable. It has a built-in catalog and straightforward integration with knowledge sources, catalogs, and dashboarding instruments.

Opponents to Ahana Cloud for Presto embody Databricks Delta Lake, Qubole, and BlazingSQL. I’ll draw comparisons on the finish of the article.

Presto and Ahana structure

Presto is not a general-purpose relational database. Relatively, it’s a instrument designed to effectively question huge quantities of information utilizing distributed SQL queries. Whereas it could exchange instruments that question HDFS utilizing pipelines of MapReduce jobs corresponding to Hive or Pig, Presto has been prolonged to function over completely different sorts of information sources together with conventional relational databases and different knowledge sources corresponding to Cassandra.

Briefly, Presto just isn’t designed for on-line transaction processing (OLTP), however for on-line analytical processing (OLAP) together with knowledge evaluation, aggregating massive quantities of information, and producing stories. It may well question all kinds of information sources, from information to databases, and return outcomes to plenty of BI and evaluation environments.

Presto is an open source project that operated under the auspices of Facebook. It was invented at Facebook and the project continues to be developed by both Facebook internal developers and a number of third-party developers under the supervision of the Presto Foundation.

Presto’s scalable, clustered architecture uses a coordinator for SQL parsing, planning, and scheduling, and a number of worker nodes for query execution. Result sets from the workers flow back to the client through the coordinator.

Ahana Cloud packages managed Presto, a Hive metadata catalog, a data lake hosted on Amazon S3, cluster management, and access to Amazon databases into what is effectively a cloud data warehouse in an open, disaggregated stack, as shown in the architecture diagram below. The Presto Hive connector manages access to ORC, Parquet, CSV, and other data files.

ahana for presto 03 Ahana

 

As implemented on AWS, Ahana Cloud for Presto places the SaaS console outside of the customer’s VPC and the Presto clusters and Hive metastore inside the customer’s VPC. Amazon S3 buckets serve as storage for data files.

The Ahana control plane takes care of cluster orchestration, logging, security and access control, billing, and support. The Presto clusters and the storage live inside the customer’s VPC.

Using Ahana Cloud for Presto

Ahana provided me with a hands-on lab that allowed me to create a cluster, connect it to sources in Amazon S3 and Amazon RDS MySQL, and exercise Presto using SQL from Apache Superset. Superset is a modern data exploration and visualization platform. I didn’t really exercise the visualization portion of Superset, as the point of the exercise was to look at SQL performance using Presto.

ahana for presto 05 IDG

When you create a Presto cluster in Ahana, you choose your instance types for the coordinator, metastore, and workers, and the initial number of workers. You can scale the number of workers up or down later. Because the datasets I was using were relatively small (only millions of rows), I didn’t bother enabling I/O caching, which is a new feature of Ahana Cloud.

ahana for presto 06 IDG

The Clusters pane of the Ahana interface shows your active, pending, and inactive clusters. The PrestoDB Console shows the status of the running cluster.

I found the process of adding data sources a bit annoying because it required me to edit URI strings and JSON configuration strings. It would have been easier if the strings had been assembled from pieces in separate text boxes, especially if the text boxes were populated automatically.

ahana for presto 07 IDG

Creating catalogs and converting from CSV to ORC format took just under a minute, for 26.2 million rows of movie ratings. Querying an ORC file is much faster than querying a CSV file. For example, counting the ORC file takes 2.5 seconds, while counting the CSV file takes 48.6 seconds.

ahana for presto 08 IDG

This federated query joins movie ratings in ORC format with movie data in a MySQL database table to create a list of ratings, counts, and popularity broken down into deciles. It took 10 seconds.

ahana for presto 09 IDG

This query computes the most popular movies in the federated database with a description that mentions weapons, and also reports the movies’ budgets. The query took 7.5 seconds.

How to integrate Ahana Presto with machine learning and deep learning

How do people integrate Ahana Presto with machine learning and deep learning? Typically, rather than using Superset as a client, they use a notebook, either Jupyter or Zeppelin. To perform the SQL query, they use a JDBC link to the Ahana Presto query engine. Then the output from the SQL query populates the appropriate structure or data frame for use in machine learning, depending on the framework used.

New features of Ahana Cloud for Presto

The version of Ahana Cloud I tested incorporated the improvements announced on March 24, 2021. These included performance improvements such as data lake I/O caching and tuned query optimization, and ease of use improvements such as automated and versioned upgrades of Ahana Compute Plane.

I didn’t use all of them myself. For example, I didn’t enable data lake I/O caching because the data lake table I was using was too small, and I didn’t spend long enough with Ahana to see a version upgrade.

Ahana Cloud for Presto vs. competitors

Overall, Ahana Cloud for Presto is a good way to turn a data lake on Amazon S3 into what is effectively a data warehouse, without moving any data. Using Ahana Cloud avoids most of the work required to set up and tune Presto and Apache Superset. SQL queries run quickly on Ahana Cloud for Presto, even when they are joining multiple heterogeneous data sources.

Databricks Delta Lake uses different technologies to accomplish some of the same things as Ahana Cloud for Presto. All the files in Databricks Delta Lake are in Apache Parquet format, and Delta Lake uses Apache Spark for SQL queries. Like Ahana Cloud for Presto, Databricks Delta Lake can speed up SQL queries with an integrated cache. Delta Lake can’t perform federated queries, however.

Qubole, a cloud-native data platform for analytics and machine learning, helps you to ingest datasets from a data lake, build schemas with Hive, query the data with Hive, Presto, Quantum, and/or Spark, and continue to your data engineering and data science. You can use Zeppelin or Jupyter notebooks, and Airflow workflows. In addition, Qubole helps you manage your cloud spending in a platform-independent way. Unlike Ahana, Qubole can run on AWS, Microsoft Azure, Google Cloud Platform, and Oracle Cloud.

BlazingSQL is an even faster way of running SQL queries, using Nvidia GPUs and running SQL on data loaded into GPU memory. BlazingSQL lets you ETL raw data directly into GPU memory as GPU DataFrames. Once you have GPU DataFrames in GPU memory, you can use RAPIDS cuML for machine learning, or convert the DataFrames to DLPack or NVTabular for in-GPU deep learning with PyTorch or TensorFlow.

Ahana Cloud for Presto is a worthwhile alternative to its competitors, and is easier to set up and maintain than an open source Presto deployment. It’s certainly worth the effort of a free trial.

Cost: $0.25/Ahana Cloud Credit (ACC) hour. See pricing calculator and table of instance charges. Example: Presto Cluster of 10 x r5.xlarge running every workday costs $256/month.

Platform: Runs on Amazon Elastic Kubernetes Service.

Copyright © 2021 IDG Communications, Inc.

Source link