Hanging a steadiness with ‘open’ at Snowflake

The relative deserves of “open” have been hotly debated in our {industry} for years. There’s a sense in some quarters that being open is helpful by default, however this view doesn’t all the time absolutely contemplate the targets being served. What issues most to the overwhelming majority of organizations are safety, efficiency, prices, simplicity, and innovation. Open ought to all the time be employed in service of these targets, not because the aim in itself.

After we develop merchandise at Snowflake, we consider the place open requirements, open codecs, and open supply can create the very best end result for our clients. We consider strongly within the constructive affect of open and we’re grateful for the open supply group’s efforts, which have propelled the large information revolution and rather more. However open will not be the reply in each occasion, and by sharing our pondering on this matter we hope to supply a helpful perspective to others creating revolutionary applied sciences.

[ Also on InfoWorld: What’s next for the cloud data warehouse ]

Open is usually understood to explain two broad components: open requirements and open supply. We’ll take a look at them every in additional element right here.

Open requirements

Open requirements embody file codecs, protocols, and programming fashions, which embody languages and APIs. Though open requirements usually present worth to customers and distributors alike, it’s essential to grasp the place they serve higher-level priorities and the place they don’t.

File codecs

We agree that open file codecs are an essential counter to the very actual drawback of vendor lock-in. The place we differ is within the assertion that these open codecs are the optimum solution to symbolize information throughout processing, and that direct file entry must be a key attribute of an information platform. 

At first look, the flexibility to instantly entry information in a typical, well-known format is interesting, nevertheless it turns into troublesome when the format must evolve. Take into account an enhancement that permits higher compression or higher processing. How will we coordinate throughout all doable customers and functions to grasp the brand new format?

Or contemplate a brand new safety functionality the place information entry is dependent upon a broader context. How will we roll out a brand new privateness functionality that causes by a broader semantic understanding of the info to keep away from re-identification of people? Is it essential to coordinate all doable customers and functions to undertake these modifications in lockstep? What occurs if one in every of these is missed?

Our lengthy expertise with these trade-offs provides us a robust conviction in regards to the superior worth of offering abstraction and indirection versus exposing uncooked information and file codecs. We strongly consider in API-driven entry to information, in higher-level constructs abstracting away bodily storage particulars. This isn’t about rejecting open; it’s about delivering higher worth for purchasers. We steadiness this with making it very simple to get information out and in in commonplace codecs.

A great illustration of the place abstracting away the main points of file codecs considerably helps finish customers is compression. A capability to transparently modify the underlying illustration of knowledge to attain higher compression interprets to storage financial savings, compute financial savings, and higher efficiency. Exposing the main points of file codecs makes it subsequent to unimaginable to roll out higher compression with out incurring lengthy migrations, breaking modifications, or added complexity for functions and builders. 

Related points come up once we take into consideration enhancements to safety, information governance, information integrity, privateness, and plenty of different areas. The historical past of database methods presents loads of examples, like iSAMS or CODASYL, exhibiting us that bodily entry to information results in an innovation lifeless finish. Extra lately, adopters of Hadoop discovered themselves managing pricey, complicated, and unsecured environments that didn’t ship the promised efficiency.

In a world with direct file entry, introducing new capabilities interprets into delays in realizing the advantages of these capabilities, complexity for utility builders, and, doubtlessly, governance breaches. That is one other level arguing for abstracting away the inner illustration of knowledge to supply extra worth to clients, whereas supporting ingestion and export of open file codecs. 

Open protocols and APIs

Information entry strategies are extra essential than file codecs. All of us agree that avoiding vendor lock-in is a key buyer precedence. However whereas some consider that open codecs are the answer, the heavy lifting in any migration is de facto about code and information entry, whether or not it’s protocols and connectivity drivers, question languages, or enterprise logic. Those that have gone by a system migration can seemingly attest that the subject of file codecs is a crimson herring.

For us, that is the place open issues most — it’s the place pricey lock-in could be prevented, information governance could be maximized, and larger innovation is feasible. Specializing in open protocols and APIs is vital to avoiding complexity for customers and enabling steady, clear innovation.

Open supply

The advantages cited for open supply embody a larger understanding of the know-how, elevated safety by transparency, decrease prices, and group growth. Open supply can ship towards a few of these targets, and does so primarily when know-how is put in on-premises, however the shift to managed companies vastly alters these dynamics.

On the subject of larger understanding of code, contemplate {that a} refined question processor is usually constructed and optimized over a number of years by dozens of Ph.D. graduates. Making the supply code obtainable is not going to magically permit its customers to grasp its interior workings, however there could also be larger worth in surfacing information, metadata, and metrics that present readability to clients.

One other side of this dialogue is the will to repeat and modify supply code. This could present worth and optionality to organizations that may make investments to construct these capabilities, however we’ve additionally seen it result in undesirable penalties, together with fragmented platforms, much less agility to implement modifications, and aggressive dysfunction. 

Elevated safety

This has historically been one of many essential arguments for open supply. When a company deploys software program inside its safety perimeter, supply code availability can certainly improve confidence about safety. However there’s a rising consciousness of the dangers in software program provide chains, and sophisticated know-how options usually combination a number of software program subsystems with out an understanding of the complete end-to-end affect on safety.

Fortunately there’s a higher mannequin, which is the deployment of know-how as managed cloud companies. Encapsulation of the interior workings of those companies permits for sooner evolution and speedy supply of innovation to clients. With extra focus, managed companies can take away the configuration burden and eradicate the hassle required for provisioning and tuning. 

Decrease value

Most organizations have acknowledged by now that not paying a software program license doesn’t essentially imply decrease prices. In addition to the price of upkeep and assist, it ignores the fee and complexity of deploying, updating, and break-fixing software program. Price must be measured when it comes to complete value and value/efficiency out of the field. Right here, too, managed companies are preferable, eradicating amongst different issues the necessity to handle variations, work round upkeep home windows, and fine-tune software program.


Probably the most highly effective elements of open supply is the notion of group, by which a gaggle of customers work collaboratively to enhance a know-how and assist each other. However collaboration doesn’t have to suggest supply code contribution. We consider group as customers serving to each other, sharing finest practices, and discussing future instructions for the know-how. 

Because the shift from on-premises to the cloud and managed companies continues, these subjects of management, safety, value, and group recur. What’s fascinating is that the unique targets of open supply are being met in these cloud environments with out essentially offering supply code for everybody—which is the place we began this dialogue. We should not lose sight of the specified outcomes by specializing in techniques which will now not be the very best path to these outcomes.

Open at Snowflake

At Snowflake, we take into consideration first rules, about desired outcomes, about supposed and unintended penalties, and, most significantly, about what’s finest for our clients. As such, we don’t consider open as a blanket, non-negotiable attribute of our platform, and we’re very intentional in selecting the place and the way we embrace it. 

Our priorities are clear: 

  1. Ship the best ranges of safety and governance; 
  2. Present industry-leading efficiency and value/efficiency by steady innovation; and 
  3. Set the best ranges of high quality, capabilities, and ease of use so our clients can give attention to deriving worth from information with out the necessity to handle infrastructure. 

We additionally wish to make sure that our clients proceed to make use of Snowflake as a result of they wish to and never as a result of they’re locked in. To the extent that open requirements, open codecs, and open supply assist us obtain these targets, we embrace them. However when open conflicts with these targets, our priorities dictate towards it.

Open requirements at Snowflake

With these priorities in thoughts, now we have absolutely embraced commonplace file codecs, commonplace protocols, commonplace languages, and commonplace APIs. We’re intentional about the place and the way we accomplish that, and now we have invested closely within the means to leverage the capabilities of our parallel processing engine in order that clients can get their information out of Snowflake rapidly ought to they want or select to. Nonetheless, abstracting away the main points of our low-level information illustration permits us to repeatedly enhance our compression and ship different optimizations in a method that’s clear to customers. 

We will additionally advance the controls for safety and information governance rapidly, with out the burden of managing direct (and brittle) entry to information. Equally, our transactional integrity advantages from our stage of abstraction and never exposing underlying information on to customers. 

We additionally embrace open protocols, languages, and APIs. This contains open requirements for information entry, conventional APIs akin to ODBC and JDBC, and likewise REST-based entry. Equally, supporting the ANSI SQL commonplace is vital to question compatibility whereas providing the ability of a declarative, higher-level mannequin. Different examples we embrace embody enterprise safety requirements akin to SAML, OAuth, and SCIM, and quite a few know-how certifications.

With correct abstractions and selling open the place it issues, open protocols permit us to maneuver sooner (as a result of we don’t have to reinvent them), permit our clients to re-use their information, and allow quick innovation on account of abstracting the “what” from the “how.” 

Open supply at Snowflake

We ship a small variety of elements that get deployed as software program options into our clients’ methods, akin to connectivity drivers like JDBC or Python connectors or our Kafka connector. For all of those we offer the supply code. Our aim is to allow most safety for our clients, and we accomplish that by delivering our core platform as a managed service, and we improve the peace of thoughts for installable software program by open supply.

Nonetheless, a misguided utility of open can create pricey complexity as a substitute of low-cost ease of use. Providing steady, commonplace APIs whereas not opening up our internals permits us to rapidly iterate, innovate, and ship worth to clients. However clients can’t create—intentionally or unintentionally—dependencies on inner implementation particulars, as a result of we encapsulate them behind APIs that comply with stable software program engineering practices. That may be a main profit for each side, and it’s key to sustaining our weekly cadence of releases, to steady innovation, and to useful resource effectivity. Prospects who’ve migrated to Snowflake inform us persistently that they recognize these selections.

The interface to our absolutely managed service, run in its personal safety perimeter, is the contract between us and our clients. We will do that as a result of we perceive each part and dedicate a large amount of sources to safety. Snowflake has been evaluated by safety groups throughout the gamut of firm profiles and industries, together with extremely regulated industries akin to healthcare and monetary companies. The system will not be solely safe, however the separation of the safety perimeter by the clear abstraction of a managed service simplifies the job of securing information and information methods for purchasers.

On a last be aware, we love our person teams, our buyer councils, and our person conferences. We absolutely embrace the worth of a vibrant group, open communications, open boards, and open discussions. Open supply is an orthogonal idea, from which we don’t shrink back. For instance, we collaborated on open sourcing FoundationDB, and made important contributions to evolving FoundationDB additional. 

Nonetheless, we don’t extrapolate from this to say there may be an inherent benefit to open supply software program. We might equally have used a distinct operational retailer and a distinct mannequin of constructing it to go well with our necessities if wanted. The FoundationDB instance illustrates our key thesis: Open is a superb assortment of initiatives and processes, nevertheless it’s one in every of many instruments. It’s not the hammer for all nails and is the only option solely in some conditions. 

Source link