Deeplearning4j: Deep learning and ETL for the JVM

Eclipse Deeplearning4j is an open source, distributed, deep learning library for the JVM. Deeplearning4j is written in Java and is compatible with any JVM language, such as Scala, Clojure, or Kotlin. The underlying computations are written in C, C++, and Cuda. Keras will serve as the Python API. Integrated with Hadoop and Apache Spark, Deeplearning4j brings AI to business environments for use on distributed GPUs and CPUs.

Deeplearning4j is actually a stack of projects intended to support all the needs of a JVM-based deep learning application. Beyond Deeplearning4j itself (the high-level API), it includes ND4J (general-purpose linear algebra,), SameDiff (graph-based automatic differentiation), DataVec (ETL), Arbiter (hyperparameter search), and the C++ LibND4J (underpins all of the above). LibND4J in turns calls on standard libraries for CPU and GPU support, such as OpenBLAS, OneDNN (MKL-DNN), cuDNN, and cuBLAS.

The goal of Eclipse Deeplearning4j is to provide a core set of components for building applications that incorporate AI. AI products within an enterprise often have a wider scope than just machine learning. The overall goal of the distribution is to provide smart defaults for building deep learning applications.

Deeplearning4j competes, at some level, with every other deep learning framework. The most comparable project in scope is TensorFlow, which is the leading end-to-end deep learning framework for production. TensorFlow currently has interfaces for  Python, C++, and Java (experimental), and a separate implementation for JavaScript. TensorFlow uses two ways of training: graph-based and immediate mode (eager execution). Deeplearning4j currently only supports graph-based execution.

PyTorch, probably the leading deep learning framework for research, only supports immediate mode; it has interfaces for Python, C++, and Java. H2O Sparkling Water integrates the H2O open source, distributed in-memory machine learning platform with Spark. H2O has interfaces for Java and Scala, Python, R, and H2O Flow notebooks.

Commercial support for Deeplearning4j can be purchased from Konduit, which also supports many of the developers working on the project.

Source link