Caffe vs TensorFlow: What’s Best for Enterprise Machine Learning?

The variety of open-source machine learning frameworks suitable for enterprise projects has consolidated into a handful of candidates over the last ten years. Among them are Keras, TensorFlow, Caffe, PyTorch, Microsoft Cognitive Toolkit (CNTK) and Apache MXNet.

Due to their open-source nature, academic provenance, and varying levels of interoperability with each other, these are not discrete or 'standalone' products. Neither are they even distinct competitors in a traditional sense: Microsoft has brought backend support for Keras into its CNTK; Facebook has integrated a new iteration of Caffe into PyTorch; and Google's TensorFlow has adopted Keras as its default API—among many other unusual relationships and interdependencies in the current machine learning research culture.

Yet all of these projects still exist in their own right as prospective base structures for enterprise software production. By what criteria then can you decide on the core technologies for your own machine learning workflows?

Adoption Factors

When considering the adoption of a library or framework for internal development or ML consulting, the feature sets and capabilities of each must be weighed against their current level of industry adoption.

The backing of key AI players such as Google, Facebook and Amazon will likely mean wider-ranging documentation and industry implementation, as well as ongoing maintenance and updating of supporting resources.

We must also make a realistic estimation of the library's future prospects; in the current climate of consolidation, even a popular framework is quite capable of being abandoned, as Theano was in 2017.

Other factors to consider include:

• The neural network architectures supported
• Ease of deployment
• Specific application areas (such as computer vision or text analytics)
• The type of programming style (imperative or symbolic) supported
• The ability to optimize for low-level execution in hardware, either natively or through secondary methods such as NVIDIA's cuDNN libraries for CUDA, or both

In this article, we compare the merits of two of the most popular machine learning frameworks: Caffe vs TensorFlow. On the one hand, this is a comparison between the interests and goals of tech giants Facebook and Google; on the other, between the development advantages of generalization and the performance benefits of low-level, layer-specific ML engineering.

TensorFlow: Overview

TensorFlow is a symbolic programmatic library developed from earlier internal research at Google Brain. It has gained widespread industry and academic adoption in the machine/deep learning sector since being released to open source in 2015.

Developed in C++ and Python, TensorFlow uses dataflow graphs to map dependencies between individual operations. This representative approach allows high-volume implementations that can access diverse algorithms and models via a common layer. TensorFlow uses a readable syntax that is ably documented.

TensorFlow's strong application areas include Time Series Forecasting, Natural Language Processing (NLP), computer vision, text classification and speech recognition.

Besides boasting notable adherents such as Airbnb, China Mobile and Intel, TensorFlow is used to power or augment many of Google's core offerings, including Search, Translate and Gmail.

TensorFlow provides dedicated APIs for C and Python, and experimental APIs for Swift, Go, C++, Java and JavaScript. Third-party API resources are available for other languages including C#, Rust and Scala. However, as we'll see shortly, the latest version of TensorFlow has committed to Keras as its API of choice.

TensorFlow offers low-level hardware support via CUDA and cloud-based Tensor Processing Units (TPUs), dedicated application-specific integrated circuits (ASICs) designed to exploit the architecture.

Recent research indicates that TensorFlow is a little ahead of Facebook's PyTorch as the fastest-growing deep learning framework. It is used primarily in research and server production.

TensorFlow: Key facts

• Released: 2015
• Created by: Google
• Official website: tensorflow.org
• Ideal for: time series forecasting, NLP, computer vision, text classification, speech recognition
• In Iflexion’s AI stack: yes

Caffe: Overview

Caffe (Convolutional Architecture for Fast Feature Embedding) is an imperative, low-level library developed by a team of researchers at the University of California, Berkeley. It gained a widespread support in the academic community under the BSD license as well as a reputation for high-speed data processing.

Caffe development relies on compiling from C++ source code. It offers bindings for Python and MATLAB.

Caffe's core objectives are strongly oriented toward image segmentation and classification—a specialization that makes it a distinct offering in the run of open-source Python-centered deep learning libraries.

Caffe has an expressive architecture that is suitable for the development and modelling of Convolutional Neural Networks (CNNs), and which allows for transparent and easy switching between GPU and CPU operations.

It's fast: even the project's longstanding claim of being able to process 60 million images in one day (equating to 1 millisecond per image for inference, or 4 milliseconds per image for learning) have been exceeded with recent updates to the library.

One of Caffe's other biggest strengths comes in the form of access to a large number of pre-trained models from the Model Zoo deep net repository.

Although Caffe lacks TensorFlow's high-level APIs and generalization, the extra work needed to implement layers makes for a lean and well-optimized release code. This is fit for mobile and cloud environments without the overhead of complex framework architectures. This capability, among other qualities, made Caffe appealing to one of the biggest players in the mobile AI/ML development space: in 2017, Facebook Research released Caffe2 to open source.

This new iteration offers quantized computation, additional hardware support, new capacity for high-volume distributed training, and better support for Recurrent Neural Networks (RNNs) than the original. It also brings its own implementation of Model Zoo. In May 2018, Caffe2 was integrated into PyTorch.

Caffe is well-suited for edge deployment, less so for abstract research. Caffe2 is heavily used within the Facebook machine learning community.

Caffe: Key facts

• Released: 2017
• Created by: University of California, Berkley; Facebook
• Official website: caffe.berkeleyvision.org
• Ideal for: image segmentation and classification
• In Iflexion’s AI stack: yes

Installation

TensorFlow can easily be deployed on Python via Pip in local hardware, or in a virtual envelope such as Anaconda. The latter approach can virtualize access to specifically-versioned CUDA libraries or other package prerequisites, and allow projects with conflicting requirements to run in the same environment.

Caffe must be compiled from source code for deployment purposes, where hardware-native builds are required. Otherwise, pre-built Docker builds are available.

Iflexion recommends: TensorFlow's framework is certainly easier to experiment with. However, the potential performance benefits of developing for a narrow mission could be worth sacrificing the initial facility and flexibility of TensorFlow in favor of Cafe.

Development

TensorFlow has historically offered a wide range of flexible high-level APIs to speed up initial development. Although this has contributed in part to its popularity and uptake, the release of TensorFlow 2 removes a great deal of the framework's API redundancy and duplication, committing to a custom fork of the increasingly popular Keras framework. Keras offers an extensible, user-friendly and modular interface to TensorFlow's capabilities.

Caffe must be developed through mid or low-level APIs, which limits the configurability of the workflow model and restricts most of the development time to a C++ environment that discourages experimentation and requires greater initial architectural mapping.

Iflexion recommends: By now, Caffe's trade-off between flexibility and high-performing, optimized code is coming into focus. Since TensorFlow 2 addresses so many long-term criticisms regarding bloat and scope, one would need an extraordinarily lean brief in order to favor Caffe's more arcane approach to development.

Hardware support

TensorFlow currently only supports NVIDIA's brand of GPUs, via the company's cuDNN libraries. It also supports CPU deployment (at greatly reduced performance) as well as offering the hardware lock-in of dedicated TPU nodes. Periodically, compatibility issues arise between version releases of TensorFlow and different NVIDIA hardware units.

TensorFlow can run two-model instances on dual GPUs or split the training of a single model over multiple GPUs.

There are various projects which aim to add non-NVIDIA GPU support to TensorFlow, primarily via the OpenCL framework. The PlaidML project provides an additional, experimental gateway to GPU-driven TensorFlow operations without NVIDIA hardware.

Caffe is also optimized for CUDA. NVIDIA supports Caffe directly, claiming a 65% speed increase over the original on its Pascal GPUs, as well as the capability to generate single-node operations over multiple GPUs. However, Caffe's CLI-driven C++ workflow generally limits its ability to switch easily between GPU configurations, as TensorFlow can.

OpenCL maintains an ongoing project to improve AMD and non-NVIDIA GPU support, as well as extra compatibility with CPU-driven operations and Windows-based environments.

The GPUopen project, which aims to develop AMD-friendly heterogeneous computing environments for machine learning and other sectors, has ported Caffe to its C++ Heterogeneous-Compute Interface for Portability (HIP) framework. The implementation claims speed parity with native CUDA operations.

Facebook's Caffe2 can use GPUs more opportunistically, offering near-linear scaling for training on the ResNet-50 neural network via NVIDIA's NCCL multi-GPU communications library.

Iflexion recommends: Surprisingly, the one clear winner in the Caffe vs TensorFlow matchup is NVIDIA. Increased uptake of the Tesla P100 in data centers seems to further cement the company's pole position as the default technology platform for machine learning research, development and production.

Performance

This is a more complicated subject when comparing Caffe vs TensorFlow. Both have fragmented into multiple branches and missions, making direct speed comparisons increasingly difficult. Even in the rare instances where objective and/or recent benchmarking is available, the most meaningful use cases are hidden behind the proprietary internal politics of Google and Facebook. After all, they are the highest-volume creators and consumers of both frameworks, respectively.

In the case of TensorFlow, one has to evaluate both the data center/desktop/VM environment and the recently-launched Lite iteration, which is arguably more comparable to the mobile-centric mission of many aspects of Facebook's Caffe2.

Over the last five years, TensorFlow has been popularly estimated to lag behind even the original version of Caffe, since the latter has been perceived to benefit from its low-level streamlined approach to neural network design. However, this judgement doesn't account for the improved performance of TensorFlow 2, the impact of which is likely to become clearer over the next 18 months.

Neither does it account for the speed advantages that scale and uptake bring to the table: when a framework reaches a certain level of diffusion, ubiquity tends to compensate for architectural bloat via dedicated industry workarounds, which come into existence through an increased market demand. These include ASIC solutions (in this case, via Google's own TPUs), edge caching and other case-specific resources available through low-latency local data centers and bespoke on-premises hardware solutions.

Like-for-like speed testing between TensorFlow and Caffe is a problem at the moment, due to increased recent activity in their release cycles, the difference in scope between various versions of both frameworks, and the fact that Caffe is still primarily used for vision-related tasks—which is an important but not pivotal element in TensorFlow.

Iflexion recommends: If your project is centered around vision-based research, and particularly if it incorporates facial recognition, it's a reasonable bet that a PyTorch/Caffe2 basis will provide a more performant solution. Facebook's vanguard position in facial recognition technologies, facilitated by its unrivalled access to high-volume social network facial image data, has inclined it toward Caffe for a reason. You'll also benefit from an improved API and more configurability and flexibility than the original Caffe.

For all other purposes, TensorFlow's scalability, pace of innovation and industry support suggests a likely edge in production-level performance across all phases of development and deployment.

Conclusion

If Facebook had not intervened to give Caffe a new lease of life by enhancing its capabilities and integrating it into the ascendant PyTorch, TensorFlow might well now be the default recommendation for enterprise machine learning. This would ring true even for projects that deal primarily with image segmentation and classification—Caffe's primary areas of specialization.

As it stands, Caffe2's improvements and new implementation of the Model Zoo could be argued to have returned the contenders to parity. Now both libraries provide a development environment with a gentler learning curve and more resources with available models to exploit as templates and examples for new and original workflows.

Nonetheless, even the friendliest Caffe environment is oriented towards hands-on, low-level development as opposed to the easier abstractions of TensorFlow. Only the aims of your project and the people who will work on it can determine whether or not that's an advantage.