In this talk weÌ¢âÂã¢ll show how weÌ¢âÂã¢re scaling our Machine Learning back-end at UnifyID, capable of servicing 1+ million users, requiring both CPU, but mostly GPU intensive Neural Network computations. WeÌ¢âÂã¢ll be demonstrating how we are running containers on EC2 GPU instances and how we have tackled some problems often seen when deploying Machine Learning clusters in production. Issues addressed include but not limited to: Horizontal scaling using GPU information from NVML. Leveraging multiple APIs for ML (TensorFlow, Caffe, Torch) by creating a uniform API for ML microservices. Running unreliable, academic quality ML code reliably in production. This is how the talk will progress; We will start by discussing the design of our auto-scalable ML back-end and the issues we tried to address. Then we will present our uniform, open-source API for Machine Learning microservices, which is available on DockerHub. Then weÌ¢âÂã¢ll discuss why GPU horizontal scaling is still unavailable for Kubernetes and Mesos. Finally we will present our in-house built auto-scaler which works based on GPU information from NVML.
