Dynamic, Event-Driven Machine Learning Pipelines with Argo Workflows

Most people would say Argo is a YAML based orchestration framework. But really, it's a highly expressive YAML based API. Recently at Arthur, we've harnessed Argo to build highly dynamic, event-driven workflows using the Golang Argo Client and the Kubernetes API. Using this setup, we can create 1000's of variations of the same workflow in seconds to configure multiple machine learning pipelines unique to each of our customers. These pipelines standup dynamic infrastructure, build and deploy containers, and setup autoscaling resources to implement flexible, scalable, and cost-effective big data processing.In this talk we will describe the journey that lead us to Argo and some of the road bumps we had with competitors along the way. In addition, we will describe how we use GO to build and run event-driven, dynamic workflows that power our machine learning as a service capabilities. By the end of this talk listeners will learn how Argo not only stands above the competition for static, scheduled workloads, but continues to power the most cutting edge, event-driven machine learning capabilities.