Big Data and Machine Learning: Two birds with one stone

Machine Learning (ML) has become a term of excitement and anticipation of what the future will look like and how our world might change with technologies like Artificial Intelligence, Augmented Reality, Virtual Reality, and all other technologies that start to all fall into place. All big tech companies have taken the road of developing their own infrastructure to support their needs and development. Same goes for Kaizen Gaming, a leading GameTechcompany, specializing in online gaming, where stakes are high and staying at the tip of the spear when it comes to innovation, is essential.

Looking back at when our ML journey started, we were following the community standards, working in local machines with limited resources and facing several data issues. Putting a model into production was a timely deployment process. That’s when we realized that we needed a well-performing model that can support automated decision-making or personalization application.

 

It’s one thing to believe you are a Machine Learning-ready company and another to actually be! And yes, the answer is in the data. In the data that your organization stores and can feed in, to the long intensive training processes, but also serve in real-time for production applications and correlate as different as it may be, to provide valuable business insights. Issues like historic data state, data scalability, accessibility, and elasticity come in the foreground and a platform that supports big data storage and highly concurrent processing becomes a requirement. The Azure Databricks platform was the enabler in our case through which we can incorporate the data lake storage, the cloud capabilities, and a number of features of their unified data platform. The data lake is used as the main source of development but also production pipelines, which allows access in very large data scales with features such as time travel that includes data versioning. The cloud resources enable seamless scaling of our applications with optimized cost, in contrast to our previous set-up which was based on dedicated VMs regardless of their usage. The unified platform allows for multiple teams to work together such as machine learning (ML), big data (BD), and business intelligence (BI) utilizing the same data sources and a single point of reference across the organization. In addition, the common environment and the multilanguage support enabled a close collaboration across teams, with data and ML engineers working closely in an efficient collaboration set-up. Versioning is a key concept in software, but it doesn’t differ when it comes to Machine Learning and data. The supported model registry allows for seamless model deployment in production with different versions and easy interchange between them. Finally, the platform enabled things like CI/CD pipelines and effortless Git integration, job scheduling and the concept of building DAG (data pipelines) as a native solution.

Overall, for a company to scale in Machine Learning and data application it is critical to adopt a cloud platform that enables storage and cloud service, but on the other hand, it is essential if it needs to differentiate from the market and truly take data driven decisions. It can be a very steep learning curve and a very costly one so investing in proven solutions goes a long way.