The insurer’s data science team was being weighed down by operational tasks and overseeing manual processes, preventing them from focusing on finding new ways to drive innovation within the organization. The team was encountering prolonged delays in monthly scoring analytics, and the looming prospect of cloud migration further complicated matters. Data availability issues were prominent, creating delays in model completion and increasing stress due to impending deadlines.
The process of model creation was hindered by a lack of clarity into the data features behind the custom logic, which had difficulty accommodating new features. One significant inefficiency was the amount of time research scientists spent on repetitive manual tasks like cohort creation, manual pipeline triggering, and general platform maintenance. Additionally, the Cloudera cluster presented its own set of challenges - from the need for manual coordination to schedule runs to consequent hardware and computational bottlenecks. The restrictive nature of certain tools, which were exclusive to the Cloudera ecosystem, further constrained their operations.