Case Studies :: Algorithm Optimization

Batch Job Scheduling

Optimization solutions are the lesser known field of Artificial Intelligence but really is one of the most powerful tools to come out of the field. It is not just an academic field but has very many real-world applications that can boost profitability for many companies. Yet, it continues to be an area that is overlooked by many people. One of our key optimization projects has been to develop an automated batch job scheduling system for one of our financial clients.

The Scenario

Each night the client would execute a large number of interlinked processing tasks that would run against data as it arrived from downstream processes. Over many years the team had created a timetable for the tasks to execute in a specific order based on their experience of monitoring the system. This all worked well as long as the data arrived from downstream systems in the expected order. On days when data arrives in different orders or even potentially segments of it don't arrive at all, then the system is incapable of coping with the scenario and the processes either complete later than required or don't complete at all. Through monitoring of the jobs each one takes an estimable amount of time to complete once the data has arrived. Also, the ordering of the jobs are linked. Process C will require the arrival of results from Process B and A in order to be triggered.

The network and hardware capacity of the machines executing the tasks was limited. Only a small set of tasks could run simultaneously.

The Solution

Each task is of a determinable length and there is a dependency between some of the tasks. Finding the optimal solution for the scheduling is an Np-Hard problem that is a perfect candidate for an AI Optimization Algorithm.

The development of a base schedule was implemented in C# and Python with the use of an Artificial Bee Colony Algorithm. This algorithm was able to find a solution that satisfied all constraints around the ordering of tasks by modelling each task as a node in a chain with a fixed duration. The chain links were joined based on the pre-known process order. The algorithm was constrained to adhere to the link ordering with the target of finding an optimal ordering of the links to achieve the fastest possible completion time.

The algorithm ran each time a new set of tasks was added to the schedule. Just by adding this optimization process to the scheduling of the overnight batches improved the consistency of the arrival time by shaving over 10% off the previous expected execution time. This meant that the team responsible for delivering the pricing data to the Front Office was able to meet their SLAs far more consistently and the Trading desks were able to get the data they needed on time far more frequently.

Algorithm Optimization is a yet insufficiently tapped process in many organizations and yet it can deliver real value to many. We can help to bring that value to your business and unlock potential earnings and savings to you and your customers.

Call or contact now for a free, no obligation Consultation

We have a team of skilled and experience Data Scientists and Machine Learning experts that are ready to assist you with the consulting and development of your ML projects. Whether it is a matter of using us to develop your MVPs or for the whole of your Data Science requirements then we can help.