How we created an image processing app in 72 hours

In November 2018, Volvo ran a hackathon that gathered talents from all over the globe to generate innovative ideas focused on undercarriage inspection automation. Sigma Software team created the best solution from scratch after 72 hours of live coding! Here’s how we did it.

This was the first hackathon by Volvo Group Connected Solutions and Volvo Construction Equipment that brought together multidisciplinary teams to hunt for the next big innovation in image intelligence. Over 80 developers, designers, and also the data scientists have joined in one place in Lindholmen Science Park to create the solution that will help with image processing of machinery parts of Volvo excavators.

The challenge

Machinery moving parts after some working time need to be periodically inspected and replaced sooner or later in case of their wear-out. In order to make it work a lot of manual operations and measurements by engineers are required. Combine it with fixing the information in technical documentation and sharing it with everyone and you’ll get a quite complex and time-consuming procedure.

Thus, the aim of the hackathon was to implement the best solution to automate and streamline this process with new technologies using a smartphone camera.

What we came up with

This is where image intelligence and machine learning comes into play. By detecting and measuring the real-time status of all the integral parts it can predict whether it should be replaced and/or fixed or not. As well as what amount of working hours is left for the part. Let’s dive deeper.

We’ve developed a sophisticated solution for the diagnosis and maintenance of construction equipment. The mobile app we’ve developed could be used by any person who doesn't have a real technical background in order to make these measurements. It has to be done to high precision and at the construction site.

So simply thanks to the cameras over, for example, the track shoe, you can detect the height of certain parts of the excavator, so that algorithms compare it with a normal height to define the wear-out status. Thus engineers can predict how many working hours left for this part or other part to be replaced based on formula calculations behind. We manually segmented over 500 pictures of the parts in order to train a deep learning model and help with the prediction of the height and object recognition.

Solution features

The final features at the end were highly related. Not only because of being able to envision and to create the end-to-end process how the information from the measurement stage could be processed with a smartphone camera but by the end phase and end-to-end process be able to share this information automatically with the service dealer.

At the live demo, we have also shown that the accuracy and precision of their object recognition forecast that was achieved were really impressive. After 72 hours we gained up to 5 mm accuracy of object recognition while the standard measurement tools by Google or Apple provide approximately only 10 mm. A measuring error of 10 mm constitutes the difference between a new and a worn-out part.

Conclusion

Our team created a deep learning model using an object with known size as a reference to ensure the desired accuracy. In the short time available for development, we’ve managed to create a working app prototype with an appealing look and feel, intuitive and handy navigation, embedded logics of integration with existing Volvo construction services.

The solution that we’ve presented is a whole new way of working with the technical information documentation. It allows to avoid manual operations with information that could probably get lost or neglected for some reason because of human factors. It automates and significantly fastens the whole process. It also enables businesses to increase their uptime of different machinery, like excavators or trucks or different kinds of heavy machinery, in a way that decreases the downtime of operations. All you need is one smartphone camera. No deep insight or kind of measurement tool is needed in a way to make this happen.

The vision of the future

We also considered how the first development of the application could further be improved and extended. The usage of this technology is amazing in terms of predictive maintenance. We had inspected just a few parts of the excavator undercarriage. However, it could be extended based on their different rights and of parts within one brand.

In order to build the platform for predictive maintenance, we could help to gain a lot of funds from it because it decreases downtime. And for everyone right now it's quite a mission in order to decrease costs. Especially in these turbulent times. It's a challenge to be able to find a way to automate the process, make this operation be quite smooth enough, and avoid any kind of downtime.

The future is really recent for these technologies and also for predictive maintenance in the terms of the global automotive industry as well.