Using Deep Sea Animal Recognition to Measure the Impact of Offshore Wind Energy

A project team was formed, which included Cal Poly’s Digital Transformation Hub (DxHub), five computer science students, four marine biology students, two faculty and one staff member at Cal Poly. Over a 10-month period, the team, with advisement from MBARI professional video annotators and engineers, created an annotation tool that would feed streaming video into a web browser to be annotated by scientists.  The application allows scientists to easily drag multiple bounding boxes onto a screen and then select which species is represented on the screen. These images are then used as training data for a Machine Learning (ML) model that will attempt to detect and classify which species are present in future videos. To further amplify images created by this process, the team added in the ability to track an organism for the duration it is on the screen from a single annotation. This can create an extra 30-100 images per annotation by a scientist, creating a large number of images to train the ML model in a short amount of time while saving valuable time for the scientists in the annotation process.

Throughout the course of this project, annotators have been able to create 20,000 labeled images in total with many species having over 5,000 labeled images individually.  With these labeled images, optimizing the ML model and using humans to further tune the model through Human-in-the-loop training, the project team anticipates being able to create a model that will be able to identify species with 90% accuracy.