FlorID – a nationwide identification service for plants from photos and habitat information

Abstract

1. Extensive monitoring of biodiversity is indispensable to assess the extent of the ongoing crisis, to develop appropriate mitigation strategies, and to evaluate their success. Citizen science observations can greatly improve such monitoring, but they often contain misidentifications. In collaboration with the National Data and Information Center on the Swiss Flora, we combined deep-learning approaches for image classification and ecological modeling to create an automated plant identification tool providing feedback to citizen scientists and quality-checking submitted observations.
2. We trained an ensemble of data-efficient vision transformers on almost 1.5 Mio photos of 3076 plant species (85% of the Swiss flora), originating from citizen science, regional atlases, expert archives, and a two-season field sampling campaign targeting difficult and underrepresented species. For optimization, we employed a marginalization loss function that leverages taxonomic relatedness between species. We additionally modeled observation probabilities from environmental conditions and season, using a deep neural network-based approach tailored to presence-only data. This ecological classifier was trained on 6.7 Mio observations and 18 25-m-resolution predictors related to climate, vegetation structure, soil conditions, terrain, and land cover. FlorID, the resulting identification service, will be available as a free-to-use application programming interface and as a support option in FlorApp, a leading mobile phone app for collecting Swiss citizen science data.
3. Evaluated on a manually selected, taxonomically stratified test set, the image classifier showed a top-1 accuracy of >80% and a top-5 accuracy of >95%. These scores improved distinctly when image and ecological predictions were combined and when multiple images per observation were considered. We found highest performance when images depicted vegetative and reproductive parts or whole plants, followed by flowers, fruits, and vegetative parts. However, considerable taxon-dependent differences remained.
4. Given its unique wealth of floristic and environmental data, Switzerland is in an excellent position to implement automated, data-hungry, and integrative solutions to improve plant biodiversity monitoring. In this context, FlorID represents a large step forward and a blueprint for similar efforts targeting different taxa or regions. Moreover, it is a basis for further developments such as automated community inventories or habitat classifications.

Riccardo de Lutio

Senior Research Scientist

I am a Senior Research Scientist at NVIDIA working on 3D vision, neural reconstruction, simulation, and world models.