When in 2021 a bunch of intrepid MWLR researchers digitised five decades of Soil News, the newsletter of the New Zealand Society of Soil Science, it doubled the amount of digital land and soils documents available through the MWLR Digital Library. While some of us enjoy digging for treasure and diving into New Zealand soils history, the vast amount of information now available in digital form spans several generations of researchers and can be overwhelming to view and search.
In 2024 an opportunity arose to develop a web-based AI tool, in collaboration with Microsoft and Callaghan Innovation. These partners set about exploring ways to tailor generative AI so that it can draw on trusted local knowledge bases. Known as retrieval-augmented generation (RAG), this approach allows AI to search authoritative sources in order to provide answers to prompted questions. The goal was to test whether modern AI techniques could unlock hidden insights from our massively expanded online soil science library.
For the pilot project, named SoilInsights, around a thousand PDF files were loaded as training material for the tool. A prompt enables interaction with the tool. It very quickly became clear that the precision and speed of information retrieval was encouraging – and at times exceeded expectations. The tool retrieves information on names, dates, events, etc. with ease. It also deals well with more complex questions, such as the soil information available for a particular region. At this stage our pilot uses historical information available up to 1995, but we are planning to connect the system to updated soil information, enabling it to respond with a broader and more current range of insights.
We are currently evaluating the performance of the tool in terms of relevance, accuracy, clarity, grounding in trusted sources, and completeness of answers. This process allows us to adjust the settings the tool uses in shaping its response to make sure it is fit for purpose.
We see real promise in these developments, because they could unlock new ways to increase the return on investment for NZ Inc from our growing collection of digital soil information and resources, cumulatively built up through current and past generations investment in publicly funded science . In the future, SoilInsights could do more than just answer questions from a single online library: it could weave together soil information from across the new Bioeconomy Science Institute, providing even greater insights for our wide range of end-users, researchers and agencies. Leveraging the pattern-recognition power of generative AI, it could help visitors navigate this breadth of science with greater ease, uncovering information that spans disciplines and generations of research.
It is important to note that this initiative not only increases the reach and utility of our digital resources, but also fosters intergenerational knowledge retrieval. SoilInsights helps preserve and share the deep expertise of long-serving soil researchers, ensuring that pre-database information, which is often at risk of being lost, can be discovered and put to use. This project builds essential GenAI capability at the Bioeconomy Science Institute, demonstrating how advanced AI methods can add new value to legacy research holdings in New Zealand’s unique environmental context.
Key contact
