Ext: UNSD Brown Bag: Free to play: Developing your own open source RAG and fine-tuning LLM applica..

Extended UNSD Brown Bag: Free to play: Developing your own open-source RAG and fine-tuning LLM application, the UN Trade and Development's experience - 6 June 2024
The United Nations Statistics Division (UNSD) hosted an Extended UNSD Brown Bag webinar titled "Free to play: Developing your own open-source RAG and fine-tuning LLM application, the UN Trade and Development's experience." The event, which attracted about 38 participants, featured Daniel Hopp, a Data Scientist with the UN Global Crisis Response Group (GCRG) at UN Trade and Development (UNCTAD).
Daniel, with a background in private sector analytics and data science, presented on the development of a free and open-source application that leverages Retrieval Augmented Generation (RAG) and fine-tuning techniques to make Large Language Models (LLMs) more useful for specific organizational domains. This is particularly important because while LLMs have gained immense popularity, their effectiveness can be limited when applied to specialized domains or recent developments not covered in their training data.
The presentation provided a primer on LLMs, RAG, and fine-tuning, and walked through the use-cases and implementation process of UNCTAD's application. The application consists of four constituent libraries available on GitHub: document processing, running the LLM, the front end, and model fine-tuning.
During the Q&A session, participants asked several insightful questions. The summary of the Q&A below only covers some questions:
• A question was raised about handling a specialized audience, corpus, and methodologies. Daniel suggested combining fine-tuning and RAG, especially when an organization has a lot of question-answer pairs, which is perfect for fine-tuning. The fine-tuned model can handle methodological questions, while RAG can handle new, incoming queries.
• Another question concerned conservativeness and relative importance in LLM responses. Daniel explained that conservativeness is controlled by the 'temperature' hyperparameter. A lower temperature (like 0) makes the model more conservative and factual, while higher temperatures introduce more randomness and creativity. For UN applications focusing on impartial, factual information, they use a temperature of 0.
• Regarding relative importance of sources, Daniel emphasized that this isn't just an LLM problem but a broader software design issue. He suggested querying preferred databases first (like expert speeches) before falling back to less authoritative sources (like blog posts), and instructing the LLM to emphasize the more authoritative sources in its response.
• Concerning the reliability of answers provided by LLMs, Daniel stressed the critical need to fact-check all LLM outputs. Even with techniques like RAG and fine-tuning, which improve accuracy, LLMs can still "hallucinate" or generate plausible but incorrect information. He noted that RAG can help mitigate this by providing relevant source documents. He emphasized that in contexts like the United Nations, where accuracy is paramount, users must always verify LLM-generated claims and citations.
This Extended UNSD Brown Bag was open to all UN System colleagues. The recording, presentation slides, and relevant links are available in the UN internal Training resources group on the Global Network of Data Officers and Statisticians. If you are not yet a member of the Global Network, then please join with your UN email address at www.yammer.com... .