The NLP-Voice Research Lab at KBTG Labs, a research facility within KASIKORN Business—Technology Group (KBTG), reports on the THaLLE LLM model customized to enhance financial capabilities, as assessed by the Chartered Financial Analyst (CFA) exam.
The introductory CFA exam is a multiple-choice assessment requiring scores above 70%. The LLM artificial intelligence model that scored higher than 70% is a large-scale model like GPT-4o, achieving 88% on the Flare CFA test, or Gemini 1.5 Pro with a score of 78%. In contrast, smaller LLM models like Qwen2-7B only scored 68%.
The KBTG team crafted their own CFA exam dataset and fine-tuned the Llama3-7B-Instruct and Qwen2-7B-Instruct models. Results showed that the Qwen2-7B-Instruct model surpassed 71.7% accuracy during testing.
The report highlights key learnings from this fine-tuning process, emphasizing the use of the LoRA technique. Utilizing a LoRA rank exceeding 32 proved ineffective, while employing an all-linear target yielded superior outcomes compared to adjusting q_proj and v_proj specifically. Additionally, incorporating system prompts in the dataset for fine-tuning improved results. Lastly, grouping data by length had a negative impact on model performance.
It is noted that the CFA exam is merely an initial assessment, and utilizing LLM as a financial advisory tool requires further study.
Source: KBTG, ArXiv
TLDR: KBTG Labs fine-tunes LLM models to enhance financial abilities based on CFA exam results, uncovering key insights for optimizing model performance in financial applications.
Leave a Comment