https://www.comp.nus.edu.sg/~kanmy/wing/project/Research ProjectsHugo Blox Builder (https://hugoblox.com)en-usMon, 01 Dec 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/media/icon_hub1b49484bccf4f586453e7d2a784a896_225489_512x512_fill_lanczos_center_3.pnghttps://www.comp.nus.edu.sg/~kanmy/wing/project/https://www.comp.nus.edu.sg/~kanmy/wing/project/selfadaptation/Mon, 01 Dec 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/selfadaptation/<p><strong>🎯 Overview</strong>: We study test-time scaling (TTS) methods for self-improving and self-adapting agents, advancing a new paradigm of artificial intelligence in which autonomous systems do not merely act, but evolve reliably through learning from experience, refining behavior at test time, and autonomously modifying their own learning mechanisms.</p> <p><strong>🧠 Abstract</strong>: Although modern foundation models (FMs) like ChatGPT are extraordinarily capable, they remain largely fragile: small variations in input (aka &lsquo;&lsquo;prompts&rsquo;&rsquo;), such as subtle phrasing changes or unintended noise, can lead to contradictory or ungrounded reasoning. This limits their broader deployment in high-stakes domains like healthcare, law, and science, where precision, reliability, and interpretability are essential.</p> <p>In this project, we aim to build the next generation of FM systems that can continually assess and correct their behavior at test time, making them more <strong>trustworthy</strong> and <strong>capable</strong>. Our studies span three main aspects:</p> <ul> <li> <p><strong>(1) Understand Fragility of FMs</strong>: How can we interpretably understand model behavior through the lens of prompts to improve FMs reliably? (<a href="https://aclanthology.org/2025.acl-long.292/" target="_blank" rel="noopener">NLPromptEval @ ACL'25</a>, <a href="https://aclanthology.org/2025.naacl-long.15/" target="_blank" rel="noopener">FormatBiasEval @ NAACL'25</a>)</p> </li> <li> <p><strong>(2) Study TTS Methods</strong>: With the above understanding, how can we develop TTS to enhance model effectively and reliably? (<a href="https://aclanthology.org/2024.emnlp-main.1135/" target="_blank" rel="noopener">Multi-Expert Prompting @ EMNLP'24</a>, <a href="https://aclanthology.org/2024.acl-long.395/" target="_blank" rel="noopener">adv-ICL @ ACL'24</a>, <a href="https://aclanthology.org/2025.coling-main.172/" target="_blank" rel="noopener">Chain-of-Opinion @ COLING'25</a>)</p> </li> <li> <p><strong>(3) Self-Improving Agents</strong>: How can we build powerful TTS-powered agentic systems that critique and refine their own behavior, ultimately enabling autonomous self-improvement? (<a href="https://aclanthology.org/2025.findings-acl.176/" target="_blank" rel="noopener">LongGuide (Self-Adapting Agent) @ ACL'25</a>, <a href="https://g-vista.github.io/" target="_blank" rel="noopener">VISTA (Self-Improving Agent)</a>)</p> </li> </ul> <p>Our findings so far have offered a foundation for building foundation model agents that reliably improve through interaction, feedback, and structure-aware test-time methods—without reliance on additional gradient-based training.</p> <p><strong>🔥 News</strong>: We are expanding our studies towards multiple directions. Please review our work above. If you have a strong interest in the topics listed and a solid background in mathematics and programming, and optionally in research, come to join us!</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/discourse/Tue, 04 Nov 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/discourse/<p>Large language models (LLMs) can now generate coherent text, but can they truly understand language? In this project, we explore discourse understanding, a foundational objective in natural language processing (NLP). In the first part, we refine the notion of understanding as faithful behavior under a series of Socratic-style questions (appeared at ACL 2024). In the second part, we take a causal approach to mechanistically interpret the key components of transformer models in discourse processing (appeared at EMNLP 2025).</p> <p>Yisong, a PhD student at WING, began his research journey by tackling the fundamental question: Do language models truly understand text? Over time, he refined this broad inquiry into the specific challenge of discourse understanding. In his recent work, advised by Prof. Min, they introduced a more precise framework for measuring discourse comprehension from three discourse-specific perspectives, while also contributing insights into broader LLM challenges such as logical consistency. Currently, Yisong and Min are delving deeper into the mechanisms behind how models process and understand discourse, and they discovered that sparse computational graphs, termed as &ldquo;discursive circuits&rdquo;, control how models process discourse relations.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/summarizing-scientific-content-with-social-media-language/Fri, 01 Aug 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/summarizing-scientific-content-with-social-media-language/<p>As social media dominates public information consumption, effectively communicating scientific knowledge through these platforms is crucial. Despite advances in document summarization, complex scientific jargon remains a barrier for general audiences, limiting access to important scientific information. This project develops methodologies to simplify scientific content while preserving accuracy, translating it into language that engages social media users. Our goal is to increase accessibility of scientific knowledge and enhance its influence on informed societal decision-making.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/ssm-interpretability/Wed, 01 Jan 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/ssm-interpretability/<p>State-space models (SSMs) have emerged as promising alternatives to Transformers, particularly for long-context tasks, due to their efficiency in modeling long-range dependencies through structured state transitions. While prior work has focused on interpreting final-layer outputs, this study investigates the feature flow across layers in SSMs. By comparing this behavior to Transformers, we show fundamental differences in how contextual information is encoded and propagated. Our analysis reveals trade-offs in efficiency and expressivity, offering a deeper understanding of learning dynamics in both architectures. This work not only advances our understanding of SSMs but also lays the foundation for designing hybrid models that combine the strengths of both paradigms.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/chatcrs/Wed, 01 Jan 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/chatcrs/<p>This project is inspired by the idea of applying LLMs for zero-shot conversational recommendation. Advanced LLMs (e.g., ChatGPT) are limited in domain-specific CRS tasks for 1. generating grounded responses with recommendation- oriented knowledge, or 2. proactively leading the conversations through different dialogue goals. In this work, we first analyze those limitations through a comprehensive evaluation, showing the necessity of external knowledge and goal guidance which contribute sig- nificantly to the recommendation accuracy and language quality. In light of this finding, we propose a novel ChatCRS framework to decompose the complex CRS task into several sub-tasks through the implementation of 1. a knowledge retrieval agent using a tool-augmented approach to reason over external knowledge bases and 2. a goal-planning agent for dialogue goal prediction. Incorporating those external inputs, LLMs proactively plan interactions and generate outputs with rich information. Experiments on two multi-goal CRS datasets reveal that ChatCRS sets new state-of-the-art benchmarks, improving language quality of informativeness by 17% and proactivity by 27%, with tenfold recommendation accuracy enhancement</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/knowledge-consistency/Wed, 01 Jan 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/knowledge-consistency/<p>Few works study the variation and cross-lingual consistency of factual knowledge embedded in multilingual models. However, cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge. We apply interpretability approaches to analyze the behavior of a model in cross-lingual contexts, discovering that multilingual models show different levels of consistency, subject to language families or linguistic factors. In addition, we identify a bottleneck in cross-lingual consistency on a particular layer. To mitigate this problem, we try vocabulary expansion, adding biases from monolingual inputs, and different types of supervision consisting of additional cross-lingual word alignment objective, instruction tuning, and code-switching training. We find that among these methods, code-switching training and cross-lingual word alignment objective show the most promising results, emphasizing the noteworthiness of cross-lingual alignment supervision for cross-lingual consistency enhancement.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/m3check/Wed, 01 Jan 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/m3check/<p>Imagine that, while we cannot simply Google a complex claim, we can still find some relevant clues to help us. The question is, how can we use these clues to verify the input claim? M^3Check is a RAG model that follows three major steps: 1) break down the claim into clues, 2) verify each clue based on articles, and 3) conclude, check, and reason the claim based on a structured, supervised, and verified chain of clues. This project leverages findings from multiple projects, offering a good example of how research outcomes can be applied to address a real-world problem.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/model-generated-code-mixed-sentences/Wed, 01 Jan 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/model-generated-code-mixed-sentences/<p>Code-switching/code-mixing is a common linguistic practice where speakers switch between multiple languages in a single discourse. Developing language models with code-switching capabilities is crucial to the interest of the large multilingual communities. However, collecting real-world code-mixed sentences for training remains challenging due to its colloquial nature, highlighting the importance of synthetic code-mixed data. Preliminary experiments shows LLMs can generate code-mixed sentences by switching entities in multiple languages. However, these model-generated sentences are often not natural limiting further leverage for training. This project explores methods to improve the naturalness of model-generated code-mixed sentences. Ultimately, we aim to build an automated pipeline capable of generating natural-sounding code-switched sentences for further downstream tasks.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/tart/Wed, 01 Jan 2025 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/tart/<p>TART is designed to overcome the <strong>limitations of LLMs in table reasoning</strong> by combining <strong>structured table processing, precise tool execution, and explainability</strong>. The key components of TART are:</p> <ol> <li><strong>Table Formatter</strong>: Prepares tables by formatting, aligning, and cleaning the data for structured input.</li> <li><strong>Tool Maker</strong>: Dynamically generates tools (Python functions) tailored for complex table operations.</li> <li><strong>Explanation Generator</strong>: Integrates structured reasoning steps and tool outputs into human-readable explanations.</li> </ol> <p>TART is evaluated on <strong>nine table-based reasoning benchmarks</strong>, including <strong>SCITAB, TabFact, FinQA, and HybridQA</strong>. The framework consistently outperforms Chain-of-Thought (CoT) prompting by integrating structured tool-assisted reasoning.</p> <p>TART has received the <strong>Best Paper Runner-Up Award</strong> at the <a href="https://table-representation-learning.github.io/NeurIPS2024/" target="_blank" rel="noopener">3rd Table Representation Learning Workshop</a> at NeurIPS 2024 and has been accepted at <strong>NAACL 2025</strong>. Its code and dataset are publicly available for further research. Researchers working on LLM explainability, table reasoning, and AI-assisted fact-checking are encouraged to contribute.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/analytical-reporting/Tue, 12 Nov 2024 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/analytical-reporting/<p>While LLMs excel in language understanding and can pass professional financial examinations, they struggle to generate expert-level analytical reasoning from real data. In this project, we explore analytical report generation using LLMs, turning complex market data and corporate disclosures into actionable, decision-relevant narratives. We first develop DataTales (EMNLP 2024), a benchmark pairing 4.9k financial market data tables with human reports to diagnose the exact gap between LLM generation and human creation. Secondly, we propose KAHAN (Findings of EMNLP 2025), a knowledge-augmented hierarchical analysis framework that coordinates analysis from entity metrics to market-wide patterns using executable code. Thirdly, we introduce AnalysisBank (EMNLP 2026), a distilled library of over 5,300 expert analysis patterns that conditions reasoning directly on observed data signals to generate novel, data-specific insights.</p> <p>Yajing, a PhD student at WING and a Senior Data Scientist at Rio Tinto, started her research driven by the real-world business need for timely daily market reports. While working alongside market professionals, she observed how human commentary takes hours to draft while existing automation defaults to shallow summaries. Advised by Prof. Min-Yen Kan, she formalized this industry pain point into a systematic study of execution failures in generative models. Together with their collaborators, they developed structured methodologies to decouple calculation from composition and condition reasoning moves directly on observed data patterns. Their work demonstrates that grounded analytical depth can be systematically achieved across financial modalities and transferred to domains like medical diagnostics and scientific writing.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/aiscientist/Mon, 26 Aug 2024 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/aiscientist/<p>Our collaboration with Joeran Beel and Moritz Baumgart of ISG Siegen originated from Joeran&rsquo;s sabbatical visit with WING in Oct 2024. Moritz, a talented undergraduate student at University of Siegen, joined us as a research assistant, working remotely which coordinated with his advisor here in NUS.</p> <p>Over the course of Joeran&rsquo;s visit, we were able to install and test the AI Scientist system advertised as an automated robo scientist by Sakana.AI. We find positive use cases, but also weaknesses that need much handholding from us.</p> <p>Haomeng, Rui Feng and Nura join our team as students working towards assessing the growing variety of LLM based research agents going into 2025.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/benchmarking-and-improving-ocr-on-sea-languages/Thu, 01 Aug 2024 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/benchmarking-and-improving-ocr-on-sea-languages/<p>While Optical Character Recognition (OCR) has been widely studied for high-resource languages such as English and Chinese, its performance on Southeast Asian (SEA) languages remains largely unexplored. This study addresses this gap by evaluating three OCR tools — EasyOCR, Tesseract, and the transformer-based General OCR Theory (GOT) — on English, Indonesian, Vietnamese, and Thai. We introduce a reusable pipeline for collecting textual data from Wikipedia and benchmarking OCR tools. Contrary to popular belief, our results show that OCR tools perform well on complex scripts like Vietnamese and Thai, with most errors arising from misclassifying characters outside the target language. Additionally, we demonstrate the effectiveness of fine-tuning GOT with limited training data, yielding notable improvements on Vietnamese and Thai.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/linguistic-gnns/Thu, 01 Aug 2024 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/linguistic-gnns/<p>This project tackles the challenge of extractive question answering (QA), where the goal is to find a precise span of text from a passage that answers a given question. Traditional models often rely on deep language representations but may overlook the underlying syntactic and semantic structure of the context. By addressing this gap, the project aims to create more interpretable and linguistically aware QA systems that can better understand how information is organized and related within text.</p> <p>To achieve this, we first parse each context passage into a constituency tree using a stack Bi-LSTM, forming the backbone of a graph representation. We then enrich this graph with semantic relationships predicted by a knowledge graph link prediction model. The combined structure is processed by a heterogeneous graph transformer integrating both syntactic and semantic cues for answer extraction. This approach allows for more explainable reasoning and provides a structured way to incorporate external knowledge into the QA process.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/emoji/Mon, 20 May 2024 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/emoji/<p>The project began when Yisong, intrigued by <a href="https://x.com/EmojiMashupBot" target="_blank" rel="noopener">the Emoji Mashup Bot</a>, explored the question: Can emojis be composed to convey meanings like English phrases? With the support of his advisor, Prof. Min, he recruited an undergraduate student, Zi Yun Yang, to collaborate on the project. Ms. Yang built an annotation platform to collect emoji compositions and their corresponding English phrases from her peers. After her graduation, Yisong and the team added another layer of annotation on composition types and formalized an emoji-based textual entailment task (EmoTE).</p> <p>The first iteration of this work was published at <a href="https://lrec-coling-2024.org" target="_blank" rel="noopener">LREC-COLING 2024</a>. Prof. Min was not listed as an author due to his role as a conference chair, but this project cannot be finished without his guidance. The project continues to gain interest, particularly among undergraduates, with many students from our local NLP course using the emoji dataset for their final projects.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/izzyviz/Fri, 17 May 2024 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/izzyviz/<p>In the field of natural language processing, large and complex neural networks, particularly transformers, have become essential due to their effectiveness in handling text data. However, despite their success, current methods for visualizing how activations of specific network units influence model outputs remain underdeveloped. This gap hinders interpretability and limits researchers’ and practitioners’ understanding of neural network behaviors during training and inference.</p> <p>This project aims to address this issue by creating a robust visualizer tailored for transformer architectures. By processing training or test data, this project seeks to track and analyze changes in attention scores and other structural components of the transformer network, visualizing their evolution within individual input sentences during inference. The project involves developing innovative visualization conventions to compactly represent the high-dimensional neural network processes in a manner that is analytical and accessible, through the usage of static diagrams.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/scitab/Wed, 06 Dec 2023 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/scitab/<p>SCITAB was developed to fill the gap in scientific fact-checking benchmarks by incorporating claims derived from actual scientific papers rather than crowd-sourced ones. These claims require multi-step reasoning to verify against evidence in tables. The dataset was created using a human–model collaboration approach, where real-world claims were filtered, counterclaims were generated using InstructGPT, and all claims were manually verified by expert annotators.</p> <p>The dataset has been extensively evaluated with state-of-the-art models, including TAPAS, TAPEX, Vicuna, and GPT-4, revealing that existing table-based models struggle with SCITAB. Only GPT-4 shows notable improvements, highlighting the dataset’s challenge and potential for advancing research in table-based reasoning.</p> <p>SCITAB was presented at <strong>EMNLP 2023</strong>, and the dataset is publicly available for further research. Researchers interested in compositional reasoning, table fact-checking, and model robustness are encouraged to explore SCITAB and contribute to its future developments.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/advanced-opacs/Fri, 04 Jun 2004 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/advanced-opacs/<p>When library patrons utilize an online public access catalog (OPAC), they tend to type very few query words. Although it has been observed that patrons often have specific information needs, current search engine usability encourages users to underspecify their queries. With an OPAC&rsquo;s fast response times and the difficulty of using more advanced query operators, users are pushed towards a probe-and-browse mode of information seeking. Additionally, patrons have adapted (or forced to adapt) to OPACs and give keywords as their queries, rather than more precise queries. As a consequence, the search results often only approximate the patron&rsquo;s information need, missing crucial resources that may have been phrased differently or offering search results that may be phrased exactly as wanted but which only address the patron&rsquo;s information need tangentially.</p> <p>One solution is to teach the library patron to use advanced query syntax and to formulate more precise queries. However, this solution is both labor-intensive for library staff and time-intensive for patrons. Furthermore, different OPACs support different levels of advanced capabilities and often represent these operators with different syntax. An alternative solution that we propose is the use of advanced query analysis and query expansion. Rather than change the behavior of the patron, a system can analyze their keyword queries to infer more precise queries that uses advanced operators when appropriate. To make these inferences, the system will not only rely on logic but also will dynamically access both a) historical query logs and b) the library catalog to assess the feasibility of its suggestions. The proposed research will target three different types of query rewriting/expansion: 1) correction of misspellings, 2) inferring the relationships between a query&rsquo;s noun and noun phrases, and 3) inferring intended advanced query syntax. The realization of these techniques will allow patrons to continue using OPACs by issuing simple keyword searchers while benefiting from more precise querying and alternative search suggestions that would originate from the implemented system.</p> <p>In our current work we have examined how to re-engineer and design the User Interface to better supoort the actual information seeking methods used by library patrons. Jesse has re-engineered the work and has incorporated our own NUS OPAC as well as Colorado State University&rsquo;s OPAC results into his tabbed, overview+details framework. If you have a library catalog with MARC21 results that can be exported we can wrap our UI around your database. Contact us for more information.</p> <p>In collaboration with Prof. Danny C. C. Poo.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/light-verb-constructions/Sat, 01 May 2004 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/light-verb-constructions/<p>Project Duration: May 2004 - July 2005.</p> <p>A light verb construction (LVC) is a verb–complement pair in which the verb has little lexical meaning and much of the semantic content of the construction is obtained from the complement. Examples of LVCs are &ldquo;make a decision&rdquo; and &ldquo;give a presentation&rdquo;, and these pose challenges for natural language processing and understanding. In this project, we investigate methods to identify LVCs from a corpus, as well as recognizing linguistic features of LVCs.</p>https://www.comp.nus.edu.sg/~kanmy/wing/project/definition-qa/Fri, 01 Aug 2003 00:00:00 +0000https://www.comp.nus.edu.sg/~kanmy/wing/project/definition-qa/<p>Joint work with PRIS group led by Prof. Tat-Seng Chua</p> <p>We explore advanced techniques in definition question answering: soft pattern matching, and boosting of IR recall and precision of extended definition sentences using external web resources and historical query logs. We also explore the construction of fluent definitions using sentence understanding and re-synthesis.</p> <h2>Project Deliverables (excluding publications)</h2> <ul> <li>DefSearch: definition searcher demonstration, based on work in TREC 2003 and a paper in WWW 2004. <li>JavaRAP: Lappin and Leass' pronominal anaphora resolution program, implemented as a Java program. Freely downloadable and installable. Published in LREC 2004. <li>Processed Gigaword and AQUAINT corpora. Fully POS tagged and parsed versions of two commonly-used corpora used in TREC research. <li>Processed DUC corpus. Canonicalized versions of the 2001-2004 corpus for internal research. </ul>