Augmenting Cognition in the AI-Accelerated Era Best Paper Award

Paper Abstract
Real-time cognitive state decoding in naturalistic settings requires portable neuroimaging technologies like functional near-infrared spectroscopy (fNIRS), but small dataset sizes have prevented development of complex deep learning models such as transformers. We investigated whether models pretrained on large functional magnetic resonance imaging (fMRI) datasets could be transferred to fNIRS through fine-tuning, leveraging the fact that both modalities measure hemodynamic responses to neural activity. We collected fNIRS data from 32 participants performing working memory (N-back) and motor tasks matching the Human Connectome Project protocol, enabling direct comparison with fMRI data from over 1,000 participants. We tested two state-of-the-art architectures: BolT (transformer-based) and BrainGNN (graph neural network). While BolT achieved strong performance on fMRI classification, transfer learning to fNIRS was not successful, with no model exceeding chance-level performance (50%) despite theoretical similarities between the modalities. BrainGNN struggled to classify either the fMRI or the fNIRS data. We identify critical factors contributing to poor classification performance and propose alternative approaches for the research community, including cross-modal translation architectures, development of larger standardized fNIRS datasets, and concurrent multimodal acquisition. Our findings provide crucial guidance for future cross-modal neuroimaging research.

The full paper is available through SpringerLink, provided that you have proper access rights.