본문 바로가기
KERD
논문KCI 등재

학습피드백으로서 보상과 처벌 관련 두뇌 활성화 연구

Learning-associated Reward and Penalty in Feedback Learning: an fMRI activation study

강은주|김진희
한국인지과학회 (The Korean Society for Cognitive Science)|2017.03|인지과학 (Korean Journal of Cognitive Science)|제28권 제1호|pp. 65-90|DOI 10.19066/cogsci.2017.28.1.004

0

조회

0

다운로드

2

인용

0

참고문헌

키워드

prefrontal cortex전전두피질오류처리error processing강화학습negative feedbackreinforcement learning부적 피드백전측도anterior insula
주제분류인지과학

초록

본 연구의 목적은 학습상황에서 피드백으로 주어지는 금전적 획득/손실(학습 피드백)과 비학습적 상황에서 우연히 제시되는 의사 피드백(무선 피드백)을 비교하는 방법을 사용하여, 금전적 보상과 처벌의 학습 피드백으로서만 가지는 정보처리에 어느 두뇌 영역이 관여하는지를 규명하는 데 있다. 이를 위해 정상 성인(n = 22)을 대상으로 fMRI scan 동안 단서 자극에 대한 범주 버튼 반응(좌/우)의 정확 여부에 따라 피드백이 제시되는 시행(학습시행)과 단서 자극의 위치판단 반응과 무관하게 피드백이 제시되는 시행(무선시행)을 사건 관련 fMRI 방략으로 제시하였다. 두 시행 간 보상과 처벌과 같은 동기적 사건에 대한 두뇌 반응이 변별적으로 나타나는지를 알아보기 위해 시행 유형(학습 vs. 무선)과 피드백 유형(보상 vs, 처벌)을 두 독립변인으로 한 반복측정 이원분산분석을 하였다(voxel-wise FWE p < .001). 그 결과, 좌측 배외측 전두피질(dorsolateral prefrontal cortex), 좌측 전측 도(anterior insular), 배내측 전두피질(dorsomedial prefrontal cortex) 등의 영역에서 유의한 상호작용 효과가 관찰되었는데, 이들 영역은 모두 학습-보상 피드백 및 무선-처벌 피드백보다 학습-처벌 피드백에 대해 증가한 두뇌 활성을 보였다. 본 연구 결과는 학습상황에서 주어지는 처벌 피드백에 대한 기존 전략의 변경이나 재평가를 위한 집행적 처리, 적절하지 못하거나 틀린 행동에 대한 오류처리 과정 그리고 실패 경험에 대한 부정적 정서처리가 위에서 언급한 피질신경망을 중심으로 이루어질 가능성을 보여준다. 따라서 학습의 처벌 피드백은 보상과 달리 위와 같은 추가적 정보처리 과정이 존재할 가능성을 시사한다.

Abstract

Rewards or penalties become informative only when contingent on an immediately preceding response. Our goal was to determine if the brain responds differently to motivational events depending on whether they provide feedback with the contingencies effective for learning. Event-related fMRI data were obtained from 22 volunteers performing a visuomotor categorical task. In learning-condition trials, participants learned by trial and error to make left or right responses to letter cues (16 consonants). Monetary rewards (+500) or penalties (-500) were given as feedback (learning feedback). In random-condition trials, cues (4 vowels) appeared right or left of the display center, and participants were instructed to respond with the appropriate hand. However, rewards or penalties (random feedback) were given randomly (50/50%) regardless of the correctness of response. Feedback-associated BOLD responses were analyzed with ANOVA [trial type (learning vs. random) x feedback type (reward vs. penalty)] using SPM8 (voxel-wise FWE p < .001). The right caudate nucleus and right cerebellum showed activation, whereas the left parahippocampus and other regions as the default mode network showed deactivation, both greater for learning trials than random trials. Activations associated with reward feedback did not differ between the two trial types for any brain region. For penalty, both learning-penalty and random-penalty enhanced activity in the left insular cortex, but not the right. The left insula, however, as well as the left dorsolateral prefrontal cortex and dorsomedial prefrontal cortex/dorsal anterior cingulate cortex, showed much greater responses for learning-penalty than for random-penalty. These findings suggest that learning-penalty plays a critical role in learning, unlike rewards or random-penalty, probably not only due to its evoking of aversive emotional responses, but also because of error-detection processing, either of which might lead to changes in planning or strategy.

참고문헌 (73)

  1. Amiez, C., "Reward encoding in the monkey anterior cingulate cortex", Cerebral Cortex, 16, pp.1040~1055 (2006)
  2. Anderson, A. K., "Dissociated neural representations of intensity and valence in human olfaction", Nature Neuroscience, 6, pp.196~202 (2003)
  3. Aron, A. R., "Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning", Journal of Neurophysiology, 92, pp.1144~1152 (2004)
  4. Balleine, B. W., "The role of the dorsal striatum in reward and decision-making", The Journal of Neuroscience, 27, pp.8161~8165 (2007)
  5. Bastin, J., "Direct recordings from human anterior insula reveal its leading role within the error-monitoring network", Cerebral Cortex, pp.bhv352~ (2016)
  6. Berridge, K. C., "What is the role of dopamine in reward : hedonic impact, reward learning, or incentive salience?", Brain Research Reviews, 28, pp.309~369 (1998)
  7. Bischoff-Grethe, A., "The influence of feedback valence in associative learning", Neuroimage, 44, pp.243~251 (2009)
  8. Bray, S., "Neural coding of reward-prediction error signals during classical conditioning with attractive faces", Journal of Neurophysiology, 97, pp.3036~3045 (2007)
  9. Breiter, H. C., "Functional imaging of neural responses to expectancy and experience of monetary gains and losses", Neuron, 30, pp.619~639 (2001)
  10. Buckner, R. L., "The brain’s default network", Annals of the New York Academy of Sciences, 1124, pp.1~38 (2008)
  11. Buzzell, G. A., "Uncertainty-dependent activity within the ventral striatum predicts task-related changes in response strategy", Cognitive, Affective & Behavioral Neuroscience, 16, pp.219~233 (2016)
  12. Carreiras, M., "Brain activation for consonants and vowels", Cerebral Cortex, 18, pp.1727~1735 (2008)
  13. Collins, A. G., "How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis", European Journal of Neuroscience, 35, pp.1024~1035 (2012)
  14. Curtis, C. E., "Persistent activity in the prefrontal cortex during working memory", Trends in Cognitive Sciences, 7, pp.415~423 (2003)
  15. Dale, A. M., "Optimal experimental design for event-related fMRI", Human Brain Mapping, 8, pp.109~114 (1999)
  16. Debener, S., "Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring", The Journal of Neuroscience, 25, pp.11730~11737 (2005)
  17. Delgado, M. R., "Dorsal striatum responses to reward and punishment: effects of valence and magnitude manipulations", Cognitive, Affective & Behavioral Neuroscience, 3, pp.27~38 (2003)
  18. Delgado, M. R., "An fMRI study of reward-related probability learning", Neuroimage, 24, pp.862~873 (2005)
  19. Fox, M. D., "The human brain is intrinsically organized into dynamic, anticorrelated functional networks", Proceedings of the National Academy of Sciences, 102, pp.9673~9678 (2005)
  20. Gu, X., "Anterior insular cortex and emotional awareness", The Journal of comparative neurology, 521, pp.3371~3388 (2013)
  21. Ham, T., "Cognitive control and the salience network : an investigation of error processing and effective connectivity", The Journal of Neuroscience, 33, pp.7091~7098 (2013)
  22. Hare, T. A., "Self-control in decision-making involves modulation of the vmPFC valuation system", Science, 324, pp.646~648 (2009)
  23. Holroyd, C. B., "The neural basis of human error processing : reinforcement learning, dopamine, and the error-related negativity", Psychological Review, 109, pp.679~709 (2002)
  24. Holroyd, C. B., "Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals", Nature Neuroscience, 7, pp.497~498 (2004)
  25. Iannaccone, R., "Conflict monitoring and error processing : New insights from simultaneous EEG–fMRI", Neuroimage, 105, pp.395~407 (2015)
  26. Kim, H., "Is avoiding an aversive outcome rewarding? Neural substrates of avoidance learning in the human brain", PLoS Biology, 4, pp.e233~ (2006)
  27. 김수현, "학습의 진행과 두뇌 피드백 정보처리의 변화", 한국심리학회지: 인지 및 생물, 27(3), pp.419~450 (2015), 한국인지및생물심리학회[DOI]
  28. Knutson, B., "Anticipation of increasing monetary reward selectively recruits nucleus accumbens", The Journal of Neuroscience, 21, pp.RC159~ (2001)
  29. Knutson, B., "A region of mesial prefrontal cortex tracks monetarily rewarding outcomes : characterization with rapid event-related fMRI", Neuroimage, 18, pp.263~272 (2003)
  30. Krawczyk, D. C., "Contributions of the prefrontal cortex to the neural basis of human decision making", Neuroscience and Biobehavioral Reviews, 26, pp.631~664 (2002)
  31. Kuhnen, C. M., "The neural basis of financial risk taking", Neuron, 47, pp.763~770 (2005)
  32. Liu, X., "Common and distinct networks underlying reward valence and processing stages : a meta-analysis of functional neuroimaging studies", Neuroscience and Biobehavioral Reviews, 35, pp.1219~1236 (2011)
  33. Lotze, M., "Activation of cortical and cerebellar motor areas during executed and imagined hand movements : an fMRI study", Journal of Cognitive Neuroscience, 11, pp.491~501 (1999)
  34. Luking, K. R., "Candy and the brain: neural response to candy gains and losses", Cognitive, Affective & Behavioral Neuroscience, 2013(13), pp.437~451 (2013)
  35. Maratos, E. J., "Neural activity associated with episodic memory for emotional context", Neuropsychologia, 39, pp.910~920 (2001)
  36. Mckiernan, K. A., "A parametric manipulation of factors affecting task-induced deactivation in functional neuroimaging", Journal of Cognitive Neuroscience, 15, pp.394~408 (2003)
  37. Menon, V., "Saliency, switching, attention and control : a network model of insula function", Brain Structure & Function, 214, pp.655~667 (2010)
  38. Metereau, E., "Cerebral correlates of salient prediction error for different rewards and punishments", Cerebral Cortex, 23, pp.477~487 (2013)
  39. Middleton, F. A., "Basal-ganglia ‘projections’ to the prefrontal cortex of the primate", Cerebral Cortex, 12, pp.926~935 (2002)
  40. Montague, P. R., "A framework for mesencephalic dopamine systems based on predictive Hebbian learning", The Journal of Neuroscience, 16, pp.1936~1947 (1996)
  41. O’Doherty, J., "Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices", The Journal of Neuroscience, 23, pp.7931~7939 (2003)
  42. O’Doherty, J., "Dissociable roles of ventral and dorsal striatum in instrumental conditioning", Science, 304, pp.452~454 (2004)
  43. O’Doherty, J., "Abstract reward and punishment representations in the human orbitofrontal cortex", Nature Neuroscience, 4, pp.95~102 (2001)
  44. Obleser, J., "Segregation of Vowels and Consonants in Human Auditory Cortex : Evidence for Distributed Hierarchical Organization", Frontiers in Psychology, 1, pp.232~ (2010)
  45. Pessiglione, M., "How the brain translates money into force : a neuroimaging study of subliminal motivation", Science, 316, pp.904~906 (2007)
  46. Peterburs, J., "The role of the human cerebellum in performance monitoring", Current Opinion in Neurobiology, 40, pp.38~44 (2016)
  47. Peters, S., "Longitudinal development of frontoparietal activity during feedback learning : Contributions of age, performance, working memory and cortical thickness", Developmental Cognitive Neuroscience, 19, pp.211~222 (2016)
  48. Poldrack, R. A., "Interactive memory systems in the human brain", Nature, 414, pp.546~550 (2001)
  49. Raichle, M. E., "A default mode of brain function", Proceedings of the National Academy of Sciences, 98, pp.676~682 (2001)
  50. Ressler, N., "Rewards and punishments, goal-directed behavior and consciousness", Neuroscience and Biobehavioral Reviews, 28, pp.27~39 (2004)
  51. Rustemeier, M., "Processing of positive and negative feedback in patients with cerebellar lesions", Cerebellum, 15, pp.425~438 (2016)
  52. Salimpoor, V. N., "Interactions between the nucleus accumbens and auditory cortices predict music reward value", Science, 340, pp.216~219 (2013)
  53. Samanez-Larkin, G. R., "Individual differences in insular sensitivity during loss anticipation predict avoidance learning", Psychological Science, 19, pp.320~323 (2008)
  54. Schultz, W., "Dopamine neurons and their role in reward mechanisms", Current Opinion in Neurobiology, 7, pp.191~197 (1997)
  55. Schultz, W., "A neural substrate of prediction and reward", Science, 275, pp.1593~1599 (1997)
  56. Seger, C. A., "How do the basal ganglia contribute to categorization? Their roles in generalization, response selection, and learning via feedback", Neuroscience and Biobehavioral Reviews, 32, pp.265~278 (2008)
  57. Seger, C. A., "The roles of the caudate nucleus in human classification learning", The Journal of Neuroscience, 25, pp.2941~2951 (2005)
  58. Seger, C. A., "Dynamics of frontal, striatal, and hippocampal systems during rule learning", Cerebral Cortex, 16, pp.1546~1555 (2006)
  59. Shackman, A. J., "The integration of negative affect, pain and cognitive control in the cingulate cortex", Nature Reviews Neuroscience, 12, pp.154~167 (2011)
  60. Singer, T., "A common role of insula in feelings, empathy and uncertainty", Trends in Cognitive Sciences, 13, pp.334~340 (2009)
  61. Skinner, B. F., "The behavior of organisms: An experimental analysis" (1938), D. Appleton-Century Company
  62. Spreng, R. N., "The fallacy of a"task-negative"network", Frontiers in Psychology, 3, pp.145~ (2012)
  63. Steele, V. R., "Neuroimaging measures of error-processing : Extracting reliable signals from event-related potentials and functional magnetic resonance imaging", Neuroimage, 132, pp.247~260 (2016)
  64. Stephani, C., "Functional neuroanatomy of the insular lobe", Brain Structure & Function, 216, pp.137~149 (2011)
  65. Stoodley, C. J., "Functional topography of the cerebellum for motor and cognitive tasks : an fMRI study", Neuroimage, 59, pp.1560~1570 (2012)
  66. Taylor, S. F., "The effect of emotional content on visual recognition memory : a PET activation study", Neuroimage, 8, pp.188~197 (1998)
  67. Tricomi, E. M., "Modulation of caudate activity by action contingency", Neuron, 41, pp.281~292 (2004)
  68. Tricomi, E. M., "Information content and reward processing in the human striatum during performance of a declarative memory task", Cognitive, Affective & Behavioral Neuroscience, 12, pp.361~372 (2012)
  69. Ullsperger, M., "Error monitoring using external feedback : specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging", The Journal of Neuroscience, 23, pp.4308~4314 (2003)
  70. Wheeler, E. Z., "The human ventromedial frontal lobe is critical for learning from negative feedback", Brain, 131, pp.1323~1331 (2008)
  71. Williams, Z. M., "Selective enhancement of associative learning by microstimulation of the anterior caudate", Nature Neuroscience, 9, pp.562~568 (2006)
  72. Xu, L., "Neural mechanism of intertemporal choice : from discounting future gains to future losses", Brain Research, 1261, pp.65~74 (2009)
  73. Zanolie, K., "Separable neural mechanisms contribute to feedback processing in a rule-learning task", Neuropsychologia, 46, pp.117~126 (2008)

관련 논문AI추천

학습의 진행과 두뇌 피드백 정보처리의 변화강은주, 김수현, 김진희한국인지및생물심리학회 · 한국심리학회지: 인지 및 생물2015 · 제27권 제3호An fMRI Study Investigating Adolescent Brain Activation by Rewards and Feedback김시경, 김영랑, 손정우, 신철진, 오종현, 이상익, 이승복, 조승우, 주가원, 최원희, 하태현대한신경정신의학회 · PSYCHIATRY INVESTIGATION2013 · 제10권 제1호피드백 학습능력의 개인차와 두뇌 회백질 부피: 형태분석 연구강은주, 김진희한국인지및생물심리학회 · 한국심리학회지: 인지 및 생물2013 · 제25권 제4호친사회적- 도덕적 동기 및 도덕적 의사결정의 신경학적 기제에 대한 개관 연구정주연, 한상훈한국감성과학회 · 감성과학2011 · 제14권 제4호조현병 환자에서 보상 및 손실의 연합 학습의 특징구민성, 김석형, 김재진, 김정환, 박선영, 박일호, 송정은대한조현병학회 · 대한조현병학회지2012 · 제15권 제2호휴지기 두뇌의 네트워크와 지연할인율의 개인차정위훈한국인지및생물심리학회 · 한국심리학회지: 인지 및 생물2021 · 제33권 제1호

이 간행물의 다른 논문 (3건)

조현형 인격 성향을 가진 대학생의 P300 국소화 분석김명선, 김보미, 안은지, 장경미, 나은찬pp. 1-26형용사의 의미가 색 구별에 미치는 영향: 스트룹 과제를 통한 검증김경호, 이형철, 김신우, 홍성균pp. 27-42과제 무관련 정서 점화가 신념편향에 미치는 영향홍영지, 우현정, 이윤형pp. 43-64
원문 보기