Attention機製最早是在視覺圖像領域提出來的,應該是在九幾年思想就提出來了,但是真正火起來應該算是google mind團隊的這篇論文《Recurrent Models of Visual Attention》[14],他們在RNN模型上使用了attention機製來進行圖像分類。隨後,Bahdanau等人在論文《Neural Machine Translation by Jointly Learning to Align and Translate》 [1]中,使用類似attention的機製在機器翻譯任務上將翻譯和對齊同時進行,他們的工作算是是第一個提出attention機製應用到NLP領域中。接著類似的基於attention機製的RNN模型擴展開始應用到各種NLP任務中。最近,如何在CNN中使用attention機製也成為了大家的研究熱點。下圖表示了attention研究進展的大概趨勢。

VIP內容

題目:Attention in Natural Language Processing

摘要:

注意力是一種越來越受歡迎的機製,在廣泛的神經結構中使用。該機製本身以各種格式實現。然而,由於這一領域的快速發展,仍然缺乏對注意力的係統概述。在本文中,我們為自然語言處理中的注意力架構定義了一個統一的模型,重點是那些設計用來處理文本數據的向量表示的模型。根據四個維度提出了注意力模型的分類:輸入的表示、兼容性函數、分布函數和輸入和輸出的多樣性。然後展示了如何在注意力模型中利用先驗信息的例子,並討論了該領域正在進行的研究工作和麵臨的挑戰。

成為VIP會員查看完整內容
0
16
0

最新論文

On social network platforms, a user's behavior is based on his/her personal interests, or influenced by his/her friends. In the literature, it is common to model either users' personal preference or their socially influenced preference. In this paper, we present a novel deep learning model SocialTrans for social recommendations to integrate these two types of preferences. SocialTrans is composed of three modules. The first module is based on a multi-layer Transformer to model users' personal preference. The second module is a multi-layer graph attention neural network (GAT), which is used to model the social influence strengths between friends in social networks. The last module merges users' personal preference and socially influenced preference to produce recommendations. Our model can efficiently fit large-scale data and we deployed SocialTrans to a major article recommendation system in China. Experiments on three data sets verify the effectiveness of our model and show that it outperforms state-of-the-art social recommendation methods.

0
0
0
下載
預覽
子主題
Top