題目:Attention in Natural Language Processing
摘要:
注意力是一種越來越受歡迎的機製,在廣泛的神經結構中使用。該機製本身以各種格式實現。然而,由於這一領域的快速發展,仍然缺乏對注意力的係統概述。在本文中,我們為自然語言處理中的注意力架構定義了一個統一的模型,重點是那些設計用來處理文本數據的向量表示的模型。根據四個維度提出了注意力模型的分類:輸入的表示、兼容性函數、分布函數和輸入和輸出的多樣性。然後展示了如何在注意力模型中利用先驗信息的例子,並討論了該領域正在進行的研究工作和麵臨的挑戰。
On social network platforms, a user's behavior is based on his/her personal interests, or influenced by his/her friends. In the literature, it is common to model either users' personal preference or their socially influenced preference. In this paper, we present a novel deep learning model SocialTrans for social recommendations to integrate these two types of preferences. SocialTrans is composed of three modules. The first module is based on a multi-layer Transformer to model users' personal preference. The second module is a multi-layer graph attention neural network (GAT), which is used to model the social influence strengths between friends in social networks. The last module merges users' personal preference and socially influenced preference to produce recommendations. Our model can efficiently fit large-scale data and we deployed SocialTrans to a major article recommendation system in China. Experiments on three data sets verify the effectiveness of our model and show that it outperforms state-of-the-art social recommendation methods.