Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

1
66
下載
關閉預覽

相關內容

Transformer是穀歌發表的論文《Attention Is All You Need》提出一種完全基於Attention的翻譯架構

知識薈萃

精品入門和進階教程、論文和代碼整理等

更多

查看相關VIP內容、論文、資訊等

摘要

Transformers 在自然語言處理、計算機視覺和音頻處理等許多人工智能領域都取得了巨大的成功。因此,自然會引起學術界和工業界研究人員的極大興趣。到目前為止,各種各樣的Transformer變種(即X-formers)已經被提出,但是,關於這些Transformer器變種的係統和全麵的文獻綜述仍然缺乏。在這項綜述中,我們提供了一個全麵的Transformer綜述。我們首先簡單介紹了普通的Transformer,然後提出了一個x-former的新分類。接下來,我們將從三個方麵介紹不同的x -former架構修改,預訓練和應用。最後,展望了未來的研究方向。

//www.webtourguide.com/paper/f03a47eb6ddb5d23c07f51662f3220a0

引言

Transformer[136]是一種出色的深度學習模型,被廣泛應用於自然語言處理(NLP)、計算機視覺(CV)和語音處理等各個領域。Transformer最初是作為一種用於機器翻譯的序列到序列模型提出的[129]。後來的工作表明,基於Transformer的預訓練模型(PTMs)[100]可以在各種任務上實現最先進的性能。因此,Transformer已經成為NLP的首選架構,特別是對於PTMs。除了語言相關的應用,Transformer也被應用於CV[13, 33, 94],音頻處理[15,31,41],甚至其他學科,如化學[113]和生命科學[109]。

由於成功,各種各樣的Transformer 變種(即x -former)在過去幾年裏被提出。這些X-formers從不同的角度改進了vanilla Transformer。

(1) 模型的效率。應用Transformer的一個關鍵挑戰是它在處理長序列時效率低下,這主要是由於自注意力模塊的計算和存儲複雜性。改進方法包括輕量級注意力(例如稀疏注意變體)和分治法(例如循環和分層機製)。

(2) 模型泛化。由於Transformer是一種靈活的體係結構,並且很少對輸入數據的結構偏差進行假設,因此很難對小規模數據進行訓練。改進方法包括引入結構偏差或正則化、對大規模無標記數據進行預處理等。

(3) 模型的適應。該工作旨在使Transformer適應特定的下遊任務和應用程序。

在這個綜述中,我們的目的是提供一個Transformer及其變體的全麵綜述。雖然我們可以根據上麵提到的觀點來組織x-former,但許多現有的x前輩可能會解決一個或幾個問題。例如,稀疏注意變量不僅降低了計算複雜度,而且在輸入數據上引入了結構先驗,緩解了小數據集上的過擬合問題。因此,將現有的各種X-formers進行分類,並根據它們改進Transformer的方式提出新的分類方法會更有條理: 架構修改、預訓練和應用。考慮到本次綜述的受眾可能來自不同的領域,我們主要關注於一般的架構變體,而隻是簡單地討論了預訓練和應用的具體變體。

到目前為止,基於普通Transformer的各種模型已經從三個角度被提出:架構修改的類型、預訓練的方法和應用。圖2給出了Transformer變種的分類說明。

盡管“x-formers”已經證明了他們在各種任務上的能力,但挑戰仍然存在。除了目前關注的問題(如效率和泛化),Transformer的進一步改進可能在以下幾個方向:

(1) 理論分析。Transformer的體係結構已被證明能夠支持具有足夠參數的大規模訓練數據集。許多工作表明,Transformer比CNN和RNN有更大的容量,因此有能力處理大量的訓練數據。當Transformer在足夠的數據上進行訓練時,它通常比CNN或RNN有更好的性能。一個直觀的解釋是,Transformer對數據結構沒有什麼預先假設,因此比CNN和RNN更靈活。然而,理論原因尚不明確,我們需要對Transformer能力進行一些理論分析。

(2) 注意力機製之外的全局交互機製更加完善。Transformer的一個主要優點是使用注意力機製來建模輸入數據中節點之間的全局依賴關係。然而,許多研究表明,對大多數節點來說,完全注意力是不必要的。在某種程度上,不可區分地計算所有節點的注意力是低效的。因此,在有效地建模全局交互方麵仍有很大的改進空間。一方麵,自注意力模塊可以看作是一個具有動態連接權的全連接神經網絡,通過動態路由聚合非局部信息; 因此,其他動態路由機製是值得探索的替代方法。另一方麵,全局交互也可以通過其他類型的神經網絡來建模,比如記憶增強模型。

(3) 多模態數據統一框架。在許多應用場景中,集成多模態數據對於提高任務性能是非常有用和必要的。此外,一般的人工智能還需要能夠捕獲跨不同模式的語義關係。由於Transformer在文本、圖像、視頻和音頻方麵取得了巨大的成功,我們有機會建立一個統一的框架,更好地捕捉多模態數據之間的內在聯係。但是,在設計中對模式內和模式間的注意還有待改進。

成為VIP會員查看完整內容
4
117
0

文本排序的目標是生成從語料庫檢索到的有序文本列表,以響應特定任務的查詢。雖然文本排序最常見的形式是搜索,但在許多自然語言處理應用程序中也可以找到該任務的實例。

本書提供了Transformer神經網絡架構的文本排序的概述,其中BERT是最著名的例子。毫不誇張地說,Transformer和自監督預訓練的結合徹底改變了自然語言處理(NLP)、信息檢索(IR)等領域。在文本排名的上下文中,這些模型在許多領域、任務和設置中產生高質量的結果。

在這項綜述中,我們提供了現有工作的綜合,作為希望更好地理解如何將transformers應用於文本排序問題的從業者和希望在這一領域繼續工作的研究人員的單一切入點。我們涵蓋了廣泛的現代技術,分為兩個高級類別:在多階段排名體係結構中執行重新排名的transformer模型,以及嚐試直接執行排名的密集表示。有許多例子屬於第一類,包括基於相關性分類的方法、來自多個文本片段的證據聚合、語料庫分析和序列到序列模型。雖然第二類方法還沒有得到很好的研究,但使用transformers進行表示學習是一個新興的和令人興奮的方向,必將引起更多的關注。在我們的調研中,有兩個主題貫穿始終:處理長文檔的技術(在NLP中使用的典型逐句處理方法之外),以及處理有效性(結果質量)和效率(查詢延遲)之間權衡的技術。

盡管transformer架構和預訓練技術是最近的創新,但它們如何應用於文本排序的許多方麵已經被比較好地理解,並代表了成熟的技術。然而,仍然存在許多開放的研究問題,因此,除了為文本排序預先設定訓練transformers的基礎之外,該調研還試圖預測該領域的發展方向。

//www.webtourguide.com/paper/fe2037d3186f4dd1fe3c3ea1fb69f79e

成為VIP會員查看完整內容
0
53
0

來自UIUC的Transformers最新教程。

Transformer 架構 architecture Attention models Implementation details Transformer-based 語言模型 language models BERT GPT Other models

Transformer 視覺 Applications of Transformers in vision

成為VIP會員查看完整內容
5
126
0

深度神經網絡在擁有大量數據集和足夠的計算資源的情況下能夠取得巨大的成功。然而,他們快速學習新概念的能力相當有限。元學習是解決這一問題的一種方法,通過使網絡學會如何學習。令人興奮的深度元學習領域正在高速發展,但缺乏對當前技術的統一、深刻的概述。這項工作就是這樣。在為讀者提供理論基礎之後,我們研究和總結了主要的方法,這些方法被分為i)度量;ii)模型;和iii)基於優化的技術。此外,我們確定了主要的開放挑戰,如在異構基準上的性能評估,以及元學習計算成本的降低。

摘要:

近年來,深度學習技術在各種任務上取得了顯著的成功,包括遊戲(Mnih et al., 2013; Silver et al., 2016),圖像識別(Krizhevsky et al., 2012; He et al., 2015)和機器翻譯(Wu et al., 2016)。盡管取得了這些進展,但仍有大量的挑戰有待解決,例如實現良好性能所需的大量數據和訓練。這些要求嚴重限製了深度神經網絡快速學習新概念的能力,這是人類智能的定義方麵之一(Jankowski等人,2011;(Lake等,2017)。

元學習被認為是克服這一挑戰的一種策略(Naik and Mammone, 1992; Schmidhuber, 1987; Thrun, 1998)。其關鍵思想是元學習主體隨著時間的推移提高自己的學習能力,或者等價地說,學會學習。學習過程主要與任務(一組觀察)有關,並且發生在兩個不同的層次上:內部和外部。在內部層,一個新的任務被提出,代理試圖快速地從訓練觀察中學習相關的概念。這種快速的適應是通過在外部層次的早期任務中積累的知識來促進的。因此,內部層關注的是單個任務,而外部層關注的是多個任務。

從曆史上看,元學習這個術語的使用範圍很廣。從最廣泛的意義上說,它概括了所有利用之前的學習經驗以更快地學習新任務的係統(Vanschoren, 2018)。這個廣泛的概念包括更傳統的機器學習算法選擇和hyperparameter優化技術(Brazdil et al ., 2008)。然而,在這項工作中,我們專注於元學習領域的一個子集,該領域開發元學習程序來學習(深度)神經網絡的良好誘導偏差。1從今以後,我們使用術語深元學習指元學習的領域。

深度元學習領域正在快速發展,但它缺乏一個連貫、統一的概述,無法提供對關鍵技術的詳細洞察。Vanschoren(2018)對元學習技術進行了調查,其中元學習被廣泛使用,限製了對深度元學習技術的描述。此外,在調查發表後,深度元學習領域也出現了許多令人興奮的發展。Hospedales等人(2020)最近的一項調查采用了與我們相同的深度元學習概念,但目標是一個廣泛的概述,而忽略了各種技術的技術細節。

我們試圖通過提供當代深度元學習技術的詳細解釋來填補這一空白,使用統一的符號。此外,我們確定了當前的挑戰和未來工作的方向。更具體地說,我們覆蓋了監督和強化學習領域的現代技術,已經實現了最先進的性能,在該領域獲得了普及,並提出了新的想法。由於MAML (Finn et al., 2017)和相關技術對該領域的影響,我們給予了格外的關注。本研究可作為深度元學習領域的係統性介紹,並可作為該領域資深研究人員的參考資料。在整個過程中,我們將采用Vinyals(2017)所使用的分類法,該分類法確定了三種深度元學習方法:i)度量、ii)模型和iii)基於優化的元學習技術。

成為VIP會員查看完整內容
2
86
0

摘要

Transformer模型架構最近引起了極大的興趣,因為它們在語言、視覺和強化學習等領域的有效性。例如,在自然語言處理領域,Transformer已經成為現代深度學習堆棧中不可缺少的主要部分。最近,提出的令人眼花繚亂的X-former模型如Linformer, Performer, Longformer等這些都改進了原始Transformer架構的X-former模型,其中許多改進了計算和內存效率。為了幫助熱心的研究人員在這一混亂中給予指導,本文描述了大量經過深思熟慮的最新高效X-former模型的選擇,提供了一個跨多個領域的現有工作和模型的有組織和全麵的概述。

關鍵詞:深度學習,自然語言處理,Transformer模型,注意力模型

介紹

Transformer是現代深度學習領域中一股強大的力量。Transformer無處不在,在語言理解、圖像處理等許多領域都產生了巨大的影響。因此,在過去的幾年裏,大量的研究致力於對該模型進行根本性的改進,這是很自然的。這種巨大的興趣也刺激了對該模式更高效變體的研究。

最近出現了大量的Transformer模型變體,研究人員和實踐者可能會發現跟上創新的速度很有挑戰性。在撰寫本文時,僅在過去6個月裏就提出了近12種新的以效率為中心的模式。因此,對現有文獻進行綜述,既有利於社區,又十分及時。

自注意力機製是確定Transformer模型的一個關鍵特性。該機製可以看作是一種類似圖的歸納偏差,它通過基於關聯的池化操作將序列中的所有標記連接起來。一個眾所周知的自注意力問題是二次時間和記憶複雜性,這可能阻礙模型在許多設置的可伸縮性。最近,為了解決這個問題,出現了大量的模型變體。以下我們將這類型號命名為“高效Transformers”。

根據上下文,可以對模型的效率進行不同的解釋。它可能指的是模型的內存占用情況,當模型運行的加速器的內存有限時,這一點非常重要。效率也可能指計算成本,例如,在訓練和推理期間的失敗次數。特別是對於設備上的應用,模型應該能夠在有限的計算預算下運行。在這篇綜述中,我們提到了Transformer在內存和計算方麵的效率,當它們被用於建模大型輸入時。

有效的自我注意力模型在建模長序列的應用中是至關重要的。例如,文檔、圖像和視頻通常都由相對大量的像素或標記組成。因此,處理長序列的效率對於Transformer的廣泛采用至關重要。

本篇綜述旨在提供這類模型的最新進展的全麵概述。我們主要關注的是通過解決自我注意力機製的二次複雜性問題來提高Transformer效率的建模進展和架構創新,我們還將在後麵的章節簡要討論一般改進和其他效率改進。

本文提出了一種高效Transformer模型的分類方法,並通過技術創新和主要用例對其進行了表征。特別地,我們回顧了在語言和視覺領域都有應用的Transformer模型,試圖對各個領域的文獻進行分析。我們還提供了許多這些模型的詳細介紹,並繪製了它們之間的聯係。

成為VIP會員查看完整內容
4
79
0

最近東北大學自然語言處理實驗室在Github上發布了自然語言處理與機器學習最新綜述論文合集,共有358篇之多,涵蓋ML&nlp眾多主題 , 是一份非常不錯的指南!

地址:https://github.com/NiuTrans/ABigSurvey#architectures

在本文中,我們調研了數百篇關於自然語言處理(NLP)和機器學習(ML)的綜述論文。我們將這些論文按熱門話題分類,並對一些有趣的問題進行簡單計算。此外,我們還顯示了論文的url列表(358篇論文)。

A Survey of Surveys (NLP & ML)

Natural Language Processing Lab., School of Computer Science and Engineering, Northeastern University

NiuTrans Research

In this document, we survey hundreds of survey papers on Natural Language Processing (NLP) and Machine Learning (ML). We categorize these papers into popular topics and do simple counting for some interesting problems. In addition, we show the list of the papers with urls (358 papers).

Categorization

We follow the ACL and ICML submission guideline of recent years, covering a broad range of areas in NLP and ML. The categorization is as follows:

To reduce class imbalance, we separate some of the hot sub-topics from the original categorization of ACL and ICML submissions. E.g., NER is a first-level area in our categorization because it is the focus of several surveys.

Statistics

We show the number of paper in each area in Figures 1-2.

Figure 1: # of papers in each NLP area.

Figure 2: # of papers in each ML area..

Also, we plot paper number as a function of publication year (see Figure 3).

Figure 3: # of papers vs publication year.

In addition, we generate word clouds to show hot topics in these surveys (see Figures 4-5).

Figure 4: The word cloud for NLP.

Figure 5: The word cloud for ML.

The NLP Paper List

Computational Social Science and Social Media

  1. Computational Sociolinguistics: A Survey.Computational Linguistics 2016paper

    Dong Nguyen, A Seza Dogruoz, Carolyn Penstein Rose, Franciska De Jong

Dialogue and Interactive Systems

  1. A Comparative Survey of Recent Natural Language Interfaces for Databases.VLDB 2019paper

    Katrin Affolter, Kurt Stockinger, Abraham Bernstein

  2. A Survey of Arabic Dialogues Understanding for Spontaneous Dialogues and Instant Message.arXiv 2015paper

    AbdelRahim A. Elmadany, Sherif M. Abdou, Mervat Gheith

  3. A Survey of Available Corpora for Building Data-Driven Dialogue Systems.arXiv 2015paper

    Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, Joelle Pineau

  4. A Survey of Document Grounded Dialogue Systems.arXiv 2020paper

    Longxuan Ma, Wei-Nan Zhang, Mingda Li, Ting Liu

  5. A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions.arXiv 2019paper

    Sashank Santhanam, Samira Shaikh

  6. A Survey on Dialog Management: Recent Advances and Challenges.arXiv 2020paper

    Yinpei Dai, Huihua Yu, Yixuan Jiang, Chengguang Tang, Yongbin Li, Jian Sun

  7. A Survey on Dialogue Systems: Recent Advances and New Frontiers.Sigkdd Explorations 2017paper

    Hongshen Chen, Xiaorui Liu, Dawei Yin, Jiliang Tang

  8. Challenges in Building Intelligent Open-domain Dialog Systems.arXiv 2019paper

    Minlie Huang, Xiaoyan Zhu, Jianfeng Gao

  9. Neural Approaches to Conversational AI.ACL 2018paper

    Jianfeng Gao, Michel Galley, Lihong Li

  10. Recent Advances and Challenges in Task-oriented Dialog System.arXiv 2020paper

    Zheng Zhang, Ryuichi Takanobu, Minlie Huang, Xiaoyan Zhu

Generation

  1. A bit of progress in language modeling.arXiv 2001paper

    Joshua T. Goodman

  2. A Survey of Paraphrasing and Textual Entailment Methods.Journal of Artificial Intelligence Research 2010paper

    Ion Androutsopoulos, Prodromos Malakasiotis

  3. A Survey on Neural Network Language Models.arXiv 2019paper

    Kun Jing, Jungang Xu

  4. Neural Text Generation: Past, Present and Beyond.arXiv 2018paper

    Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, Yong Yu

  5. Pre-trained Models for Natural Language Processing : A Survey.arXiv 2020paper

    Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang

  6. Recent Advances in Neural Question Generation.arXiv 2019paper

    Liangming Pan, Wenqiang Lei, Tat-Seng Chua, Min-Yen Kan

  7. Recent Advances in SQL Query Generation: A Survey.arXiv 2020paper

    Jovan Kalajdjieski, Martina Toshevska, Frosina Stojanovska

  8. Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation.Journal of Artificial Intelligence Research 2018paper

    Albert Gatt,Emiel Krahmer

Information Extraction

  1. A Survey of Deep Learning Methods for Relation Extraction.arXiv 2017paper

    Shantanu Kumar

  2. A Survey of Event Extraction From Text.IEEE 2019paper

    Wei Xiang, Bang Wang

  3. A Survey of Neural Network Techniques for Feature Extraction from Text.arXiv 2017paper

    Vineet John

  4. A Survey on Open Information Extraction.COLING 2018paper

    Christina Niklaus, Matthias Cetto, André Freitas, Siegfried Handschuh

  5. A Survey on Temporal Reasoning for Temporal Information Extraction from Text (Extended Abstract).arXiv 2019paper

    Artuur Leeuwenberg, Marie-Francine Moens

  6. Automatic Extraction of Causal Relations from Natural Language Texts: A Comprehensive Survey.arXiv 2016paper

    Nabiha Asghar

  7. Content Selection in Data-to-Text Systems: A Survey.arXiv 2016paper

    Dimitra Gkatzia

  8. Keyphrase Generation: A Multi-Aspect Survey.FRUCT 2019paper

    Erion Cano, Ondrej Bojar

  9. More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction.arXiv 2020paper

    Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou:

  10. Relation Extraction : A Survey.arXiv 2017paper

    Sachin Pawar, Girish K. Palshikar, Pushpak Bhattacharyya

  11. Short Text Topic Modeling Techniques, Applications, and Performance: A Survey.arXiv 2019paper

    Jipeng Qiang, Zhenyu Qian, Yun Li, Yunhao Yuan, Xindong Wu

Information Retrieval and Text Mining

  1. A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques.arXiv 2017paper

    Mehdi Allahyari, Seyed Amin Pouriyeh, Mehdi Assefi, Saied Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, Krys Kochut

  2. A survey of methods to ease the development of highly multilingual text mining applications.language resources and evaluation 2012paper

    Ralf Steinberger

  3. Opinion Mining and Analysis: A survey.IJNLC 2013paper

    Arti Buche, M. B. Chandak, Akshay Zadgaonkar

Interpretability and Analysis of Models for NLP

  1. Analysis Methods in Neural Language Processing: A Survey.NACCL 2018paper

    Yonatan Belinkov, James R. Glass

  2. Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop.EMNLP 2019paper

    Afra Alishahi, Grzegorz Chrupala, Tal Linzen

  3. Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models.arXiv 2020paper

    Viktor Schlegel, Goran Nenadic, Riza Batista-Navarro

  4. Visualizing Natural Language Descriptions: A Survey.ACM 2016paper

    Kaveh Hassani, Won-Sook Lee

  5. When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People?.ACL 2020paper

    Kenneth Joseph, Jonathan H. Morgan

Knowledge Graph

  1. A survey of techniques for constructing chinese knowledge graphs and their applications.mdpi 2018paper

    Tianxing Wu, Guilin Qi, Cheng Li, Meng Wang

  2. A Survey on Knowledge Graphs: Representation, Acquisition and Applications.arXiv 2020paper

    Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, Philip S. Yu:

  3. Knowledge Graph Embedding for Link Prediction: A Comparative Analysis.arXiv 2016paper

    Andrea Rossi, Donatella Firmani, Antonio Matinata, Paolo Merialdo, Denilson Barbosa

  4. Knowledge Graph Embedding: A Survey of Approaches and Applications.IEEE 2017paper

    Quan Wang, Zhendong Mao, Bin Wang, Li Guo

  5. Knowledge Graphs.arXiv 2020paper

    Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard de Melo, Claudio Gutierrez, José Emilio Labra Gayo, Sabrina Kirrane, Sebastian Neumaier, Axel Polleres, Roberto Navigli, Axel-Cyrille Ngonga Ngomo, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan F. Sequeda, Steffen Staab, Antoine Zimmermann

Language Grounding to Vision and Robotics and Beyond

  1. Emotionally-Aware Chatbots: A Survey.arXiv 2018paper

    Endang Wahyu Pamungkas

  2. Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods.arXiv 2019paper

    Aditya Mogadala, Marimuthu Kalimuthu, Dietrich Klakow

Linguistic Theories and Cognitive Modeling and Psycholinguistics

  1. Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing.Comput. Linguistics 45(3) 2019paper

    Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, Anna Korhonen

  2. Survey on the Use of Typological Information in Natural Language Processing.COLING 2016paper

    Helen O'Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, Anna Korhonen

Machine Learning for NLP

  1. A comprehensive survey of mostly textual document segmentation algorithms since 2008.Pattern Recognition 2017paper

    Sébastien Eskenazi, Petra Gomez-Kramer, Jean-Marc Ogier

  2. A Primer on Neural Network Models for Natural Language Processing.arXiv 2015paper

    Yoav Goldberg

  3. A Survey Of Cross-lingual Word Embedding Models.Journal of Artificial Intelligence Research 2019paper

    Sebastian Ruder, Ivan Vulic, Anders Sogaard

  4. A Survey of Neural Networks and Formal Languages.arXiv 2020paper

    Joshua Ackerman, George Cybenko

  5. A Survey of the Usages of Deep Learning in Natural Language Processing.IEEE 2018paper

    Daniel W. Otter, Julian R. Medina, Jugal K. Kalita

  6. A Survey on Contextual Embeddings.arXiv 2020paper

    Qi Liu, Matt J. Kusner, Phil Blunsom

  7. Adversarial Attacks and Defense on Texts: A Survey.arXiv 2020paper

    Aminul Huq, Mst. Tasnim Pervin

  8. Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey.arXiv 2019paper

    Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, Chenliang Li

  9. An Introductory Survey on Attention Mechanisms in NLP Problems.IntelliSys 2019paper

    Dichao Hu

  10. Attention in Natural Language Processing.arXiv 2019paper

    Andrea Galassi, Marco Lippi, Paolo Torroni

  11. From static to dynamic word representations: a survey.ICMLC 2020paper

    Yuxuan Wang, Yutai Hou, Wanxiang Che, Ting Liu

  12. From Word to Sense Embeddings: A Survey on Vector Representations of Meaning.Journal of Artificial Intelligence Research 2018paper

    Jose Camachocollados, Mohammad Taher Pilehvar

  13. Natural Language Processing Advancements By Deep Learning: A Survey.arXiv 2020paper

    Amirsina Torfi, Rouzbeh A. Shirvani, Yaser Keneshloo, Nader Tavvaf, Edward A. Fox

  14. Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering.COLING 2018paper

    Wuwei Lan,Wei Xu

  15. Recent Trends in Deep Learning Based Natural Language Processing.IEEE 2018paper

    Tom Young, Devamanyu Hazarika, Soujanya Poria, Erik Cambria

  16. Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey.arXiv 2017paper

    Lorenzo Ferrone, Fabio Massimo Zanzotto

  17. Towards a Robust Deep Neural Network in Texts: A Survey.arXiv 2020paper

    Wenqi Wang, Lina Wang, Run Wang, Zhibo Wang, Aoshuang Ye

  18. Word Embeddings: A Survey.arXiv 2019paper

    Felipe Almeida, Geraldo Xexéo

Machine Translation

  1. A Brief Survey of Multilingual Neural Machine Translation.arXiv 2019paper

    Raj Dabre, Chenhui Chu, Anoop Kunchukuttan

  2. A Comprehensive Survey of Multilingual Neural Machine Translation.arXiv 2020paper

    Raj Dabre, Chenhui Chu, Anoop Kunchukuttan

  3. A Survey of Deep Learning Techniques for Neural Machine Translation.arXiv 2020paper

    Shuoheng Yang, Yuxin Wang, Xiaowen Chu

  4. A Survey of Domain Adaptation for Neural Machine Translation.COLING 2018paper

    Chenhui Chu, Rui Wang

  5. A Survey of Methods to Leverage Monolingual Data in Low-resource Neural Machine Translation.arXiv 2019paper

    Ilshat Gibadullin, Aidar Valeev, Albina Khusainova, Adil Mehmood Khan

  6. A Survey of Multilingual Neural Machine Translation.arXiv 2020paper

    Raj Dabre, Chenhui Chu, Anoop Kunchukuttan

  7. A Survey of Word Reordering in Statistical Machine Translation: Computational Models and Language Phenomena.Comput Linguistics 2016paper

    Arianna Bisazza, Marcello Federico

  8. A Survey on Document-level Machine Translation: Methods and Evaluation.arXiv 2019paper

    Sameen Maruf, Fahimeh Saleh, Gholamreza Haffari

  9. Machine Translation Approaches and Survey for Indian Languages.arXiv 2017paper

    Nadeem Jadoon Khan, Waqas Anwar, Nadir Durrani

  10. Machine Translation Evaluation Resources and Methods: A Survey.arXiv 2018paper

    Lifeng Han

  11. Machine Translation using Semantic Web Technologies: A Survey.Journal of Web Semantics 2018paper

    Diego Moussallem, Matthias Wauer, Axelcyrille Ngonga Ngomo

  12. Machine-Translation History and Evolution: Survey for Arabic-English Translations.arXiv 2017paper

    Nabeel T. Alsohybe, Neama Abdulaziz Dahan, Fadl Mutaher Baalwi

  13. Neural Machine Translation and Sequence-to-Sequence Models: A Tutorial.arXiv 2017paper

    Graham Neubig

  14. Neural Machine Translation: A Review.arXiv 2019paper

    Felix Stahlberg

  15. Neural Machine Translation: Challenges, Progress and Future.arXiv 2020paper

    Jiajun Zhang, Chengqing Zong

  16. The Query Translation Landscape: a Survey.arXiv 2019paper

    Mohamed Nadjib Mami, Damien Graux, Harsh Thakkar, Simon Scerri, Soren Auer, Jens Lehmann

Natural Language Processing

  1. A Survey and Classification of Controlled Natural Languages.Comput. Linguistics 2014paper

    Tobias Kuhn

  2. Jumping NLP curves: A review of natural language processing research.IEEE 2014paper

    Erik Cambria ; Bebo White

  3. Natural Language Processing - A Survey.arXiv 2012paper

    Kevin Mote

  4. Natural Language Processing: State of The Art, Current Trends and Challenges.arXiv 2017paper

    Diksha Khurana, Aditya Koli, Kiran Khatter, Sukhdev Singh

NER

  1. A survey of named entity recognition and classification.Lingvistic Investigationes 2007paper

    David Nadeau, Satoshi Sekine

  2. A Survey of Named Entity Recognition in Assamese and other Indian Languages.arXiv 2014paper

    Gitimoni Talukdar, Pranjal Protim Borah, Arup Baruah

  3. A Survey on Deep Learning for Named Entity Recognition.arXiv 2018paper

    Jing Li, Aixin Sun, Jianglei Han, Chenliang Li

  4. A Survey on Recent Advances in Named Entity Recognition from Deep Learning models.COLING 2019paper

    Vikas Yadav, Steven Bethard

  5. Design Challenges and Misconceptions in Neural Sequence Labeling.COLING 2018paper

    Jie Yang, Shuailong Liang, Yue Zhang

  6. Neural Entity Linking: A Survey of Models based on Deep Learning.arXiv 2020paper

    Ozge Sevgili, Artem Shelmanov, Mikhail Arkhipov, Alexander Panchenko, Chris Biemann

NLP Applications

  1. A Comprehensive Survey of Grammar Error arXivection.arXiv 2020paper

    Yu Wang, Yuelin Wang, Jie Liu, Zhuo Liu

  2. A Short Survey of Biomedical Relation Extraction Techniques.arXiv 2017paper

    Elham Shahab

  3. A Survey on Natural Language Processing for Fake News Detection.LREC 2020paper

    Ray Oshikawa, Jing Qian, William Yang Wang

  4. Automatic Language Identification in Texts: A Survey.J. Artif. Intell. Res. 65 2019paper

    Tommi Jauhiainen

  5. Disinformation Detection: A review of linguistic feature selection and classification models in news veracity assessments.arXiv 2019paper

    Jillian Tompkins

  6. Extraction and Analysis of Fictional Character Networks: A Survey.ACM 2019paper

    Xavier Bost (LIA), Vincent Labatut (LIA)

  7. Fake News Detection using Stance Classification: A Survey.arXiv 2019paper

    Anders Edelbo Lillie, Emil Refsgaard Middelboe

  8. Fake News: A Survey of Research, Detection Methods, and Opportunities.ACM 2018paper

    Xinyi Zhou, Reza Zafarani

  9. Image Captioning based on Deep Learning Methods: A Survey.arXiv 2019paper

    Yiyu Wang, Jungang Xu, Yingfei Sun, Ben He

  10. SECNLP: A Survey of Embeddings in Clinical Natural Language Processing.J. Biomed. Informatics 2019paper

    Kalyan KS, S Sangeetha

  11. Survey of Text-based Epidemic Intelligence: A Computational Linguistic Perspective.ACM 2019paper

    Aditya Joshi, Sarvnaz Karimi, Ross Sparks, Cecile Paris, C Raina MacIntyre

  12. Text Detection and Recognition in the Wild: A Review.arXiv 2020paper

    Zobeir Raisi, Mohamed A. Naiel, Paul Fieguth, Steven Wardell, John Zelek

  13. Text Recognition in the Wild: A Survey.arXiv 2020paper

    Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, Tianwei Wang

Question Answering

  1. A survey on question answering technology from an information retrieval perspective.Information Sciences 2011paper

    Oleksandr Kolomiyets, Marie-Francine Moens:

  2. A Survey on Why-Type Question Answering Systems.arXiv 2019paper

    Manvi Breja, Sanjay Kumar Jain:

  3. Core techniques of question answering systems over knowledge bases: a survey.SpringerLink 2017paper

    Dennis Diefenbach, Vanessa Lopez, Kamal Singh & Pierre Maret

  4. Introduction to Neural Network based Approaches for Question Answering over Knowledge Graphs.arXiv 2019paper

    Nilesh Chakraborty,Denis Lukovnikov,Gaurav Maheshwari,Priyansh Trivedi,Jens Lehmann,Asja Fischer:

  5. Survey of Visual Question Answering: Datasets and Techniques.arXiv 2017paper

    Akshay Kumar Gupta

  6. Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey.arXiv 2020paper

    Zahra Abbasiyantaeb, Saeedeh Momtazi:

  7. Tutorial on Answering Questions about Images with Deep Learning.arXiv 2016paper

    Mateusz Malinowski, Mario Fritz:

  8. Visual Question Answering using Deep Learning: A Survey and Performance Analysis.arXiv 2019paper

    Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, Snehasis Mukherjee:

Reading Comprehension

  1. A Survey on Machine Reading Comprehension Systems.arXiv 2020paper

    Razieh Baradaran, Razieh Ghiasi, Hossein Amirkhani:

  2. A Survey on Neural Machine Reading Comprehension.arXiv 2019paper

    Boyu Qiu, Xu Chen, Jungang Xu, Yingfei Sun:

  3. Machine Reading Comprehension: a Literature Review.arXiv 2019paper

    Xin Zhang, An Yang, Sujian Li, Yizhong Wang

  4. Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond.arXiv 2020paper

    Zhuosheng Zhang, Hai Zhao, Rui Wang

  5. Neural Machine Reading Comprehension: Methods and Trends.arXiv 2019paper

    Shanshan Liu, Xin Zhang, Sheng Zhang, Hui Wang, Weiming Zhang:

Recommender Systems

  1. A review on deep learning for recommender systems: challenges and remedies.SpringerLink 2019paper

    Zeynep Batmaz, Ali Yurekli, Alper Bilge, Cihan Kaleli:

  2. A Survey on Knowledge Graph-Based Recommender Systems.arXiv 2020paper

    Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong, Qing He

  3. Adversarial Machine Learning in Recommender Systems: State of the art and Challenges.ACM 2020paper

    Yashar Deldjoo, Tommaso Di Noia, Felice Antonio Merra

  4. Cross Domain Recommender Systems: A Systematic Literature Review.ACM 2017paper

    Muhammad Murad Khan,Roliana Ibrahim,Imran Ghani

  5. Deep Learning based Recommender System: A Survey and New Perspectives.ACM 2019paper

    Shuai Zhang, Lina Yao, Aixin Sun, Yi Tay:

  6. Deep Learning on Knowledge Graph for Recommender System: A Survey.ACM 2020paper

    Yang Gao, Yi-Fan Li, Yu Lin, Hang Gao, Latifur Khan

  7. Explainable Recommendation: A Survey and New Perspectives.arXiv 2020paper

    Yongfeng Zhang, Xu Chen:

  8. Sequence-Aware Recommender Systems.ACM 2018paper

    Massimo Quadrana,Paolo Cremonesi,Dietmar Jannach

  9. Use of Deep Learning in Modern Recommendation System: A Summary of Recent Works.arXiv 2017paper

    Ayush Singhal, Pradeep Sinha, Rakesh Pant:

Resources and Evaluation

  1. A Short Survey on Sense-Annotated Corpora.LREC 2020paper

    Tommaso Pasini, José Camacho-Collados:

  2. A Survey of Current Datasets for Vision and Language Research.EMNLP 2015paper

    Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao (Kenneth) Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, Margaret Mitchell:

  3. A Survey of Word Embeddings Evaluation Methods.arXiv 2018paper

    Amir Bakarov

  4. Critical Survey of the Freely Available Arabic Corpora.arXiv 2017paper

    Wajdi Zaghouani:

  5. Distributional Measures of Semantic Distance: A Survey.arXiv 2012paper

    Saif Mohammad, Graeme Hirst:

  6. Measuring Sentences Similarity: A Survey.Indian Journal of Science and Technology 2019paper

    Mamdouh Farouk:

  7. Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches.arXiv 2020paper

    Shane Storks, Qiaozi Gao, Joyce Y. Chai

  8. Survey on Evaluation Methods for Dialogue Systems.arXiv 2019paper

    Jan Deriu, álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, Mark Cieliebak:

  9. Survey on Publicly Available Sinhala Natural Language Processing Tools and Research.arXiv 2019paper

    Nisansa de Silva

Semantics

  1. Diachronic word embeddings and semantic shifts: a survey.COLING 2018paper

    Andrey Kutuzov, Lilja Ovrelid, Terrence Szymanski, Erik Velldal

  2. Evolution of Semantic Similarity -- A Survey.arXiv 2020paper

    Dhivya Chandrasekaran, Vijay Mago

  3. Semantic search on text and knowledge bases.Foundations and trends in information retrieval 2016paper

    Hannah Bast , Bjorn Buchhold, Elmar Haussmann

  4. Semantics, Modelling, and the Problem of Representation of Meaning -- a Brief Survey of Recent Literature.arXiv 2014paper

    Yarin Gal

  5. Survey of Computational Approaches to Lexical Semantic Change.arXiv 2019paper

    Nina Tahmasebi, Lars Borin, Adam Jatowt

  6. Word sense disambiguation: a survey.ACM 2015paper

    Alok Ranjan Pal, Diganta Saha

Sentiment Analysis and Stylistic Analysis and Argument Mining

  1. A Comprehensive Survey on Aspect Based Sentiment Analysis.arXiv 2020paper

    Kaustubh Yadav

  2. A Survey on Sentiment and Emotion Analysis for Computational Literary Studies.arXiv 2018paper

    Evgeny Kim, Roman Klinger

  3. Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research.arXiv 2020paper

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Rada Mihalcea

  4. Deep Learning for Aspect-Level Sentiment Classification: Survey, Vision, and Challenges.IEEE 2019paper

    Jie Zhou, Jimmy Xiangji Huang, Qin Chen, Qinmin Vivian Hu, Tingting Wang, Liang He

  5. Deep Learning for Sentiment Analysis : A Survey.Wiley Interdisciplinary Reviews-Data Mining and Knowledge Discovery 2018paper

    Lei Zhang, Shuai Wang, Bing Liu

  6. Sentiment analysis for Arabic language: A brief survey of approaches and techniques.arXiv 2018paper

    Mo'ath Alrefai, Hossam Faris, Ibrahim Aljarah

  7. Sentiment Analysis of Czech Texts: An Algorithmic Survey.ICAART 2019paper

    Erion Cano, Ondrej Bojar

  8. Sentiment Analysis of Twitter Data: A Survey of Techniques.arXiv 2016paper

    Vishal.A.Kharde, Prof. Sheetal.Sonawane

  9. Sentiment Analysis on YouTube: A Brief Survey.arXiv 2015paper

    Muhammad Zubair Asghar, Shakeel Ahmad, Afsana Marwat, Fazal Masud Kundi

  10. Sentiment/Subjectivity Analysis Survey for Languages other than English.Social Netw. Analys. Mining 2016paper

    Mohammed Korayem, Khalifeh Aljadda, David Crandall

  11. Word Embeddings for Sentiment Analysis: A Comprehensive Empirical Survey.arXiv 2019paper

    Erion Cano, Maurizio Morisio

Speech and Multimodality

  1. A Comprehensive Survey on Cross-modal Retrieval.arXiv 2016paper

    Kaiye Wang

  2. A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis.arXiv 2019paper

    Jorge Agnese, Jonathan Herrera, Haicheng Tao, Xingquan Zhu

  3. A Survey of Code-switched Speech and Language Processing.arXiv 2019paper

    Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, Alan W. Black

  4. A Survey of Recent DNN Architectures on the TIMIT Phone Recognition Task.TSD 2018paper

    Josef Michálek, Jan Vanek

  5. A Survey of Voice Translation Methodologies - Acoustic Dialect Decoder.arXiv 2016paper

    Hans Krupakar, Keerthika Rajvel, Bharathi B, Angel Deborah S, Vallidevi Krishnamurthy

  6. Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures.IJCAI 2017paper

    Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank

  7. Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems.arXiv 2019paper

    Laura von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, Jannis Schuecker

  8. Multimodal Machine Learning: A Survey and Taxonomy.IEEE 2019paper

    Tadas Baltrusaitis, Chaitanya Ahuja, Louis-Philippe Morency

  9. Speech and Language Processing.Stanford University 2019paper

    Dan Jurafsky and James H. Martin

Summarization

  1. A Survey on Neural Network-Based Summarization Methods.arXiv 2018paper

    Yue Dong

  2. Abstractive Summarization: A Survey of the State of the Art.AAAI 2019paper

    Hui Lin, Vincent Ng

  3. Automated text summarisation and evidence-based medicine: A survey of two domains.arXiv 2017paper

    Abeed Sarker, Diego Mollá Aliod, Cécile Paris

  4. Automatic Keyword Extraction for Text Summarization: A Survey.arXiv 2017paper

    Santosh Kumar Bharti, Korra Sathya Babu

  5. From Standard Summarization to New Tasks and Beyond: Summarization with Manifold Information.arXiv 2020paper

    Shen Gao, Xiuying Chen, Zhaochun Ren, Dongyan Zhao, Rui Yan

  6. Neural Abstractive Text Summarization with Sequence-to-Sequence Models: A Survey.arXiv 2018paper

    Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, Chandan K. Reddy

  7. Recent automatic text summarization techniques: a survey.Artificial Intelligence Review 2016paper

    Mahak Gambhir, Vishal Gupta

  8. Text Summarization Techniques: A Brief Survey.IJCAI 2017paper

    Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, Krys Kochut

Tagging Chunking Syntax and Parsing

  1. A Neural Entity Coreference Resolution Review.arXiv 2019paper

    Nikolaos Stylianou, Ioannis Vlahavas

  2. A survey of cross-lingual features for zero-shot cross-lingual semantic parsing.arXiv 2019paper

    Jingfeng Yang, Federico Fancellu, Bonnie L. Webber

  3. A Survey on Semantic Parsing.AKBC 2019paper

    Aishwarya Kamath, Rajarshi Das

  4. The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers.arXiv 2018paper

    Dongxiang Zhang, Lei Wang, Nuo Xu, Bing Tian Dai, Heng Tao Shen

Text Classification

  1. A Survey of Naive Bayes Machine Learning approach in Text Document Classification.IJCSIS 2010paper

    K. A. Vidhya, G. Aghila

  2. A survey on phrase structure learning methods for text classification.IJNLC 2014paper

    Reshma Prasad, Mary Priya Sebastian

  3. Deep Learning Based Text Classification: A Comprehensive Review.arXiv 2020paper

    Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, Jianfeng Gao

  4. Text Classification Algorithms: A Survey.arXiv 2019paper

    Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura E. Barnes, Donald E. Brown

The ML Paper List

Architectures

  1. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects.arXiv 2020paper

    Zewen Li, Wenjie Yang, Shouheng Peng, Fan Liu

  2. A Survey of End-to-End Driving: Architectures and Training Methods.arXiv 2020paper

    Ardi Tampuu, Maksym Semikin, Naveed Muhammad, Dmytro Fishman, Tambet Matiisen

  3. A Survey on Latent Tree Models and Applications.Journal of Artificial Intelligence Research 2013paper

    Raphael Mourad, Christine Sinoquet, Nevin L. Zhang, Tengfei Liu, Philippe Leray

  4. An Attentive Survey of Attention Models.arXiv 2019paper

    Sneha Chaudhari, Gungor Polatkan, Rohan Ramanath, Varun Mithal

  5. Binary Neural Networks: A Survey.Pattern Recognition 2020paper

    Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, Nicu Sebe

  6. Deep Echo State Network (DeepESN): A Brief Survey.arXiv 2017paper

    Claudio Gallicchio, Alessio Micheli

  7. Recent Advances in Convolutional Neural Networks.Pattern Recognition 2018paper

    Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, Tsuhan Chen

  8. Sum-product networks: A survey.arXiv 2020paper

    Iago París, Raquel Sánchez-Cauce, Francisco Javier Díez

  9. Survey on the attention based RNN model and its applications in computer vision.arXiv 2016paper

    Feng Wang, David M. J. Tax

  10. Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks.arXiv 2019paper

    Ralf C. Staudemeyer, Eric Rothstein Morris

AutoML

  1. A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions.arXiv 2020paper

    Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, Xin Wang

  2. A Survey on Neural Architecture Search.arXiv 2019paper

    Martin Wistuba, Ambrish Rawat, Tejaswini Pedapati

  3. AutoML: A Survey of the State-of-the-Art.arXiv 2019paper

    Xin He, Kaiyong Zhao, Xiaowen Chu

  4. Benchmark and Survey of Automated Machine Learning Frameworks.arXiv 2020paper

    Marc-André Zoller, Marco F. Huber

  5. Neural Architecture Search: A Survey.Journal of Machine Learning Research 2019paper

    Thomas Elsken, Jan Hendrik Metzen, Frank Hutter

Bayesian Methods

  1. A survey of non-exchangeable priors for Bayesian nonparametric models.IEEE 2015paper

    Nicholas J. Foti, Sinead Williamson

  2. Bayesian Nonparametric Space Partitions: A Survey.arXiv 2020paper

    Xuhui Fan, Bin Li, Ling Luo, Scott A. Sisson

  3. Towards Bayesian Deep Learning: A Survey.arXiv 2016paper

    Hao Wang, Dityan Yeung

Classification Clustering and Regression

  1. A Survey of Classification Techniques in the Area of Big Data.arXiv 2015paper

    Praful Koturwar, Sheetal Girase, Debajyoti Mukhopadhyay

  2. A Survey of Constrained Gaussian Process Regression: Approaches and Implementation Challenges.arXiv 2020paper

    Laura P. Swiler, Mamikon Gulian, Ari Frankel, Cosmin Safta, John D. Jakeman

  3. A Survey on Multi-View Clustering.arXiv 2017paper

    Guoqing Chao, Shiliang Sun, Jinbo Bi

  4. Deep learning for time series classification: a review.arXiv 2019paper

    Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller

  5. How Complex is your classification problem? A survey on measuring classification complexity.ACM 2019paper

    Ana Carolina Lorena, Luis P F Garcia, Jens Lehmann, Marcilio C P Souto, Tin K Ho

Curriculum Learning

  1. Automatic Curriculum Learning For Deep RL: A Short Survey.arXiv 2020paper

    Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer

  2. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey.arXiv 2020paper

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone

Data Augmentation

  1. A survey on Image Data Augmentation for Deep Learning.Journal of Big Data 2019paper

    Connor Shorten

  2. Time Series Data Augmentation for Deep Learning: A Survey.arXiv 2020paper

    Qingsong Wen, Liang Sun, Xiaomin Song, Jingkun Gao, Xue Wang, Huan Xu

Deep Learning - General Methods

  1. A Survey of Neuromorphic Computing and Neural Networks in Hardware.arXiv 2017paper

    Catherine D. Schuman, Thomas E. Potok, Robert M. Patton, J. Douglas Birdwell, Mark E. Dean, Garrett S. Rose, James S. Plank

  2. A Survey on Deep Hashing Methods.arXiv 2020paper

    Xiao Luo, Chong Chen, Huasong Zhong, Hao Zhang, Minghua Deng, Jianqiang Huang, Xiansheng Hua

  3. A survey on modern trainable activation functions.arXiv 2020paper

    Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, Roberto Prevete

  4. Convergence of Edge Computing and Deep Learning: A Comprehensive Survey.IEEE 2020paper

    Xiaofei Wang, Yiwen Han, Victor C.M. Leung, Dusit Niyato, Xueqiang Yan, Xu Chen

  5. Deep learning.nature 2015paper

    Yann LeCun

  6. Deep Learning on Graphs: A Survey.IEEE 2018paper

    Ziwei Zhang, Peng Cui, Wenwu Zhu

  7. Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective.arXiv 2019paper

    Guan-Horng Liu, Evangelos A. Theodorou

  8. Geometric Deep Learning: Going beyond Euclidean data.IEEE 2016paper

    Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, Pierre Vandergheynst

  9. Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey.arXiv 2020paper

    Andrea Borghesi, Federico Baldo, Michela Milano

  10. Review: Ordinary Differential Equations For Deep Learning.arXiv 2019paper

    Xinshi Chen

  11. Survey of Dropout Methods for Deep Neural Networks.arXiv 2019paper

    Alex Labach, Hojjat Salehinejad, Shahrokh Valaee

  12. Survey of Expressivity in Deep Neural Networks.arXiv 2016paper

    Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohldickstein

  13. Survey of reasoning using Neural networks.arXiv 2017paper

    Amit Sahu

  14. The Deep Learning Compiler: A Comprehensive Survey.arXiv 2020paper

    Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Depei Qian

  15. The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches.arXiv 2018paper

    Zahangir Alom, Tarek M Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S Awwal, Vijayan K Asari

  16. Time Series Forecasting With Deep Learning: A Survey.arXiv 2020paper

    Bryan Lim, Stefan Zohren

Deep Reinforcement Learning

  1. A Brief Survey of Deep Reinforcement Learning.arXiv 2017paper

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, Anil A Bharath

  2. A Short Survey On Memory Based Reinforcement Learning.arXiv 2019paper

    Dhruv Ramani

  3. A Short Survey on Probabilistic Reinforcement Learning.arXiv 2019paper

    Reazul Hasan Russel

  4. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress.arXiv 2018paper

    Saurabh Arora, Prashant Doshi

  5. A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments.arXiv 2020paper

    Sindhu Padakandla

  6. A Survey of Reinforcement Learning Informed by Natural Language.IJCAI 2019paper

    Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktaschel

  7. A Survey of Reinforcement Learning Techniques: Strategies, Recent Development, and Future Directions.arXiv 2020paper

    Amit Kumar Mondal

  8. A survey on intrinsic motivation in reinforcement learning.arXiv 2019paper

    Aubret, Arthur, Matignon, Laetitia, Hassas, Salima

  9. A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots.arXiv 2019paper

    Nicolai A. Lynnerup, Laura Nolling, Rasmus Hasle, John Hallam

  10. Deep Reinforcement Learning: An Overview.arXiv 2017paper

    Yuxi Li

  11. Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementations.IEEE 2019paper

    Dimitri P. Bertsekas

Federated Learning

  1. A Survey towards Federated Semi-supervised Learning.FRUCT 2020paper

    Yilun Jin, Xiguang Wei, Yang Liu, Qiang Yang

  2. Advances and Open Problems in Federated Learning.arXiv 2019paper

    Peter Kairouz, H Brendan Mcmahan, Brendan Avent, Aurelien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G L Doliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary A Garrett, Adria Gascon, Badih Ghazi, Phillip B Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecny, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrede Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Ozgur, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramer, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X Yu, Han Yu, Sen Zhao

  3. Threats to Federated Learning: A Survey.CoRL 2019 2020paper

    Lingjuan Lyu, Han Yu, Qiang Yang

  4. Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective.arXiv 2020paper

    Yilun Jin, Xiguang Wei, Yang Liu, Qiang Yang

Few-Shot and Zero-Shot Learning

  1. A Survey of Zero-Shot Learning: Settings, Methods, and Applications.ACM 2019paper

    Wei Wang,Vincent W. Zheng,Han Yu,Chunyan Miao

  2. Few-shot Learning: A Survey.arXiv 2019paper

    Yaqing Wang, Quanming Yao

  3. Generalizing from a Few Examples: A Survey on Few-Shot Learning.ACM 2019paper

    Yaqing Wang, Quanming Yao, James Kwok, Lionel M. Ni

General Machine Learning

  1. A survey of dimensionality reduction techniques.arXiv 2014paper

    C.O.S. Sorzano, J. Vargas, A. Pascual Montano

  2. A Survey of Predictive Modelling under Imbalanced Distributions.arXiv 2015paper

    Paula Branco, Luis Torgo, Rita Ribeiro

  3. A Survey on Activation Functions and their relation with Xavier and He Normal Initialization.arXiv 2020paper

    Leonid Datta

  4. A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective.arXiv 2018paper

    Yuji Roh, Geon Heo, Steven Euijong Whang

  5. A survey on feature weighting based K-Means algorithms.Journal of Classification 2016paper

    Renato Cordeiro de Amorim

  6. A Survey on Graph Kernels.Applied Network Science 2020paper

    Nils M. Kriege, Fredrik D. Johansson, Christopher Morris

  7. A Survey on Multi-output Learning.IEEE 2019paper

    Donna Xu, Yaxin Shi, Ivor W. Tsang, Yew-Soon Ong, Chen Gong, Xiaobo Shen

  8. A Survey on Resilient Machine Learning.arXiv 2017paper

    Atul Kumar, Sameep Mehta

  9. A Survey on Surrogate Approaches to Non-negative Matrix Factorization.Vietnam journal of mathematics 2018paper

    Pascal Fernsel, Peter Maass

  10. A Tutorial on Network Embeddings.arXiv 2018paper

    Haochen Chen, Bryan Perozzi, Rami Al-Rfou, Steven Skiena

  11. Adversarial Examples in Modern Machine Learning: A Review.arXiv 2019paper

    Rey Reza Wiyatno, Anqi Xu, Ousmane Dia, Archy de Berker

  12. Algorithms Inspired by Nature: A Survey.arXiv 2019paper

    Pranshu Gupta

  13. Deep Tree Transductions - A Short Survey.INNSBDDL 2019paper

    Davide Bacciu, Antonio Bruno

  14. Graph Representation Learning: A Survey.APSIPA Transactions on Signal and Information Processing 2019paper

    Fenxiao Chen, Yuncheng Wang, Bin Wang, C.-C. Jay Kuo

  15. Heuristic design of fuzzy inference systems: A review of three decades of research.Engineering Applications of Artificial Intelligence 2019paper

    Varun Ojha, Ajith Abraham, Vaclav Snasel

  16. Hierarchical Mixtures-of-Experts for Exponential Family Regression Models with Generalized Linear Mean Functions: A Survey of Approximation and Consistency Results.arXiv 2013paper

    Wenxin Jiang, Martin A. Tanner

  17. Hyperbox based machine learning algorithms: A comprehensive survey.arXiv 2019paper

    Thanh Tung Khuat, Dymitr Ruta, Bogdan Gabrys

  18. Imbalance Problems in Object Detection: A Review.IEEE 2019paper

    Kemal Oksuz, Baris Can Cam, Sinan Kalkan, Emre Akbas

  19. Learning Representations of Graph Data -- A Survey.arXiv 2019paper

    Mital Kinderkhedia

  20. Machine Learning at the Network Edge: A Survey.arXiv 2020paper

    M.G. Sarwar Murshed, Christopher Murphy, Daqing Hou, Nazar Khan, Ganesh Ananthanarayanan, Faraz Hussain

  21. Machine Learning for Spatiotemporal Sequence Forecasting: A Survey.arXiv 2018paper

    Xingjian Shi, Dit-Yan Yeung

  22. Machine Learning in Network Centrality Measures: Tutorial and Outlook.Association for Computing Machinery 2018paper

    Felipe Grando, Lisandro Zambenedetti Granville, Luís C. Lamb

  23. Machine Learning Testing: Survey, Landscapes and Horizons.arXiv 2019paper

    Jie M. Zhang, Mark Harman, Lei Ma, Yang Liu

  24. Machine Learning with World Knowledge: The Position and Survey.arXiv 2017paper

    Yangqiu Song, Dan Roth

  25. Mean-Field Learning: a Survey.arXiv 2012paper

    Hamidou Tembine, Raúl Tempone, Pedro Vilanova

  26. Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey.Autonomous Agents and Multi-Agent Systems 2020paper

    Roxana Radulescu, Patrick Mannion, Diederik M. Roijers, Ann Nowé

  27. Narrative Science Systems: A Review.International Journal of Research in Computer Science 2015paper

    Paramjot Kaur Sarao, Puneet Mittal, Rupinder Kaur

  28. Network Representation Learning: A Survey.IEEE 2020paper

    Daokun Zhang, Jie Yin, Xingquan Zhu, Chengqi Zhang

  29. Relational inductive biases, deep learning, and graph networks.arXiv 2018paper

    Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gül?ehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, Razvan Pascanu

  30. Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey.JMLR 2019paper

    Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, Pascal Poupart

  31. Statistical Queries and Statistical Algorithms: Foundations and Applications.arXiv 2020paper

    Lev Reyzin

  32. Structure Learning of Probabilistic Graphical Models: A Comprehensive Survey.arXiv 2011paper

    Yang Zhou

  33. Survey on Feature Selection.arXiv 2015paper

    Tarek Amr Abdallah, Beatriz de La Iglesia

  34. Survey on Five Tribes of Machine Learning and the Main Algorithms.Software Guide 2019paper

    LI Xu-ran, DING Xiao-hong

  35. Survey: Machine Learning in Production Rendering.arXiv 2020paper

    Shilin Zhu

  36. The Benefits of Population Diversity in Evolutionary Algorithms: A Survey of Rigorous Runtime Analyses.arXiv 2018paper

    Dirk Sudholt

  37. Tutorial on Variational Autoencoders.arXiv 2016paper

    Carl Doersch

  38. Unsupervised Cross-Lingual Representation Learning.ACL 2019paper

    Sebastian Ruder, Anders Sogaard, Ivan Vulic

  39. Verification for Machine Learning, Autonomy, and Neural Networks Survey.arXiv 2018paper

    Weiming Xiang, Patrick Musau, Ayana A. Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, Taylor T. Johnson

Generative Adversarial Networks

  1. A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications.arXiv 2020paper

    Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, Jieping Ye

  2. A Survey on Generative Adversarial Networks: Variants, Applications, and Training.arXiv 2020paper

    Abdul Jabbar, Xi Li, Bourahla Omar

  3. Generative Adversarial Networks: A Survey and Taxonomy.arXiv 2019paper

    Zhengwei Wang, Qi She, Tomas E Ward

  4. Generative Adversarial Networks: An Overview.IEEE 2018paper

    Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, Anil A Bharath

  5. How Generative Adversarial Nets and its variants Work: An Overview of GAN.arXiv 2017paper

    Yongjun Hong, Uiwon Hwang, Jaeyoon Yoo, Sungroh Yoon

  6. Stabilizing Generative Adversarial Network Training: A Survey.arXiv 2020paper

    Maciej Wiatrak, Stefano V. Albrecht, Andrew Nystrom

  7. Stabilizing Generative Adversarial Networks: A Survey.arXiv 2019paper

    Maciej Wiatrak, Stefano V. Albrecht, Andrew Nystrom

Graph Neural Networks

  1. A Comprehensive Survey on Graph Neural Networks.IEEE 2019paper

    Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, Philip S. Yu

  2. A Survey on The Expressive Power of Graph Neural Networks.arXiv 2020paper

    Ryoma Sato

  3. Adversarial Attack and Defense on Graph Data: A Survey.arXiv 2018paper

    Lichao Sun, Ji Wang, Philip S. Yu, Bo Li

  4. Bridging the Gap between Spatial and Spectral Domains: A Survey on Graph Neural Networks.arXiv 2020paper

    Zhiqian Chen, Fanglan Chen, Lei Zhang, Taoran Ji, Kaiqun Fu, Liang Zhao, Feng Chen, Chang-Tien Lu

  5. Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A survey.arXiv 2020paper

    Joakim Skarding, Bogdan Gabrys, Katarzyna Musial

  6. Graph embedding techniques, applications, and performance: A survey.Knowledge Based Systems 2017paper

    Palash Goyal, Emilio Ferrara

  7. Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspective.arXiv 2020paper

    Luis C. Lamb, Artur Garcez, Marco Gori, Marcelo Prates, Pedro Avelar, Moshe Vardi

  8. Graph Neural Networks: A Review of Methods and Applications.arXiv 2018paper

    Maosong Sun, Zhengyan Zhang, Ganqu Cui, Cheng Yang, Jie Zhou, Zhiyuan Liu

  9. Introduction to Graph Neural Networks.IEEE 2020paper

    Zhiyuan Liu, Jie Zhou

  10. Tackling Graphical NLP problems with Graph Recurrent Networks.arXiv 2019paper

    Linfeng Song

Interpretability and Analysis

  1. A Survey Of Methods For Explaining Black Box Models.ACM 2018paper

    Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, Dino Pedreschi

  2. A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability.arXiv 2018paper

    Xiaowei Huang

  3. Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation.Sigkdd Explorations 2020paper

    Raha Moraffah, Mansooreh Karami, Ruocheng Guo, Adrienne Raglin, Huan Liu

  4. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI.Information Fusion 2020paper

    Alejandro Barredo Arrieta, Natalia Diazrodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gillopez, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera

  5. Explainable Reinforcement Learning: A Survey.CD-MAKE 2020 2020paper

    Erika Puiutta, Eric M. S. P. Veith

  6. Foundations of Explainable Knowledge-Enabled Systems.Knowledge Graphs for eXplainable Artificial Intelligence: Foundations, Applications and Challenges/arXiv 2020paper

    Shruthi Chari

  7. How Generative Adversarial Networks and Their Variants Work: An Overview.ACM 2017paper

    Yongjun Hong, Uiwon Hwang, Jaeyoon Yoo, Sungroh Yoon

  8. Language (Technology) is Power: A Critical Survey of "Bias" in NLP.Association for Computational Linguistics 2020paper

    Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

  9. Survey & Experiment: Towards the Learning Accuracy.arXiv 2010paper

    Zeyuan Allen Zhu

  10. Understanding Neural Networks via Feature Visualization: A survey.arXiv 2019paper

    Anh Nguyen, Jason Yosinski, Jeff Clune

  11. Visual interpretability for deep learning: a survey.Journal of Zhejiang University Science C 2018paper

    Quanshi Zhang, Songchun Zhu

  12. Visualisation of Pareto Front Approximation: A Short Survey and Empirical Comparisons.CEC 2019paper

    Huiru Gao, Haifeng Nie, Ke Li

Meta Learning

  1. A Comprehensive Overview and Survey of Recent Advances in Meta-Learning.arXiv 2020paper

    Huimin Peng

  2. Meta-Learning in Neural Networks: A Survey.arXiv 2020paper

    Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, Amos J. Storkey

  3. Meta-Learning: A Survey.arXiv 2018paper

    Joaquin Vanschoren

Metric Learning

  1. A Survey on Metric Learning for Feature Vectors and Structured Data.arXiv 2013paper

    Aurélien Bellet, Amaury Habrard, Marc Sebban

  2. A Tutorial on Distance Metric Learning: Mathematical Foundations, Algorithms and Experiments.arXiv 2018paper

    Juan Luis Suárez, Salvador García, Francisco Herrera

ML Applications

  1. A Survey of Adaptive Resonance Theory Neural Network Models for Engineering Applications.Neural Networks 2019paper

    Leonardo Enzo Brito da Silva, Islam Elnabarawy, Donald C. Wunsch II

  2. A Survey of Machine Learning Methods and Challenges for Windows Malware Classification.arXiv 2020paper

    Edward Raff, Charles Nicholas

  3. A survey on deep hashing for image retrieval.arXiv 2020paper

    Xiaopeng Zhang

  4. A Survey on Deep Learning based Brain-Computer Interface: Recent Advances and New Frontiers.arXiv 2019paper

    Xiang Zhang, Lina Yao, Xianzhi Wang, Jessica J M Monaghan, David Mcalpine, Yu Zhang

  5. A Survey on Deep Learning in Medical Image Analysis.Medical Image Analysis 2017paper

    Geert J S Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud A A Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A W M Van Der Laak, Bram Van Ginneken, Clara I Sanchez

  6. Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial.IEEE 2019paper

    Mingzhe Chen, Ursula Challita, Walid Saad, Changchuan Yin, Mérouane Debbah

  7. How Developers Iterate on Machine Learning Workflows -- A Survey of the Applied Machine Learning Literature.arXiv 2018paper

    Doris Xin, Litian Ma, Shuchen Song, Aditya G. Parameswaran

  8. Machine Learning Aided Static Malware Analysis: A Survey and Tutorial.arXiv 2018paper

    Andrii Shalaginov, Sergii Banin, Ali Dehghantanha, Katrin Franke

  9. Machine Learning for Survival Analysis: A Survey.arXiv 2017paper

    Ping Wang, Yan Li, Chandan K. Reddy

  10. The Creation and Detection of Deepfakes: A Survey.arXiv 2020paper

    Yisroel Mirsky, Wenke Lee

Model Compression and Acceleration

  1. A Survey of Model Compression and Acceleration for Deep Neural Networks.arXiv 2017paper

    Yu Cheng, Duo Wang, Pan Zhou, Tao Zhang

  2. A Survey on Methods and Theories of Quantized Neural Networks.arXiv 2018paper

    Yunhui Guo

  3. An Overview of Neural Network Compression.arXiv 2020paper

    J O Neill

  4. Knowledge Distillation: A Survey.arXiv 2020paper

    Jianping Gou, Baosheng Yu, Stephen John Maybank, Dacheng Tao

  5. Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey.arXiv 2020paper

    Jiayi Liu, Samarth Tripathi, Unmesh Kurup, Mohak Shah

Multi-Task and Multi-View Learning

  1. A Brief Review on Multi-Task Learning.Multimedia Tools and Applications 2018paper

    Kimhan Thung, Chong Yaw Wee

  2. A Survey on Multi-Task Learning.arXiv 2017paper

    Yu Zhang, Qiang Yang

  3. A Survey on Multi-view Learning.arXiv 2013paper

    Chang Xu, Dacheng Tao, Chao Xu

  4. An overview of multi-task learning.National Science Review 2018paper

    Yu Zhang, Qiang Yang

  5. An Overview of Multi-Task Learning in Deep Neural Networks.arXiv 2017paper

    Sebastian Ruder

  6. Revisiting Multi-Task Learning in the Deep Learning Era.arXiv 2020paper

    Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Dengxin Dai, Luc Van Gool

Online Learning

  1. A Survey of Algorithms and Analysis for Adaptive Online Learning.Journal of Machine Learning Research 2017paper

    H. Brendan McMahan

  2. Online Learning: A Comprehensive Survey.arXiv 2018paper

    Steven C.H. Hoi, Doyen Sahoo, Jing Lu, Peilin Zhao

  3. Preference-based Online Learning with Dueling Bandits: A Survey.arXiv 2018paper

    Robert Busa-Fekete, Eyke Hüllermeier, Adil El Mesaoudi-Paul

Optimization

  1. A Survey of Optimization Methods from a Machine Learning Perspective.arXiv 2019paper

    Shiliang Sun, Zehui Cao, Han Zhu, Jing Zhao

  2. A Systematic and Meta-analysis Survey of Whale Optimization Algorithm.Comput. Intell. Neurosci. 2019paper

    Hardi M. Mohammed, Shahla U. Umar, Tarik A. Rashid

  3. An overview of gradient descent optimization algorithms.arXiv 2017paper

    Sebastian Ruder

  4. Convex Optimization Overview.IEEE 2008paper

    Nikos Komodakis

  5. Gradient Boosting Machine: A Survey.arXiv 2019paper

    Zhiyuan He, Danchen Lin, Thomas Lau, Mike Wu

  6. Optimization for deep learning: theory and algorithms.arXiv 2019paper

    Ruoyu Sun

  7. Optimization Models for Machine Learning: A Survey.arXiv 2019paper

    Claudio Gambella, Bissan Ghaddar, Joe Naoum-Sawaya

  8. Particle Swarm Optimization: A survey of historical and recent developments with hybridization perspectives.Machine Learning and Knowledge Extraction 2019paper

    Saptarshi Sengupta, Sanchita Basak, Richard Alan Peters II

Semi-Supervised and Unsupervised Learning

  1. A brief introduction to weakly supervised learning.arXiv 2018paper

    Zhihua Zhou

  2. A Survey on Semi-Supervised Learning Techniques.arXiv 2014paper

    V. Jothi Prakash, Dr. L.M. Nithya

  3. Improvability Through Semi-Supervised Learning: A Survey of Theoretical Results.arXiv 2019paper

    Alexander Mey, Marco Loog

  4. Learning from positive and unlabeled data: a survey.Machine Learning 2020paper

    Jessa Bekker, Jesse Davis

Transfer Learning

  1. A Comprehensive Survey on Transfer Learning.arXiv 2019paper

    Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, Qing He

  2. A Survey of Unsupervised Deep Domain Adaptation.arXiv 2020paper

    Garrett Wilson, Diane J. Cook

  3. A Survey on Deep Transfer Learning.ICANN 2018paper

    Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, Chunfang Liu

  4. A survey on domain adaptation theory: learning bounds and theoretical guarantees.arXiv 2020paper

    Ievgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, Younès Bennani

  5. Evolution of transfer learning in natural language processing.arXiv 2019paper

    Aditya Malte, Pratik Ratadiya

  6. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.arXiv 2019paper

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

  7. Neural Unsupervised Domain Adaptation in NLP---A Survey.arXiv 2020paper

    Alan Ramponi, Barbara Plank

  8. Transfer Adaptation Learning: A Decade Survey.arXiv 2019paper

    Lei Zhang

Trustworthy Machine Learning

  1. A Survey on Bias and Fairness in Machine Learning.arXiv 2019paper

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, Aram Galstyan

  2. Differential Privacy and Machine Learning: a Survey and Review.arXiv 2014paper

    Zhanglong Ji, Zachary C. Lipton, Charles Elkan

  3. Tutorial: Safe and Reliable Machine Learning.arXiv 2019paper

    Suchi Saria, Adarsh Subbaswamy

Team Members

Ziyang Wang, Nuo Xu, Bei Li, Yinqiao Li, Quan Du, Tong Xiao, and Jingbo Zhu

Please feel free to contact us if you have any questions (wangziyang [at] stumail.neu.edu.cn or libei_neu [at] outlook.com).

We would like to thank the people who have contributed to this project. They are

Xin Zeng, Laohu Wang, Chenglong Wang, Xiaoqian Liu, Xuanjun Zhou, Jingnan Zhang, Yongyu Mu, Zefan Zhou, Yanhong Jiang, Xinyang Zhu, Xingyu Liu, Dong Bi, Ping Xu, Zijian Li, Fengning Tian, Hui Liu, Kai Feng, Yuhao Zhang, Chi Hu, Di Yang, Lei Zheng, Hexuan Chen, Zeyang Wang, Tengbo Liu, Xia Meng, Weiqiao Shan, Shuhan Zhou, Tao Zhou, Runzhe Cao, Yingfeng Luo, Binghao Wei, Wandi Xu, Yan Zhang, Yichao Wang, Mengyu Ma, Zihao Liu

成為VIP會員查看完整內容
8
70
0

題目

NLP注意力機製綜述論文翻譯,Attention, please! A Critical Review of Neural Attention Models in Natural Language Processing

關鍵詞

注意力機製,自然語言處理,深度學習,人工智能

簡介

注意力是一種廣泛用於神經體係結構的越來越流行的機製。由於該領域的快速發展,仍然缺少對注意力的係統概述。 在本文中,我們為自然語言處理的注意力體係結構定義了一個統一的模型,重點是旨在與文本數據的矢量表示一起工作的體係結構。 我們討論了提案不同的方麵,注意力的可能用途,並繪製了該領域的主要研究活動和公開挑戰。

作者

Andrea Galassi

A.GALASSI@UNIBO.IT

Department of Computer Science and Engineering (DISI),

University of Bologna, Bologna, Italy ;

Marco Lippi

MARCO.LIPPI@UNIMORE.IT

Department of Sciences and Methods for Engineering (DISMI),

University of Modena and Reggio Emilia, Reggio Emilia, Italy ;

Paolo Torroni

PAOLO.TORRONI@UNIBO.IT

Department of Computer Science and Engineering (DISI),

University of Bologna, Bologna, Italy

成為VIP會員查看完整內容
Attention, please! A Critical Review of Neural Attention Models in NLP.pdf
9
67
0

The tutorial is written for those who would like an introduction to reinforcement learning (RL). The aim is to provide an intuitive presentation of the ideas rather than concentrate on the deeper mathematics underlying the topic. RL is generally used to solve the so-called Markov decision problem (MDP). In other words, the problem that you are attempting to solve with RL should be an MDP or its variant. The theory of RL relies on dynamic programming (DP) and artificial intelligence (AI). We will begin with a quick description of MDPs. We will discuss what we mean by “complex” and “large-scale” MDPs. Then we will explain why RL is needed to solve complex and large-scale MDPs. The semi-Markov decision problem (SMDP) will also be covered.

The tutorial is meant to serve as an introduction to these topics and is based mostly on the book: “Simulation-based optimization: Parametric Optimization techniques and reinforcement learning” [4]. The book discusses this topic in greater detail in the context of simulators. There are at least two other textbooks that I would recommend you to read: (i) Neuro-dynamic programming [2] (lots of details on convergence analysis) and (ii) Reinforcement Learning: An Introduction [11] (lots of details on underlying AI concepts). A more recent tutorial on this topic is [8]. This tutorial has 2 sections: • Section 2 discusses MDPs and SMDPs. • Section 3 discusses RL. By the end of this tutorial, you should be able to • Identify problem structures that can be set up as MDPs / SMDPs. • Use some RL algorithms.

成為VIP會員查看完整內容
3
79
0

With the rise and development of deep learning, computer vision has been tremendously transformed and reshaped. As an important research area in computer vision, scene text detection and recognition has been inescapably influenced by this wave of revolution, consequentially entering the era of deep learning. In recent years, the community has witnessed substantial advancements in mindset, approach and performance. This survey is aimed at summarizing and analyzing the major changes and significant progresses of scene text detection and recognition in the deep learning era. Through this article, we devote to: (1) introduce new insights and ideas; (2) highlight recent techniques and benchmarks; (3) look ahead into future trends. Specifically, we will emphasize the dramatic differences brought by deep learning and the grand challenges still remained. We expect that this review paper would serve as a reference book for researchers in this field. Related resources are also collected and compiled in our Github repository:https://github.com/Jyouhou/SceneTextPapers.

成為VIP會員查看完整內容
2
48
0

Differentiable Graphics with TensorFlow 2.0

Deep learning has introduced a profound paradigm change in the recent years, allowing to solve significantly more complex perception problems than previously possible. This paradigm shift has positively impacted a tremendous number of fields with a giant leap forward in computer vision and computer graphics algorithms. The development of public libraries such as Tensorflow are in a large part responsible for the massive growth of AI. These libraries made deep learning easily accessible to every researchers and engineers allowing fast advances in developing deep learning techniques in the industry and academia. We will start this course with an introduction to deep learning and present the newly released TensorFlow 2.0 with a focus on best practices and new exciting functionalities. We will then show different tips, tools, and algorithms to visualize and interpret complex neural networks by using TensorFlow. Finally, we will introduce a novel TensorFlow library containing a set of graphics inspired differentiable layers allowing to build structured neural networks to solve various two and three dimensional perception tasks. To make the course interactive we will punctuate the presentations with real time demos in the form of Colab notebooks. Basic prior familiarity with deep learning will be assumed.** Deep learning has introduced a profound paradigm change in the recent years, allowing to solve significantly more complex perception problems than previously possible. This paradigm shift has positively impacted a tremendous number of fields with a giant leap forward in computer vision and computer graphics algorithms. The development of public libraries such as Tensorflow are in a large part responsible for the massive growth of AI. These libraries made deep learning easily accessible to every researchers and engineers allowing fast advances in developing deep learning techniques in the industry and academia. We will start this course with an introduction to deep learning and present the newly released TensorFlow 2.0 with a focus on best practices and new exciting functionalities. We will then show different tips, tools, and algorithms to visualize and interpret complex neural networks by using TensorFlow. Finally, we will introduce a novel TensorFlow library containing a set of graphics inspired differentiable layers allowing to build structured neural networks to solve various two and three dimensional perception tasks. To make the course interactive we will punctuate the presentations with real time demos in the form of Colab notebooks. Basic prior familiarity with deep learning will be assumed.

成為VIP會員查看完整內容
3
22
0
小貼士
相關論文
Arxiv
13+閱讀 · 2020年12月23日
Arxiv
15+閱讀 · 2020年9月16日
AutoML: A Survey of the State-of-the-Art
Arxiv
49+閱讀 · 2019年8月14日
Arxiv
12+閱讀 · 2019年4月5日
Arxiv
10+閱讀 · 2018年9月5日
相關資訊
分布式並行架構Ray介紹
CreateAMind
6+閱讀 · 2019年8月9日
A Technical Overview of AI & ML in 2018 & Trends for 2019
待字閨中
10+閱讀 · 2018年12月24日
NIPS 2017:貝葉斯深度學習與深度貝葉斯學習(講義+視頻)
機器學習研究會
31+閱讀 · 2017年12月10日
計算機視覺近一年進展綜述
機器學習研究會
6+閱讀 · 2017年11月25日
【論文】變分推斷(Variational inference)的總結
機器學習研究會
24+閱讀 · 2017年11月16日
【論文】圖上的表示學習綜述
機器學習研究會
9+閱讀 · 2017年9月24日
【推薦】深度學習目標檢測全麵綜述
機器學習研究會
17+閱讀 · 2017年9月13日
【推薦】GAN架構入門綜述(資源彙總)
機器學習研究會
9+閱讀 · 2017年9月3日
【推薦】全卷積語義分割綜述
機器學習研究會
17+閱讀 · 2017年8月31日
Top
微信掃碼谘詢專知VIP會員
Top