Google Blog, GitHub (tensorflow/tensor2tensor, Unofficial PyTorch), PapersWithCode, Harvard NLP: The Annotated Transformer (updated)\n\n","updatedAt":"2023-11-22T08:09:55.018Z","author":{"avatarUrl":"/avatars/58a0b486fd0bec90887388ec9bf26b17.svg","fullname":"Avneesh Mishra","name":"TheProjectsGuy","type":"user","isPro":false,"isHf":false,"isMod":false}},"numEdits":0,"editors":["TheProjectsGuy"],"reactions":[{"reaction":"๐ฅ","users":["ravikumarmn"],"count":1}],"identifiedLanguage":{"language":"en","probability":0.7787639498710632},"isReport":false}},{"id":"65b761ef8bb894c96ce1dc6f","author":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isMod":false},"createdAt":"2024-01-29T08:29:35.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Heterogeneous Encoders Scaling in the Transformer for Neural Machine Translation](https://huggingface.co/papers/2312.15872) (2023)\n* [Quick Back-Translation for Unsupervised Machine Translation](https://huggingface.co/papers/2312.00912) (2023)\n* [MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining](https://huggingface.co/papers/2312.17482) (2023)\n* [TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing](https://huggingface.co/papers/2312.05605) (2023)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\n
The following papers were recommended by the Semantic Scholar API
\n","updatedAt":"2024-06-08T20:54:34.292Z","author":{"avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isMod":false}},"numEdits":0,"editors":["blanchon"],"reactions":[],"identifiedLanguage":{"language":"en","probability":0.5290509462356567},"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"1706.03762","authors":[{"_id":"6411c77d6b75ddced38902b6","user":{"_id":"60fe1b231a3e6f5129776bf9","avatarUrl":"/avatars/c80edad5267c6ed9ecde9b056993d5c3.svg","isPro":false,"fullname":"Ashish Vaswani","user":"ashishvaswanigoogle","type":"user"},"name":"Ashish Vaswani","status":"extracted_pending","statusLastChangedAt":"2023-03-15T09:40:25.803Z","hidden":false},{"_id":"6411c77d6b75ddced38902b7","name":"Noam Shazeer","hidden":false},{"_id":"6411c77d6b75ddced38902b8","user":{"_id":"60fee8e1465daccb9f332e2f","avatarUrl":"/avatars/ea92bf15e181b4f13a70f18ee3ba7a51.svg","isPro":false,"fullname":"Niki Parmar","user":"nikip","type":"user"},"name":"Niki Parmar","status":"extracted_pending","statusLastChangedAt":"2023-03-15T09:40:25.803Z","hidden":false},{"_id":"6411c77d6b75ddced38902b9","name":"Jakob Uszkoreit","hidden":false},{"_id":"6411c77d6b75ddced38902ba","name":"Llion Jones","hidden":false},{"_id":"6411c77d6b75ddced38902bb","name":"Aidan N. Gomez","hidden":false},{"_id":"6411c77d6b75ddced38902bc","name":"Lukasz Kaiser","hidden":false},{"_id":"6411c77d6b75ddced38902bd","name":"Illia Polosukhin","hidden":false}],"publishedAt":"2017-06-12T17:57:34.000Z","title":"Attention Is All You Need","summary":"The dominant sequence transduction models are based on complex recurrent or\nconvolutional neural networks in an encoder-decoder configuration. The best\nperforming models also connect the encoder and decoder through an attention\nmechanism. We propose a new simple network architecture, the Transformer, based\nsolely on attention mechanisms, dispensing with recurrence and convolutions\nentirely. Experiments on two machine translation tasks show these models to be\nsuperior in quality while being more parallelizable and requiring significantly\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014\nEnglish-to-German translation task, improving over the existing best results,\nincluding ensembles by over 2 BLEU. On the WMT 2014 English-to-French\ntranslation task, our model establishes a new single-model state-of-the-art\nBLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction\nof the training costs of the best models from the literature. We show that the\nTransformer generalizes well to other tasks by applying it successfully to\nEnglish constituency parsing both with large and limited training data.","upvotes":41,"discussionId":"641192343ea54b1aa7e2f084"},"canReadDatabase":false,"canManageCommunity":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62600854000b49a547738c98","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62600854000b49a547738c98/ywyX6Ye6RaN1eGrDacNMs.jpeg","isPro":false,"fullname":"Shreyas Dixit","user":"SRDdev","type":"user"},{"_id":"63765e6b2361581ceb232cc8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1675652727502-63765e6b2361581ceb232cc8.jpeg","isPro":false,"fullname":"Luke Cheng","user":"chenglu","type":"user"},{"_id":"6545ef0a407bb19ff55357db","avatarUrl":"/avatars/6f47d4f337e316961aaa1fb9e95b452f.svg","isPro":false,"fullname":"huph22","user":"huph22","type":"user"},{"_id":"64bd994ef8f28a19b0d0acbe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64bd994ef8f28a19b0d0acbe/igR6xy19rpgl2uqeh3iPa.jpeg","isPro":true,"fullname":"Snehasish Barman","user":"sbarman25","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"63ce875d199b36f7552d4f07","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ce875d199b36f7552d4f07/NrTlnhUTAp0j0ZsMTsWIu.jpeg","isPro":false,"fullname":"Marc Sun","user":"marcsun13","type":"user"},{"_id":"655ac762cb17ec19ef82719b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655ac762cb17ec19ef82719b/1kDncYrGLYS_2SR8cNdAL.png","isPro":false,"fullname":"Welcome to matlok","user":"matlok","type":"user"},{"_id":"63d10d4e8eaa4831005e92b5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63d10d4e8eaa4831005e92b5/7p7-OmWM6PqqCs7ZStPGD.jpeg","isPro":false,"fullname":"Aymeric Roucher","user":"m-ric","type":"user"},{"_id":"62ded294dcdc9043efaa4fc3","avatarUrl":"/avatars/1294078b8cd5c1901e4021253d877357.svg","isPro":false,"fullname":"Peter J Liu","user":"peterjliu","type":"user"},{"_id":"6551fce8ea9be8f1e6facf99","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6551fce8ea9be8f1e6facf99/yWHIimvBNxvLpEdKAMT2x.jpeg","isPro":false,"fullname":"James Aymer","user":"qwerty87","type":"user"},{"_id":"6514fcd72f236e3633b7038e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hZ6UKfY6HrcuBF26rOamk.jpeg","isPro":false,"fullname":"Davit","user":"davidbuniat","type":"user"},{"_id":"6340651b388c3fa40f9a5bc0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6340651b388c3fa40f9a5bc0/av1C4_S7bHGxAzOu8lOmG.jpeg","isPro":false,"fullname":"Adam Molnar","user":"lunarflu","type":"user"}],"acceptLanguages":["*"]}">
The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks in an encoder-decoder configuration. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer, based
solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show these models to be
superior in quality while being more parallelizable and requiring significantly
less time to train. Our model achieves 28.4 BLEU on the WMT 2014
English-to-German translation task, improving over the existing best results,
including ensembles by over 2 BLEU. On the WMT 2014 English-to-French
translation task, our model establishes a new single-model state-of-the-art
BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction
of the training costs of the best models from the literature. We show that the
Transformer generalizes well to other tasks by applying it successfully to
English constituency parsing both with large and limited training data.
Introduces language transformer architecture for natural language processing (NLP). Transformer: Input (encoded) added with positional encodings (1D sine and cosine of different frequencies) given to encoder layers (multi-headed self-attention and linear layers in a residual setup), then the encoder output along with previous output (encoding added with positional embeddings) is given to decoder layers (masked multi-headed self-attention, multi-headed cross attention from encoder output, and linear layers in a residual setup). The residual setup has layer normalization. Attention is formulated as an information retrieval problem: Project (the same) input to key, query, and value embeddings; dot-product of query and key give the values to be retrieved (take softmax of dot product, normalize, and then multiply with value for retrieval). For multi-head: split and do this in parallel, along with concatenation. Trained for language translation (English to French and German), good BELU at low training cost. Attention head visualization in appendix. From Google, University of Toronto.