lingo.lol is one of the many independent Mastodon servers you can use to participate in the fediverse.
A place for linguists, philologists, and other lovers of languages.

Server stats:

64
active users

#historyofai

0 posts0 participants0 posts today

Building on the 90s, statistical n-gram language models, trained on vast text collections, became the backbone of NLP research. They fueled advancements in nearly all NLP techniques of the era, laying the groundwork for today's AI.

F. Jelinek (1997), Statistical Methods for Speech Recognition, MIT Press, Cambridge, MA

#NLP #LanguageModels #HistoryOfAI #TextProcessing #AI #historyofscience #ISE2025 @fizise @fiz_karlsruhe @tabea @enorouzi @sourisnumerique

Summarizing our very brief #HistoryOfAI which was published here for several weeks in a series of toots , let's have a look at the popularity dynamics of symbolic vs subsymbolic AI put into perspective with historical AI hay-days and winters via the Google ngram viewer.
books.google.com/ngrams/graph?

#ISE2024 #AI #ontologies #machinelearning #neuralnetworks #llms @fizise @sourisnumerique @enorouzi #semanticweb #knowledgegraphs

In 2022 with the advent of ChatGPT, large language models and AI in general gained an unprecedented popularity. It combined InstructGPT, a GPT-3 model complemented and fine-tuned with reinforcement learning feedback, Codex text2code, plus a massive engineering effort.

N. Lambert, et al. (2022). Illustrating Reinforcement Learning from Human Feedback (RLHF). huggingface.co/blog/rlhf

#HistoryOfAI #AI #ISE2024 @fizise @sourisnumerique @enorouzi #llm #gpt #llms

Higher, faster, farther... in 2021 Generative AI gains momentum with the advent of DaLL-E, a GPT-3 based zero-shot text2image model, and other major milestones, as e.g., GitHub CoPilot, Open AI Codex, WebGPT, and Google LaMDA.

Codex: Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code, arxiv.org/abs/2107.03374
DaLL-E: Ramesh, A.et al. (2021). Zero-Shot Text-to-Image Generation, arxiv.org/abs/2107.03374

#HistoryOfAI #AI #ISE2024 @fizise @sourisnumerique @enorouzi #llm #gpt

In 2020, GPT-3 was released by OpenAI, based on 45TB data crawled from the web. A “data quality” predictor was trained to boil down the training data to 550GB “high quality” data. Learning from the prompt (few-shot learning) was also introduced.

T. B. Brown et al. (2020). Language models are few-shot learners. NIPS 2020, pp.1877–1901. proceedings.neurips.cc/paper/2

#HistoryOfAI #AI #ISE2024 #llms #gpt #lecture @enorouzi @sourisnumerique @fizise

In 2019, OpenAI released GPT-2 as a direct scale-up of GPT, comprising 1.5B parameters and trained on 8M web pages.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.
d4mucfpksywv.cloudfront.net/be
OpenAI blog post: openai.com/index/better-langua
GPT-2 on HuggingFace: huggingface.co/openai-communit

#HistoryOfAI #AI #llm #ISE2024 @fizise @enorouzi @sourisnumerique #gpt

In 2018, Generative Pre-trained Transformers (GPT, by OpenAI) and Bidirectional Encoder Representations from Transformers (BERT, by Google) are introduced.

Radford, A. et al (2018). Improving language understanding by generative pre-training, s3-us-west-2.amazonaws.com/ope

J. Devlin et al (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, ACL 2019, aclanthology.org/N19-1423

#HistoryOfAI #ISE2024 #AI #llm @fizise @enorouzi @sourisnumerique

In 2014 Attention mechanisms were introduced by Bahdanau, Cho, and Bengio, which allow models to selectively focus on specific parts of the input. In 2017, the Transformer model introduced by Ashish Vaswani et al. followed, which learns to encode and decode sequential information especially effective for tasks like machine translation and #NLP.

Attention: arxiv.org/pdf/1409.0473
Transformers: arxiv.org/pdf/1706.03762

#HistoryOfAI #AI #ISE2024 @fizise @sourisnumerique @enorouzi #transformers

In 2013, Mikolov et al. (from Google) published word2vec, a neural network based framework to learn distributed representations of words as dense vectors in continuous space, aka word embeddings.

T. Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781
arxiv.org/abs/1301.3781

#HistoryOfAI #AI #ise2024 #lecture #distributionalsemantics #wordembeddings #embeddings @sourisnumerique @enorouzi @fizise

In 1999, NVIDIA introduced the first Graphical Processing Unit (GPU) card Nvidia Geforce 256, enabling an unprecedented speedup for parallel computations as required for machine learning. This innovation paved the way for the rapid advancement of deep learning algorithms.

John Peddie, Famous Graphics Chips: Nvidia’s GeForce 256, IEEE Computer Society.
computer.org/publications/tech

#HistoryOfAI #ISE2024 #AI #deeplearning #machinelearning #lecture @sourisnumerique @enorouzi @fizise @fiz_karlsruhe

In 1996, Long Short-Term Memory (LSTM) Recurrent Neural Networks are introduced by Sepp Hochreiter and Jürgen Schmidhuber, which efficiently enabled #neuralnetworks to process sequences of data (instead of single data points) being able to learn from data and to generate text.

Hochreiter, Sepp; Schmidhuber, Juergen (1996). LSTM can solve hard long time lag problems. Advances in NIPS, pp. 473–479.
dl.acm.org/doi/10.5555/2998981

#HistoryOfAI #AI #ISE2024 #lecture @sourisnumerique @enorouzi @fizise

In 1994, Tim Berners-Lee introduced the #SemanticWeb in his plenary presentation at the 1st WWW conference in Geneva, Switzerland.

“I have a dream for the Web [in which computers] become capable of analyzing all the data on the Web – the content, links, and transactions between people and computers. A Semantic Web, which makes this possible, has yet to emerge..."

Slides from TBL, 1994: w3.org/Talks/WWW94Tim/

#HistoryOfAI #AI #knowledgegraphs #lexture #ISE2024 @sourisnumerique @enorouzi

In 1968, Terry Winograd introduced SHRDLU, a natural language understanding agent that was able to plan and execute directives in rudimentary 'block world'. In particular, SHRDLU emphasized the importance of user-friendly interfaces for HCI.

T. Winograd (1970). Procedures as a Representation for Data in a Computer Program for Understanding Natural Language", MIT AI Technical Report 235. web.archive.org/web/2020100321

#HistoryOfAI #AI #lecture #ISE2024 @fiz_karlsruhe @sourisnumerique @enorouzi

During Cold War, rule-based machine translation from English to Russian and vice versa was a hot topic. However, for translating languages with rules, you have to explicitly cover an innumerable amount of exceptions. Thus, government funding for Machine Translation was cut in 1966, leading to the first AI winter.

W.J. Hutchins (1985) Machine Translation: Past, Present, and Future, Longman. p.5
archive.org/details/machinetra

#AI #HistoryOfAI #ISE2024 @sourisnumerique @enorouzi @fizise #lecture

Two millennia after Aristotle, Gottfried Wilhelm Leibniz adopted the idea to represent knowledge with a (mathematical) universal language and proposed the calculus ratiocinator to reason over this knowledge.

G. W. Leibniz (1676), De arte characteristica ad perficiendas scientias ratione nitentes uni-muenster.de/Leibniz/DatenV

lecture slides: docs.google.com/presentation/d

#HistoryOfAI #AI #ISE2024 #lecture @fizise @enorouzi @sourisnumerique #leibniz #philosophy #calculemus

Knowledge Representation and Symbolic Reasoning as another AI discipline are much older than machine learning. Already in the 4th century BCE greek philosopher Aristotle suggested ten universal categories under which to place every object of human apprehension.

Studtmann, P.. Aristotle's Categories. In Zalta, E.N. (ed.). Stanford Encyclopedia of Philosophy. plato.stanford.edu/entries/ari

#HistoryOfAI #AI #ISE2024 #knowledgerepresentation #symbolicAI #philosophy @sourisnumerique @enorouzi @fizise

AI as a scientific discipline started with the 1956 Dartmouth Summer Workshop initiated by John McCarthy together with Marvin Minsky, Allen Newell, Herbert Simon, and others.
"An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves."
Proposal for the Darrtmouth Summer Project on #AI: raysolomonoff.com/dartmouth/bo

#HistoryOfAI #AI #ISE2024 #lecture @fizise @sourisnumerique @enorouzi