All programmes > Natural Language Processing

Natural Language Processing (B-KUL-H02B1A)

4 ECTSEnglish33 First termCannot be taken as part of an examination contract
de Lhoneux Miryam
POC Artificial Intelligence

Aims

The course focuses on methods and algorithms for building computer software that understands, generates and manipulates human language. We study the algorithms and models while introducing core tasks in natural language processing (NLP), including language modelling, morphological processing, POS tagging, syntactic analysis, semantic interpretation, machine translation, coreference resolution, discourse analysis, and dialogue modelling. We illustrate the methods and technologies with current applications in real world settings.

After following this course, the student has acquired theoretical and practical understanding of contemporary machine learning models designed for processing human language and of the underlying computational properties of NLP models. The student will have learned how underlying linguistic phenomena, that is, the linguistic features, can be modelled and automatically learned from data using deep learning techniques.

Previous knowledge

This course focuses on the algorithms, mathematical modeling and machine learning methods for processing human language. We rely on a good understanding of the machine learning foundations. Hence there is the prerequisite to have successfully passed or to follow in parallel a machine learning course (e.g., Machine Learning and Inductive Inference - B-KUL-H02C1A, Principles of Machine Learning - B-KUL-H0E98A, Artificial Neural Networks and Deep Learning - B-KUL-H02C4A). Knowledge of the basics of linear algebra and of probability theory is required.

Students who also want to deepen their knowledge of the linguistic aspects of natural language processing are recommended to follow this advanced natural language processing course and the course Linguistics and Artificial Intelligence (B-KUL-H02B6A) in parallel.

Is included in these courses of study

Activities

3.5 ects. Natural Language Processing: Lecture (B-KUL-H02B1a)

3.5 ECTSEnglishFormat: Lecture20 First term
de Lhoneux Miryam
POC Artificial Intelligence

Content

1. Introduction

  • What is natural language processing (NLP)?
  • Current state-of-the-art of NLP
  • Ambiguity
  • Other challenges
  • Representing words, phrases and sentences
     

2. Segmentation and tokenization

  • Regular expressions
  • Word tokenization, lemmatization and stemming
  • Sentence segmentation
  • Subword tokenization
     

3. Language Modelling

  • N-gram language models
  • perplexity
  • maximum likelihood estimation
  • smoothing
     

4. Neural Language Modelling

  • Word embeddings
  • Vector space models for NLP
  • Recurrent neural network (RNN) for language modelling
  • Transformer architecture for language modelling
  • Use of language models in downstream tasks: fine-tuning and pretraining
     

5. Part-of-Speech (POS) Tagging

  • Hidden Markov model and viterbi
  • Conditional Random Fields
  • (Bi)LSTM for POS tagging
  • Encoder-decoder architecture for sequence-to-sequence labeling
     

6. Morphological analysis

  • Inflection and derivation
  • Finite state morphology
  • Sequence-to-sequence neural models of morphological inflection
     

7. Syntactic Parsing

  • Universal Dependencies
  • Dependency parsing: Graph based dependency parsing, transition based dependency parsing
  • Constituent parsing with a (probabilistic) context free grammar ((P)CFG) and the Cocke-Younger-Kasami (CYK) algorithm
     

8. Semantics (lexical and compositional)

  • Word sense disambiguation
  • Semantic role labelling
     

9. Discourse: Coreference Resolution

  • Discourse coherence
  • Algorithm of Hobbs
  • Neural end-to-end coreference resolution
     

10. Question Answering

  • Evolution of QA systems from rule-based to neural 
  • Complex pipelines to end-to-end to retrieval-free
  • Closed-domain vs open-domain
  • Text-only vs multimodal
     

11. Neural Machine Translation

  • Encoder-decoder architecture (e.g., RNN, transformer-based)
  • Attention models
  • Improvements and alternative architectures that deal with limited parallel training data
     

12. Conversational Dialogue Systems and Chatbots

  • Task oriented dialog agents: Rule based versus neural based approaches
  • Chatbots: End-to-end sequence-to-sequence neural models
     

Course material

Handbooks

Daniel Jurafsky and James H. Martin. 2024. Speech and Language Processing: An Introduction to Natural
Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition.

Jacob Eisenstein. 2019. Introduction to Natural Language Processing. MIT Press.

Yoav Goldberg. 2016. A Primer on Neural Network Models for Natural Language Processing.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.

  
+ recent articles: e.g., of the proceedings of the Meetings of the ACL, AAAI, NeurIPS.

Format: more information

Interactive lectures with short exercises.

Is also included in other courses

0.5 ects. Natural Language Processing: Exercises (B-KUL-H00G0a)

0.5 ECTSEnglishFormat: Practical13 First term
de Lhoneux Miryam
POC Artificial Intelligence

Content

  • Exercises on tokenization and segmentation
  • Exercises on language modelling and POS tagging
  • Exercises on syntactic parsing
  • Exercises on semantic and discourse processing
  • Exercises on machine translation
  • Exercises on question answering

Is also included in other courses

Evaluation

Evaluation: Natural Language Processing (B-KUL-H22B1a)

Type : Exam during the examination period
Description of evaluation : Written
Type of questions : Open questions, Closed questions
Learning material : Calculator, Course material

Explanation


Open book written exam featuring a mixture of theory and exercise questions.