Natural Language Processing

Natural Language Processing #

Natural Language Processing (NLP) studies how computers can analyse, understand, represent, and generate human language.

It combines ideas from linguistics, computer science, machine learning, and deep learning to work with text and language-based information.

Natural Language Processing = Linguistics + Computation + Machine Learning

The learning path begins with language understanding and vector representations, progresses through language modelling, tagging, and parsing, and then moves towards transformers, knowledge graphs, Retrieval-Augmented Generation, and modern NLP applications.

N-gram Language Modelling

N-gram Language Modelling #

A language model assigns probabilities to sequences of words. It can compare complete sentences or predict which word is likely to come next.

Key ideas include:

• word prediction and sequence probability
• the chain rule and Markov assumption
• unigram, bigram and trigram models
• Maximum Likelihood Estimation
• unseen sequences and smoothing
• interpolation and backoff
• intrinsic and extrinsic evaluation
• perplexity

Learning Objectives #

• Explain what a language model represents.
• Calculate simple unigram and bigram probabilities.
• Explain why unseen N-grams create zero probabilities.
• Distinguish smoothing, interpolation and backoff.
• Interpret perplexity correctly.

Big Picture #

flowchart TD
    A["Training Corpus"] --> B["Count N-grams"]
    B --> C["Estimate Probabilities"]
    C --> D["Handle Unseen Events"]
    D --> E["Score Word Sequences"]
    E --> F["Evaluate Model"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9

1. What Is a Language Model? ☆ #

A language model estimates how probable a sequence of words is.

NN and Neural Language Modelling

Neural Networks and Neural Language Modelling #

Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.

Learning Objectives #

• Explain the computation performed by a neural unit.
• Describe why hidden layers and nonlinear activations are needed.
• Explain how feed-forward networks support NLP classification.
• Trace the flow through a feed-forward neural language model.
• Compare N-gram and neural language models.

Big Picture #

flowchart TD
    A["Context Words"] --> B["One-hot Inputs"]
    B --> C["Embedding Lookup"]
    C --> D["Combined Context"]
    D --> E["Hidden Layer"]
    E --> F["Softmax"]
    F --> G["Next-word Probabilities"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Neural Network Units ☆ #

A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.

LLM and Prompt Engineering

LLMs and Prompt Engineering #

A Large Language Model extends neural language modelling through much larger datasets, many more parameters, broad pretraining and adaptation to many downstream tasks. Its central operation remains next-token prediction.

Learning Objectives #

• Explain how neural language modelling develops into an LLM.
• Describe the meaning of large, general-purpose and pretrained.
• Explain how a prompt guides generation.
• Distinguish zero-shot and few-shot prompting.
• Compare prompting with model adaptation.

Big Picture #

flowchart TD
    A["Broad Text Data"] --> B["Large-scale Pretraining"]
    B --> C["General Language Model"]
    C --> D["Prompt or Adaptation"]
    D --> E["Task Output"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE

1. From Neural Language Models to LLMs ☆ #

A neural language model learns a conditional probability for the next token:

Part-of-Speech Tagging and Hidden Markov Models

Part-of-Speech Tagging and Hidden Markov Models #

Part-of-Speech tagging assigns a grammatical category to each word in a sequence. Because many words can play different grammatical roles, a tagger must use surrounding context rather than examine each word independently.

Learning Objectives #

• Identify common English word classes and Penn Treebank tags.
• Explain why POS tagging is a sequence-labelling problem.
• Describe the Markov assumption.
• Distinguish a Markov Chain from a Hidden Markov Model.
• Explain how an HMM represents POS tagging.

Big Picture #

flowchart TD
    A["Word Sequence"] --> B["Use Context"]
    B --> C["Infer Hidden Tags"]
    C --> D["Tagged Sequence"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6

1. What Is Part-of-Speech Tagging? ☆ #

A part of speech describes the grammatical role played by a word in a sentence. POS tagging assigns one tag to every word in a sequence.

Statistical, ML and Neural Models of POS Tagging

Statistical, ML and Neural Models of POS Tagging #

‘HMM Inference: Forward and Viterbi Algorithms’ covers the portion:

• Forward Algorithm
• Viterbi Algorithm
• HMM inference for POS tagging

The complete Topic also includes:

• Maximum Entropy Markov Models
• Bidirectionality
• Neural-network models for POS tagging

HMM Inference: Forward and Viterbi Algorithms #

Hidden Markov Models create two closely related inference problems:

• Likelihood: How probable is an observed sequence under the model?
• Decoding: Which hidden-state sequence most probably generated the observations?

The Forward Algorithm solves the likelihood problem, while the Viterbi Algorithm solves the decoding problem. Both use dynamic programming and a trellis, but they combine paths differently.