Vector Semantics and Embedding

NLP - Vector Semantics and Embedding #

• Lexical semantics and word meaning.
• Lemmas, senses, and semantic relationships.
• Distributional hypothesis.
• Vector semantics and word embeddings.
• Document and word vectors.
• Dot product and cosine similarity.
• Term Frequency–Inverse Document Frequency (TF-IDF).
• Prediction-based word embeddings and self-supervision.
• Word2Vec using Skip-gram with Negative Sampling and CBOW.
• Embedding matrices, context-window choices, analogies, visualisation, and bias.
• GloVe and global word–word co-occurrence statistics.

Learning Objectives #

• Explain lexical semantics and distinguish a lemma from a word sense.
• Compare synonymy, similarity, relatedness, antonymy, and connotation.
• Explain the distributional hypothesis and its role in modelling meaning.
• Describe how words and documents can be represented as vectors.
• Construct and interpret word–document and word–context matrices.
• Calculate dot product and cosine similarity between vectors.
• Explain why raw word frequency can be misleading.
• Calculate TF, IDF, and TF-IDF weights.
• Explain how Word2Vec learns embeddings from a prediction task.
• Construct positive and negative Skip-gram training pairs.
• Explain how sigmoid, negative sampling, and gradient descent train SGNS.
• Compare Skip-gram with CBOW.
• Explain how context-window size affects the relationships captured.
• Interpret word analogies and two-dimensional embedding visualisations.
• Explain how GloVe combines global counts with learned dense vectors.
• Recognise how social biases can be encoded in word embeddings.

Big Picture #

flowchart TD
    A[Words and Documents] --> B[Observe Their Context]
    B --> C[Represent Them as Vectors]
    C --> D[Compare Vector Directions]
    D --> E[Estimate Semantic Similarity]
    E --> F[Search, Classify, Retrieve or Generate]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9

1. Lexical Semantics ☆ #

Lexical semantics is the linguistic study of word meaning and the relationships between word meanings.