Deep Reinforcement Learning

Deep Reinforcement Learning #

Deep Reinforcement Learning (DRL) studies how an agent learns to make a sequence of decisions by interacting with an environment and receiving feedback through rewards.

Reinforcement learning provides the framework for sequential decision-making. Deep learning extends this framework with powerful function approximators that can handle large or complex state and action spaces.

Deep Reinforcement Learning = Reinforcement Learning + Deep Neural Networks

Classical reinforcement learning foundations → progresses towards → value-based deep learning → policy-gradient methods → model-based approaches → imitation learning.

Natural Language Processing

Natural Language Processing #

Natural Language Processing (NLP) studies how computers can analyse, understand, represent, and generate human language.

It combines ideas from linguistics, computer science, machine learning, and deep learning to work with text and language-based information.

Natural Language Processing = Linguistics + Computation + Machine Learning

The learning path begins with language understanding and vector representations, progresses through language modelling, tagging, and parsing, and then moves towards transformers, knowledge graphs, Retrieval-Augmented Generation, and modern NLP applications.

Artificial and Computational Intelligence

Artificial and Computational Intelligence #

Artificial and Computational Intelligence studies how agents perceive, reason, search, optimise, learn and act in complex environments.

Core Learning Pages #

Modular Structure #

1. Introduction #

• Artificial Intelligence foundations
• Overview of modern AI
• AI application domains

2. Introduction to Intelligent Agents #

• Notion of agents and environments
• Rationality
• Nature of environments
• Structure of agents

3. Problem-Solving Agents Using Search #

• Problem formulation
• Uninformed and informed search algorithms
• Heuristics
• Greedy Best-First Search
• A* Search and optimality of A*
• Heuristic accuracy and algorithm performance
• Admissible heuristics from relaxed problems
• Pattern databases and experience
• Learning heuristics
• Local search and optimisation
• Hill Climbing
• Local Beam Search
• Genetic Algorithms
• Ant Colony Optimisation
• Neural Architecture Search
• Neuroevolution

4. Game Playing #

• Minimax Algorithm
• Alpha-Beta Pruning
• Monte Carlo Tree Search
• Stochastic Games

5. Knowledge Representation Using Logic #

• Propositional and Predicate Logic
• TT-Entail and theorem proving
• Logic representation of intelligent agents
• Proof by resolution
• DPLL Algorithm
• Agents based on Propositional Logic
• Unification
• Forward Chaining
• Backward Chaining
• Resolution

6. Multi-Agent Decision Making #

• Properties of multi-agent environments
• Multi-agent planning
• Non-Cooperative Game Theory
• Cooperative Game Theory
• Collective decision making

7. Probabilistic Representation and Reasoning #

• Representing knowledge in uncertain domains
• Semantics of Bayesian Networks
• Exact inference in Bayesian Networks
• Approximate inference in Bayesian Networks

8. Probabilistic Reasoning Over Time #

• Time and uncertainty
• Inference in temporal models
• Hidden Markov Models
• Learning HMMs
• Dynamic Bayesian Networks

9. Ethics in AI #

• Explainable AI
• Logically Explained Networks
• Explainable Bayesian Networks

Experiments #

Practical Tools #

• Programming languages: Python, Prolog
• Tools and libraries: Jupyter, NumPy, SciPy, Pandas, pgmpy, NLTK
• Environments: Google Colab, SWI-Prolog Online

Book References #

Primary Textbook #

1. Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 4th Edition, Pearson Education, 2020.

Reference Books #

1. Ryszard S. Michalski, Jaime G. Carbonell and Tom M. Mitchell, Machine Learning: An Artificial Intelligence Approach, Elsevier, 2014.
2. Dan W. Patterson, Introduction to AI and Expert Systems, Prentice Hall of India, New Delhi, 2010.
3. Elaine Rich and Kevin Knight, Artificial Intelligence, 2nd Edition, Tata McGraw Hill Publishing Company, New Delhi, 2003.

Home | Artificial Intelligence

AI Stages: ANI, AGI, ASI

July 4, 2024

AI Development Stages: ANI → AGI → ASI #

Artificial Intelligence is often described in three stages, based on capability and scope:

• ANI: Task-specific intelligence (today’s AI)
• AGI: Human-level general intelligence (future goal)
• ASI: Beyond human intelligence (theoretical)

ANI — Artificial Narrow Intelligence #

• also called Weak AI
• designed to perform one specific task
• Operates within a predefined environment
• Cannot generalise beyond its training
• Most AI systems today are ANI

examples

Artificial Intelligence

Artificial Intelligence #

Artificial Intelligence (AI) is concerned with building systems that can perceive, reason, learn, decide and act in ways that achieve useful goals.

This page introduces the foundations of AI, its major application areas, four classic ways of thinking about intelligence, important milestones in its development, and some of the risks that accompany increasingly capable AI systems.

• Artificial Intelligence: Foundations, Overview of Modern AI & Application Domains.
• What is Artificial Intelligence: Acting Humanly, Thinking humanly, Thinking rationally, Acting Rationally
• Foundations of AI , Brief Overview of Modern AI & Application Domains

Learning Objectives #

• explain what intelligence and artificial intelligence mean in practical terms
• identify the major disciplines that contributed to AI
• distinguish thinking humanly, acting humanly, thinking rationally and acting rationally
• explain the idea behind the Turing Test
• describe why the rational-agent view is central to modern AI
• recognise important application areas and risks of AI

Big Picture #

flowchart TD
    A["Artificial Intelligence"] --> B["Human-like"]
    A --> C["Rational"]
    B --> D["Think like humans"]
    B --> E["Act like humans"]
    C --> F["Think rationally"]
    C --> G["Act rationally"]

    style A fill:#C8E6C9
    style B fill:#E1F5FE
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#EDE7F6
    style F fill:#C8E6C9
    style G fill:#C8E6C9

1. What Is Intelligence? #

Intelligence is not a single ability. It involves several capabilities working together, including:

Basic Statistics

Basic Statistics #

Statistics: describes data (what you see).
Probability: models uncertainty (what you don’t know yet).

• Summarise a dataset using central tendency and variability
• Explain core probability ideas using simple examples
• Apply the axioms of probability
• Distinguish mutually exclusive vs independent events

flowchart TD
    A[Dataset] --> B[Central Tendency]
    A --> C[Variability]
    B --> B1[Mean]
    B --> B2[Median]
    B --> B3[Mode]
    C --> C1[Range]
    C --> C2[Variance]
    C --> C3[Standard Deviation]
    C --> C4[IQR]

Measures of Central Tendency #

Central tendency tells you where the “middle” of the data is. Describes a set of scores with a single number that describes the PERFORMANCE of the group.

M1: Foundations

M1: Foundations #

This module establishes how to measure ML system performance, identify bottlenecks, and decide which work can run in parallel. It connects algorithmic complexity with the capabilities and limits of CPUs, GPUs, and clusters.

Module coverage:

• Performance Metrics and Complexity Analysis: time and space complexity, throughput, latency, and speedup through Amdahl’s Law.
• Parallelisation Paradigms: data-level, task-level, and algorithm-specific parallelism.
• Modern Hardware Architectures: multi-core CPUs, GPGPUs, GPU/CPU clusters, and resource management.

Learning Objectives #

• Distinguish complexity from measured running time, throughput, and latency.
• Explain the limits of speedup and the cost of communication.
• Choose between data, task, and request parallelism for a workload.
• Relate memory hierarchy, bandwidth, and hardware organisation to performance.

Pages #

Topic Outline #

1. ML and DL System Performance #

• Time complexity of algorithms and running time
• Performance scaling and tuning
• Training versus deployment
• Distributed and cloud environments, embedded devices, and mobile systems
• Throughput and response time

2. Parallel and Distributed Algorithms #

• Systems and performance
• Speedup: approaches, issues, and Amdahl’s Law ☆
• Data parallelism, task parallelism, and request parallelism
• Scale-out clusters: communication cost and its impact on speedup
• Scale-up versus scale-out

3. Parallel Programming Models #

• Parallel programming models and the MapReduce pattern
• Task-parallel and request-parallel execution
• Multi-core and GPGPU parallelism: SIMD versus MIMD
• Memory hierarchies and bandwidth considerations
• CPU-only and GPU-accelerated distributed clusters
• Horizontal and vertical data sharding
• Parameter-server and all-reduce paradigms

How This Module Connects #

Performance measurement and parallel execution provide the basis for M2: Distributed ML Algorithms. Communication and memory limits remain relevant throughout the later modules.

ML and DL System Performance

ML and DL System Performance #

Machine learning system optimisation begins with measurement. Before changing an algorithm, adding processors, or moving work to a GPU, we need to understand what is slow, which resource is limiting performance, and how performance changes as the workload grows.

Course coverage:

1. Metrics: time complexity, running time, space, throughput, latency, and response time
2. Performance scaling and tuning: measuring bottlenecks and predicting how a workload grows
3. Training versus deployment: different objectives, workloads, and constraints
4. Deployment environments: distributed and cloud systems, embedded devices, and mobile systems

Learning Objectives #

By the end of this page, you should be able to:

Reinforcement Learning

Reinforcement Learning #

Reinforcement Learning (RL) is a learning approach in which an agent improves its behaviour by interacting with an environment and observing the rewards produced by its actions.

Unlike supervised learning, the agent is not given a correct action label for every situation. It must discover useful behaviour through trial, feedback, and repeated interaction.

Why Reinforcement Learning? #

RL is useful when decisions affect what happens next and the quality of an action may only become clear later.

Understanding and Generation

NLP - Understanding and Generation #

• The Study of Language.
• Applications of Natural Language Understanding.
• Evaluating Language Understanding Systems.
• Different Levels of Language Analysis.
• Organisation of Natural Language Understanding Systems.

Learning Objectives #

• Explain what Natural Language Processing studies and how it relates to artificial intelligence and linguistics.
• Describe why human language is difficult for computers to process.
• Recognise the main applications and stages of an NLP pipeline.
• Distinguish morphological, lexical, syntactic, semantic, pragmatic, and discourse analysis.
• Explain the relationship between natural language understanding and natural language generation.
• Identify suitable ways to evaluate different NLP systems.

Big Picture #

flowchart TD
    A[Human Language] --> B[Acquire and Prepare Text or Speech]
    B --> C[Analyse Language Structure]
    C --> D[Represent Meaning and Context]
    D --> E[Model or Reason]
    E --> F[Understand, Predict or Generate]
    F --> G[Evaluate and Improve]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. What Is Natural Language Processing? ☆ #

Natural language processing (NLP) is the discipline of building machines that can manipulate human language - or data that resembles human language - in the way that it is written, spoken, and organised.