Thus far, you might've come to the conclusion that feature-based RL is the clear winner for designing most agents, and in general, these approaches have seen impressive performance.

Here's a depiction of the Approximate, Feature-Based Q-Learning we've been using up until now:

That said, there's a fairly large assumption that these feature-based agent's have had even beginning the learning process; what is that assumption?

That the features are already hand-picked, and the feature-extractor hand-crafted!

Plainly, this is too general an assumption, and for many domains, hand-picking and crafting the features can be a major drawback:

• We may not even know where to begin with choices of features, especially if the state-action space is too complex.

• Even if we could choose some good, representative set of features, encoding the feature-extractor can be impractical or time-consuming.

This may feel like a familiar dilemma from 485 -- wherein did it occur and what tools did we start to use to address it?

We saw this with some of our classification models, wherein it may be difficult to choose features and so instead rely on a neural network to, in some representation, learn them for us!

As such, for our next topic, we'll examine deep reinforcement learning, and how application of neural networks can apply to our tasks at hand!

Deep Q-Learning describes a class of approximate q-learners that, instead of a linear-q-value function, deploy some neural-network architecture to approximate q-values instead.

The benefits of using a NN for this task are that we no longer have to hand-pick features, but can instead rely on the network to focus on facets of the primitive state for us.

To understand a bit about how these techniques operate, we'll divert into a small review on their underlying tool.

Tiny Neural Network Review

---

Neural networks are, succinctly stated, general non-linear function approximators!

In brief: if you know what your inputs \(X\) are, and you know what constitutes "the right answer" \(Y_i\) to each input \(X_i\), you can deploy a neural network to learn the \(f\) in \(f(x) = y\)

Why does the property of being a non-linear function approximator sound appealing to us in the present task?

We can use it to estimate \(Q(s, a)\); it's a function after all!

Deep Q-Networks (DQNs) have many variant structures, but generally, possess an input layer that accepts the primitive state, some number of hidden layers to learn and weight features, and a final output estimating Q-values.

Notable parts of the above structure:

• Inputs: encode the primitive state. These can be as rich as the Pacman gameboard (when even those higher-level representations are available) or as low-level as the pixels composing the screen of a game.

• Hidden Layers: the "deep" in DQN, generally required to be many layers deep for more complex problem spaces; here's where the feature learning occurs (when it can), though is particularly opaque.

• Outputs: estimates for each Q-value.

• Weights: each edge in orange is accompanied by a weight, similar to those attached to features in our approximate q-learning.

• Activations: each unit \(Z_i\) has an output that is some differentiable, non-linear function of the weighted sum of its inputs.

The general form of learning in a neural network is still in finding the weights that best approximate the function trying to be learned, as mechanized by the backpropagation algorithm, through which differences / errors in the network's output vs. the expected target are used to tweak the weights to close the gap:

However, if we were attempt to naively deploy back-propagation in our online-Q-iteration algorithm, we run into some issues with DQNs:

Big takeaway: Q-learning (though it looks like it) is not the same as gradient-descent! Definition of a clear "target" is not well defined, because in the task of online exploration, our estimate of Q-values is always changing with new experiences!

Learning in a DQN

---

So, to address these issues, we turn to some clever tricks:

Decorrelating Samples: in order to prevent an issue known as "catastrophic forgetting" whereby lessons learned along a particular trajectory (correlated episodes) can be lost by learning later trajectories, we can deploy a replay buffer, a storage location for *all* episodes that can be sampled at random.

Effectively, this means that you can stash all of your agent's experiences in a memory bank, and then periodically reflect on them to learn!

Fixing a Target: in order to enable a gradient with a fixed target, we can introduce a second DQN known as the target network with a separate set of weights \(\phi'\) such that the target network provides a stationary bullseye, and is then periodically updated as the OG network learns.

Depicted, we see an iterative process by which the OG network is updated while sampling, and then essentially "leapfrogs" its weights with the target network.

Putting it all together leads us to the following algorithm (Summary credit to Dr. Sergey Levine):

Powerful stuff!

The drawbacks of deep-q-learning (in a nutshell):

• Neural Networks overfit (that's their job!), so transportability is an issue with Q-values estimated in this way.

• Interpretability of "learned" features from the network is low, so it's difficult to determine just what higher-order features the network finds important from the lower-level state.

• In practice, in order for weights to converge during training, DQNs require a well-oiled machine, from having good sampling, reward function, and a variety of the other concerns we've discussed in the past lectures.

There are entire classes devoted to Deep-Q-Learning that warrant a graduate-level coverage, but you should look into some of these tools on your own!

Still, there are others who believe that Deep-Q-Learning may be another passing phase in which different tools are needed to overcome their transportability issues, so while these might be some of the most modern approaches, just note that they are not the ultimate form of RL -- no one knows that yet!

BUT! DQNs were just the start of the deep-learning-applied-to-RL pipeline... we'll take a look at some more modern developments next time.