Baseline Team

To kickstart your agent design, we have provided you with a team of two baseline agents, defined in baselineTeam.py. They are quite bad. The OffensiveReflexAgent simply moves toward the closest food on the opposing side. The DefensiveReflexAgent wanders around on its own side and tries to chase down invaders it happens to see.

File Format

You should include your agents in a file of the same format as myTeam.py. Your agents' logic (training and otherwise) must be completely contained in this one file, with the exception of a weights file (format of your choosing), which can contain any post-training data that is used to inform your agents' actions.

Interface

The GameState in capture.py should look familiar, but contains new methods like getRedFood, which gets a grid of food on the red side (note that the grid is the size of the board, but is only true for cells on the red side with food). Also, note that you can list a team's indices with getRedTeamIndices, or test membership with isOnRedTeam.

Finally, you can access the list of noisy distance observations via getAgentDistances. These distances are within 6 of the truth, and the noise is chosen uniformly at random from the range [-6, 6] (e.g., if the true distance is 6, then each of {0, 1, ..., 12} is chosen with probability 1/13). You can get the likelihood of a noisy reading using getDistanceProb.

Note that getAgentDistances returns one such noisy value to each of the agents (by index) from the perspective of the agent whose turn it was when it was called.

Distance Calculation

To facilitate agent development, we provide code in distanceCalculator.py to supply shortest path maze distances.

CaptureAgent Methods

To get started designing your own agent, we recommend subclassing the CaptureAgent class. This provides access to several convenience methods. Some useful methods (which can be called via self.methodName from within agents you design in the myTeam.py module) are:

GameState Methods

Many methods are parameterized by a GameState object, representing the current board-state or that of a successor following an action taken by the agent.

Here are a variety of helpful methods you can call on GameState objects:

The above represents only a snippet of the many methods you might find useful from the capture.py module; make sure to fully explore the methods available there for more info!

Maze Layout

The following is the maze layout in which the tournament / grading tests will occur, with key:

• % = walls

• = open tile

• . = standard pellet -- eat these and return to your side to score points!

• o = power pellet -- eat this to turn the defenders vulnerable!

• 1,2,3,4 = agent starting positions by agent index

The dividing line between the red and blue side is between X = 16 (red side) and X = 17 (blue side).

             1111111111222222222233
   01234567890123456789012345678901
0  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1  %..o%.%.%.%.%      %% ...  %o..%
2  %..%%       %  % % .       %%..%
3  %      %%%  .  %%% %% %%%      %
4  %  %%  ..%  %  %% %%  %..  %%  %
5  %   %  %%%  %%        %%%  %   %
6  %%% %%       % %%%%       %% %%%
7  %1 .%..% %%%  .  .% %%% %..%. 4%
8  %o..%..% %..        ..% %..%..o%
9  %3 .%..% %%% %.  .  %%% %..%. 2%
10 %%% %%       %%%% %       %% %%%
11 %  %%%        %%  %%%  %   %
12 %%  ..%  %% %%  %  %..  %%  %
13 %      %%% %% %%%  .  %%%      %
14 %..%%       . % %  %       %%..%
15 %..o%  ... %%      %.%.%.%.%o..%
16 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Training vs. Testing

• At the top of myTeam.py you'll find a space for Constants that will be used during training and deployment of your agent.

• Chiefly, note the constant named TRAINING, which is set to True if the agent is in training mode, or False if it is not. The implications of these settings are as follows:

  • If TRAINING = True, your agent will attempt to learn some weights / whatever data you'd like it to learn through repeated trials, which should be stored persistently in some data file called weights_TEAMNAME.json. The format of this file can be whatever you'd like, though a common suggestion is to store some JSON map of features / properties to their learned values (including record-keeping data like the schedule of exploration reduction during learning). During play, an agent that is training will take some exploratory moves, and update the weights as it learns; you may check this flag's state when choosing actions. Additionally:

    • If the agent is training and the weight file is *missing* from the same directory, you should create it and initialize the weights / values to 0 or some small random values.

      Important: your weights MUST begin at either all zeros or small random values; if you hand-code any starting weight values, you will be heavily penalized and disqualified from the tournament!

    • If the agent is training and the weight file is *present / found* in the same directory, you will continue training using those values.

  • If TRAINING = False, your agent will expect the weight file to be present, use these weights for all of its decisions, and will not take exploratory actions (unless you want some nondeterminism in its moves, but that's still part of the policy). The weight file will not be updated while the flag is set to False.

  • WARNING: there is a time.sleep(0.1) command in game.py that is useful in the animation of games, but can be set to time.sleep(0) while TRAINING to accelerate learning.

• Lastly, you will document whatever commands you used to execute the training (i.e., with what arguments you supplied the execution of capture.py). Save this in your config.txt file specified in the Deliverables section below.

• Note that to let an agent train across many games, you should run capture.py with the "quiet" flag -q (on some systems, -Q), which runs games without the GUI interface, and by specifying the -n # flag, specifying how many of these silent games you'd like to run.

• IMPORTANT: Your final submission (on GitHub Classroom) should contain your fully trained weight file, and have TRAINING = False by default.

Tasks and Challenges

To design your agents, whose behaviors must be developed through Reinforcement Learning, the following are common tasks that you'll want to perform:

• Design the training loop such that, whenever TRAINING = True, you have a well-designed infrastructure for where your agents will update their weights, choose to explore vs. exploit, etc.

• Create a feature extractor that, given the primitive state-action space, decomposes it into useful features that your agents can attend to, learn from, and weight.

• Design a reward function such that rewards are neither granted too densely nor too sparsely, and inform your agent as to what constitutes "good" and "bad" decisions given some decomposition of the state. This will, for some actions, depend on taking a "pre" and "post" action snapshot of the state to determine the reward, like figuring out if an attacking Pacman died from a move such that it was reset to the starting position after running into a defending ghost.

• Consider the exploration / exploitation tradeoff: you may want to involve a small exploration bonus to rewards during the formative days of your agents' training.

• Save your weights such that after each trial / game of training, "lessons learned" by way of any updates to the agent's policy are saved. Note: for convergence guarantees, weights should not exceed the range of \([-1, 1]\), but it's not a problem if they do so long as they're not exploding to huge numbers (commonly, can just choose rewards that are between \([-1, 1]\), feature values in the same range, and have a small learning rate to ensure this).

• [Optional] Read the noise such that your agents can combine noisy readings of opponent locations into usable estimates. Doing so will require some tools from 3300!

Model Tradeoffs

When considering just how you wish to design your agent(s), there are a variety of options for the architecture you use. Here is a quick rundown of approaches alongside some of their strengths and weaknesses.

FAQ

Here are some typical problems people face during their design, and some tips for getting around them:

I'm not quite sure where to start with the feature design, especially because some of the baseline features are NOT action-features--any tips?

During training, my weights are blowing up out of control, flipping from positive to negative, or otherwise updating weirdly; what gives?

In the BaselineAgents, their weights aren't between [-1, 1], so how come it works for them and not us?

Notice that they're called "ReflexAgents" because those weights are hard-coded and only encode the programmer's beliefs about the best relative weighting between them -- crucially, these weights are *not* learned. The explosion of weights happens from the q-learning update rule as it's learning, which is why *we* have to worry about weights outside of that range, and they don't. Functionally, it's fine if our weights exceed that range so long as they're not exploding to huge values that may eventually become infinity.

I'm not sure what arguments should go into my reward function, nor how to structure it? Tips on \(R(s, a, s')\)?

My agents are stuck in the starting hallway, and can't ever get to the rewards to learn -- how to push them out of the house?!

A few things to try: (1) a more dense reward function (i.e., more trickles of rewards when they take actions that move them towards some objective), (2) the optimistic sampling discussed in lecture (largely applies to exact Q-learners, but you can spoof the larger rewards for rarely explored action-states).

My agent looks like it's doing well, but then just stays at home right after it dies! What happened?

You're dealing with the attribution problem! Your agent probably died while it was in pursuit of a pellet, so the big negative reward it got gets associated with the feature sensitive to the pellet's proximity. You can get around this by having *multiple* reward and q-functions that emulate the different types of rewards that humans experience, for instance hunger, thirst, comfort, etc. with potentially different features that are predictive of each. This is what we referred to as "reward splitting" in the lecture, wherein it's a decent idea to do this for the partition of negative vs. positive rewards. Alternately, you can *selectively disable* (i.e., set to 0) certain features that you don't want to be active for certain rewards, e.g., setting food-sensitive features to 0 when near a ghost.

My agent has features that compute some distance to a target, and the weight associated with it is positive after training and being sensitive to some reward, but it doesn't move towards the target as expected -- what gives?

Warning: What follows is a semi-elaborate workaround that gets people unstuck in some circumstances, but features like this can *usually* be better engineered into some binary/one-hot encoding format to avoid numerical issues like the following. It's likely that your feature associated with the distance from a given location to the target is getting smaller the closer the agent gets to the target, which (if the weight is positive) means that states *farther* from the target will have a *larger* feature value, and thus appear more preferable. A quick trick is to instead change the feature to grow as the agent gets closer to its destination, which can be accomplished by taking C - dist(s, a) where C is some constant like the maximum maze distance, and dist(s, a) is the distance to some target from state s' after taking action a in state s. It's generally also a good idea to normalize this feature by C to avoid weight explosions.

My game crashes with a weird error message about some window when using the -n launch parameter to play n games in series -- what gives?

On some Mac systems, this is caused by an issue with the tk graphics library used for rendering. In graphicsUtils.py, uncomment line 78 and try again, this often fixes things (but on other systems would actually break it lul).

Restrictions

You are free to design any agent you want. However, you will need to respect the provided APIs if you want to participate in the competition. Agents which compute during the opponent's turn will be disqualified. In particular, any form of multi-threading is disallowed, because we have found it very hard to ensure that no computation takes place on the opponent's turn.