Professor Andrew Forney • Andrew.Forney[at]lmu.edu

Cognitive Systems are loosely defined as automated agents that exhibit animalistic or higher-level human intellectual capacities such as the ability to explain their choices, think counterfactually, and learn from examples, rewards, and punishment. The ever-diversifying field of AI boasts two primary subdisciplines that investigate how to develop, research, and theorize over the nature of such agents. This course examines these two fields from a formal, computational standpoint, while weaving in some considerations from the philosophy of science as well; in particular, we will explore the theory, applications, and opportunities to concert:

1. Reinforcement Learning: a paradigm for training sophisticated, but somewhat animalistic, agents based on the principles of positive and negative reinforcement from online, active experimentation.

2. Causal Inference: a theoretical framework built atop probability and statistics that allows for an interpretable and rich set of inference queries that follow from modeling systems of causes and effects.

Learning Outcomes

By the course's end, students will:

• Gain a deep understanding of modern theories of Causal Inference (CI), including the causal hierarchy, do-calculus, causal modeling, structural counterfactuals, and transportability.

• Obtain a practical understanding of some problems and approaches to Reinforcement Learning, including Multi-Armed Bandit problems (and applications to CI), Markov Decision Processes, Q-Learning, and Deep Q Networks.

• Appreciate the conceptual bridge that connects human cognition, attempts to automate it, and portions of experimental design as they relate to knowledge acquisition on an individual-basis.

• Experience working with popular reinforcement learning environments like with Pytorch and AI Gym, in causal modeling with Structural Causal Models, and a deepened understanding of probability and statistics.

• Identify some potential avenues for research, especially at the intersection of causal inference and reinforcement learning.

Prerequisites

Note: before taking this course, it is assumed that you have a matured grasp on the Python programming language, as well as the conceptual topics from Algorithms (CMSI 2130) and Artificial Intelligence (CMSI 3300). We will spend little time in-class reviewing each!

• Algorithms / AI Refresher: need a review? You can consult my course archives for the last offerings of CMSI 2130, 3300.

  In particular, you should *at least* be familiar with the following concepts:

  • CMSI 2130: Classical Search, Minimax, Dynamic Programming, Hillclimbing.

  • CMSI 3300: Probability Theory, Bayesian Networks, Sampling, Machine Learning (esp. learning algorithms like SGD).

• Python Refresher: consult your CMSI 1010 instructor's materials or any number of online tutorials like Berkeley's listed below. For practice, see our LMU CMSI resource page, also linked below.

• Git Refresher: you should be comfortable with the fundamentals of Git and managing GitHub repositories as GitHub Classroom will be used to distribute and collect all assignments in the course. Need a refresher? See the Git links below.

Texts

Given that there are two main, and rather distinct, topics covered in this class, we will employ 2 optional textbooks:

• Causal Inference [CI] Text: Causal Inference in Statistics: A Primer, by Pearl, Glymour, and Jewell

• Reinforcement Learning [RL] Text: Reinforcement Learning: An Introduction [Free]

Although these specific texts are not mandatory for the course, they contains many concepts that we will be covering in great detail, and your assignments will expect that you understand these concepts.

Additionally, I will provide abbreviated course notes for our lectures on my course page (see Notes tab)

Resources will be provided throughout the course detailing any additional topics that are not sufficiently contained in the above text. These resources will be free and publicly available.

Lecture Attendance

• Not mandatory to attend: lecture attendance is strongly advised since you'll otherwise miss out on all of our inside-jokes. Less importantly, you'll be missing out on lecture content that may not be summarized in the course notes, like questions asked by peers, classwork exercises, and hints for homework and exams.

• No need to inform me of missed lecture: I'll just assume you'll be making things up by reading online / asking friends.

• Lectures are participatory: come prepared to ask and answer questions to make the most out of your education!

• Laptops discouraged, notetaking encouraged: take notes by hand and summarize the important points -- don't copy verbatim.

Technology permitting, I will attempt to record the lectures, but you should not rely on lecture recordings for keeping up with content!

Workload Expectations

The university's workload expectations are such that every unit in which you are enrolled translates to 3 hours of weekly work, including time in-class, spent studying, and working on assignments.

• As this is a 4-Unit class, it is expected that you will be allocating an average of 9 hours outside of lectures per week working on course assignments, reading, and studying. You should manage your time appropriately.

• Some weeks will have less material to keep you busy, and others more, but be aware that this is a work-intensive course that may require you to spend a lot of time studying and programming!

LMU CMSI has a ton of resources and people who are thrilled to help you succeed! See the following opportunities and avenues for getting help in this class' material.

Slack Messaging

Alongside email exchanges, you may reach me via the Slack messaging service (Slack is a chat client that can be downloaded here) in the LMUCS channel (lmucs.slack.com) and username forns.

• Feel free to reach out any time -- in the worst case, I have to step away from my desk and will answer your message when I return!

• Please try to send your note / question in a single message rather than splitting it into like 9 different messages that make my computer sound like it's making popcorn.

Student / Office Hours are times during which I'm available for nothing but you, the students! I can help you: review course material, debug your programs, provide guidance on further exploration, and more!

• TR 10:15am - 11:45am, DOO 201A or 217 (they're right next to each other)

• W 10:00am - 1:00pm, via Zoom (see link in course welcome email)

• Questions by Slack ANY time!

Steps for Remote Office Hours Attendance:

1. I do not publicly post my Zoom office-hours link, but just send me a note on Slack *in advance of office hours* to see where I'm at (the "Starter Pack" email I send at the beginning of the semester also has this information).

2. My office hours tend to be well attended, so if you join into a waiting room, it means I'm helping other students and will admit you as soon as I'm free.

3. Once you've been admitted, you / your group will be placed into a Zoom Breakout Room with priority given to the order in which you joined. Wait in here and I'll be with you as soon as I can!

Teaching Assistants (TAs)

Teaching Assistants (TAs) are student employees who are on-staff during certain hours to help you review material, debug code, and answer questions about the class!

TA Roster and Schedule

Warning: this schedule is subject to change and may fluctuate with weekly TA availability; check the #tutoring channel on the LMUCS Slack for the latest updates to the TAs' schedules!

Respecting our TAs! Remember that TAs are students like you with their own obligations and schedules; here are some tips for how to make respectful use of their generous time and skill.

Bottom Line: remember that TAs are teaching assistants, but if you ever have questions or concerns that you don't wish to discuss with TAs, you should always feel free to contact the professor!

The following constitutes the tentative schedule for topics to be covered in the course. This is largely an approximation for what we will cover each lecture, and is subject to change. For each topic, the relevant chapter of the textbook is also given, however, you are only required to understand the sections of each chapter that correlate with the material in the course notes -- the rest are for your curiosity and edification!

* CN = Course Notes, indicating topics that will be covered and listed on our course website but are not found in the textbook, even though *all* topics will have an accompanying page in our course notes.

Grade Decomposition

Grades will be assigned based on the following weighted coursework:

• [70%] Assignments: Catch-all category for classwork + homework combo exercises, which will be assigned roughly every 3 weeks and be weighted equivalently. You can expect roughly 5 large assignments throughout the course of the semester, some of which will overlap with the period in which you will be completing your final project (see below).

• [30%] Final Project: a final project primarily in the domain of reinforcement learning but concerting elements of causal inference. The final project spec is available at the start of the course, but you will have the tools to begin work roughly 1/2 way through the semester.

Final Grades

Final letter grades are given based on the university scale of grade percentages:

• A: 93 - 100

• A-: 90 - 92

• B+: 86 - 89

• B: 83 - 85

• B-: 80 - 82

• C+: 76 - 79

• C: 73 - 75

• C-: 70 - 72

• D: 65 - 69

• F: 64 and below

Fractional grade percentages at 0.5 or over will be rounded up, so an 89.5% will be considered a 90% (A-), but an 89.4% will be considered an 89% (B+) on the above scale.

That said, these are only the guaranteed grade assignments: if your "final" grade is an 82%, you are guaranteed a B- or better, but you might still get an A if 82% is the top score.

Extra Credit

Extra credit opportunities will be sporadically available during lectures / exams; if you are present and can hand in an attempt at the extra credit opportunities, you will receive bonus points!

If you receive all extra credit opportunities, you can gain a maximum +2% on your final grade. These bonuses are applied post-curve, so no student will be punished for not attending lectures relative to their peers.

Submission Standards

Each of your submitted homeworks and projects are subject to the following constraints:

• Group Size Limit: You are free to work in groups for the course's assignments. The maximum group size for this semester is 4 students.

• Assignment posting: all assignments (classwork, homework, or otherwise) will be announced in class and posted on this site with due dates. You are responsible for attending the lectures / checking this site / your email for any updates to assignments and their associated deadlines.

• Late Policy: assignments are due at exactly the time indicated by the method specified. Assignments that are late by 0 - 24 hours over the deadline receive only 80% of the credit they otherwise would have. Assignments late by more than 24 hours receive a 0. There are no exceptions to this rule, plan accordingly.

  • Time management is one of the most important meta-skills to learn in this class -- avoid the risk and try to finish at *least* one day in advance of each deadline.

  • Good habits: read the specs the *day* they come out to let the topics simmer, put blocks in your calendar that you plan on working on the assignments.

• Submission Errors: "I forgot to push to GitHub" is not a valid excuse for late assignments; always double check your remote repository to make sure that your submission is present before the deadline. Moreover, assignments that contain syntax errors or deviate from given skeletons will receive a 0.

• Style: assignments which are sloppily submitted without proper formatting, style, and comments are subject to penalties. We will discuss what constitutes such offenses in the first classwork, though you should be familiar with clean coding standards from your introductory courses.

Academic Honesty

Students are encouraged to talk and think about problems in groups, but each submission should be their own, containing no code that has been copy-pasted from another student. We take plagiarism very seriously, and each submission will be run through a similarity-checking mechanism to ensure fairness. Code or homework which we believe to be shared innapropriately between students or copied from the internet is subject to severe disciplinary action (not that you would, just sayin'). The following are examples of unacceptable behavior in this course:

• Copying non-trivial amounts of code from the ANY other source into an assignment (excluding small things from StackOverflow like how to use Python list comprehension to simplify some code). This includes code found on the internet or through the use of a generative AI model like ChatGPT.

• Copying code from another student's submission, sharing detailed pseudocode, or having them verbally explain their submission line-by-line and you copying that.

• Copying another group's solutions on a classwork exercise (sharing between group members is encouraged... that's what it means to... you know... group up).

If you plan to copy anything from the internet, ask first!

The following slides provide more CMSI-specific standards for academic honesty; it is expected that by reading this syllabus you are aware of these standards.

CMSI Academic Honesty Standards

Rule of thumb: if anyone but you is responsible for your submitted work, you are liable for academic dishonesty! Remember that it is YOU taking this course to learn the tools of your chosen trade, your degree will be useless otherwise!

Note: this is an abnormally impacted section of this class! Your patience and hard work are appreciated in advance; we'll make the most out of this semester together!

With that said, here are some things you can expect from this class, and what I expect from you:

Community Agreement

Just as I have certain expectations of you in this course, you may have certain expectations of me. Here is a high-level list of each:

Tips for Success in this Class

Here are a few tips for succeeding in this class (and college, in general); they're not meant to sound patronizing, but rather, are simply pearls of wisdom to hand down from someone who has recently trodden your path!

• Ask questions: this is a brave space where inquiry is the heart of your education -- don't waste the opportunity!

  • Chances are good that someone else has the same question, free karma!

  • I might have made a mistake / typo and your question can correct it -- we're all human (I think)!

  • Don't feel like asking in-class? Jot a ? in your notes to ask later via email, Slack, or in Office Hours!

• Fear not the specter of error: it may not feel great to be wrong, especially publicly in a college class -- but I'm here to tell you there's no better time to be wrong!

  • You'll never be wrong about what you were wrong about again -- free studying!

  • In-class is the perfect time to be wrong -- there are no stakes for being wrong, unlike in an interview, exam, or while you're coding on the job!

  • We celebrate being wrong in here because everyone can learn from mistakes -- you'll be admired as the brave one who volunteered.

• Indulge your curiosity: and no, that's not some tagline stolen from a Vegas day-spa (actually, it might be, don't quote me on that).

  • We only have time to scratch the surface of class topics -- don't let your learning stop there, especially on topics that resonate!

  • Google related projects, experiment with questions of "what if" on different algorithms -- go forth and explore!

• Remember that you belong: no matter what you look like, believe, etc. remember that if you have a passion for computing and the hunger to learn, you are a welcome member of this class, department, and institution.

  • This includes having a growth mindset: even if you feel like you're behind or unprepared, know that everyone who has the drive can learn computing -- we believe in you!

  • No need to feel like impostors -- this material is challenging but will help you grow -- if it were easy, you wouldn't get anything from it!

  • Always feel free to let me know if you feel otherwise on any of the above: if you're feeling excluded for ANY reason, let me know, because if I can't help you, I'll find someone who can.

University Resources

Here are some LMU links for University Resources that might be of interest:

• Computer Science Department - Student Guide: There are some great review and practice guides in here too!

• Academic Degree Requirements and Policies: Pay particular attention to honor code and process.

• Disability Support Services (DSS)

• Student Codes and Policies

• Academic Resource and Learning Center