While this post is dated August 27th 2021, this was actually posted on December 12th 2022. After I graduated from my masters in November of this year, I thought it would be a good idea to go back and reflect upon my experiences and found that I actually have already written this reflection piece about the first year! For simplicity, I’ve put the date on this post to be under the original date when I first wrote most if not all of it. I did make some edits in December but they only consisted of minor changes for polish.

Having finished the first year of my master’s program a few months ago, I thought this would be a good time to reflect and think about how my experience was and some of the learnings I had in my first year.

Entering the Master’s Program

For some context, I think it’s best to describe a little bit of how I ended up in the current program in the first place. During my undergraduate time at Georgia Tech, I had opportunities to work on research projects with my professors. Compared to classes, research projects were free form and it was amazing learning about and investigating issues that I was curious about. It felt like you were on the frontier of knowledge and making steps to try new things, which was in vast contrast to classes which focused more on learning about existing knowledge and applying it. I was interested in math, computer graphics, and reinforcement learning and research really gave me the opportunity to dive deep, really understand these topics in-depth, and attempt to make contributions to what essentially is human knowledge. Furthermore, I have always been a sucker for tough problems and puzzles, and that is exactly what doing research consisted of. In contrast to internships that I was doing which felt more like only applying the knowledge I learned, research also felt important and consequential, in comparison to working on products that I didn’t have much of interests in.

During my undergraduate time, there were professors looking for undergrads to do more manual work. I chose explicitly not to pursue these opportunities because I wanted to see what it was really like to be doing the research rather than working strictly on implementation. Luckily, I had opportunities to do research projects with multiple professors that felt like real research. The downside is that research is also hard! And that I didn’t end up with any publications to show for my hard work. It’s a little sad, but in the end I really appreciated the opportunities and it motivated me to continue pursing research.

Out of all the topics that I was interested in, I was fascinated with reinforcement learning. It was such an interesting problem to me to be creating AI algorithms that can learn and adapt to environment in a reasonable fashion. I’m the person that wrote strategy guides to games that I played, so this idea of trying to distill down the process of interacting with the world to form knowledge into an algorithm was totally within my wheelhouse. RL research can often get you to think about your own learning process and put a microscope in how complex the “simple” act of interacting with the every day world is. Just as studying art can make you appreciate the beauty of the world, I think studying RL made me better appreciate the flexibility and adaptability of creatures and their ability to perform even simple actions that we have trouble getting a robot to do.

So, along with the advice of several professors that I chatted with, I applied to PhD programs! I applied only to RL PhD programs because that is where I felt I was really interested in. And because RL faculty seemed relatively sparse, I also applied to master’s programs in Canada (I learned that in the rest of the world, doing a Masters before a PhD is required). Here’s a list of schools that I applied to:

1. Stanford
2. UC Berkeley
3. CMU
4. MIT
5. University of Washington
6. UBC
7. McGill
8. UofA

Each and every school had professors that was working in RL which was what made me apply in the first place! Sadly, but also luckily, I only got accepted by UofA. I thought my chances was higher given I had worked in previous research projects before with likely good recommendations, but perhaps it was due to my lack of publications or other reasons that I wasn’t accepted to any other school. That is not to diminish the fact that RL is big in UofA with many faculty working in the area, so I can’t really complain.

Thoughts on People

Even though I done my first year remotely at UofA, the people that I’ve worked with are pretty awesome. I’m the guy that talks in class to help the professor out and just to interact in general but I found many other like-minded individuals that are also pursuing interesting topics in RL research. My advisor has been awesome so far in being accommodating and communicating the expectations for a master’s student.

One thing that I really liked about everyone at RLAI is that they really care about doing good research and advancing towards their longer term research goals. What disheartens me about the field of RL is the lack of progress that seems to have happen with difficult to reproduce papers that are all racing to create the next state of the art algorithm, often in exchange for massive computational cost or engineering efforts that isn’t clear from their papers. Faculty at UofA feels like there is a sharper focus on research towards a specific vision of what AI or RL is in the future, while being more forward with the limitations of the experiments performed. I think that focus moves their research towards understanding,and running more precise experiments to help inform research directions and development. That dedication to a research direction and honesty about the limitations of our own work and others work is something I really appreciate.

Thoughts on Work

I think I did not really come to realize how I felt about my first year of my master’s program until I was at Riot for the summer. Combining classes, research projects, TA, and the self-imposed pressure of wanting to do more, I was definitely overwhelmed. I managed to push through to where I am now, but it was not without its difficulties.

The elephant in the room is definitely online classes. As everyone will tell you, online classes sucks. For me, I felt that online classes had this particular draining effect on me that I didn’t quite feel quite strongly during my in-person undergraduate classes. I think partially it is due to needing to be out and about to go to classes in the first place, and that the lack of physical exercise, going outside and taking a walk to the lecture hall, made classes particularly draining for me. I feel that sitting in front of a screen for isn’t really an issue, but that online classes accentuated the passive nature of classes and made classes feel less interactive and more tiring. After attending a lecture, I would be so motivationally drained and be so tired that it would be difficult for me to do work afterwards. So instead, I would spend time watching videos and playing games. Thinking back, maybe there were things that I could have done to refresh myself, but I really do not think there are much I could have done in that state.

Things were a little more bearable in the fall semester because my two classes were both on Tuesday Thursday since I did not have that motivational drain on my other days. On the flip side, it made Tuesday and Thursdays really bad days for me. For my winter semester, I had to attend one class Monday through Thursday, which I felt made the whole ordeal tougher.

Graduate school so far definitely makes me think that I should be doing a lot more than I am right now. For courses, I was expected to read a mountain of class readings that I’m not sure how the professor expects us to have time to read and complete long and difficult class assignments in addition to course projects. Aside from this, I felt like I should also be looking for an advisor, working on side research projects, making sure the various degree requirements are getting done on time, reading papers in my field of interest, performing TA duties, and exercising to keep yourself healthy. I felt like I had so much to do that I was always dropping the ball somewhere, and the sheer amount of things that occupied my mind at times made it paralyzing to work on any particular project because I would always be thinking about something else. This is accentuated by the fact that many of these duties have rather nebulous definitions of done. And since there would always be more that you can do, it made these tasks both easy to avoid and difficult to feel accomplished when I did muster up the motivation to put in some work.

I think the overwhelming amount of things I was focused on, along with online classes, were a major blow to my mental health especially during my second semester. For me, it was just really hard to feel productive during that time. There was that pressure to put in at least 8 hours of work per day, there was online classes putting a drain on my energy and motivation, and there was a mountain of nebulous work waiting for me around every corner that just kept coming. The lack of structure also did not help. I felt tired during the morning, only motivated to work from the guilt of not working. I stopped exercising because I felt like there was always something school related that I should be doing instead the moment I started, or that I didn’t have the energy for it. My work day would feel super long because I would be unmotivated and bored during work, but also feel super short because the amount of work that got done was so nebulous so it was hard to understand whether you were actually productive or not. On top of that, I’d beat myself up for not working as much as I can during the day, get tired and sleepy during classes, and then go play games and watch videos till late at night. Reading what I wrote during the peak of my stress, I can retrospectively tell that I was definitely depressed.

Experiments

In brief terms, working on experiment SUCKS. I cannot begin to describe how much running experiments and analyzing experiments can be one of the most time wasting and unproductive thing I felt compared to any other type of work that I’ve had to do. Let me lay out the reasons why I think so:

1. The first thing is that experiments can often take forever to run. It is like a way worse version of waiting your code to compile for a big project. It can take sometimes hours to just run a single experiment and you’ll be twiddling with your thumb all the while, only to find out hours later that you missed some crucial step and have to rerun everything.
2. Organizing data can be a pain. In a project, you’ll likely want to tweak different parameters again and again, making tweaks to all aspects of the algorithm, and then understanding which tweaks matter which do not. It is critically important to know where how a piece of data is generated in order to properly organize it. However, there will be hundreds of thousands of runs, and you will never fully capture everything you need the first time as the assumptions and parameters of the project evolves. This means that what you are tracking for each run, both the data needed to signify the process that generated this data, and the run itself, will change over time. This means that either all the previous data you have gathered is invalid, or they need to be modified in order for them to not be wasted. The sheer amount of data that you will need to gather, organize, and process, is kind of insane.
3. The large computational cost of experiments means that you will likely need to run it on another more powerful computer. At UofA, we generally use Compute Canada (CC), and that comes with a whole host of issues (which might or might not be specific to CC): figuring out how to transfer data efficiently across systems, setting up the cluster with the proper packages and software, and more. The error messages that Compute Canada gives you can be at times unhelpful. What’s worse is that some problems can’t be debugged without scheduling a job that could take a few hours in the queue, and when something goes wrong in the job, it might just freeze and crash without any error messages at all. All of this makes working with the cluster time consuming and frustrating.
4. Bugs can be almost impossible to spot in experiments because your algorithm will just be not working. You need very targeted logging to catch any bugs, and its often hard to understand each intermediate steps of a complex algorithm in order to debug what it is supposed to do. You will also need to make sure you are not gathering these debugging data when performing larger experiments because otherwise your hard drive will get filled up real quick.
5. The lack of standardizations means that you will likely be implementing everything from scratch, which is error prone and pain staking to do. A lot of frameworks also are just super hard to understand, and not flexible enough. They also often not really designed for researchers, making it very hard to tinker and tweak the various parts of the algorithm which we often do in research.

All in all, writing code for and running experiments is a frustrating and time consuming process. The feedback loop is terrible, the data management aspect is difficult, and really, it is very hard to feel productive when running experiments because these experiments can take days to implement and run, only for the data generated to be useless and thrown away.

Anyways, that’s it for this one. Hopefully, I didn’t come across as overly negative in this reflection. Year 2 will certainly be quite different from this past year, with me actually being in Edmonton, and being able to focus solely on research without needing to take any classes. Can’t wait to see what will be next!