Alt Title: The Uncomfortable Truth About Hard Problems

The Discomfort of Hard Problems

At the beginning of my second year in my Ph.D., I remember having a discussion with my PI where he told me to run a validation for my experiment. Instead, I dragged my feet and chose to clean up my code (adding logging, modularizing functions and general refactoring). I told myself that my cleanup was worthwhile: when the validation showed positive results, I’d need scalable, correct code for larger simulations. But I was really avoiding the validation because I knew that if my experiment failed the validation step, I’d be back to the drawing board after “wasting” a year. Instead of embracing the possibility of failure and doing the emotionally hard work, I spent time on busy-but-productive tasks. As my PI suspected, the validation killed that experiment. Even though I knew experiments fail for countless reasons and that failure wasn’t a reflection of my abilities, I couldn’t help but feel like I had failed.

Having said that, that’s not the story I’m trying to tell here - what I described happens all the time in research and is often talked about. What I’m here to discuss is what happened later, over a year later in fact, where I found a new substrate to test the idea and that the code cleanup actually saved me time; cleaning up code (especially messy code) over a year later is nightmarish and bug-prone.

This experience reinforced something I’d been thinking about for a while: hard problems aren’t just difficult because they require specialized knowledge, or involve many complex steps; they’re hard because they exist in a liminal space where you genuinely can’t tell if you’re making progress or just spinning your wheels. We’re trained to break down problems into manageable steps, but what happens when you can’t tell if those steps are worthwhile? Is the work you’re putting in going to pay off down the line, or is it all wasted effort?

The Yak Shaving Paradox

In the tech industry, there’s a concept called “shaving the yak” that perfectly captures this dillema; it has two almost contradictory definitions:

1. Any apparently useless activity which, by allowing one to overcome intermediate difficulties, allows one to solve a larger problem.

2. A less useful activity done consciously or subconsciously to procrastinate about a larger but more useful task.

This blog post has been the longest post (time-wise) I’ve worked on, because it was so hard to put into words what I was experiencing. It wasn’t until I was talking to one of my friends when that phrase about shaving the yak popped into my head, and everything flowed. That conversation made me realize that part of what makes research so difficult is that the same work can either be essential groundwork or elaborate procrastination (the busy-but-productive work), and you often can’t tell which you were doing until after the fact.

In my original example, I was clearly shaving the yak. If you had asked me after I killed that experiment, I would have lamented the time I had wasted and said that I clearly fell into the second camp. Yet even then, those improvements I made to my code have made revisiting the problem far easier. This ambiguity is what makes hard problems so uniquely uncomfortable. Let’s consider what happened immediately after I killed that experiment. I had to write a report on my research from that first year and I now faced another dilemma: I had “killed” that experiment by that point, so do I put in a lot of time writing that essay by generating and polishing figures, rerunning simulations that would take a while, and doing a more thorough literature review -OR- do I throw in the towel, say that the results weren’t what I expected and move on to the next problem? Both choices felt like they could be yak shaving. Polishing a failed experiment might be pointless busy work to avoid the difficult task of identifying a new research problem, or it might be essential documentation of what I’d learned. Cutting my losses and moving on to the next project might be productive progress, or it might be my way of avoiding the hard work of understanding why the first approach failed beyond just throwing my hands up and saying “biology is messy”. The reason this ambiguity persists is that hard problems unfold over such long timescales that you’re making decisions with incomplete information about what will actually matter.

I can’t help but think of reinforcement learning and the sparse reward problem, where we only get feedback after a long time, as well as the credit assignment problem, where it is difficult to disambiguate the exact steps we took that led to our current outcome.

When Our Ego Gets Involved

The uncertainty becomes even more uncomfortable when we get emotionally invested in our approach. As I mentioned in my previous post, I can’t imagine going back:

What I mean is this: you need a sort of “balance” where you have to care about your research, and I mean deeply care about your research - it has to fill you with wonder and make you really contemplate the hard questions - but at the same time you have to be able to take your failures (and there will be many) on the chin and try the next idea

Many of us do what Cal Newport dubs “knowledge work”, where it is work that benefits from thinking deeply about problems and solving them. Unsurprisingly, our work naturally ties our self-worth to our ideas, methods and solutions to the challenges we face; not unlike how someone who works as a carpenter might tie their self-worth to the furniture they craft and the design decisions they make given a client’s requirements. When you can’t tell if your work is productive or just elaborate avoidance, questioning your process and methodds feels like questioning your judgment and intelligence. Most of us took some form of a science class during our schooling years, where we learn about how to construct hypotheses by searching through literature, forming a testable question, and most importantly, defending our position.

My advisor asking me to carry out that validation scared me, because in a sense, that project was a part of who I was - it not only symbolized a year’s worth of time and effort, but was the product of everything I was, a scientist. Thankfully, because I had already established that sense of self-worth outside of academia, I shrugged it off quickly, but that experience stuck with me. Learning to separate your self-worth from individual experiments or projects is imperative in navigating this ambiguity. Instead of being precious with your ideas, you accept that failure is multi-faceted and multi-factored; failure can happen because the biology is just too complicated, the recordings weren’t of high-enough resolution, or there was human error in the data collection.

Going back to School

That conversation I mentioned at the start was a bitter pill to swallow - it was emotionally painful to kill an experiment I’d spent a year working on. But it taught me something important about not being precious with ideas and learning to reflect on which form of yak shaving I’m doing (even if it’s an uncertain activity). In my previous blog post, I mentioned going back to academia because the research I had my eye on was progressing quickly, but that’s not the whole truth. I went back because I wanted to develop the skill of navigating both forms of yak shaving in a “sink or swim” environment.

I don’t think the discomfort I mentioned at the start, the push and pull between productive and unproductive work, ever fully goes away. What’s worse is how easy it is to forget this dynamic and get wrapped up in busy work - it happens all the time, and even to the best of us. But I do think you can develop better judgment about when to push through and when to step back, though it’s not something you can easily learn alone. You need honest mentors and a strong support network who will give you fair but direct feedback about your work. The skill isn’t eliminating ambiguity but learning to navigate it with guidance and practice.

This challenge isn’t exclusive to academia and I’m sure it applies across fields where long-term, uncertain work is the norm. To tie things back to the tech industry, there’s a reason junior engineers don’t work on “bigger picture” projects: it’s hard to know where a field might turn and how to pivot seamlessly when you can’t yet distinguish necessary preparation from elaborate avoidance by for example, moving to the new shiny tech stack.