Alt Title: The Uncomfortable Truth About Hard Problems

The Discomfort of Hard Problems

At the beginning of my second year in my Ph.D. I remember having a discussion with my PI, where he told me to try out an experiment/validation and I dragged my feet, instead choosing to refactor my code by adding logging, refactoring functions, and modularizing my code. I dragged my feet on it, because I knew that if the idea failed, I’d be back to the drawing board and would have had “wasted” a year. I told myself this was necessary work: if the experiment worked, I’d need scalable infrastructure for larger experiments on the supercomputer. Instead of embracing the suck and doing the (emotionally) hard work i.e. the validation, I dragged my feet by spending time on busy-productive work. Even though I knew at the time, that experiments fail for a multitude of reasons, and that it wasn’t a reflection on me, I couldn’t help but feel like someone more capable could have gotten to where I was faster. And yet, over a year later (now) I found a new substrate to test the idea and that refactored code turned out to be exactly what I needed, because cleaning up code (especially messy code) over a year later is nightmarish.

This experience reinforced something I’d been thinking about for a while: hard problems aren’t hard just because they require some esoteric knowledge, or involve many complex steps; they’re also hard because tackling those hard problems comes with a specific kind of discomfort that many of us aren’t used to dealing with. The steps to solving these problems can exist in a liminal space where you can’t tell whether you’re making progress or just spinning your wheels. Is the work I’m putting in going to pay off at some point down the line? Or is it all wasted effort?

The Yak Shaving Paradox

In the tech industry, there’s a concept called “shaving the yak” that perfectly captures this dillema; it has two almost contradictory definitions:

1. Any apparently useless activity which, by allowing one to overcome intermediate difficulties, allows one to solve a larger problem.

2. A less useful activity done consciously or subconsciously to procrastinate about a larger but more useful task.

This blog post has been the longest post (time-wise) I’ve worked on, because it was so hard to put into words what I was experiencing. It wasn’t until I was talking to one of my friends when that phrase about shaving the yak popped into my head, and everything flowed. That conversation made me realize that part of what makes research so difficult is that the same work can either be essential groundwork or elaborate procrastination, and you often can’t tell which you were doing until after the fact.

In my original example, I was clearly shaving the yak. If you had asked me after I killed that experiment, I would have lamented the time I had wasted and said that I clearly fell into the second camp. Yet even then, those improvements I made to my code have made revisiting the problem far easier. This ambiguity is what makes hard problems so uniquely uncomfortable. Let’s consider what happened immediately after. I had to write a report on my research from that first year. I was now faced with another dilemma: I had “killed” that experiment by that point, so do I put in a lot of time writing that essay by generating and polishing figures, rerunning simulations that would take a while, and doing a more thorough literature review -OR- do I throw in the towel, say that the results weren’t what I expected and move on to the next problem? Both choices felt like they could be yak shaving. Polishing a failed experiment might be pointless busy work to avoid the difficult task of identifying a new research problem, or it might be essential documentation of what I’d learned. Cutting my losses and moving on to the next project might be productive progress, or it might be my way of avoiding the hard work of understanding why the first approach failed beyond just throwing my hands up and saying “biology is messy”. The reason this ambiguity persists is that hard problems unfold over such long timescales that you’re making decisions with incomplete information about what will actually matter.

I can’t help but think of the sparse reward problem, where we only get feedback after a long time, and the credit assignment problem, where it is difficult to disambiguate the exact steps we took that led to our current outcome.

When Our Ego Gets Involved

The uncertainty becomes even more uncomfortable when we get emotionally invested in our approach. As I mentioned in my previous post, I can’t imagine going back:

What I mean is this: you need a sort of “balance” where you have to care about your research, and I mean deeply care about your research - it has to fill you with wonder and make you really contemplate the hard questions - but at the same time you have to be able to take your failures (and there will be many) on the chin and try the next idea

Many of us do what Cal Newport dubs “knowledge work”, where it is work that benefits from thinking deeply about problems and solving them. Unsurprisingly, our work naturally ties our self-worth to our ideas, methods and solutions to the challenges we face; not unlike how someone who works as a carpenter might tie their self-worth to the furniture they craft and the design decisions they make. When you can’t tell if your work is productive or just elaborate avoidance, questioning the work feels like questioning your judgment and intelligence; after all, does your lathe really need to be oiled? Or are you just avoiding that tenon and mortise joint? When you’ve been trained to have rock-solid evidence for every hypothesis, having your work questioned feels like having your competence questioned. My advisor asking me to carry out that validation scared me, because in a sense, that project was a part of who I was - it symbolized a year’s worth of time and effort. Thankfully, because I had already established that sense of self-worth outside of academia, I shrugged it off quickly, but that experience stuck with me. Learning to separate your self-worth from individual experiments or projects is crucial for navigating this ambiguity. Instead of being precious with your ideas, you accept that failure is multi-faceted and multi-factored; failure can happen because the biology is just too complicated, the recordings weren’t of high-enough resolution, or there was human error in the data collection.

Going back to School

That conversation I mentioned at the start was a bitter pill to swallow - it was emotionally painful to kill an experiment I’d spent a year working on. But it taught me something important about not being precious with ideas and learning to reflect on which form of yak shaving I’m doing (even if it’s an uncertain activity). In my previous blog post, I mentioned going back to academia because the research was progressing quickly, but that’s not the whole truth. I went back because I wanted to develop this exact skill of learning to navigate both forms of yak shaving in a “sink or swim” environment.

I don’t think that discomfort, the push and pull between productive and unproductive work, ever fully goes away. What’s worse is how easy it is to forget this dynamic and get wrapped up in busy work. But I do think you can develop better judgment about when to push through and when to step back, though it’s not something you can easily learn alone. You need honest mentors and a strong support network who will give you fair but direct feedback about your work. The skill isn’t eliminating ambiguity but learning to navigate it with guidance and practice.

This challenge isn’t exclusive to academia and I’m sure it applies across fields where long-term, uncertain work is the norm. To tie things back to the tech industry again, there’s a reason junior engineers don’t work on “bigger picture” projects: it’s hard to know where a field might turn and how to pivot seamlessly when you can’t yet distinguish necessary preparation from elaborate avoidance.