In one of my favorite movies, No Reservations, there’s a scene where a young girl played by Abigail Breslin is prepping vegetables in a restaurant kitchen alongside her aunt, a chef played by Catherine Zeta-Jones. Seeing truffles on the cutting board, she sniffs them, winces in disgust, and throws them in the trash.

Who could blame her? Truffles kind of look like turds, and they smell like musty dirt. My own first encounters with truffles came when my brother had returned from Belgium with a bottle of this golden oil in a tiny bottle. He told me how expensive the small amount of truffle oil was, and when we dashed it on some french fries, I instantly caught the aroma—strong, rich, and earthy. It was good, but in the same way a really rich chocolate cake is good. Have one bite, it’s amazing. But after a few, it’s too much.

A few months ago, I had the opportunity to taste a black truffle for the first time. As a food writer, I was at a press dinner trying some dishes that featured truffles, and the chef took a shiny little tool and shaved off slivers, revealing an inside that was speckled, almost marbled. It was beautiful, and when I placed the slice in my mouth, it melted elegantly, but… there was honestly no release of flavor, or even texture. I didn’t detect any earthy mushroom taste, nor did I get that hit of richness or pungent aroma I’d experienced from the oil years ago.

I turned to a representative from the truffle oil company and expressed confusion. “Is it just me or does this not taste like much?” I asked. Was I not picking up on something? That’s when she let loose with what is now the Big Truffle Lie, in my mind: Truffles have to be mixed with fat to get that rich flavor, and there are a lot of people and products in the industry that replicate that richness synthetically, while still charging a pretty penny.

Why are truffles so special and expensive in the first place, if they taste like nada? A truffle is a mushroom, yes, but it’s not just any type of fungus. These shrooms grow under tree roots, mostly in Italy, and they get so pricy because they are impossible to cultivate, meaning they have to be foraged seasonally in the wild, usually with the help of hogs or dogs who have been specially trained for years. That scarcity drives up the price, and over centuries truffles became associated with luxury in European cuisine.

If you’re completely annoyed that something expensive and rare is being faked in oil, you aren’t alone. Peruse any list of “foods chefs actually hate,” and you’ll see truffle oil again and again. One of those chefs is Hassan Musselmani, a Detroit-based alumnus of Hell’s Kitchen and Chopped. “I mean, if you've ever had a real shaved truffle, you know that it tastes nothing like the perfume in a bottle that they give you,” he tells Delish.

That annoyance within the food world stretches back even farther. In 2007, chef and restauranteur Daniel Patterson penned an op-ed for the New York Times criticizing the use of chemically-enhanced truffle oil. “[T]heir one-dimensional flavor is also changing common understanding of how a truffle should taste.” That prediction was made nearly 15 years ago, and here I was at that dinner table experiencing the same disconnect.

Is the mere association with luxury enough to keep customers ponying up season after season—for fake oils, dishes infused with fake oil, or even for the real thing? Apparently so.

I talked to Megan Elias, director of the Gastronomy Program at Boston University, about how food can serve purposes aside from filling your belly. “I would argue that every food carries some kind of message about who you are,” she said. “People say okay, this is something tasty, but then the tastiness is compounded by the difficulty of getting it.” Meaning if only the wealthy can get it, it must be delicious, because the wealthy can get whatever they want. “It's very hard to tell how much you actually like it, or if you know you have to have a so-called ‘sophisticated palate’ to like it.”

So to the elite, truffles can be a sign of casual wealth, but for the rest of us, it’s mostly a special occasion splurge. They're a food choice that represents celebration, like lobster or caviar or Dom Pérignon. Truffles can be a treat—if they taste incredible to you.

Because here’s the thing: Everybody should determine their own special occasion food. When I’m celebrating, I’m always going to spend a little lavishly on oysters and a glass of fancy white wine. These items are worth it to me, because I love them and they make me happy, even if they do cost a little more. Do what makes your tongue happy! Just make sure you’re spending on the real stuff.