Conflict in a story arises from desire. The basic structure of any story is that the protagonist has a desire and there are forces or people who oppose their attaining that desire. The story proceeds as they attempt to fulfil that desire and face increasing obstacles leading to a moment of truth in which we discover just what they are willing to do to attain that desire, what they are willing to give up, how much they are willing to bleed.
Sometimes that desire is for something they don't have. Sometimes it is to retain something they do have that someone wants to take away from them. Sometimes in the course of the story they come to understand that something else matters more to them than the thing they thought was their greatest desire. But the basic arc of a story is formed by their desire, the things that stand in the way of that desire, and what they will do (or not do) to attain that desire.
Conflict comes from the forces that stand in the way of the protagonist attaining their desire. Conflict also comes when the protagonist must decide what they are willing to do to attain that desire, and the changes that may bring to their relationship with others. Conflict is not an incidental to spice up a story, therefore. It is of the essence of story. If there is no conflict there is no story because there is either no desire or nothing stands in the way of that desire -- or at least nothing that challenges the protagonist to stretch themselves to the limit to attain that desire.
At the same time this means that you don't have to do a thing to create conflict in you story. If you have the basic elements of story: the desire, the forces that oppose that desire, and the character of the protagonist who pursues that desire, conflict will inevitably fall out of character's pursuit of their desire.
Multiple characters may have their own story arcs, and sometimes the conflict may arise from the interference of one character's pursuit of their desire with another character's pursuit of their own desire. Multiple intersecting story arcs are often the sources of conflict in a complex work, but the basic formula is always the same: desire creates conflict between the protagonist and the forces that oppose their desire.
So, to your specific questions:
Is there any way to know when conflict should happen, so it doesn't feel too sparse or conversely over burdened with drama? -- Yes, conflict is the result of desire. Establish what your hero's desire is and what forces oppose that desire. Then start them on the quest to achieve that desire. The result is conflict. (At each step in the story, the hero will logically try the least costly thing they think might work, so conflict escalates through the story as they try more and more costly things, culminating in the highest cost they are willing to pay.)
Why should conflict happen? Is there a trick or formula to creating conflict? -- Yes, conflict is a result of desire and its frustrations. Find the desire. Find the reasons why that desire cannot be achieved. Bingo. You have conflict.
Once I've figured out when and why, are there any rules to actually writing it? -- No, conflict is a natural consequence of a hero pursuing their desire in the face of opposition. Find the desire, find the opposing forces, and march your hero forward. Given that your protagonist wants X, and given who they are, and given the the antagonist wants Y, which is incompatible with the protagonist attaining X, and given who the antagonist is, what does each of them do next.