Span text pushes other elements (buttons) to the right and left-CodePudding

I am trying to create an voting system for my website but am currently struggling with upvote and downvote counters styling. Here is how it looks right now:

As you can see, the problem is that whenever the number on the right or left side of the buttons gets large, it pushes other elements. I want the two buttons to stay in the center and the upvote counter to grow to the left, and downvote to the right. This way, everything will be centered.

Here is my current CSS code:

.upvoteButtonsContainer{
  display: flex;
  justify-content: center;
  margin-top: 15px;
}

.upvoteButton{
  margin-right: 2px;
  border: none;
  background-color: transparent;
  box-shadow: rgba(0, 0, 0, 0.24) 0px 3px 8px;
  border-radius: 5px;
  font-size: 25px;
}

.downvoteButton{
  margin-left: 2px;
  border: none;
  background-color: transparent;
  box-shadow: rgba(0, 0, 0, 0.24) 0px 3px 8px;
  border-radius: 5px;
  font-size: 25px;
}

.upvoteCounterContainer{
  display: block;
  direction: rtl;
  text-align: right;
}

.downvoteCounterContainer{
  display: block;
  direction: ltr;
  text-align: left;
}

.upvoteCounter{
  margin-right: 5px;
}

.downvoteCounter{
  margin-left: 5px;
}

And my HTML code:

<div className="homePage">
      {imageLists.map((image) => {
        return (
          <div className="container">
            <div className="meme">
              <img src={image} className="meme-img"></img>
            </div>
            <div className="upvoteButtonsContainer">
              <div className="upvoteCounterContainer">
                <span className="upvoteCounter">120</span>
              </div>
              <button type="button" className="upvoteButton">