Unity AI doesn't take speed variable in regard when following GameObject

Question

I am trying to make a Pong AI that follows the ball on the Y axis in order to defend from getting scored on.

The problem is, no matter what value I set it's speed variable to, it will always go as fast as it has to in order to defend the ball from reaching the goal, which isn't what I'm aiming for because this obviously doesn't let the player ever score.

Can someone tell me why it doesn't take in regard the speed variable?

public class EnemyController : MonoBehaviour
{
    [SerializeField] private GameObject ball;
    [SerializeField] private float speed = 1.75f;
    private Vector2 ballYPos;

    void Start()
    {

    }

    void Update()
    {
        float step = speed * Time.deltaTime;
        ballYPos = new Vector2(transform.position.x, ball.transform.position.y).normalized;
        transform.position = Vector2.MoveTowards(transform.position, ballYPos, step);
    }
}

I tried changing the speed variable but nothing ever changed. It still went as fast as it had to in order to defend.