Lowering the capacity of a list involves a new backing array and copying the data across, so it's a relatively expensive operation.
In your particular case I would say it is not worth it unless you start hitting memory problems.
One strategy that can be employed if it were to become a real problem is to create a 'chunked' implementation of IList<>
which uses not one array, but multiple, each of preconfigured size , with additional chunks (fixed size arrays) added as the previous fills up. This also allows the list to shrink relatively inexpensively by releasing unused chunks as items are removed, whilst minimizing the memory overhead to just one non-full chunk (the last).
This approach adds a performance overhead to all operations on the list though, as the list has to calculate which chunk an items resides and create new chunks as required. So it is not useful unless you truly have a memory problem and a list that truly changes size dramatically over time.