Decreasing all values of a dictionary without loop. Or alternative storage possibility

Question

I'm storing some data in a Math.net vector, as I have to do some calculations with it as a whole. This data comes with a time information when it was collected. So for example:

Initial = 5, Time 2 = 7, Time 3 = 8, Time 4 = 10

So when I store the data in a Vector it looks like this.

stateVectorData = [5,7,8,10]

Now sometimes I need to extract a single entry of the vector. But I don't have the index itself, but a time Information. So what I try is a dictionary with the information of the time and the index of the data in my stateVector.

Dictionary<int, int> stateDictionary = new Dictionary<int, int>(); //Dict(Time, index)

Everytime I get new data I add an entry to the dictionary(and of course to the stateVector). So at Time 2 I did:

stateDictionary.Add(2,1);

Now this works as long as I don't change my vector. Unfortunately I have to delete an entry in the vector when it gets too old. Assume time 2 is too old I delete the second entry and have a resulting vector of:

stateVector = [5,8,10]

Now my dictionary has the wrong index values stored.
I can think of two possible solutions how to solve this.

1. To loop through the dictionary and decrease every value (with key > 2) by 1.
2. What I think would be more elegant, is storing a reference to an vectorentry in the dictionary instead of the index.

So something like

Dictionary<int, ref int> stateDictionary =
    new Dictionary<int, ref int>(); //Dict(Time, reference to vectorentry)
stateDictionary.Add(2, ref stateVector[1]);

Using something like this, I wouldn't care about deleting some entrys in the vector, as I still have the reference to the rest of the vectorentries. Now I know it's not possible to store a reference in C#.

So my question is, is there any alternative to looping through the whole dictionary? Or is there another solution without a dictionary I don't see at the moment?

Edit to answer juharr: Time information doesn't always increase by one. Depends on some parallel running process and how long it takes. Probably increasing between 1 to 3. But also could be more. There are some values in the vector which never get deleted. I tried to show this with the initial value of 5 which stays in the vector.

Edit 2:
Vector stores at least 5000 to 6000 elements. Maximum is not defined at the moment, as it is restricted by the elements I can handle in real time, so in my case I have about 0.01s to do my further calculations. This is why I search an effective way, so I can increase the number of elements in the vector (or increase the maximum "age" of my vectorentries).
I need the whole vector for calculation about 3 times the number I need to add a value.
I have to delete an entry with the lowest frequency. And finding a single value by its time key will be the most often case. Maybe 30 to 100 times a second.

I know this all sounds very undefined, but the frequency of finding and deleting part depends on an other process, which can vary a lot.

Though hope you can help me. Thanks so far.

Edit 3:
@Robinson
The exact number of times I need the whole vector also depends on the parallel process. Minimum would be two times every iteration (so twice in 0.01s), maximum at least 4 to 6 times every iteration.
Again, the size of the vector is what I want to maximize. So assumed to be very big.

Edit Solution:
First thanks to all, who helped me.
After experimenting a bit, I'm using the following construction.
I'm using a List, where I save the indexes in my state vector. Additionally I use a Dictionary to assign my Time-key to the List Entry. So when I delete something in the state vector, I loop only over the List, which seems to be much faster than looping the dictionary.

So it is:

stateVectorData = [5,7,8,10]
IndexList = [1,2,3];
stateDictionary = { Time 2, indexInList = 0; Time 3, indexInList = 1; Time 4, indexInList = 2 }
TimeKey->stateDictionary->indexInList -> IndexList -> indexInStateVector -> data

Answer 1

You can try this:

public class Vector
{
    private List<int> _timeElements = new List<int>();

    public Vector(int[] times)
    {
        Add(times);
    }

    public void Add(int time)
    {
        _timeElements.Add(time);
    }

    public void Add(int[] times)
    {
        _timeElements.AddRange(time);
    }

    public void Remove(int time)
    {
        _timeElements.Remove(time);
        if (OnRemove != null)
            OnRemove(this, time);
    }

    public List<int> Elements { get { return _timeElements; } }

    public event Action<Vector, int> OnRemove;
}

public class Vectors
{
    private Dictionary<int, List<Vector>> _timeIndex;

    public Vectors(int maxTimeSize)
    {
        _timeIndex = new Dictionary<int, List<Vector>>(maxTimeSize);
        for (var i = 0; i < maxTimeSize; i++)
            _timeIndex.Add(i, new List<Vector>());

        List = new List<Vector>();
    }

    public List<Vector> FindVectorsByTime(int time)
    {
        return _timeIndex[time];
    }

    public List<Vector> List { get; private set; }

    public void Add(Vector vector)
    {
        List.Add(vector);
        vector.Elements.ForEach(element => _timeIndex[element].Add(vector));
            vector.OnRemove += OnRemove;
    }

    private void OnRemove(Vector vector, int time)
    {
        _timeIndex[time].Remove(vector);
    }
}

To use:

var vectors = new Vectors(maxTimeSize: 6000);

var vector1 = new Vector(new[] { 5, 30, 8, 20 });
var vector2 = new Vector(new[] { 25, 5, 23, 11 });

vectors.Add(vector1);
vectors.Add(vector2);

var findsTwo = vectors.FindVectors(time: 5);

vector1.Remove(time: 5);

var findsOne = vectors.FindVectors(time: 5);

The same can be done for adding times, also the code is just for illustration purposes.