Any object that exists with memory usage like a generator but can return a numpy array when necessary?

Question

I have some time data which starts at T0 and goes to T1 in steps of dt. This data is in small increments of dt and is currently stored as a numpy array and as such uses a lot of space. A more efficient way to store this would be to store T0, T1 and dt, e.g. using a generator. However generators don't work with many functions, e.g. numpy functions, arithmetic and plotting. I want something which works like a generator, i.e. only stores the 3 necessary values, and then generates a numpy array when necessary to use for use in some function or so.

Is there any object that already exists that works like this? I.e. Storing the necessary data (3 values) in a generator and then returning/representing itself as a numpy array when used in functions or arithmetic? The memory would only then be used while within the scope of the function and be released when it falls out of scope.

EDIT with solution: I created an mplementation what I wanted. Copying generators turned out to be tricky, see here, so I instead store the start, stop and step and create and return generators or numpy arrays as required.

The code for this is as follows:

import numpy as _np

class frange():
    """
    Return an object can be used to generate a generator or an array 
    of floats from start (inclusive) to stop (exclusive) by step. 
    This object stores the start, stop, step and length of
    the data. Uses less memory than storing a large array.

    Example
    -------
    An example of how to use this class to generate some data is
    as follows for some time data between 0 and 2 in steps of
    1e-3 (0.001)::

        $ time = frange(0, 2, 1e-3)

        $ printlen(time) # prints length of frange, just like an array or list

        $ generator = time.get_generator() # gets a generator instance
        $ for i in generator: # iterates through printing each element
        $     print(i)

        $ array = time.get_array() # gets an array instance
        $ newarray = 5 * array # multiplies array by 5


    """
    def __init__(self, start, stop, step):
        """
        Intialises frange class instance. Sets start, top, step and 
        len properties.

        Parameters
        ----------
        start : float
            starting point
        stop : float
            stopping point 
        step : float
            stepping interval
        """
        self._slice = slice(start, stop, step)
        self.len = self.get_array().size
        return None

    def get_generator(self):
        """
        Returns a generator for the frange object instance.

        Returns
        -------
        gen : generator
            A generator that yields successive samples from start (inclusive)
            to stop (exclusive) in step steps.
        """
        s = self._slice
        gen = drange(s.start, s.stop, s.step) # intialises the generator
        return gen

    def get_array(self):
        """
        Returns an numpy array containing the values from start (inclusive)
        to stop (exclusive) in step steps.

        Returns
        -------
        array : ndarray
            Array of values from start (inclusive)
            to stop (exclusive) in step steps.
        """
        s = self._slice        
        array = _np.arange(s.start, s.stop, s.step)
        return array

    def __len__(self):
        return self.len      

def drange(start, stop, step):
    """
    A generator that yields successive samples from start (inclusive)
    to stop (exclusive) in step intervals.

    Parameters
    ----------
    start : float
        starting point
    stop : float
        stopping point 
    step : float
        stepping interval

    Yields
    ------
    x : float
        next sample
    """
    x = start
    if step > 0:
        while x + step <= stop: # produces same behaviour as numpy.arange
            yield x
            x += step
    elif step < 0:
        while x + step >= stop: # produces same behaviour as numpy.arange
            yield x
            x += step
    else:
        raise ZeroDivisionError("Step must be non-zero")

Answer 1

Python already has a class that stores start, stop, step attributes, a slice

In [523]: s = slice(0, 1, .1)

np.lib.index_tricks has a class that can expand slices. In this case it uses arange:

In [524]: np.r_[s]
Out[524]: array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9])
In [525]: np.arange(s.start, s.stop, s.step)
Out[525]: array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9])
In [526]: np.arange(0, 1, .1)
Out[526]: array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9])

slice just stores its atttributes; any calculation is done by the code that uses it. np.r_ uses this trick to invoke np.linspace if the step value is imaginary.

In [527]: np.r_[slice(0,1,11j)]
Out[527]: array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ])

I don't see how the generators discussed in the other answer are an improvement over running arange or linspace on the fly.

If you are developing your own indexing classes, it will be worth you while to study the index_tricks.py file.

Answer 2

Every generator can be turned to a list or a numpy array with list comprehension, like so:

def mygenerator(T0, T1, dt):
    while T0<T1:
        T0 += dt
        yield T0

def gen2numpy(gen):
    return np.array([item for item in gen])

I used a very naive generator that assumes much about T0, T1, dt but that is because you provided no code that would help with those assumptions... Either way, look at how I turned a generator to a numpy array

If you are looking for a single object that can do both, define a class and use those functions as it's methods

But be careful - iterating over the generator, even partially, and only then generating the list will give only a partial list or an error. If you have to iterate before creating the list, I would recommend creating 2 identical (but seperate) generators, one for iteration and the other one in case a complete list is needed later...

Answer 3

A slight improvement over using raw slices would be to create a class that can be coerced to array with __array__:

class range_array(object):
    def __init__(*args):
        self._slice = slice(*args)

    def __array__(self):
        s = self._slice
        return np.arange(s.start, s.stop, s.step)

Which means that code like this will work:

a = range_array(T0, T1, dt)
res = np.dot(a, a)

You could go a bit further and implement __array_ufunc__ in numpy 1.13:

class range_array(np.lib.mixins.NDArrayOperatorsMixin):
    def __init__(start, stop, step):
        sl = slice(*args)
        self._start = sl.start
        self._stop = sl.stop
        self._step = sl.step

    def __array__(self):
        return np.arange(self._start, self._stop, self._step)

    def __array_ufunc__(self, ufunc, method, args, kwargs):
        # special case np.add(range, 1) to just add to stop and start, etc