1

I am storing crypto-currency data into a Django data model (using Postgres database). The vast majority of the records are saved successfully. But, on one record in particular I am getting an exception decimal.InvalidOperation.

The weird thing is, I can't see anything different about the values being saved in the problematic record from any of the others that save successfully. I have included a full stack trace on paste bin. Before the data is saved, I have outputted raw values to the debug log. The following is the data model I'm saving the data to. And the code that saves the data to the data model.

I'm stumped! Anyone know what the problem is?

Data Model

class OHLCV(m.Model):
    """ Candles-stick data (open, high, low, close, volume) """

    # class variables
    _field_names = None
    timeframes = ['1m', '1h', '1d']

    # database fields
    timestamp = m.DateTimeField(default=timezone.now)
    market = m.ForeignKey('bc.Market', on_delete=m.SET_NULL, null=True, related_query_name='ohlcv_markets', related_name='ohlcv_market')
    timeframe = m.DurationField() # 1 minute, 5 minute, 1 hour, 1 day, or the like
    open = m.DecimalField(max_digits=20, decimal_places=10)
    high = m.DecimalField(max_digits=20, decimal_places=10)
    low = m.DecimalField(max_digits=20, decimal_places=10)
    close = m.DecimalField(max_digits=20, decimal_places=10)
    volume = m.DecimalField(max_digits=20, decimal_places=10)

Code Which Saves the Data Model

@classmethod
def fetch_ohlcv(cls, market:Market, timeframe:str, since=None, limit=None):
    """
    Fetch OHLCV data and store it in the database

    :param market:
    :type market: bc.models.Market
    :param timeframe: '1m', '5m', '1h', '1d', or the like
    :type timeframe: str
    :param since:
    :type since: datetime
    :param limit:
    :type limit: int
    """

    global log
    if since:
        since = since.timestamp()*1000

    exchange = cls.get_exchange()
    data = exchange.fetch_ohlcv(market.symbol, timeframe, since, limit)
    timeframe = cls.parse_timeframe_string(timeframe)
    for d in data:
        try:
            timestamp = datetime.fromtimestamp(d[0] / 1000, tz=timezone.utc)
            log.debug(f'timestamp={timestamp}, market={market}, timeframe={timeframe}, open={d[1]}, high={d[2]}, low={d[3]}, close={d[4]}, volume={d[5]}')
            cls.objects.create(
                timestamp=timestamp,
                market=market,
                timeframe=timeframe,
                open=d[1],
                high=d[2],
                low=d[3],
                close=d[4],
                volume=d[5],
            )
        except IntegrityError:
            pass
        except decimal.InvalidOperation as e:
            error_log_stack(e)
djvg
  • 11,722
  • 5
  • 72
  • 103
MikeyE
  • 1,756
  • 1
  • 18
  • 37

1 Answers1

2

Have a look at your data and check if it fits within the field limitations:

  1. The mantissa must fit in the max_digits;
  2. The decimal places should be less than decimal_places;
  3. And according to the DecimalValidator : the number of whole digits should not be greater than max_digits - decimal_places;

Not sure how your fetch_ohlcv function fills the data array, but if there is division it is possible that the number of decimal_digits is greater than 10. The problem I had, that brought me here, was too many digits in the integer part therefore failing the last requirement.

Check this answer for more information on a similar issue.

Ivan Tonkov
  • 36
  • 3