2

We have a custom database that handle serialized indexed data and we have reached 70% of the capacity of the database (which is int.MaxValue or 2^31 elements). We started analyzing if it would be possible to break this limit by making our own implementation of a Collection that can handle a possible infinite number of elements (limited by the memory of the computer of course).

Currently we are delegating as much as possible to WPF to handle the UI virtualization and we handle only the data virtualization from our side with the implementation of IList. However the IList.Count property is limited to 2^31 by its type (integer). Meaning that this is our primary limit.

We thought about making our own interface for the iteration but therefore we would lose the UI Virtualization of the DataGrid which is heavily tied to the IList interface.

Even though I am (more or less) capable of making our own UserControl I would like to know if something like this is not already done for WPF. I assume that we are not the first ones who want to make accessible that much of data for their users.

jazb
  • 5,498
  • 6
  • 37
  • 44
mwryl
  • 664
  • 7
  • 16
  • 1
    Could you partition the table into several tables (by date, by ID, or by some other meaningful meta information)? Is it possible to identify and remove/archive elements which are no longer needed? – Axel Kemper Sep 03 '19 at 08:12
  • 1
    what's the db?? – jazb Sep 03 '19 at 08:22
  • @AxelKemper this is not very possible, user have two modes: one accessing only the last data, and one allowing him to see every evolution of every elements (each element can have multiple revisions). But the problem is not on the side of database, but more on the side of the UI and how it would handle the UI virtualization (and if we have to make it ourselves). – mwryl Sep 03 '19 at 08:27
  • @JohnB the database is a self-contained db that we made because we were not able to achieve what we wanted with the existing databases. – mwryl Sep 03 '19 at 08:27
  • 2
    Blimey, that's a lot of data! Maybe you should look into data virtualisation, not UI virtualisation. You will have access to every record in the db but it will be displayed or retrieved in chunks. Those chunks can have (2^31 - 1) elements :-) if you really want to. – XAMlMAX Sep 03 '19 at 08:58
  • @XAMlMAX yes, it seems that paginated data will be the easiest solution to implement a way to access to all data without doing a specific user control for that. – mwryl Sep 05 '19 at 07:48

1 Answers1

1

When it comes to dealing with a lot of records from database it would be easier to Virtualise data rather than creating custom List and Controls to accommodate 2^31 items from your database.
I take it your database is prepared to handle that amount of data and it self will survive when you reach 9,223,372,036,854,775,807 rows, which is Int64.MaxValue MSDN.

XAMlMAX
  • 2,268
  • 1
  • 14
  • 23