0

I would like to know what is the best way to manage a large 3d array with something like :

x = 1000 y = 1000 z = 100

=> 100000000 objects

And each cell is an object with some amount of data.

Simple methods are very loooooong even if all data are collapsed (I a first tryed an array of array of array of objects)

class Test

    def initialize
        @name = "Test"
    end

end

qtt = 1000*1000*100
Array.new(qtt).each { |e| e = Test.new }

I read somewhere that DB could be a good thing for such cases.

What do you think about this ?

What am I trying to do ?

This "matrix" represents a world. And each element is a 1mx1mx2m block who could be a different kind (water, mud, stone, ...) Some block could be empty too.

But the user should be able to remove blocks everywhere and change everything around (if they where water behind, it will flow through the hole for exemple.

In fact what I wish to do is not Minecraft be a really small clone of DwarfFortress (http://www.bay12games.com/dwarves/)

Other interesting things

In my model the groud is at level 10. It means that [0,10] is empty sky in most of cases. Only hills and parts of mountains could be present on those layers.

Underground is basicaly unknown and not dug. So we should not have to add instances for unused blocks.

What we should add from the beginning to the model : gems, gold, water who could stored without having to store the adjacent stone/mood/earth blocks.

At the beginning of the game, 80% of the cube doesn't need to be loaded in memory.

Each time we dig we create new blocks : the empty block we dug and the blocks around.

The only things we should index is :

  • underground rivers
  • underground lakes
  • lava rivers
MARTIN Damien
  • 996
  • 2
  • 15
  • 36
  • all depends on what you want to do with!If you wants some persistence, so yes DB could be a good thing, if you just want to do simple work with, maybe you should wrap them in a class. The question is a little bit too vague. – louiscoquio Jul 09 '12 at 15:28
  • Can you make your question clearer? And if you really need high performance with a TON of data, you need a different language. – Linuxios Jul 09 '12 at 15:32
  • I added more informations of what I wish to do. – MARTIN Damien Jul 09 '12 at 15:34
  • Can you give an example of what your methods are doing? In case the methods run on the entire matrix and touch every cell, then this is indeed a big problem. However, in case you have many similar objects in your cells and you do not need that each will have its own specific characters, there are some design patterns you can use to make things go faster. In any case, with this size of data, a naive approach won't work. – davidrac Jul 09 '12 at 15:42
  • For exemple every tick the water causes erosion on the adjacent cubes, so the impacted cubes will see their hardness decreased. This will be calculated on all the borders of water blocks (not realy often but it will). The most common events will be when a character will digg and remove a cube : I'll have to control all the cubes arround. It is more a "cascading effect" and will certainly never have repercusions on all the board in one shot. – MARTIN Damien Jul 09 '12 at 15:47

1 Answers1

1

Holding that many objects in memory is never a good thing. A flat-file or database-centric approach would be a lot more efficient and easier to maintain.

What I would do - The object-oriented approach

Store the parameters of the blocks as simple data and construct the objects dynamically.

Create a Block class to represent a block in the game, and give it variables to hold the parameters of that particular block:

class Block
  # location of the Block
  attr_accessor :x, :y, :z

  # an individual id for the Block
  attr_accessor :id

  # to define the block type (rock, water etc.)
  attr_accessor :block_type

  # and add any other attributes of a Block...
end

I'd then create a few methods that would enable me to serialise/de-serialise the data to a file or database.

As you've stated it works on a board, you'd also need a Board class to represent it that would maintain the state of the game as well as perform actions on the Block objects. Using the x, y, z attributes from each Block you can determine its location within the game. Using this information you can then write a method in the Block class that locates those blocks adjacent to the current one. This would enable you to perform the "cascading" effects you talk about where one Block is affected by actions on another.

Accessing the data efficiently

This will rely entirely on how you choose to serialise the Block objects. I would probably choose a binary format to reduce unnecessary data reads and store the objects via their id parameter, and then use something like MMIO to quickly do random-access reads/writes on a large data file in an Array-like manner. This will allow you to access the data quickly and efficiently, without the memory overhead. How you read the data will relate to your adjacent blocks method above.

You can of course also choose the DB storage route which will allow you to isolate the Blocks and do lookups on particular blocks in a higher-level manner, however that might give you a bit of extra overhead.

It sounds like an interesting project, I hope this helps a bit! :)

P.S With regards to the comment above by @Linuxious about choosing a different language. Yes this might be true in some cases, but a skilled programmer never blames his tools. A program is only as efficient as the programmer makes it...unless you're writing it in Java ;)

xiy
  • 808
  • 6
  • 13
  • Thank you very much for this great answer :) I think about my problem and found some things : Does I need to have an instance for all blocks ? no ! On ly the ground and the caves and it should represent 30% max of the volume... The ground I positionned at level 10 by default so [0,10] is nearly empty (just some hills and maybe parts of mountains). Underground will be dug but not on all levels and not on all the surface. That why I tryed to initialize a cube full of `nil` and with only a few real `block` and it is far faster. Now I will take a look at what you say. it is very interseting :) – MARTIN Damien Jul 09 '12 at 20:09