1

I'm scanning a large data source, currently about 8 million entries, extracting on string per entry, which I want in alphabetical order.

Currenlty I put them in an array then sort an index to them using qsort() which works fine.

But out of curiosity I'm thinking of instead inserting each string into a data structure that maintains them in alphabetical order as I scan them from the data source, partly for the experience of emlplementing one, partly because it will feel faster without the wait for the sort to complete after the scan has completed (-:

What data structure would be the most straightforward to implement in C?

UPDATE

To clarify, the only operations I need to perform are inserting an item and dumping the index when it's done, by which I mean for each item in the original order dump an integer representing the order it is in after sorting.

SUMMARY

  • The easiest to implement are binary search trees.
  • Self balancing binary trees are much better but nontrivial to implement.
  • Insertion can be done iteratively but in-order traversal for dumping the results and post-order traversal for deleting the tree when done both require either recursion or an explicit stack.
  • Without implementing balancing, runs of ordered input will result in the degenerate worst case which is a linked list. This means deep trees which severely impact the speed of the insert operation.
  • Shuffling the input slightly can break up ordered input significantly and is easier to implement that balancing.
hippietrail
  • 15,848
  • 18
  • 99
  • 158

4 Answers4

3

Binary search trees. Or self-balancing search trees. But don't expect those to be faster than a properly implemented dynamic array, since arrays have much better locality of reference than pointer structures. Also, unbalanced BSTs may "go linear", so your entire algorithm becomes O(n²), just like quicksort.

Fred Foo
  • 355,277
  • 75
  • 744
  • 836
  • On going linear, I have noticed there are quite a few sections in the data with huge tracts already sorted. – hippietrail Apr 11 '11 at 17:31
  • 1
    @hippietrail: then a self-balancing tree would be the ideal structure. Note that `qsort` is commonly quicksort, so that may go quadratic on such data as well. Maybe try to implement mergesort? – Fred Foo Apr 11 '11 at 17:33
  • @larsmans isn't the probability that `quicksort` (if implemented properly) will take quadratic time neglectable for all practical purposes? – abeln Apr 11 '11 at 17:45
  • @abeln: that probability depends very strongly on the distribution of the input. If that's likely near-sorted, then quicksort (and unbalanced tree sort) is likely quadratic. – Fred Foo Apr 11 '11 at 17:47
  • @hippietrail, concerning your update: I think a good dynamic array implementation plus a tailored sort algorithm will be optimal. But implementing BSTs just to see how well they behave can be a lot of fun, and you can skip the stupid deletion algorithm :) – Fred Foo Apr 11 '11 at 17:48
  • @larsmans: does that behavior apply to the "median of three" or "pick a random pivot" implementations too? – abeln Apr 11 '11 at 17:51
  • @abeln: no, but whether `qsort` uses median-of-three or not depends on the platform. (While all implementations I've seen *are* quicksort implementations.) Besides, `qsort`'s genericity-through-indirection can also make it slow. – Fred Foo Apr 11 '11 at 17:57
  • @larsmans: Yes optimal performance is not my objective so much as learning, that's why I asked "easiest to implement in C" since things like pointers and the relatively low-level memory management can be pretty trick in this language. – hippietrail Apr 12 '11 at 04:06
2

You are already using the optimal approach. Sort at the end will be much cheaper than maintaining an online sorted data structure. You can get the same O(logN) with a rb-tree but the constant will be much worse, not to mention significant space overhead.

That said, AVL trees and rb-trees are much simpler to implement if you don't need to support deletion. Left-leaning rb tree can fit in 50 or so lines of code. See http://www.cs.princeton.edu/~rs/talks/LLRB/ (by Sedgewick)

Igor Nazarenko
  • 2,184
  • 13
  • 10
0

You could implement a faster sorting algorithm such us Timsort or other sorting algorithms with a nlog(n) worst case and just search it using Binary search since its faster if the list is sorted.

Tom
  • 176
  • 1
  • 11
  • In fact I don't even need a search operation. Just insert, then at the end I want to dump the index. I might edit the question to state that. – hippietrail Apr 11 '11 at 17:33
0

you should take a look at Trie datastructure wikilink i think this will serve what you want

ashmish2
  • 2,885
  • 8
  • 40
  • 54