Most efficiently insert a number in a maintained large sorted variable

Question

I need to most efficiently insert a number in a maintained large sorted variable. Is there a better method than test1?

test1 is quite a bit faster vs test2 which is just to append a variable then resort.

      q←1000000⍴0 ⋄ q←10 9 8 7 6 5 4 3 2,q  ⍝q is kept sorted

      test1←{
        y←⍺(⍳∘1≤)⍵ ⍝ very fast
        (y↑⍺),⍵,(y↓⍺)  ⍝ is there a tacit version here and without copying?
      }

      10↑q test1 6
10 9 8 7 6 6 5 4 3 2
      cmpx 'q test1  6'
3.2E¯4

      test2←{y←⍵,⍺ ⋄ y[⍒y]}
      10↑q test2 6
10 9 8 7 6 6 5 4 3 2

      cmpx 'q test2  6'
1.5E¯3

I tried presorted variable. With test1 is quicker than appending then sorting. Perhaps test1 refactored with better tacit?

I tried several different strategies, and found that your `test1` always comes out on top. Was able to eek out a tad bit more performance by rewriting it as `{(n↑⍺),⍵,⍺↓⍨n←1⍳⍨⍺<⍵}`, however. Here are the different functions I tested against: `test3←{(⍺/⍨~m),⍵,⍺/⍨m←⍺<⍵}`, `test4←{h t←⍺⊆⍨1+⍺<⍵ ⋄ h,⍵,t}`, and a tacitification of your `test2` with `test5←(⊂∘⍒⌷⊢)⍤,`. — B. Wilson, Dec 31 '22 at 05:55
I am confused by the initial comment "q is kept sorted", because the initial value of q is not sorted - is there something missing from the problem description? In general, continuously appending to a list is best avoided in APL. The Dyalog interpreter will optimise the use of modified assignments using ,←, allocating extra storage beyond the end so that repeated catenations do not re-allocate memory for the entire list on every operation (as in "keys,←6"). However, that won't allow you to keep the keys sorted. Perhaps you can tell us more about the underlying use case? — Morten Kromberg, Dec 31 '22 at 10:32
@B.Wilson Note that ↓⍨ will only be faster on small arguments — , Dec 31 '22 at 11:09
You can write `(y↑⍺),⍵,(y↓⍺)` as `y(↑,⍵,↓)⍺` and sorting is faster if using an idiom like `{⍵[⍒⍵]}y` but `test1` is still fastest. — Adám, Dec 31 '22 at 17:25
oops that is q←1000000⍴0 ⋄ q←10 9 8 7 6 5 4 3 2,q ⍝q is kept sorted — creatural, Dec 31 '22 at 22:42

score 1 · Answer 1 · answered Jan 02 '23 at 07:41

Possibly not the answer you are looking for, but in a production application, if access to these sorted keys with frequent appends was an important performance consideration in a Dyalog APL application, you might resort to something like the following class. The strategy is to have an unsorted variable data which can be appended to efficiently using a method called Append. Sorting is done on demand, if needed (there is room for further optimisation by checking whether the appended value is greater than the last element in the list, which would be worthwhile if that was a common case).

   :Class Sorted
   
    :Property Values 
    :Access Public       

      ∇Set value
      data←value
      sorted←0
      ∇

      ∇r←Get value      
      :If ~sorted
          sorteddata←data[⍒data]
          sorted←1
      :EndIf     
      r←sorteddata
      ∇     
      
    :EndProperty

    ∇ Make initial
      :Implements Constructor
      :Access Public
      data←initial
      sorted←0
    ∇

    ∇ r←Append values
      :Access Public
      data,←values
      r←sorted←0
    ∇

:EndClass

Usage would be along the lines of:

     s←⎕NEW Sorted (10 9 8 7 6 5 4 3 2,1E6⍴0)
     s.Append 6
     s.Append 7
     ≢s.Values
100011

Most efficiently insert a number in a maintained large sorted variable

1 Answers1