1

In thinkscript charts and scans, any script gets executed many times once for each bar. Program state between such executions is stored in array variables which are accessed directly or by an offset via [] or GetValue(). Many scripts involving some kind of pattern recognition need to initialize such variables at the first bar by assigning them initial values. When executing on subsequent bars, the script either copies the previous value or creates new values. Then depending on some other condition, the script may check whether the current array entry still equals the first array entry to discover whether something interesting happened in the data.

The following test script follows this simple pattern. Its single purpose is to use the scanner to verify that the first entry of a single variable retains its value.

A scan executes a single plot statement at the last bar. The script has as as the single criterion that the tested variable holds its value and is not changed by something else. Changing variables or variable array entries in thinkscript is illegal and should never happen. However, this script shows that the first entry of a variable gets overwritten. How does this happen, and how can it be avoided?

def index;
def myVar;
if (BarNumber() == 1) {
    index = -1;
    myVar = close;
} else {
    if (close > myVar[1]) {
        myVar = close;
        index = index[1];
    } else {
        if (close <= myVar[1]) {
            index = 1;
        } else {
            index = index[1];
        }
        myVar = myVar[1];
    }
}
plot scan = GetValue(index, BarNumber() -1) == -1;
user250343
  • 1,163
  • 1
  • 15
  • 24
  • Looks like the same discussion at [Hahn-Tech](https://www.hahn-tech.com/ans/thinkscript-scan-debugging/) where Pete responds >"I will admit that I never run across this issue in my coding. Perhaps because I am used to applying the tools in a way that avoids this and other so-called bugs. ... I do tons and tons of custom projects and never run into this." – leanne Apr 03 '21 at 18:26

3 Answers3

4

Separately, for anyone who came here looking for, actually, how to create a variable that retains its value - meaning the value doesn't change across bars because you don't want it to - here are 3 methods...

Use a recursive variable, as follows:

Method 1: if statement

# declare variable without initializing
def myVar;

# set up `if` condition for which variable should be set
if ( BarNumber() == 1 ) {
  
  # set the value you want when the condition is met
  # in this case, the variable is set to the close value at bar 1
  myVar = close;
}
# thinkScript always requires an `else` 
else {

  # now, here, if you want the variable to change, enter the desired value,
  # or, if you want it to always stay the same, then...
  myVar = myVar[1];

}

Method 2: if expression

# same as above, really, but more compact; use parens as desired for clarity
def myVar = if BarNumber() == 1 then close else myVar[1];

Method 3: CompoundValue()

Note: Generally, CompoundValue() works the same as the above examples, so one would not need to use it; however, it sometimes must be used. See below the code for more detail.

def myVar = CompoundValue(1,
                          if BarNumber() == 2 then close[1] else myVar[1],
                          0);

plot test = myVar;

Some CompoundValue() Details

First, parameters, as the descriptions in the docs can be confusing:

The first parameter is the length. It represents the bar number after which CompoundValue() will utilize the expression in the 2nd parameter.

1 is the minimum (and default) value accepted for this argument.

The 2nd parameter is "visible data". The value here must be an expression, usually a calculation (though I used an if expression for this example).

This is the data manipulation CompoundValue() will use once the BarNumber() value is past the length. Eg, for this example, since I had a 1 in the first parameter, the if expression won't take effect until thinkScript is processing the 2nd bar. At bars 0 and 1, the 3rd parameter value will come into play.

Note that I used an offset for close in the expression. Since I had to put a minimum of 1 in the length argument, thinkScript would be at bar 2 before it started using the expression. Thus, I had to indicate the close value for the prior bar (close[1]) was really what I wanted.

The 3rd parameter, "historical data", is the value that will be used for the bar number before and including the length (bar number) value in the 1st parameter.

For this example, I put 0, but I also could've used Double.NaN. It wouldn't matter in my case because I didn't care about setting any values prior to the calculation point.

The CompoundValue() docs give a Fibonacci example that now is easy to understand. It also shows why one might want to set a value for prior bars:

def x = CompoundValue(2, x[1] + x[2], 1);
plot FibonacciNumbers = x;

length = 2, so the 2nd argument's calculation won't take place until the 3rd bar comes around.

"visible data" = x[1] + x[2], the calculation that will take place for every bar after the 2nd bar (ie, from the 3rd bar forward).

"historical data" = 1, so that for bars 1 and 2, the constant value 1 will be used in place of the calculation in the 2nd parameter. That works for a Fibonacci calculation!


As for reasons why one would use CompoundValue() instead of the first two methods above, the main one is that CompoundValue is required when plotting items with multiple lengths or "offsets". In short, thinkScript will change all plotted offsets to equal the largest offset. CompoundValue, unlike other plotted variables, keeps its stated offset values.

For details, see the thinkScript tutorial's Chapter 12. Past/Future Offset and Prefetch, as well as my answer to the SO question Understanding & Converting ThinkScripts CompoundValue Function

leanne
  • 7,940
  • 48
  • 77
0

It is not possible to avoid this because it is a defect, a bug in the scan engine as of 2019-06-13. Let me provide a proof in a few simple steps, all code executed by the scan engine for All Symbols, to get maximum coverage.

def myLowest = LowestAll(BarNumber());
plot scan = myLowest == 1;

This returns the entire set, and it proves that in the first bar in all symbols scanned has BarNumber() == 1; Always.

Again, we start for All Symbols with

def myHighest = HighestAll(BarNumber());
plot scan = BarNumber() == myHighest;

This returns the entire set.

It proves that in all scans, the single plot statement is executed only once on the highest bar, regardless how many bars each symbol has. So it is calculating HighestAll(BarNumber()) all by itself without us needing to do it.

With the above, we have some tools to use the scan engine itself to test some basic conditions. This is important, because if we want to identify a bug, we need to have a reliable way to check actual vs expected values. That is so because we cannot debug the scan engine - we have to use this indirect method, a solid method.

Now we are using this knowledge to test the successful execution of a user written "if" statement by the scan engine.

def index;
if (BarNumber() == 1) {
    index = -1;
} else {
   index = 3;
}
    
plot scan = GetValue(index, BarNumber() -1) == -1;

The GetValue() function allows us to use a variable offset for indexing depending on the number of the bars that each symbol has. We expect to compare the first entry of index where we can verify the content as the number -1, and it works as expected because the scan returns all symbols in the set.

As the last step, we are expanding the code of the if statement to show scan engine failure. We are still executing the same test in the plot statement. However, now the scan engine corrupts the first entry of index as a side effect of the new code. The test now fails. The scan engine sets the value of index at BarNumber() == 1 to 0. There is no user code that does this - the user code sets it to -1.

def index;
def myVar;
if (BarNumber() == 1) {
    index = -1;
    myVar = close;
} else {
    if (close > myVar[1]) {
        myVar = close;
        index = index[1];
    } else {
        if (close <= myVar[1]) {
            index = 1;
        } else {
            index = index[1];
        }
        myVar = myVar[1];
    }
}
plot scan = GetValue(index, BarNumber() -1) == -1;

So with a small number of small steps we can progressively show that the scan engine has a defect / bug because it fails to keep a value in a variable.

Following is another example:

def sum;
if (BarNumber() == 1) {
    sum = 1;
} else {
    if (BarNumber() < 5) {
        sum = sum[1] + 1;
    } else {
        sum = sum[1]; # This causes the problem.
        #sum = Double.NaN;# alternative: does not corrupt previous value but useless.
    }
}
plot scan = GetValue(sum, BarNumber() -1) == 1;

Let us conclude from the observed behavior what might be happening in the compiler: In a nutshell, the thinkscript if statement does not guard against out of bounds array indexing. That is a defect.

The details would be as follows: There are two conditions to keep in mind, knowing that the chart or scan window has a left edge which is what we see as programmers, where data SEEMS to begin:

  1. The time series may have more history beyond the left side from the left chart boundary.
  2. The time series may begin at the left chart boundary - no more data.

Case 2) is what we are looking at.

So if at bar 1 we would execute index[1] or close[1], that would fail because of out of bounds array indexing.

To prevent that, we use an if-else construct as follows:

def index
if (BarNumber() == 1) {
    index = -1;
}else{
    index = index[1];
}

We must assume that each branch of the if statement executes valid code when it needs to. However, we observe that this is not the case. We observe that the index[1] term in the else branch gets evaluated even under the condition (BarNumber() == 1) where it must fail. That causes the entire if statement to fail in the if branch which should NOT be executing any evaluation in the else branch at that point. We can prove that this is actually the case by replacing the term index[1] with GetValue(index, 1) which works.

user250343
  • 1,163
  • 1
  • 15
  • 24
  • I haven't considered your entire issue yet, @user250343; however, there is a flaw in the logic at `def myHighest = HighestAll(BarNumber()); plot scan = BarNumber() == myHighest;` -- since the loop runs for every bar, at bar 1, myHighest will be bar 1. The scan will then locate every item that has a bar 1 - ie, the full set of results. However, if you replace the value of `myHighest` with an actual number, say `3`, the scan will not display the full set of results. In fact, the `plot` statement is executed at *every* bar; however, in a scan, the results will be returned only at the last bar. – leanne Mar 23 '21 at 18:32
  • @leanne Not a flaw based on your logic which is flawed. Your first assertion that at bar 1, myHighest will be bar 1 is incorrect. The sole purpose of the `HighestAll()` function is to return the highest bar number in the chart not the highest bar number left from and including the current bar, in your case bar 1. – user250343 Mar 28 '21 at 21:26
  • I stand corrected: `HighestAll( BarNumber() )` does indeed return the highest bar number, regardless of which bar is currently being processed. – leanne Mar 29 '21 at 14:48
-1

I am testing this as of 30 March 2021, so it is possible the OP's original issue is fixed

First, in order to see what ThinkOrSwim is doing with this code, I took the first example and put it into a chart study. Using labels and chart bubbles, I can demonstrate what is happening as thinkScript processes each bar.

  • Test Code:
# OP's example code
def index;
def myVar;
if (BarNumber() == 1) {
    index = -1;
    myVar = close;
} else {
    if (close > myVar[1]) {
        myVar = close;
        index = index[1];
    } else {
        if (close <= myVar[1]) {
            index = 1;
        } else {
            index = index[1];
        }
        myVar = myVar[1];
    }
}
#plot scan = GetValue(index, BarNumber() -1) == -1;


# labels; do non-variable offset values show correct values based on code?
def numBars = HighestAll( BarNumber() );
AddLabel(yes, " numBars: " + numBars + " ", Color.CYAN);

def barNum = if BarNumber() == 0 then 0 
             else if BarNumber() == 1 then 1 else -6;
AddLabel(yes,
 " Bar 1: " + "index[5]: " + index[5] + ", GetValue(index, 5): " + GetValue(index, 5) + " ",
 Color.LIGHT_ORANGE );
AddLabel(yes,
 " Bar 2: " + "index[4]: " + index[4] + ", GetValue(index, 4): " + GetValue(index, 4) + " ",
 Color.LIGHT_ORANGE );
AddLabel(yes,
 " Bar 3: " + "index[3]: " + index[3] + ", GetValue(index, 3): " + GetValue(index, 3) + " ",
 Color.LIGHT_ORANGE );
AddLabel(yes,
 " Bar 4: " + "index[2]: " + index[2] + ", GetValue(index, 2): " + GetValue(index, 2) + " ",
 Color.LIGHT_ORANGE );
AddLabel(yes,
 " Bar 5: " + "index[1]: " + index[1] + ", GetValue(index, 1): " + GetValue(index, 1) + " ",
 Color.LIGHT_ORANGE );
AddLabel(yes,
 " Bar 6: " + "index[0]: " + index[0] + ", GetValue(index, 0): " + GetValue(index, 0) + " ",
 Color.LIGHT_ORANGE );


# chart bubbles; displaying variable values - are they what we expect?
AddChartBubble(yes, high,
  "Bar Number: " + BarNumber() + "\nclose: " + close + "\nmyVar: " + myVar + "\nindex: " + index, 
  Color.YELLOW, if BarNumber() % 2 == 0 then no else yes);

# yes! the first entry of both variables actually remain the same
AddChartBubble(yes, low,
  "BarNumber() -1 == " + (BarNumber() -1) + "\nGetValue(index, " + (BarNumber() -1) + ") == " + GetValue(index, BarNumber() -1) + "\nGetValue(myVar, " + (BarNumber() -1) + ") == " + GetValue(myVar, BarNumber() -1), 
  Color.YELLOW,  if BarNumber() % 2 == 0 then yes else no);


I set up the chart to display 6 (daily) bars, so the results would be easy to see.

  • Here are the results on the TRCH chart from 19 Mar 2021 - 26 Mar 2021:

TRCH chart, 19 Mar 2021 - 26 Mar 2021, with labels and chart bubbles

One can see that the values of both index and myVar from the first bar did indeed keep their values. They also did change as expected during the processing of each bar.


Now to test the scan: the scan function is a filter that returns any stocks that meet the coded conditions. Instead of basing our test on the number of results returned, for a scan we can create a column that will show a specified value.

  • So, first I created a study specifically containing only the example code:
def index;
def myVar;
if (BarNumber() == 1) {
    index = -1;
    myVar = close;
} else {
    if (close > myVar[1]) {
        myVar = close;
        index = index[1];
    } else {
        if (close <= myVar[1]) {
            index = 1;
        } else {
            index = index[1];
        }
        myVar = myVar[1];
    }
}
plot scan = GetValue(index, BarNumber() -1);


  • I then created a custom column based on that study. When I add a plot pointing to a custom study, thinkScript will pull in the current code and set up the column code. The finished column code in this case looks like:
script SO_ScanProofOfConcept_Column {
def index;
def myVar;
if (BarNumber() == 1) {
    index = -1;
    myVar = close;
} else {
    if (close > myVar[1]) {
        myVar = close;
        index = index[1];
    } else {
        if (close <= myVar[1]) {
            index = 1;
        } else {
            index = index[1];
        }
        myVar = myVar[1];
    }
}
plot scan = GetValue(index, BarNumber() -1);

}
plot scan = SO_ScanProofOfConcept_Column().scan;

Now, I can see that everything in the custom column is, in fact, -1.0. Browse the 4th column from the left, starting with "Custom04-...":

ThinkOrSwim scan results showing custom column


So, at least as of this date, the scan engine is retaining the value of the variable as we would expect. There appear to be no corruptions of the data in the data array.

leanne
  • 7,940
  • 48
  • 77