Indicators on Live Market Data

Distinction in Execution on Live Market and on Historical Market Data

When the indicator is being executed on a historical market data, the Update function is called only once for each bar. The bar is completely formed at that moment.

When the indicator is being executed on a live market data, the Update function is called for the most recent (current) candle as many times as many ticks come. The most recent candle close price is equal to the last tick received and high/low price may also be changed if higher or lower ticks was received past the last update.

If the indicator is developed without proper handling of the Live Market update, it may work well on historical data, but when you leave it applied on live market data the wrongful output may be proceed.

Let's look at the simple oscillator which just outputs green bar if the current candle median price is higher than previous candle and red bar if it is lower. The code of such indicator is very simple:

 function Init()
     indicator:name("Test");
     indicator:description("Test Indicator");
     indicator:requiredSource(core.Bar);
     indicator:type(core.Oscillator);
 end
 
 local red;
 local green;
 local first;
 local median;
 
 function Prepare(onlyName)
     local name = profile:id() .. "(" .. instance.source:name() .. ")";
     instance:name(name);
     if onlyName then
         return ;
     end
 
     median = instance.source.median;
     first = median:first() + 1;
 
     red = instance:addStream("R", core.Bar, name .. ".R", "R", core.rgb(255, 0, 0), first);
     green = instance:addStream("G", core.Bar, name .. ".G", "G", core.rgb(0, 255, 0), first);
 end
 
 function Update(period, mode)
     if period >= first then
         if median[period] < median[period - 1] then
             red[period] = -1;
         elseif median[period] > median[period - 1] then
             green[period] = 1;
         end
     end
 end

Now run the indicator in pre-deliver mode (in other words, apply it on the historical price). The image looks great:

But if we run the indicator in non-pre-deliver mode (in other words, apply it on live price data), very strange result can be seen:

You can see that many candles has both red and green bars. How is it possible if we set red bar only when median is lower and green only when median is higher?! Does it mean that median is higher and is lower in the same time?

Let's run the indicator in debug mode. Add the following watches:

 core.now()
 period
 median[period] - median[period - 1]
 green[period]
 red[period]

Also, set break points at these lines:

 function Update(period, mode)
     if period >= first then
         if median[period] < median[period - 1] then
             red[period] = -1;
         elseif median[period] > median[period - 1] then
             green[period] = 1;
         end
     end
 end

Using the default EUR-USD-H1 set we will get the situation we need right at the second period. Look:

We are about to set the red bar. The condition is correct, the current median is smaller than the previous. Well, let's continue:

Stop, stop, stop... The period is still 2, but now we are about set green bar. And the red bar is already set just a few seconds ago for the same period!!!. But somehow now the current median is higher than the previous. What is the magic?

Please pay attention on core.now() trace point. The time is changed. The period 2 is the last, alive period, so every new tick changes the current candle!. You can easily see it if you switch into chart view.

The chart at the first stop point:

The chart at the second stop point:

At the second chart we have the same 7:00 candle, but changed by coming ticks. The red bar is already set on the first of the ticks of this candle, but now the candle grows up and we (reasonable!!!) are about to set a green bar.

So, the condition may be changed for the same candle if it is on alive market, so our mistake is that we only put the bar by the condition, but not clear the bar which is possible was set on previous tick for the same candle!!! Do you see our mistake in that supposition: Does it mean that median is higher and is lower in the same time? . The time is not the same!.

Lets fix:

 function Update(period, mode)
     if period >= first then
         if median[period] < median[period - 1] then
             red[period] = -1;
             green[period] = 0;
         elseif median[period] > median[period - 1] then
             green[period] = 1;
             red[period] = 0;
         end
     end
 end

Now the indicator works correctly as on the historical market as well as on the live market prices.

So, How to Develop Properly?

Process All Branches in Conditions

If there is any condition which chooses to fill one of the output stream, or fill stream in one case and leave default (0) value in another, always clean other streams or set the default value explicitly.

Example:

 -- Incorrect!!!!
 if close[period] < close[period - 1] then
    lt[period] = 1;
 elseif close[period] > close[period - 1] then 
    gt[period] = 1;
 else
    eq[period] = 1;
 end

 -- Correct 
 lt[period] = 0;
 gt[period] = 0;
 eq[period] = 0;
 if close[period] < close[period - 1] then
    lt[period] = 1;
 elseif close[period] > close[period - 1] then 
    gt[period] = 1;
 else
    eq[period] = 1;
 end

 -- Correct and Optimal
 if close[period] < close[period - 1] then
    rel[period] = -1;
 elseif close[period] > close[period - 1] then 
    rel[period] = 1;
 else
    rel[period] = 0;
 end

Make Indicator Stateless

The update process shall not store any result of previous calls in the global variables. This mistake is usually made for such things as state or previous calculation result kept for optimization.

Lets see an example of improper state handling:

 local direction = 0;
 
 function Update(period, mode)
    if direction == 1 then
       if close[period] < close[period - 1] then
          direction = -1;
       end
       ...
    end
 end

The relation between current and previous close values may be changed, possible more than one time for each bar on live market. So, the wrongful changes of the state may appear in that case.

The correct way is to keep ALL such data as states or previous result calculations in the internal streams. These are the streams exactly like the regular output streams but not visible to the user.

So, the correct state handling is:

 local direction;
 
 function Prepare()
    ...
    direction = instance:addInternalStream(0, 0);
 end 
 
 function Update(period, mode)
    direction[period] = direction[period - 1];
    if direction[period] == 1 then
       if close[period] < close[period - 1] then
          direction[period] = -1;
       end
       ...
    end
 end

So, in other words, every time when we are called for a particular bar, we shall process it as if we are called for them for the first time.

Being Stateless When Complex Data is Required

The internal stream may be used to keep numeric data. So, it is good enough also, for example to booleans (you may just keep zero/non-zero values), but what about complex data? What if you need the tables?

Well. In that case you may use the serial number of the source as a key for a keeping the state inside Lua table.

 function newState()
   ...
   return state;
 end
 
 function copyState(state);
   ...
 end
 
 local states = {};
 
 function Update(period, mode)
   local state = nil;
   if period > 0 then
      state = copyState(states[source:serial(period - 1)])
   end
   
   if state == nil then
      state = newState();
   end
   ...
   states[source:serial(period)] = state;
   ...
 end

Always Test in Both Deliver Mode

To avoid mistakes please always test your indicator in both modes, with pre-deliver price equal to "Yes" and to "No". Ideally, you should save the indicator output streams for the both test runs on the same data in both pre-deliver modes, and these output streams must match in every digit.

See Also

Debugging Indicator