Problems with using multiple price sources

Section for discussions related to indicators, use of indicators, and building of trading stategies using indicators.

Moderator: admin

Problems with using multiple price sources

Postby tllewell » Thu Sep 11, 2014 9:15 am

I have been working on indicators using multiple price sources, and I've been using getHistory and getSyncHistory to get all the data. Many times the last six bars or so of the indicator are skewed because not all of the price sources have data. Sometimes, though, it works just fine. To me this looks like a bug. Here is an example of the code I am using (this doesn't actually do much of anything, but on my system it displays the error):

Code: Select all
function Init()
   indicator:name("BSKT");
   indicator:description("Index of a basket including this pair.");
   indicator:requiredSource(core.Bar);
   indicator:type(core.Oscillator);
   indicator:setTag("group", "Trend");
   --indicator:setTag("replaceSource", "t");

   indicator.parameters:addGroup("Basket");
   indicator.parameters:addString("BC", "Base, or Counter", "Base (1st) or counter (2nd) currency.", "base");
   indicator.parameters:addStringAlternative("BC", "base", "", "base");
   indicator.parameters:addStringAlternative("BC", "counter", "", "counter");
   
end

local first;
local source = nil;
local source1 = nil;
local source2 = nil;
local source3 = nil;
local source4 = nil;
local source5 = nil;
local loading1 = false;
local loading2 = false;
local loading3 = false;
local loading4 = false;
local loading5 = false;
local basket;
local basket_array = {};
local open, high, low, close; --candle elements, open high, low,close

function Prepare()
   source = instance.source;
   first = source:first();

   if instance.parameters.BC == "base" then
      basket = string.match(source:instrument(), "(%u*)/%u*");
   elseif instance.parameters.BC == "counter" then
      basket = string.match(source:instrument(), "%u*/(%u*)");
   end

   local name = profile:id() .. "(" .. basket .. ")";
   instance:name(name);

   local enum, row;
   enum = core.host:findTable("Offers"):enumerator();
   while true do
      row = enum:next();
      if row == nil or
      row == NULL then
         break;
      end
      if row.Instrument ~= source:instrument() then
         if basket == string.match(row.Instrument, "(%u*)/%u*") then
            table.insert(basket_array, #basket_array + 1, row.Instrument);
         elseif basket == string.match(row.Instrument, "%u*/(%u*)") then
            table.insert(basket_array, #basket_array + 1, row.Instrument);
         end
      end
   end
   for i = 1, #basket_array do
      if i == 1 then
         source1 = core.host:execute("getSyncHistory", basket_array[1], source:barSize(), source:isBid(), 0, 1, 11);
      end
      if i == 2 then
         source2 = core.host:execute("getSyncHistory", basket_array[2], source:barSize(), source:isBid(), 0, 2, 12);
      end
      if i == 3 then
         source3 = core.host:execute("getSyncHistory", basket_array[3], source:barSize(), source:isBid(), 0, 3, 13);
      end
      if i == 4 then
         source4 = core.host:execute("getSyncHistory", basket_array[4], source:barSize(), source:isBid(), 0, 4, 14);
      end
      if i == 5 then
         source5 = core.host:execute("getSyncHistory", basket_array[5], source:barSize(), source:isBid(), 0, 5, 15);
      end
   end
   open = instance:addStream("OPEN", core.Line, name .. ".OPEN", "OPEN", core.rgb(255,0,0), 0);
   high = instance:addStream("HIGH", core.Line, name .. ".HIGH", "HIGH", core.rgb(255,0,0), 0);
   low = instance:addStream("LOW", core.Line, name .. ".LOW", "LOW", core.rgb(255,0,0), 0);
   close = instance:addStream("CLOSE", core.Line, name .. ".CLOSE", "CLOSE", core.rgb(255,0,0), 0);

   instance:createCandleGroup(name, "BSKT", open, high, low, close);
end

-- Indicator calculation routine
function Update(period, mode)
   
   if loading1 or
   loading2 or
   loading3 or
   loading4 or
   loading5 then
      return;
   end
   
   open[period] = 0;
   high[period] = 0;
   low[period] = 0;
   close[period] = 0;
   
   if basket == string.match(source:instrument(), "(%u*)/%u*") then
      open[period] = open[period] + source.open[period];
      high[period] = high[period] + source.high[period];
      low[period] = low[period] + source.low[period];
      close[period] = close[period] + source.close[period];
   elseif basket == string.match(source:instrument(), "%u*/(%u*)") then
      open[period] = open[period] - source.open[period];
      high[period] = high[period] - source.high[period];
      low[period] = low[period] - source.low[period];
      close[period] = close[period] - source.close[period];
   end
   for i = 1, #basket_array do
      if i == 1 then
         if not source1:hasData(period) then
            core.host:trace("No data for " .. basket_array[i] .. " at period " .. period .. ".\n");
            return;
         end
         if basket == string.match(basket_array[1], "(%u*)/%u*") then
            open[period] = open[period] + source1.open[period];
            high[period] = high[period] + source1.high[period];
            low[period] = low[period] + source1.low[period];
            close[period] = close[period] + source1.close[period];
         elseif basket == string.match(basket_array[1], "%u*/(%u*)") then
            open[period] = open[period] - source1.open[period];
            high[period] = high[period] - source1.high[period];
            low[period] = low[period] - source1.low[period];
            close[period] = close[period] - source1.close[period];
         end
      elseif i == 2 then
         if not source2:hasData(period) then
            core.host:trace("No data for " .. basket_array[i] .. " at period " .. period .. ".\n");
            return;
         end
         if basket == string.match(basket_array[2], "(%u*)/%u*") then
            open[period] = open[period] + source2.open[period];
            high[period] = high[period] + source2.high[period];
            low[period] = low[period] + source2.low[period];
            close[period] = close[period] + source2.close[period];
         elseif basket == string.match(basket_array[2], "%u*/(%u*)") then
            open[period] = open[period] - source2.open[period];
            high[period] = high[period] - source2.high[period];
            low[period] = low[period] - source2.low[period];
            close[period] = close[period] - source2.close[period];
         end
      elseif i == 3 then
         if not source3:hasData(period) then
            core.host:trace("No data for " .. basket_array[i] .. " at period " .. period .. ".\n");
            return;
         end
         if basket == string.match(basket_array[3], "(%u*)/%u*") then
            open[period] = open[period] + source3.open[period];
            high[period] = high[period] + source3.high[period];
            low[period] = low[period] + source3.low[period];
            close[period] = close[period] + source3.close[period];
         elseif basket == string.match(basket_array[3], "%u*/(%u*)") then
            open[period] = open[period] - source3.open[period];
            high[period] = high[period] - source3.high[period];
            low[period] = low[period] - source3.low[period];
            close[period] = close[period] - source3.close[period];
         end
      elseif i == 4 then
         if not source4:hasData(period) then
            core.host:trace("No data for " .. basket_array[i] .. " at period " .. period .. ".\n");
            return;
         end
         if basket == string.match(basket_array[4], "(%u*)/%u*") then
            open[period] = open[period] + source4.open[period];
            high[period] = high[period] + source4.high[period];
            low[period] = low[period] + source4.low[period];
            close[period] = close[period] + source4.close[period];
         elseif basket == string.match(basket_array[4], "%u*/(%u*)") then
            open[period] = open[period] - source4.open[period];
            high[period] = high[period] - source4.high[period];
            low[period] = low[period] - source4.low[period];
            close[period] = close[period] - source4.close[period];
         end
      elseif i == 5 then
         if not source5:hasData(period) then
            core.host:trace("No data for " .. basket_array[i] .. " at period " .. period .. ".\n");
            return;
         end
         if basket == string.match(basket_array[5], "(%u*)/%u*") then
            open[period] = open[period] + source5.open[period];
            high[period] = high[period] + source5.high[period];
            low[period] = low[period] + source5.low[period];
            close[period] = close[period] + source5.close[period];
         elseif basket == string.match(basket_array[5], "%u*/(%u*)") then
            open[period] = open[period] - source5.open[period];
            high[period] = high[period] - source5.high[period];
            low[period] = low[period] - source5.low[period];
            close[period] = close[period] - source5.close[period];
         end
      end
   end
end

function AsyncOperationFinished(cookie, success, message)
   for i = 1, #basket_array do
      if cookie == 1 then
         if success then
            loading1 = false;
         end
      end
      if cookie == 2 then
         if success then
            loading2 = false;
         end
      end
      if cookie == 3 then
         if success then
            loading3 = false;
         end
      end
      if cookie == 4 then
         if success then
            loading4 = false;
         end
      end
      if cookie == 5 then
         if success then
            loading5 = false;
         end
      end
   end
   
   for i = 1, #basket_array do
      if cookie == 11 then
         if success then
            loading1 = true;
         end
      end
      if cookie == 12 then
         if success then
            loading2 = true;
         end
      end
      if cookie == 13 then
         if success then
            loading3 = true;
         end
      end
      if cookie == 14 then
         if success then
            loading4 = true;
         end
      end
      if cookie == 15 then
         if success then
            loading5 = true;
         end
      end
   end

   if not loading1 and
   not loading2 and
   not loading3 and
   not loading4 and
   not loading5 then
      instance:updateFrom(0);
   end
end


I have also attached a picture of the output with GBP/NZD, GBP/AUD, GBP/CAD, EUR/GBP, GBP/JPY, and GBP/USD currencies used as inputs.
Attachments
GBP_basket.png
Picture of indicator on my computer.
tllewell
 
Posts: 76
Joined: Thu Mar 01, 2012 4:06 pm

Re: Problems with using multiple price sources

Postby Valeria » Fri Sep 12, 2014 7:32 am

Hi tllewell,

Thank you for the report. This really looks like a bug. We should further investigate the issue to make sure whether this is a bug or not. I will let you know about the results as soon as possible.
Valeria
 

Re: Problems with using multiple price sources

Postby Apprentice » Sun Sep 14, 2014 4:11 am

Test.png

I believe that this is not a bug.
Perhaps Indicator values ​​are outside the visible area of the screen.

Please Double-check your math for indicator output.
User avatar
Apprentice
FXCodeBase: Confirmed User
 
Posts: 17537
Joined: Thu Dec 31, 2009 11:59 am
Location: Zagreb, Croatia

Re: Problems with using multiple price sources

Postby tllewell » Sun Sep 14, 2014 10:16 pm

I am not sure what the screen shot you sent is supposed to show. If is is based on what I sent, did you look at the Events for the traces?

The indicator I sent doesn't actually do much of anything except collect the price histories. If you look at it you'll see all it does is a simple addition or subtraction. I'm not sure what you think could go wrong. I think it does show that the error has nothing to do with the calculation.

If you'll look at the code, I put traces in to indicate if price histories were missing bars, which as I understand it should not happen with getSyncHistory. The screen shot I sent was for a run where there were traces in the Events screen that said the price history for GBP/CAD was missing data for period 345 over and over.

Also, this error appears and disappears without any change in the code. And why would it show up in the last six bars of the indicator so often (as in the screen shot I sent)?
tllewell
 
Posts: 76
Joined: Thu Mar 01, 2012 4:06 pm

Re: Problems with using multiple price sources

Postby Valeria » Thu Sep 18, 2014 11:24 am

Hi tllewell,

We have investigated the issue. There is a bug in your code. The thing is that you can define just the beginning of the trace for the indicator source by using getSyncHistory. This is why the dates of the first and the last bars are the same in the trace of the indicator source. But this is not necessarily that the amount of bars will be the same. There can be gaps in the trace and, as the result, when the indicator is reset there is a gap in the end of the trace.
Here is an example of a trace after the download:

Code: Select all
GBP/USD TEST    Trace: 'syncHistoryStart=41898.696527778, sourceStart=41898.696527778, syncHistoryEnd=41899.186111111, sourceEnd=41899.186111111, syncHistorySize=699, sourceSize=702'. 17.09.2014 20:28:23


Here is the difference in the amount of bars of three bars. So you should search the bar by the date within the downloaded trace.
Valeria
 

Re: Problems with using multiple price sources

Postby tllewell » Fri Sep 19, 2014 7:31 pm

The doc on getSyncHistory says:
The method return the stream which ends at the same time when the indicator source ends (and is subscribed to the updates if the indicator source is subscribed)


This makes it sound like it is the indicator equivalent of ExtSubscribe in strategies, and this is pretty much how I've used it. Now you say that it does not actually work the way it is described since it returns a different number of bars than there are in the source.

Could explain how to use it a little more completely? It sounds like you are saying you can use it to start the price history of additional sources of a different method has to be used to keep up with live data as time goes on. It also sounds like you need to manually find each bar in the history before you use it because the histories are not actually synced.

Could you explain a little more, maybe supply a code example of correct usage? Do I have to code in some other method on the indicator source?
tllewell
 
Posts: 76
Joined: Thu Mar 01, 2012 4:06 pm

Re: Problems with using multiple price sources

Postby TakisGen » Mon Sep 22, 2014 10:50 am

Hi tllewell,

Keep your code to the minimum possible even if you have to think for hours how to do it.
This helps for the program to be more readable and makes identifying logic errors easier.
To do that use tables extensively.

I work with multiple price streams for years and never had problems. Price servers are doing a good job.

Here is a code abstract that might help you. Works fine for me ;)

Code: Select all
--   Prapare section
   for kI,vI in pairs(tInstruments) do
      SOURCE[kI]=core.host:execute("getSyncHistory",vI,source:barSize(),source:isBid(),0,kI,-kI)
   end

   CANDLES.close = instance:addStream("close", core.Line, name .. ".close", "", core.colors().White, first);
   CANDLES.open = instance:addStream("open", core.Line, name .. ".open", "", core.colors().White, first);
   CANDLES.high = instance:addStream("high", core.Line, name .. ".high", "", core.colors().White, first);
   CANDLES.low = instance:addStream("low", core.Line, name .. ".low", "", core.colors().White, first);
   CANDLES.volume = instance:addStream("volume", core.Line, name .. ".volume", "", core.colors().White, first);
   instance:createCandleGroup(name, "OC", CANDLES.open, CANDLES.high, CANDLES.low, CANDLES.close, CANDLES.volume);

--   Update section
   local h,l,c,v=0,0,0,0
   local p=period
   for kF,vF in pairs(SOURCE) do
      if not vF.loaded then return end
      local pS=core.findDate(vF,source:date(p),false)
      if vF and pS~=-1 then
         h=h+toPips(vF.high,pS,source)
         l=l+toPips(vF.low,pS,source)
         c=c+toPips(vF.close,pS,source)
      else
         h=CANDLES.close[p-1]
         l=CANDLES.close[p-1]
         c=CANDLES.close[p-1]
      end
   end
   CANDLES.open[period]=CANDLES.close[period-1]
   CANDLES.high[period]=h
   CANDLES.low[period]=l
   CANDLES.close[period]=c

function toPips(s,p,instanceSource)  --this is to make the ruler of the chart usable
   if not p then p=s:size()-1 end
   return s[p]/s:pipSize()*instanceSource:pipSize()
end

--   AsyncOperationFinished section
   local kI=math.abs(cookie)
   if cookie>0 then
      SOURCE[kI].loaded=true
      for k,v in pairs(SOURCE) do
         if not v.loaded then
            return
         end
      end
      instance:updateFrom(source:first())
   elseif cookie<0 then
      SOURCE[kI].loaded=false
   end
   return core.ASYNC_REDRAW ;


Cheers
TakisGen
 
Posts: 34
Joined: Sat May 28, 2011 10:09 am

Re: Problems with using multiple price sources

Postby Valeria » Mon Sep 22, 2014 11:21 am

Hi tllewell,

Different sources can have different amount of bars. You can see that on overlay charts. Here is a sample of code:

Code: Select all
function Update(period, mode)
...
    local date = source:date(period);
    local source1_period = core.findDate(source1, date, true);
    if source1_period == -1 then
        --no such a bar
    else
        if not source1:hasData(source1_period) then
            core.host:trace("No data for " .. basket_array[i] .. " at period " .. source1_period .. ".\n");
            return;
        end
        if basket == string.match(basket_array[1], "(%u*)/%u*") then
            open[period] = open[period] + source1.open[source1_period]; --note: open and source1 have different periods
        ...
    end
Valeria
 

Re: Problems with using multiple price sources

Postby tllewell » Mon Sep 22, 2014 9:05 pm

That's reasonably close to what I ended up doing. I tried to brute force looking through the price streams for the correct index, but that seems to cause problems.

I ended up modifying first so that it doesn't fall before the shortest price stream. I also check each price stream for hasData before I use it (belt and suspenders approach). Lastly, for the most part the streams do match up in most cases. So I just compare the date of the next price stream to see if it matches the date of the source, and if if doesn't I return out, like

Code: Select all
if source:date(period) ~= source1:date(period) then
     return;
end
tllewell
 
Posts: 76
Joined: Thu Mar 01, 2012 4:06 pm

Re: Problems with using multiple price sources

Postby TakisGen » Wed Sep 24, 2014 9:54 am

hey tllewell,

Do not trust comparisons using period directly as an index when dealing with multicurrency streams.
For example, if source is eur/usd and source1 is xau/usd, this will give you a headache as xau/usd is not always tradeable during the day (which means missing bars).
If you use findDate with false option set you will not even have to think about such problems.
You are ready to sum prices for your custom candles.

Cheers
TakisGen
 
Posts: 34
Joined: Sat May 28, 2011 10:09 am

Next

Return to Discussions

Who is online

Users browsing this forum: No registered users and 2 guests