Using Other Time Frame Price Data in Indicator (Yesterday Close Value indicator)

Why Other Time Frame Data is Required?

Sometimes an indicator requires data of a bigger time frame. For example, you could use the close of the previous day for calculations. However, it’s possible to try and find this data in the price history the indicator is applied to. To get yesterday’s high/low, the whole yesterday’s data must be loaded. If the current history is a minute history, in the worst case, you would need to have at least 2880 bars to be able find that data (1440 minute bars for today and then 1440 minute bars for the whole yesterday). This is a huge amount to load and then to calculate but the system has access to the day data, so why don’t you use it? You can do it.

However, there are a number of points you should handle, carefully. You can develop an indicator which shows yesterday’s close price on the chart and discuss each point in detail.

What is “Yesterday”?

The handling of the date in indicators is quite simple. A date value is a number, which is just a number of days past midnight of Dec, 30 1899. The format is also known as Windows Ole Automation date format. The integer part of this number is a number of days and the fractional part is a part of the day. So, to find a calendar midnight date and time, you should just floor the value (cut away the fractional part). To shift the date to N days ago, you should just subtract N from the date. So, if FXCM used the “calendar clock” for the trading days (i.e. the trading day starts at midnight), the finding of yesterday for bar x of the source would be pretty simple: math.floor(source:date(x)) - 1.

Unfortunately, FXCM’s trading day does not match the calendar trading day. Currently, the trading day starts at 17:00EST yesterday and lasts until 17:00EST today. For different configurations, this date can differ. Hopefully, the indicator core helps you to handle calculation of the candle borders for a particular date. There is a core.getcandle function which returns the date and time of the candle of the specified time frame to which the specified date and time belong. As you can see, this function requires two parameters which let you “tune” this function for a certain configuration: an offset of the trading day and an offset of the trading week against calendar. You should use host:execute("getTradingDayOffset") and host:execute("getTradingWeekOffset") host’s calls to get the settings of the trading day and trading week for particular host application and connection.

So, the function to get yesterday’s date properly can look as follows:

function getYesterday(date)
    local offset;
    local weekoffset;

    offset = core.host:execute("getTradingDayOffset");
    weekoffset = core.host:execute("getTradingWeekOffset");

    local today, yesterday;
    today = core.getcandle("D1", date, offset, weekoffset);
    yesterday = today - 1;
    return yesterday;
end

And yet, there is another point you should handle. There are “non-trading” periods when neither trading is possible nor prices are received. So, during a week, only Monday trough Friday candles exist. There are no “Saturday” candles at all. So, when your minute belongs to a Monday day candle, the function returns, to you, a Sunday candle which just does not exist. So, for Monday’s data you should take the previous Friday’s close price.

To handle that correctly, there is another useful function: core.isnontrading which checks whether the date belongs to a non-trading period (Friday 17:00EST-Sunday 17:00EST) and returns the date and time when this non-trading period is finished. So, to find the proper “yesterday” candle you have to check whether the calendar yesterday falls into non-trading period and shift the “yesterday” date into past to skip this non-trading period.

You can modify your function, a bit:

function getYesterday(date)
    local offset;
    local weekoffset;

    offset = core.host:execute("getTradingDayOffset");
    weekoffset = core.host:execute("getTradingWeekOffset");

    local today, yesterday;
    today = core.getcandle("D1", date, offset, weekoffset);
    yesterday = today - 1;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(yesterday, offset);
    if nontrading then
        -- if the calendar yesterday date is non-trading (Friday 17:00 EST - Sunday 17:00 EST)
        -- shift three days back against the non-trading period end (Sunday 17:00 EST).
        yesterday = nontradingend - 3;
    end

    return yesterday;
end

So, now, having the getYesterday() function, you can easily find how many days you need. The oldest day is getYesterday(source:date(source:first())), i.e. the “yesterday” bar which corresponds to the oldest available bar. And what about the newest one? There are two cases. The source collection could be either a historical (i.e. a collection loaded until the specified date which does not change for the chart window) or “alive” (i.e. loaded till “now” and continuously growing as new bars come). According to these two cases, the “to” date could be either a “yesterday” day of the latest available bar (getYesterday(source:date(source:size() - 1)) or … now as well. To specify “now”, you use 0 value.

Loading the Data

Using Host to Load Data

Ok, now you know which data you need. The next step is to get this data. Marketscope provides a host interface which lets you execute an application-specific functions. One of these functions is “getHistory”. By this call Marketscope loads the specified instrument in the specified time frame and in the specified time range. See host:execute(“getHistory”, …) for details.

local days;     -- a variable to keep days

-- the function loads the proper day history
function loadData()
    if source:size() < source:first() then
        return ;
    end

    local from, to;
    -- load the data from the date of proper yesterday for
    -- the first available bar of the source
    from = getYesterday(source:date(source:first()));
    -- load to "now" if the source is up to "now" or
    -- to the day which corresponds to the latest
    -- day of the collection.
    if source:isAlive() then
        to = 0;
    else
        to = getYesterday(source:date(source:size() – 1));
    end
    days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
end

It is important that after the “getHistory” call, Marketscope only starts to load the data. It takes some time but Marketscope does not wait until the data is completely loaded to avoid hanging of the indicator. When data loading is finished, Marketscope notifies the indicator by calling the AsyncOperationFinished function. Moreover, Marketscope is not affected by the indicator status and continues calling indicator to be updated.

So, there are two things to do.

Protect Indicator from Updating While the Data Is Loaded

First, you should prevent the indicator from updating while the data is loaded.

To prevent the indicator from being recalculated, you can return from the update function while data is being loaded. Add the “loading” global variable so that all functions could “know” that data is being loaded.

local days;         -- day source data
local loading;      -- the loading day data flag

function Prepare()
    ...
    days = nil;
    loading = false;
    ...
end

function Update(period, mode)
    if loading then 
        -- do nothing if data is being loaded
        return ;
    end

    if days == nil then
        loadData();
        return ;
    end
    ...
end        

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
    end
    return 0;
end

function loadData()
    if loading then
        return ;
    end

    local from, to;

    -- load the data from the date of proper yesterday for
    -- the first available bar of the source
    from = getYesterday(source:date(source:first()));
    -- load to "now" if the source is up to "now" or
    -- to the day which corresponds to the latest
    -- day of the collection.
    if source:isAlive() then
        to = 0;
    else
        to = getYesterday(source:date[source:size() - 1]);
    end
    loading = true;
    days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
end

Forcing Update When Data Is Loaded

Now, you need to make the indicator update when data is loaded. There is function instance:updateFrom() which 'forces' the indicator to be recalculated from the specified period. Also, when AsyncOperationFinished returns the core.ASYNC_REDRAW value, Marketscope redraws the indicator in the chart window.

So, since you haven’t calculated anything yet, you have to update your indicator right from the oldest data. Just fix AsyncOperationFinished a bit.

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
        instance:updateFrom(source:first());
        return core.ASYNC_REDRAW;
    end
    return 0;
end

Getting Data

It is almost finished. The last question is how to get the data. You can calculate the day date. Of course, you can just loop through the day history and find the date but it is a bit long. There is a function, however, that finds the date just for log₂(n) operations - core.findDate().

So, you can build the first version of your indicator. You can do so and try it.

function Init()
    indicator:name("Show Yesterday Close");
    indicator:description("The indicator shows yesterday's close value on the chart");
    indicator:requiredSource(core.Tick);
    indicator:type(core.Indicator);

    indicator.parameters:addColor("clr", "Indicator Line Color", "", core.rgb(255, 255, 0));
end

local source;       -- indicator source data
local first;        -- first available source data
local days;         -- day source data
local loading;      -- the loading day data flag
local loadedFrom;   -- the date from which the days was loaded
local SYC;          -- yesterday close output

local host;
local offset;
local weekoffset;


function Prepare()
    local name;

    name = profile:id() .. "(" .. instance.source:name() .. ")";
    instance:name(name);

    source = instance.source;
    first = source:first();
    loading = false;
    days = nil;

    host = core.host;
    offset = host:execute("getTradingDayOffset");
    weekoffset = host:execute("getTradingWeekOffset");

    SYC = instance:addStream("SYC", core.Line, name, "SYC", instance.parameters.clr, first);
end

function Update(period, mode)
    if period < first then
        return ;
    end
    if loading then
        return ;
    end

    -- load the data if there is no day
    if days == nil then
        SYC:setBookmark(1, source:first());
        loadData();
        return ;
    end

    -- try to find the value in the days collection
    local yesterday;
    yesterday = getYesterday(source:date(period));
    if yesterday < loadedFrom then
        SYC:setBookmark(1, period);
        loadData();
        return ;
    end

    local i;
    i = core.findDate(days, yesterday, false);
    if i >= 0 then
        SYC[period] = days.close[i];
    end
end

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
        local i = SYC:getBookmark(1);
        if i < source:first() then
            i = source:first();
        end
        instance:updateFrom(i);
        return core.ASYNC_REDRAW;
    end
    return 0;
end

-- the function loads the proper day history
function loadData()
    if source:size() < source:first() then
        return ;
    end

    if loading then
        return ;
    end

    local from, to;

    if days == nil then
        -- initial data loading

        -- load the data from the date of proper yesterday for
        -- the first available bar of the source
        from = getYesterday(source:date(source:first()));
        -- load to "now" if the source is up to "now" or
        -- to the day which corresponds to the latest
        -- day of the collection.
        if source:isAlive() then
            to = 0;
        else
            to = getYesterday(source:date(source:size() - 1));
        end
        loading = true;
        loadedFrom = from;
        days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
    else
        from = getYesterday(source:date(source:first()));
        to = days:date(0);
        loading = true;
        loadedFrom = from;
        host:execute("extendHistory", 1, days, from, to);
    end
end

-- returns the beginning of the "yesterday" date for
-- the specified date
function getYesterday(date)
    local today, yesterday;

    -- get beginning of the today's candle and round it up to the second
    today = core.getcandle("D1", date, offset, weekoffset);
    today = math.floor(today * 86400 + 0.5) / 86400;

    -- get calendar yesterday
    yesterday = today - 1;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(yesterday, offset);
    if nontrading then
        -- if the calendar yesterday date is non-trading (Friday 17:00 EST - Sunday 17:00 EST)
        -- shift three days back against the non-trading period end (Sunday 17:00 EST).
        yesterday = nontradingend - 3;
    end
    return yesterday;
end

Now apply the indicator to the chart.

It works, so you can stop. When you try to load more data into past it apparently stopped to work.

How To Process the Scroll Back

What happened? Well, remember that you loaded the data starting from yesterday for the oldest of the existing candles. When you started to scroll your chart back, more data was loaded. No one updated your collection of day bars. So, you have to force Marketscope to load new data when the source collection is extended into past. Good news is that there is another cool call of the host interface: host:execute(“extendHistory”) which helps you just load more data into already loaded history instead of loading the whole history again. Since the original source history can be extended only into past, it is too easy to check whether new data is loaded. When it happens, the “yesterday” value you are trying to find is earlier than the date that you previously loaded the day history from.

So, do the following modification in your code:

1. Keep the “from” date that you loaded the day history earlier.
2. In event that the day collection already exists, extend it, instead of loading older data.

So, add new loadedFrom global variable and change the Update and loadData functions.

There is a new indicator version:

function Init()
    indicator:name("Show Yesterday Close");
    indicator:description("The indicator shows yesterday's close value on the chart");
    indicator:requiredSource(core.Tick);
    indicator:type(core.Indicator);

    indicator.parameters:addColor("clr", "Indicator Line Color", "", core.rgb(255, 255, 0));
end

local source;       -- indicator source data
local first;        -- first available source data
local days;         -- day source data
local loading;      -- the loading day data flag
local loadedFrom;   -- the oldest yesterday bar we already loaded
local SYC;          -- yesterday close output

local host;
local offset;
local weekoffset;

function Prepare()
    local name;

    name = profile:id() .. "(" .. instance.source:name() .. ")";
    instance:name(name);

    source = instance.source;
    first = source:first();
    loading = false;
    days = nil;

    host = core.host;
    offset = host:execute("getTradingDayOffset");
    weekoffset = host:execute("getTradingWeekOffset");
    SYC = instance:addStream("SYC", core.Line, name, "SYC", instance.parameters.clr, first);
end

function Update(period, mode)
    if period < first then
        return ;
    end
    if loading then
        return ;
    end

    -- load the data if there is no day
    if days == nil then
        loadData();
        return ;
    end

    -- try to find the value in the days collection
    local yesterday;
    yesterday = getYesterday(source:date(period));

    -- check whether such day bar is not loaded yet
    if yesterday < loadedFrom then
        loadData();
        return ;
    end

    -- find the day data
    local i;
    i = core.findDate(days, yesterday, false);
    if i >= 0 then
        SYC[period] = days.close[i];
    end
end

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
        instance:updateFrom(source:first());
        return core.ASYNC_REDRAW;
    end
    return 0;
end

-- the function loads the proper day history
function loadData()
    if source:size() < source:first() then
        return ;
    end

    if loading then
        return ;
    end

    local from, to;

    if days == nil then
        -- initial data loading

        -- load the data from the date of proper yesterday for
        -- the first available bar of the source
        from = getYesterday(source:date(source:first()));
        -- load to "now" if the source is up to "now" or
        -- to the day which corresponds to the latest
        -- day of the collection.
        if source:isAlive() then
            to = 0;
        else
            to = getYesterday(source:date(source:size() - 1));
        end
        loading = true;
        loadedFrom = from;
        days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
    else
        from = getYesterday(source:date(source:first()));
        to = days:date(0);
        loading = true;
        loadedFrom = from;
        host:execute("extendHistory", 1, days, from, to);
    end
end

-- returns the beginning of the "yesterday" date for
-- the specified date
function getYesterday(date)
    local today, yesterday;

    -- get beginning of the today's candle and round it up to the second
    today = core.getcandle("D1", date, offset, weekoffset);
    today = math.floor(today * 86400 + 0.5) / 86400;

    -- get calendar yesterday
    yesterday = today - 1;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(yesterday, offset);
    if nontrading then
        -- if the calendar yesterday date is non-trading (Friday 17:00 EST - Sunday 17:00 EST)
        -- shift three days back against the non-trading period end (Sunday 17:00 EST).
        yesterday = nontradingend - 3;
    end
    return yesterday;
end

Install the indicator and try it. Now, when the previous version stops working during the scroll, the new version works. Congratulations!

Notes

How to Debug Such Indicators Using SDK Debugger?

First, you must prepare the data. You can save price data files using Marketscope. Just open some instrument in both resolutions (for example 30 minutes and 1 day) and then save both histories in the Indicore SDK data format (File->Export to Indicore…). You can also download the data automatically via SDK Debugger using the quotes manager server (Tools->Load Quotes).

When you start the indicator, choose the smaller timeframe history in the indicator parameters. Please, also check the “Predeliver Data” checkbox in the indicator parameters to let the debugger show the whole available history to the indicator, like Marketscope does. Without this option, the debugger shows the history only up to the currently simulated bar. When the indicator executes “getHistory” call, the prompt for data appears. Choose the day data previously saved from Marketscope, in that prompt, or use the data from the quotes manager server.

How to Handle Timeframes Other than 1 Day?

To make the example indicator simpler, I used a 1-day time frame. However, you would want to do the same for other time frame data, for example, use 1-hour close price instead of a day close price. To do that, you can:

• Let the user choose the time frame. Add a string parameter to the indicators parameter list and then set the time frame selector flag to that parameter. As a result, in the Prepare function, the corresponding parameter contains the code of the time frame chosen by the user.

function Init()
    ...
    indicator.parameters:addString("TF", "Timeframe", "", "m1");
    indicator.parameters:setFlag("TF", core.FLAG_PERIODS);
    ...
end

• Calculate the length of a candle of the specified time frame in day. Use the [c]core.getcandle()[/c] function to get the start date of any candle and the start date of the next candle. The difference between these two values is the length of the candle in days. It is also strongly recommended that you calculate all of the dates in seconds, rather than in days, to avoid rounding errors.

local candleLength;  -- length of candle in seconds

function Prepare()
    ...
    offset = host:execute("getTradingDayOffset");
    weekoffset = host:execute("getTradingWeekOffset");
    ...
    local thisCandle, nextCandle;
    thisCandle, nextChandle = core.getcandle(instance.parameters.TF, core.now(), offset, weekoffset);
    candleLength = math.floor((nextCandle - thisCandle) * 86400 + 0.5);
    ...
end

• Use previously calculated candle length instead of 1 day value in your “getPreviousBar” function. Please note that while the candle length could be different, the non-trading period is still in days. So, handle it carefully.

function getPreviousBar(date)
    local curr, prev;

    -- get beginning of the today's candle and round it up to the second
    curr = core.getcandle(instance.parameters.TF, date, offset, weekoffset);
    today = math.floor(today * 86400 + 0.5);

    -- get calendar yesterday
    prev = (curr – candleLength) / 86400;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(prev, offset);
    if nontrading then
        prev = prev - 2;
        prev = math.floor(prev * 86400 + 0.5);

        -- get calendar yesterday
        prev = (prev – candleLength) / 86400;
    end
    return prev;
end