Optimizing/Fixing Heat Map Indicator (Elder Impulse System)
Posted: Wed Sep 14, 2011 8:08 am
The new release of the Marketscope significantly extended the size of the prices to be shown. The regular chart can contain up to 15,000 candles and backtester chart can contain up to 1,000,000 (one million candles, e.g. 3 years in 1-minute resolution). To avoid handing of the application the indicator must be ready to process such amount of the data in reasonable time.
The most resource eating indicators are bigger timeframe/multi timeframe (such as heat map) indicators. Let's take Elder Impulse System Heat Map as an example (original post).
The indicator has the following performance problems:
1) it checks the time ranges of the other time frame collections every time when access to those collections happens. So, if you load the indicator on 15,000 bar collection this check will be performance 15,000 times. But when the collection is extended backward, the update from the oldest available period will performed. So, we can use it as a flag that more data may be loaded and can check the other time frame range and initiate the loading.
2) it performs search the date in other time frame collection (which is fast, but still take time up to log2N, where N is the number of bars). However, if the top-level candle hasn't been changed comparing the previous period we can just copy the data. Which is much faster.
3) Perform moving the data from bigger frame collection only when the latest of periods is requested to be updated, so all stream is made "at once".
4) it uses the text output which is pretty heavy to handle on big charts, such as backtester windows.
Also, the indicator has the logic problem.
1) The last (active) bar of the Elder Impulse System may change it's value when new price comes. In this case, the heat map made for the active bar can be inconsistent. The indicator must detect the fact that the active bar has been changed and re-draw the data for the active bar of higher time frame.
The original Elder Impulse System indicator also has two problems:
1) It uses three streams instead of coloring the one output.
2) It ignores the fact that EMA smoothing becomes reliable in n * 1.5 bars. However, since this release EMA produces the data not from the first (oldest available) bar, but from n * 1.5 bar.
So, let's fix Elder Impulse System first.
1) Make only one output stream and change the color depending on the condition instead of putting data in three different streams.
2) Make proper handling of the "first available" data flag.
The code of the changed indicator is below:
Now fix the heat map indicator.
1) We must separate the logic of other time collection extension from getting the period and call it only when the oldest bar is updated.
2) We must cache already found index of the other time frame candle in internal stream and use it when candle is not changed.
3) We must check whether the lastest (active) candle has been changed and update the whole other time frame data.
4) We must change the text output to numeric output. To provide the similar behavior let's use 5-pixel wide line.
The fixed code with detailed comments is below. Please use this code as the template for heat map indicators.
Download indicators:
BTW, the indicator is still too heavy for Backtester (applying on 1-minute data in 1-year range (300K candles) takes about 2-3 seconds), but, at least, it does not hang the application.
The most resource eating indicators are bigger timeframe/multi timeframe (such as heat map) indicators. Let's take Elder Impulse System Heat Map as an example (original post).
The indicator has the following performance problems:
1) it checks the time ranges of the other time frame collections every time when access to those collections happens. So, if you load the indicator on 15,000 bar collection this check will be performance 15,000 times. But when the collection is extended backward, the update from the oldest available period will performed. So, we can use it as a flag that more data may be loaded and can check the other time frame range and initiate the loading.
2) it performs search the date in other time frame collection (which is fast, but still take time up to log2N, where N is the number of bars). However, if the top-level candle hasn't been changed comparing the previous period we can just copy the data. Which is much faster.
3) Perform moving the data from bigger frame collection only when the latest of periods is requested to be updated, so all stream is made "at once".
4) it uses the text output which is pretty heavy to handle on big charts, such as backtester windows.
Also, the indicator has the logic problem.
1) The last (active) bar of the Elder Impulse System may change it's value when new price comes. In this case, the heat map made for the active bar can be inconsistent. The indicator must detect the fact that the active bar has been changed and re-draw the data for the active bar of higher time frame.
The original Elder Impulse System indicator also has two problems:
1) It uses three streams instead of coloring the one output.
2) It ignores the fact that EMA smoothing becomes reliable in n * 1.5 bars. However, since this release EMA produces the data not from the first (oldest available) bar, but from n * 1.5 bar.
So, let's fix Elder Impulse System first.
1) Make only one output stream and change the color depending on the condition instead of putting data in three different streams.
2) Make proper handling of the "first available" data flag.
The code of the changed indicator is below:
- Code: Select all
-- Elder Impulse System
-- ----------------------------------------------------------------------------------------------------
-- Copyright © 2007, http://finance.groups.yahoo.com/group/MetaTrader_Experts_and_Indicators/
-- http://finance.groups.yahoo.com/group/MetaTrader_Experts_and_Indicators/
-- MT4 info: I used code from Macd_Correct by David W. Thomas and eSignal code supplied by bentleybrian <bentleybrian@yahoo.com>
-- to build this indicator . Built by transport_david .
-- ----------------------------------------------------------------------------------------------------
function Init()
indicator:name("Elder Impulse System");
indicator:description("");
indicator:requiredSource(core.Tick);
indicator:type(core.Oscillator);
indicator.parameters:addGroup("Calculation");
indicator.parameters:addInteger("EMA", "EMA periods for study", "", 13, 1, 100);
indicator.parameters:addInteger("MACDF", "MACD periods fast", "", 12, 1, 100);
indicator.parameters:addInteger("MACDS", "MACD periods slow", "", 26, 1, 100);
indicator.parameters:addGroup("Style");
indicator.parameters:addColor("UP", "Color of up signal", "", core.rgb(0, 255, 0));
indicator.parameters:addColor("DOWN", "Color of down signal", "", core.rgb(255, 0, 0));
indicator.parameters:addColor("NE", "Color of neutral signal", "", core.rgb(0, 0, 255));
indicator.parameters:addString("Width", "Bar width", "", "F");
indicator.parameters:addStringAlternative("Width", "Regular", "", "R");
indicator.parameters:addStringAlternative("Width", "Full Bar", "", "F");
end
local first, MACDfirst;
local UP;
local DOWN;
local NE;
local OUT;
local EMA;
local MACDF;
local MACDS;
local MACD;
local SIGNAL;
local alpha;
local alpha1;
function Prepare()
alpha = 2.0 / (9 + 1.0);
alpha1 = 1.0 - alpha;
EMA = core.indicators:create("EMA", instance.source, instance.parameters.EMA);
MACDF = core.indicators:create("EMA", instance.source, instance.parameters.MACDF);
MACDS = core.indicators:create("EMA", instance.source, instance.parameters.MACDS);
MACDfirst = instance.source:first() + instance.parameters.MACDS * 2;
MACD = instance:addInternalStream(MACDS.DATA:first(), 0);
SIGNAL = instance:addInternalStream(0, 0);
first = math.max(MACDfirst + 18, instance.source:first() + instance.parameters.EMA * 2);
local name = profile:id() .. "(" .. instance.source:name() .. ", " .. instance.parameters.EMA .. ", " .. instance.parameters.MACDF .. ", " .. instance.parameters.MACDS .. ")";
instance:name(name);
OUT = instance:addStream("OUT", core.Bar, name .. "OUT", "OUT", instance.parameters.NE, first);
UP = instance.parameters.UP;
DOWN = instance.parameters.DOWN;
NE = instance.parameters.NE;
if instance.parameters.Width == "F" then
OUT:setWidth(160);
end
OUT:addLevel(120, core.LINE_NONE, 1, core.rgb(0, 0 , 0));
OUT:addLevel(0, core.LINE_NONE, 1, core.rgb(0, 0 , 0));
end
function Update(period, mode)
EMA:update(mode);
MACDF:update(mode);
MACDS:update(mode);
if period >= MACDfirst then
MACD[period] = MACDF.DATA[period] - MACDS.DATA[period];
if period == MACDfirst then
SIGNAL[period] = MACD[period];
else
SIGNAL[period] = alpha * MACD[period] + alpha1 * SIGNAL[period - 1];
end
end
if period >= first then
local s1v1, s1v2;
local s2v1, s2v2;
s1v1 = EMA.DATA[period];
s1v2 = EMA.DATA[period - 1];
s2v1 = MACD[period] - SIGNAL[period];
s2v2 = MACD[period - 1] - SIGNAL[period - 1];
OUT[period] = 100;
if s1v1 > s1v2 and s2v1 > s2v2 then
OUT[period] = 100;
OUT:setColor(period, UP);
elseif s1v1 < s1v2 and s2v1 < s2v2 then
OUT[period] = 100;
OUT:setColor(period, DOWN);
else
OUT[period] = 100;
OUT:setColor(period, NE);
end
end
end
Now fix the heat map indicator.
1) We must separate the logic of other time collection extension from getting the period and call it only when the oldest bar is updated.
2) We must cache already found index of the other time frame candle in internal stream and use it when candle is not changed.
3) We must check whether the lastest (active) candle has been changed and update the whole other time frame data.
4) We must change the text output to numeric output. To provide the similar behavior let's use 5-pixel wide line.
The fixed code with detailed comments is below. Please use this code as the template for heat map indicators.
- Code: Select all
--
-- initialization function
--
function Init()
indicator:name("Multitimeframe Elder Impulse System Heat Map");
indicator:description("");
indicator:requiredSource(core.Bar);
indicator:type(core.Oscillator);
indicator.parameters:addGroup("Calculation");
addMvaParam(1, "H1", 13, 12, 26);
addMvaParam(2, "H2", 13, 12, 26);
addMvaParam(3, "H4", 13, 12, 26);
addMvaParam(4, "H6", 13, 12, 26);
addMvaParam(5, "H8", 13, 12, 26);
indicator.parameters:addString("Price", "Price", "", "close");
indicator.parameters:addStringAlternative("Price", "open", "", "open");
indicator.parameters:addStringAlternative("Price", "close", "", "close");
indicator.parameters:addStringAlternative("Price", "high", "", "high");
indicator.parameters:addStringAlternative("Price", "low", "", "low");
indicator.parameters:addStringAlternative("Price", "typical", "", "typical");
indicator.parameters:addStringAlternative("Price", "median", "", "median");
indicator.parameters:addStringAlternative("Price", "weighted", "", "weighted");
indicator.parameters:addGroup("Display");
indicator.parameters:addColor("clrUP", "Up color", "", core.rgb(0, 255, 0));
indicator.parameters:addColor("clrDN", "Down color", "", core.rgb(255, 0, 0));
indicator.parameters:addColor("clrNE", "Neutral color", "", core.rgb(0, 0, 255));
indicator.parameters:addColor("clrLBL", "Label color", "", core.COLOR_LABEL);
end
-- Add a group of time-frame related parameters
function addMvaParam(id, frame, EMA, MACDF, MACDS)
indicator.parameters:addString("B" .. id, "Time frame for avegage " .. id, "", frame);
indicator.parameters:setFlag("B" .. id, core.FLAG_PERIODS);
indicator.parameters:addInteger("EMA" .. id, "EMA " .. id .. "EMA ", "", EMA);
indicator.parameters:addInteger("MACDF" .. id, "MACDF " .. id .. "MACDF ", "", MACDF);
indicator.parameters:addInteger("MACDS" .. id, "MACDS " .. id .. "MACDS ", "", MACDS);
end
-- list of streams
local streams = {} -- list of attached streams
local source; -- source prices
local day_offset; -- offset of the trading day against calendar midnight
local week_offset; -- offset of the trading week against Sunday
local dummy; -- dummy stream
local host; -- a reference to host (perf. issue)
local PriceType; -- price type used
local clrUP, -- up color
clrDN, -- down color
clrNE; -- neutral color
local source_first; -- first bar of the source
local second = 1.0 / 86400.0; -- one second length
-- prepare the indicator for execution
function Prepare(onlyName)
-- cache the data
source = instance.source;
source_first = source:first();
host = core.host;
PriceType=instance.parameters.Price;
day_offset = host:execute("getTradingDayOffset");
week_offset = host:execute("getTradingWeekOffset");
-- validate parameters
checkBarSize(1);
checkBarSize(2);
checkBarSize(3);
checkBarSize(4);
checkBarSize(5);
-- make the indicator label
local i;
local name = profile:id() .. "(" .. source:name() .. ",".. PriceType .. ",";
for i = 1, 5, 1 do
name = name .. "(" .. instance.parameters:getInteger("EMA" .. i) .. " " .. instance.parameters:getInteger("MACDF" .. i) .. " " .. instance.parameters:getInteger("MACDS" .. i) .. ")";
end
name = name .. ")";
instance:name(name);
if onlyName then
return ;
end
-- colorize the indicator label and set range
dummy = instance:addStream("D", core.Line, name .. ".D", "D", instance.parameters.clrLBL, 0);
dummy:addLevel(0);
dummy:addLevel(120);
-- create output stream for timeframe data
for i = 1, 5, 1 do
stream = registerStream(i, instance.parameters:getString("B" .. i),
300,
instance.parameters:getString("B" .. i) .. " " .. instance.parameters:getString("EMA" .. i).. " " .. instance.parameters:getString("MACDF" .. i).. " " .. instance.parameters:getString("MACDS" .. i) .. ")");
end
clrUP = instance.parameters.clrUP;
clrDN = instance.parameters.clrDN;
clrNE = instance.parameters.clrNE;
end
-- update the indicator values
function Update(period, mode)
-- first call or further pre-loading of the data
if period == source_first then
for i = 1, 5, 1 do
updateStream(i);
end
end
if period == source:size() - 1 then
for i = 1, 5, 1 do
local stream = streams[i];
local label = stream.label;
if streams[i].loading then
label = label .. "," .. "(loading)";
end
host:execute("drawLabel", i, source:date(period), (6 - i) * 20, label);
if stream.data:size() > 0 then
updateOutput(stream, (6 - i) * 20, mode);
end
if not loading then
-- if the current value has been changed - update the whole candle backward
if source:isAlive()
and stream.external and period > source_first
and math.abs(stream.ref_candle[period] - stream.ref_candle[period - 1]) < second
and stream.output:colorI(period) ~= stream.output:colorI(period - 1) then
local t = period - 1;
local output = stream.output;
local color = stream.output:colorI(period);
local v = (6 - i) * 20;
while t > output:first() and math.abs(stream.ref_candle[period] - stream.ref_candle[t]) < second do
output[t] = v;
output:setColor(t, color);
t = t - 1;
end
end
end
end
end
end
-- the function is called when the async operation is finished
function AsyncOperationFinished(cookie)
registerDataLoaded(cookie);
end
-- validate the size of the chosen time frame
function checkBarSize(id)
local s, e, s1, e1;
s, e = core.getcandle(source:barSize(), core.now(), 0, 0);
s1, e1 = core.getcandle(instance.parameters:getString("B" .. id), core.now(), 0, 0);
assert ((e - s) <= (e1 - s1), "The chosen time frame must be equal to or bigger than the chart time frame!");
end
-- register a stream for further processing
-- @param id The identifier of the stream
-- @param barSize Stream's bar size
-- @param extent The size of the required exten
-- @param label The label of the stream
function registerStream(id, barSize, extent, label)
local stream = {};
local s1, e1, length;
local from, to;
s1, e1 = core.getcandle(barSize, core.now(), 0, 0);
length = math.floor((e1 - s1) * 86400 + 0.5);
stream.barSize = barSize; -- name of the time frame
stream.label = label; -- label
stream.length = length; -- length of the bar in seconds
stream.extent = extent; -- extent of the data in bars
if barSize == source:barSize() then
-- if the size of the timeframe requested is equal to the size of the source
-- use the indicator source
stream.external = false;
stream.loading = false;
stream.data = source;
else
-- else prepare everything for further update data loading
stream.data = nil;
stream.external = true;
stream.loading = false;
end
-- create an output
stream.output = instance:addStream("O" .. id, core.Line, "O" .. id, "O" .. id, instance.parameters.clrNE, 0);
stream.output:setWidth(5);
-- if stream is external prepare the cache for data
if stream.external then
stream.ref_candle = instance:addInternalStream(0, 0);
stream.fullUpdate = false;
else
stream.fullUpdate = true;
end
-- the place for the indicator
stream.indicator = nil;
streams[id] = stream;
end
-- The function checks whether the range of the source stream and additional streams are synchronized
function updateStream(id)
local stream = streams[id];
assert(stream ~= nil, "Stream is not registered");
if stream.external then
-- if stream is being already loaded - just wait until the stream loading is finished
if stream.loading then
return ;
end
-- try to sychronize the data of the source and the external streams:
local candle, from, to;
-- get the oldest candle of the source
candle = core.getcandle(stream.barSize, source:date(source_first), day_offset, week_offset);
if stream.data == nil then
from = getFromToLoad(candle, stream.length, stream.extent);
stream.requestedFrom = from;
stream.dataFrom = candle;
stream.loading = true;
if (source:isAlive()) then
to = 0;
else
t, to = core.getcandle(stream.barSize, source:date(source:size() - 1), day_offset, week_offset);
end
stream.data = host:execute("getHistory", id, source:instrument(), stream.barSize, from, to, source:isBid());
elseif candle < stream.dataFrom then
from = getFromToLoad(candle, stream.length, stream.extent);
stream.requestedFrom = from;
stream.dataFrom = candle;
stream.loading = true;
host:execute("extendHistory", id, stream.data, from, stream.data:date(0));
end
else
stream.fullUpdate = true;
end
-- if the indicator is not created yet, creat id
if stream.indicator == nil and stream.data ~= nil then
stream.indicator = core.indicators:create("ELDER_IMPULSE_SYSTEM",
stream.data[instance.parameters.Price],
instance.parameters:getInteger("EMA" .. id),
instance.parameters:getInteger("MACDF" .. id),
instance.parameters:getInteger("MACDS" .. id),
instance.parameters.clrUP,
instance.parameters.clrDN,
instance.parameters.clrNE);
end
end
-- the function is called when the external data is being loaded
function registerDataLoaded(id)
local stream = streams[id];
assert(stream ~= nil, "Stream is not registered");
if stream.external then
stream.loading = false;
stream.fullUpdate = true;
end
-- if all the streams has been loaded - update the indicator
instance:updateFrom(source:size() - 1);
end
-- updates the indicator output according the stream
-- stream - the stream to be updated
function updateOutput(stream, level, mode)
local i, from, to, candle_from, candle_to, indi_color, last_bf_idx;
stream.indicator:update(mode);
if stream.fullUpdate then
from = 0;
else
from = stream.output:size() - 1;
end
to = stream.output:size() - 1;
if stream.external then
candle_to = 0;
indi_color = nil;
last_bf_idx = -1;
for i = from, to, 1 do
local date = source:date(i);
if date >= candle_to then
candle_from, candle_to = core.getcandle(stream.barSize, date, day_offset, week_offset);
if last_bf_idx > 0 and last_bf_idx < source:size() - 1 and math.abs(stream.data:date(last_bf_idx + 1) - candle_from) < 1.157407407407407e-5 then
last_bf_idx = last_bf_idx + 1;
else
last_bf_idx = core.findDate(stream.data, candle_from, true);
end
if last_bf_idx < 0 then
indi_color = nil;
else
indi_color = stream.indicator.DATA:colorI(last_bf_idx);
end
end
stream.ref_candle[i] = candle_from;
if indi_color ~= nil then
stream.output[i] = level;
stream.output:setColor(i, indi_color);
else
stream.output:setNoData(i);
end
end
else
for i = from, to, 1 do
if stream.indicator.DATA:hasData(i) then
stream.output[i] = level;
stream.output:setColor(i, stream.indicator.DATA:colorI(i));
else
stream.output:setNoData(i);
end
end
end
stream.updateFull = false;
end
-- get date/time of the oldest candle which shall be requested
-- @param candle - the oldest candle of the chosen time frame
-- @param length - length of the bar in seconds
-- @param extent - the required extent in bars for history data
function getFromToLoad(candle, length, extent)
local loadFrom;
local nontrading, nontradingend;
-- calculate the date and time of the candle "extent" bar prior to the oldest requested candle
loadFrom = math.floor(candle * 86400 - length * extent + 0.5) / 86400;
-- check whether the candle found is inside the non-trading period
nontrading, nontradingend = core.isnontrading(loadFrom, day_offset);
if nontrading then
-- if it is non-trading, shift for two days to skip the non-trading periods
loadFrom = math.floor((loadFrom - 2) * 86400 - length * extent + 0.5) / 86400;
end
return loadFrom;
end
Download indicators:
BTW, the indicator is still too heavy for Backtester (applying on 1-minute data in 1-year range (300K candles) takes about 2-3 seconds), but, at least, it does not hang the application.