Optimizing averages.lua: Optimization by example

Optimizing averages.lua: Optimization by example

The old implementation of averages.lua (viewtopic.php?f=17&t=2430) is a good example on how to do not write the indicator to have it fast and efficient. Every rule of the specified in the http://fxcodebase.com/wiki/index.php/In ... timization article is broken. So, let see the highlight which helped to make it 2-6 times faster.

Generic indicator structure optimization:

1) String comparison in Update() function. You know that Update() is called every time when a new tick comes to the live market snapshot. So, we can recognize the Update function as called inside a loop. Now, the whole Update() function is just a big sequence of string comparisons to choose the proper function for the Method chosen during initialization. But the method does NOT change. Why do we need make that selection all the time? The better way is to store the proper function once, in the Prepare() method into the global variable and then call it.

was:
Code: Select all
function Update(period, mode)
local i;
if (period>=first) then
if Method=="MVA" then
MA[period]=MVA(source,period,MA,Period);
elseif Method=="EMA" then
MA[period]=EMA(source,period,MA,Period);
...

now:
Code: Select all
local UpdateFunction;

function Prepare(onlyName)
if Method=="MVA" then
UpdateFunction=_G["MVA"];
elseif Method=="EMA" then
UpdateFunction=_G["EMA"];
...
end

function Update(period, mode)
local i;
if (period>=first) then
UpdateFunction(source,period,MA,Period);
...

2) Needless internal streams. The indicator always creates five internal stream, even these streams aren't required at all. Handling of every, even unused, stream takes time on every tick. So, just create the streams ONLY in case they are actually required.

Algorithm optimization:

1) A lot of algorithms, such as simple moving average, linear-weighted moving average, least-square moving average are implemented in core in the mathex table. This table is implemented in C++, so it is faster than Lua.

Was:
Code: Select all
local Weight=0;
local Sum=0;
local i;
for i=0,Period_-1,1 do
Weight=Weight+(Period_-i);
Sum=Sum+Price[shift-i]*(Period_-i);
end
if Weight>0 then
return Sum/Weight;
else
return 0;
end
else
return 0;
end

now
Code: Select all
return mathex.lwma(Price, Period - Period_ + 1, Period);

Also, it is a good idea to understand the formula beneath the somebody's source first, because a part of the formula may be optimized. For example, for ILRS the slope may be calculate using mathex:

was:
Code: Select all
function ILRS(Price,shift,MAarray,Period_)
local sum=Period_*(Period_-1)*0.5;
local sum2=(Period_-1)*Period_*(2*Period_-1)/6;
local sum1=0;
local sumy=0;
local i;
for i=0,Period_-1,1 do
sum1=sum1+i*Price[shift-i];
sumy=sumy+Price[shift-i];
end
local num1=Period_*sum1-sum*sumy;
local num2=sum*sum-Period_*sum2;
local slope;
if num2~=0 then
slope=num1/num2;
else
slope=0;
end
return slope+MVA(Price,shift,MAarray,Period_);
end

now:
Code: Select all
params.buffer[period] = mathex.lregSlope(params.source, from, period) + mathex.avg(params.source, from, period);

2) Needless calculation

Even if algorithm is not implemented in full or partially in the mathex, there is no need to do repeating calculations which have exactly the same result for every tick. For example weights for SineWMA are the same and depends on the offset to the last bar only.

Code: Select all
function SineWMA(Price,shift,MAarray,Period_)
if shift>=first then
local Sum=0;
local Weight=0;
local i;
r=core.range(math.max(shift-Period_+1,first),shift);
for i=r.from,shift,1 do
Weight=Weight+math.sin(math.pi*(shift-i+1)/(Period_+1));
Sum=Sum+Price[i]*math.sin(math.pi*(shift-i+1)/(Period_+1));
end
if Weight>0 then
return Sum/Weight;
else
return 0;
end
end
end

now
Code: Select all
--
-- SineWMA: Sine weighted moving average
--
function SineWMAInit(source, n)
local p = {};
p.source = source;
p.n = n;
p.offset = n - 1;
p.sine = {};
p.first = source:first() + n - 1;

local i, w;
w = 0;
for i = 1, n, 1 do
p.sine[i] = math.sin(math.pi * (n - i + 1) / (n + 1));
w = w + p.sine[i];
end

p.weight = w;
p.alwaysZero = (w == 0);

return p;
end

function SineWMAUpdate(params, period, mode)
local sum = 0;
if not params.alwaysZero then
local src = params.source;
local sine = params.sine;
local n = params.n;
local p = period - n;
for i = 1, n, 1 do
sum = sum + src[p + i] * sine[i];
end
sum = sum / params.weight;
end
params.buffer[period] = sum;
end

So, as you can see all things are pretty simple, but provides a good performance improvements. When we want to use our indicators in 1-year long backtesting, and, especially, optimization, every additional second counts. 6 times optimization of the indicator may make 10-minute backtesting just a 1.5 minute long. A few hours of indicator optimization worths it, right?
Nikolay.Gekht

Posts: 1235
Joined: Wed Dec 16, 2009 6:39 pm
Location: Cary, NC

Re: Optimizing averages.lua: Optimization by example

Hello Nikolay,

1.I wanted to ask you if there is in any way you can explain to me how there is a way we can actually use C+ database. I.E. for the optimizations to the algorithms implemented you said used C+ calculation in order to improve the speed due to the fact C+ is faster. Is there an index of operations you can share which you find using C+ functions and run smoother?

2. This is about the string values.

Code: Select all
function VAMAInit(source, n)
function ARSIInit(source, n)

What is the purpose of "n"?
cnikitopoulos94

Posts: 47
Joined: Sat Sep 12, 2015 8:10 am

Re: Optimizing averages.lua: Optimization by example

cnikitopoulos94,

You can see the description of mathex and all of its methods here:
http://www.fxcodebase.com/documents/IndicoreSDK/mathex.html
Georgiy
FXCodeBase: Initiate

Posts: 150
Joined: Tue Jul 29, 2014 4:49 am