Optimizing averages.lua: Optimization by example

Moderator: admin

Optimizing averages.lua: Optimization by example

Postby Nikolay.Gekht » Wed Mar 30, 2011 1:19 pm

The old implementation of averages.lua (viewtopic.php?f=17&t=2430) is a good example on how to do not write the indicator to have it fast and efficient. Every rule of the specified in the http://fxcodebase.com/wiki/index.php/In ... timization article is broken. So, let see the highlight which helped to make it 2-6 times faster.

Generic indicator structure optimization:

1) String comparison in Update() function. You know that Update() is called every time when a new tick comes to the live market snapshot. So, we can recognize the Update function as called inside a loop. Now, the whole Update() function is just a big sequence of string comparisons to choose the proper function for the Method chosen during initialization. But the method does NOT change. Why do we need make that selection all the time? The better way is to store the proper function once, in the Prepare() method into the global variable and then call it.

was:
Code: Select all
function Update(period, mode)
 local i;
   if (period>=first) then
    if Method=="MVA" then
     MA[period]=MVA(source,period,MA,Period);
    elseif Method=="EMA" then
     MA[period]=EMA(source,period,MA,Period);
...


now:
Code: Select all
local UpdateFunction;

function Prepare(onlyName)
    if Method=="MVA" then
     UpdateFunction=_G["MVA"];
    elseif Method=="EMA" then
     UpdateFunction=_G["EMA"];
...
end

function Update(period, mode)
 local i;
   if (period>=first) then
     UpdateFunction(source,period,MA,Period);
...


2) Needless internal streams. The indicator always creates five internal stream, even these streams aren't required at all. Handling of every, even unused, stream takes time on every tick. So, just create the streams ONLY in case they are actually required.

Algorithm optimization:

1) A lot of algorithms, such as simple moving average, linear-weighted moving average, least-square moving average are implemented in core in the mathex table. This table is implemented in C++, so it is faster than Lua.

Was:
Code: Select all
  local Weight=0;
  local Sum=0;
  local i;
  for i=0,Period_-1,1 do
   Weight=Weight+(Period_-i);
   Sum=Sum+Price[shift-i]*(Period_-i);
  end
  if Weight>0 then
   return Sum/Weight;
  else
   return 0;
  end
 else
  return 0;
 end


now
Code: Select all
  return mathex.lwma(Price, Period - Period_ + 1, Period);


Also, it is a good idea to understand the formula beneath the somebody's source first, because a part of the formula may be optimized. For example, for ILRS the slope may be calculate using mathex:

was:
Code: Select all
function ILRS(Price,shift,MAarray,Period_)
 local sum=Period_*(Period_-1)*0.5;
 local sum2=(Period_-1)*Period_*(2*Period_-1)/6;
 local sum1=0;
 local sumy=0;
 local i;
 for i=0,Period_-1,1 do
  sum1=sum1+i*Price[shift-i];
  sumy=sumy+Price[shift-i];
 end
 local num1=Period_*sum1-sum*sumy;
 local num2=sum*sum-Period_*sum2;
 local slope;
 if num2~=0 then
  slope=num1/num2;
 else
  slope=0;
 end
 return slope+MVA(Price,shift,MAarray,Period_);
end


now:
Code: Select all
params.buffer[period] = mathex.lregSlope(params.source, from, period) + mathex.avg(params.source, from, period);


2) Needless calculation

Even if algorithm is not implemented in full or partially in the mathex, there is no need to do repeating calculations which have exactly the same result for every tick. For example weights for SineWMA are the same and depends on the offset to the last bar only.

Code: Select all
function SineWMA(Price,shift,MAarray,Period_)
 if shift>=first then
  local Sum=0;
  local Weight=0;
  local i;
  r=core.range(math.max(shift-Period_+1,first),shift);
  for i=r.from,shift,1 do
   Weight=Weight+math.sin(math.pi*(shift-i+1)/(Period_+1));
   Sum=Sum+Price[i]*math.sin(math.pi*(shift-i+1)/(Period_+1));
  end
  if Weight>0 then
   return Sum/Weight;
  else
   return 0;
  end
 end
end


now
Code: Select all
--
-- SineWMA: Sine weighted moving average
--
function SineWMAInit(source, n)
    local p = {};
    p.source = source;
    p.n = n;
    p.offset = n - 1;
    p.sine = {};
    p.first = source:first() + n - 1;

    local i, w;
    w = 0;
    for i = 1, n, 1 do
        p.sine[i] = math.sin(math.pi * (n - i + 1) / (n + 1));
        w = w + p.sine[i];
    end

    p.weight = w;
    p.alwaysZero = (w == 0);

    return p;
end

function SineWMAUpdate(params, period, mode)
    local sum = 0;
    if not params.alwaysZero then
        local src = params.source;
        local sine = params.sine;
        local n = params.n;
        local p = period - n;
        for i = 1, n, 1 do
            sum = sum + src[p + i] * sine[i];
        end
        sum = sum / params.weight;
    end
    params.buffer[period] = sum;
end


So, as you can see all things are pretty simple, but provides a good performance improvements. When we want to use our indicators in 1-year long backtesting, and, especially, optimization, every additional second counts. 6 times optimization of the indicator may make 10-minute backtesting just a 1.5 minute long. A few hours of indicator optimization worths it, right?
Nikolay.Gekht
FXCodeBase: Site Admin
 
Posts: 1235
Joined: Wed Dec 16, 2009 6:39 pm
Location: Cary, NC

Re: Optimizing averages.lua: Optimization by example

Postby cnikitopoulos94 » Mon Oct 03, 2016 9:59 pm

Hello Nikolay,

I just wanted to ask some questions about your codings in this article (which is magnificent by the way)

1.I wanted to ask you if there is in any way you can explain to me how there is a way we can actually use C+ database. I.E. for the optimizations to the algorithms implemented you said used C+ calculation in order to improve the speed due to the fact C+ is faster. Is there an index of operations you can share which you find using C+ functions and run smoother?

2. This is about the string values.

Code: Select all
function VAMAInit(source, n)
function ARSIInit(source, n)


What is the purpose of "n"?
cnikitopoulos94
 
Posts: 47
Joined: Sat Sep 12, 2015 8:10 am

Re: Optimizing averages.lua: Optimization by example

Postby Georgiy » Tue Oct 04, 2016 9:02 am

cnikitopoulos94,

You can see the description of mathex and all of its methods here:
http://www.fxcodebase.com/documents/IndicoreSDK/mathex.html
Georgiy
FXCodeBase: Initiate
 
Posts: 150
Joined: Tue Jul 29, 2014 4:49 am


Return to Indicator Development

Who is online

Users browsing this forum: No registered users and 8 guests