Welcome to Forex-TSD!, one of the largest Forex forums worldwide, where you will be able to find the most complete and reliable Forex information imaginable.
From the list below, select the forum that you want to visit and register to post, as many times you want. It’s absolutely free. Click here for registering on Forex-TSD.
Exclusive Forum
The Exclusive Forum is the only paid section. Once you subscribe, you will get free access to real cutting-edge Trading Systems (automated and not), Indicators, Signals, Articles, etc., that will help and guide you, in ways that you could only imagine, with your Forex trading.
Elite Section
Get access to private discussions, specialized support, indicators and trading systems reported every week.
Advanced Elite Section
For professional traders, trading system developers and any other member who may need to use and/or convert, the most cutting-edge exclusive indicators and trading systems for MT4 and MT5.
Simba,
those are very convincing results. I'm not sure just what I'm looking at, but the curves are very similar for all TF's. It seems that there is not really an aliasing problem, however, I'm lost to understand why not. The H1 series should have significantly more noise, than the M1 series. Apparently there is no sustained high frequency cyclical component in any if the lower TF's, and there is little or no noise. This seems incredible to me, and since I went to some trouble to try to understand the possibility of aliasing before you posted, I would like to post my thoughts.
My concern about aliasing may not be clear, so let me illustrate the effect of aliasing better than I have. For ease in illustration, let's assume that the spectrum of an M1 price series consists of two periodic components embedded in noise. The noise has a triangular spectrum, one of the components is at a low frequency (long period, shown in blue) and the other is at a high frequency (short period, shown in red). Since we are dealing with an M1 price series, the spectrum must end at the sample rate/2 (fs/2) (period = 2 minutes). ( If higher frequency components exist in the transactions, they will be aliased into the spectrum below fs/2). Further, assume that both components have reasonably good signal to noise ratio. This is all shown in the top half of the first figure below. The effect of subsampling this price series at 5 minute intervals (M5) is shown in the bottom half of the figure. This can be graphically computed by noting that the spectrum of a sampled signal can be found by convolving the spectrum of the original signal with a series of impulses at the sampling frequency. ( Since the M5 period is 5 times the M1 period, the sampling frequency is 1/5 the 1 minute sampling frequency as shown. )This involves the simple steps of overlaying the initial spectrum at each impulse. It's pretty clear what a mess this makes of the original spectrum. Notice that the high frequency component is aliased down near the low frequency component, but the low frequency component is not affected. aliased build.JPG
The effective spectrum of the M5 price series derived by subsampling the M1 series is shown in the figure below. On the left I tried to show how the noise builds up due to aliasing. On the right I show the final spectrum of the M5 signal. The high frequency component has been aliased down near the low frequency component, and the noise has built up so that the signal to noise ratio is now pretty bad. aliased result.JPG
Now suppose that there were several high frequency cyclical components in the original spectrum, and suppose that we subsample it once every hour, or once every 4 hours. The resulting mess should have cyclical components all over the place, and the signal to noise ratio should be terrible.
Since any cyclical components in the final (say H1) signal must have been in the original M1 signal, but with a better signal to noise ratio, it seems to me that one should use the M1 signal to extract cyclical components. Of course the problem with this is that those components that are at say 20 hour periods will be very hard to extract from M1 data because the M1 period will require 60 times as many samples at M1. On the other hand, there may be many components that are at high frequencies in the M1 series, that alias into the H1 series several times causing more peaks than are really there.
The only simple way to investigate this is to look at the spectrum of an M1 signal and an H1 signal over the same (absolute) period, e.g. 200 hours or so. This cannot be done with the R_MESA tools currently available because the length required at M1 exceeds the capability of the algorithm as coded.
It seems that you have already looked at the various TF's and are satisfied that the cyclical components are present at a low enough frequency that they are not affected by subsampling. By using something like the Goertzel algorithm, (or simply a set of narrow band filters, which is equivalent) one can apparently ignore the noise added by aliasing. This is good news. I am convinced that the components are there. Don't know why. So I will proceed to look at a phase locked tracking filter.
I would like to hear more about fractal filtering if you have some sources.
Thanks ...MadCow...
Since any cyclical components in the final (say H1) signal must have been in the original M1 signal, but with a better signal to noise ratio, it seems to me that one should use the M1 signal to extract cyclical components. Of course the problem with this is that those components that are at say 20 hour periods will be very hard to extract from M1 data because the M1 period will require 60 times as many samples at M1. On the other hand, there may be many components that are at high frequencies in the M1 series, that alias into the H1 series several times causing more peaks than are really there.
The only simple way to investigate this is to look at the spectrum of an M1 signal and an H1 signal over the same (absolute) period, e.g. 200 hours or so. This cannot be done with the R_MESA tools currently available because the length required at M1 exceeds the capability of the algorithm as coded.
MadCow...
Madcow,
Yes to both,one should use the M1 signal and compare it with the H1,M5,15,M30..over the same absolute period...that is exactly what I did(starting by M5) and showed on the pics ...I scanned 540 h1 bars in h1,m30(1080 m30 bars),m15,m5...for the same range of absolute H1frequencies 1/90 to 1/180..so,I looked for cycles with periodicity 90 to 180 H1 periods(180 to 360 m30 periods,etc,etc)..and I found the same 2 cycles at all timeframes.
If I redo the scan to look for cycles 8 to 80 h1 periods,over an absolute span of 240 h1 bars,what I will find is a very different picture,everytimeframe is giving me back a different number of cycles,periods,amplitudes and phases...it is an untradable mess as you can see in the pic...So,using the right frequency bands,the ones that give you back clear cycles is one of the keys of working with cycles...to do so,you need a scanner that can scan at least 7k -9k m1 bars and give you back cyclic periods of up to 2k -3k m1 bars.
Fractal cycles...Basically:The number of iterations or passing of time before a fractal repeats(caveat emptor:It can repeat at the same OR at a different timeframe)....the vertical lines in the pic mark the later iterations of a fractal,fractal cycle is 22 or 23 bars(it is fractal )..BTW,at the close of Friday we completed a fractal iteration...I am extremely intrigued by today`s opening directional move for the next 22 bars.
Simba,
I've read Lammert's stuff, and so far I can't make heads or tails of it. Seems a lot like astrology, but a lot of people believe him apparently, so I must dig deeper. He appears to use the definition of fractal used in the MT4 "fractal.mq4", essentially extrema. How does that relate to fractal cycles?
Simba,
I've read Lammert's stuff, and so far I can't make heads or tails of it. Seems a lot like astrology, but a lot of people believe him apparently, so I must dig deeper. He appears to use the definition of fractal used in the MT4 "fractal.mq4", essentially extrema. How does that relate to fractal cycles?
But let me begin to prove myself useful:
1. Do not use the higher TF price series when looking for cyclic components. They will be noisier than M1. You appear to be doing this already... how about the rest of the forum?
2. Use an anti-aliasing filter on M1 data and sub-sample it's output to reduce computational complexity when searching for cyclical components of long periodicity. Match the aa filter to the frequencies you search for.
3. Do not "end point flatten" the price series when applying the FFT or Geortzel.
The first two items I have discussed above.
Consider the third. One can show that the periodogram is the maximum likelihood detector of a sinusoid in stationary white gaussian noise. I know that FX noise is not stationary, white or even Gaussian, but there is still an argument that the periodogram is a good thing to use. The periodogram is essentially the square of the Fourier transform of the signal, and suffers from the fact that it's variance does not decrease as the length of the sequence processed increases. When it is used to estimate spectral density, the data is windowed to reduce this "end point" effect. But we are interested in detecting the component, not in estimating the spectrum. If we were to use a set of bandpass filters to find the signal, we would not first endpoint flatten the price series. That would clearly be counterproductive, since it would decrease the weight of the most recent samples. So don't end point flatten the price series before applying the FFT or Geortzel.
4. The FFT or Geortzel are equivalent when adjusted for resolution. The Geortzel is more efficient when searching over a small set of frequencies. Use these interchangably depending on computational efficiency determined by the band being searched and the required resolution.
5. The MESA is good to estimate the spectrum, and in some cases (particularly for time varying sinusoids) may be better than the FFT. However, estimating the spectrum and detecting sinusoids are different tasks, and if the sinusoid lasts for the duration of the FFT, MESA is no better than the FFT in the maximum likelihood sense, and most probably not as good. So use the FFT and match it's resolution to the expected duration of the sinusoid. Forget the MESA, in spite of RC's excellent software.
These are only my opinions, but they can be backed mathematically if the model is simplified as stated. I also know that not everyone will agree, and that RC and Ehlers would particularly disagree, but I'm only trying to be useful as requested.
Regards ...MadCow...
I don`t know about the rest of the Forum,a few of them may be doing some interesting things,others will probably wait to be spoon fed.
I want to apologize for abruptly editing my previous post. (You can read it in entirety in Simba'a reply). My reason was that I had discovered that end point flattening was not what I thought. It merely sets the two end points to 0 and adds a small amount to each sample between to compensate for this. It essentially adds a very long period component, that we can ignore. It does not distort the current output for sufficiently high frequency. Also I was misspelling Goertzel.
And I no longer feel that the FFT or Goertzel is the best way to find cyclic components.
In looking at end point flattening and the Goertzel algorithm I found that there may be a better way to find the spectral content when our goal is to use the estimated spectrum to predict the next few samples. We might want to use variable length Goertzel filters instead of the fixed length filters in all the Goertzel implementations. The attached picture shows the difference betwen the fixed length Goertzel implemented as Goertzel_v1.mq4 and a variable length Goertzel. Quite a difference. Variable length G EUH1.JPG
I have a habit of writing notes to myself as I analyze a problem. As I grow older, I find that this helps with short term recall problems, which plague me (hence MadCow...). The attached document is my set of notes on the variable length Goertzel, and why I prefer it. I would appreciate any feedback.Variable Length Goertzel Filters.doc
I want to apologize for abruptly editing my previous post. (You can read it in entirety in Simba'a reply). My reason was that I had discovered that end point flattening was not what I thought. It merely sets the two end points to 0 and adds a small amount to each sample between to compensate for this. It essentially adds a very long period component, that we can ignore. It does not distort the current output for sufficiently high frequency. Also I was misspelling Goertzel.
And I no longer feel that the FFT or Goertzel is the best way to find cyclic components.
In looking at end point flattening and the Goertzel algorithm I found that there may be a better way to find the spectral content when our goal is to use the estimated spectrum to predict the next few samples. We might want to use variable length Goertzel filters instead of the fixed length filters in all the Goertzel implementations. The attached picture shows the difference betwen the fixed length Goertzel implemented as Goertzel_v1.mq4 and a variable length Goertzel. Quite a difference. Attachment 91120
I have a habit of writing notes to myself as I analyze a problem. As I grow older, I find that this helps with short term recall problems, which plague me (hence MadCow...). The attached document is my set of notes on the variable length Goertzel, and why I prefer it. I would appreciate any feedback.Attachment 91121
Regards ...MadCow...
MadCow,
As I see it from your .doc
1-The basis of all your thinking is that Goertzel needs 3*max period sample rates,so,if you want to scan for cycles up to 200 periods,you need at least 600 samples...Right.
2-Consequently,the low periodicity cycles(high freq) may not be representative of what really is happening at the moment...example,a 13 bars cycle found over a span of the last 600 bars may not be there now,and ,yes you are absolutely right.
3-Conceptually,there may be a simpler solution though,and that is just to scan separately for several frequency bands...so if you want to confirm the 13 periods cycle you found,you have to redo a scan over the last 39 bars..or,even better,do a scan over the last 60 bars for cycles with freqs 1/10 to 1/5.
Conclusion..you need to do different scans..this was hinted by JM Hurst in his 1600 page course,and explained by Brian Millard in detail....or,you can build a multifreq,multispan scanner that scans in "blocks" of related "max period vs length of scan"...that will be really original and innovative
1-The basis of all your thinking is that Goertzel needs 3*max period sample rates,so,if you want to scan for cycles up to 200 periods,you need at least 600 samples...Right.
2-Consequently,the low periodicity cycles(high freq) may not be representative of what really is happening at the moment...example,a 13 bars cycle found over a span of the last 600 bars may not be there now,and ,yes you are absolutely right.
3-Conceptually,there may be a simpler solution though,and that is just to scan separately for several frequency bands...so if you want to confirm the 13 periods cycle you found,you have to redo a scan over the last 39 bars..or,even better,do a scan over the last 60 bars for cycles with freqs 1/10 to 1/5.
Conclusion..you need to do different scans..this was hinted by JM Hurst in his 1600 page course,and explained by Brian Millard in detail....or,you can build a multifreq,multispan scanner that scans in "blocks" of related "max period vs length of scan"...that will be really original and innovative
Regards
S
Clearly you're way ahead of me on this. It is nice to have confirmation that I can still think logically.
I looked at the use of different scans, and decided that trying to find short cycles in long blocks might miss them, so I might never go back to confirm that they were there. Also it would take a lot of extra computation to look at many blocks. So I looked at the code and decided that I could modify the Goertzel_v1 code to do the multifreq, multispan scanner directly. The code actually is simpler and takes fewer computation cycles than the original _v1. Innovation, or just luck? The modified Goertzel_v1 (now _v2) was used in my previous post.
Attached are two modifications, _v2 does the job without end point flattening, though it can be added by removing some comments in the code. _v3 does the whole job including end point flattening for each block/period. You can vary the minimum number of cycles to look for using the numcycles variable. This changes the resolution of the filters, with more cycles giving better resolution, but requiring more samples, so allowing for past signals to interfere with current signals. You also can use the median price instead of the close by setting the useclose to false. It does not make any difference that I can see.
My coding skills are pretty rusty, so there may be a coding problem with the end point flattening, but I think not. If there's no code problem, then EPF does not help for multi freq scanning.
The next few steps I will take are to decide how to select the components to use for prediction, and how to estimate the longevity of the prediction. I am inclined to use a measure of the SNR in each filter holding a peak to select peaks, rather than just the amplitude, but I have as yet no clue about prediction longevity.. perhaps experimentation will help. Any suggestions?
Clearly you're way ahead of me on this. It is nice to have confirmation that I can still think logically.
I looked at the use of different scans, and decided that trying to find short cycles in long blocks might miss them, so I might never go back to confirm that they were there. Also it would take a lot of extra computation to look at many blocks. So I looked at the code and decided that I could modify the Goertzel_v1 code to do the multifreq, multispan scanner directly. The code actually is simpler and takes fewer computation cycles than the original _v1. Innovation, or just luck? The modified Goertzel_v1 (now _v2) was used in my previous post.
Attached are two modifications, _v2 does the job without end point flattening, though it can be added by removing some comments in the code. _v3 does the whole job including end point flattening for each block/period. You can vary the minimum number of cycles to look for using the numcycles variable. This changes the resolution of the filters, with more cycles giving better resolution, but requiring more samples, so allowing for past signals to interfere with current signals. You also can use the median price instead of the close by setting the useclose to false. It does not make any difference that I can see.
My coding skills are pretty rusty, so there may be a coding problem with the end point flattening, but I think not. If there's no code problem, then EPF does not help for multi freq scanning.
The next few steps I will take are to decide how to select the components to use for prediction, and how to estimate the longevity of the prediction. I am inclined to use a measure of the SNR in each filter holding a peak to select peaks, rather than just the amplitude, but I have as yet no clue about prediction longevity.. perhaps experimentation will help. Any suggestions?
1-How to decide which components to take for prediction:Use 3 of your v3, set each one of them to (MaxPer-MinPeriod):50-10,100-50,200-100.
2-Then select just one cycle per v3,the one with the highest amplitude.
3-Then using,periodicity,amplitude and phase of each one of the 3 cycles,add them into a composite.
4-Length of prediction?..Besides computing the Lyapunov exponent ? rule of thumb,IMO,is to use one third to half the bars of the shortest cycle ..
Now,You may want to believe me or not,and,sincerely,you don`t have any reason,because I won`t explain why,besides telling you that I have been there and done that...
5-SNR:Forget about it..It doesn`t work,you can try as many ways as you want..they don`t work for selecting good cycles.
6-Using ALL the cycles?.....Try it and you will start believing me..IT DOESN`T WORK...prediction length will be less than 1 bar.
7-Using 3 cycles,but just the highest amplitude ones from 10-200?It can work,usually doesn`t,you will end up losing all the short periodicity cycles .
8-You want to use many cycles?It doesn`t work,but it can work if you use squared amplitudes,in any case ,don`t use more than 8 cycles.
Now..the best counsel I can give you..By far,the best way to TRADE(not predict) these cycles is to do as I wrote in number 1 and 2 and then...
9- Just trade on 3 cycles alignment(forget about composites)..I use phases in degrees not in radians,so when the 3 cycles align above 180 degrees,you can expect an upmove,when they align between 0 and 179 degrees you can expect a downmove...in practice,the rule I use is above 180 and below 270(for longs)..and above 0 and below 90(for shorts)...then,once you have the alignment,wait for price confirmation,specially trendline breakouts,before entering the trade,use a target related to the ATR of the timeframe you are trading and use a stop below previous low/above previous high...if the TP is >than the SL then take the trade,if it is not,then pass....Oh,and a small detail,always trade in the direction of the larger timeframe
By the way..October 16th is the turn date for sp500,then November 2nd and then,DA TOP..Between November 13th and December 9th(sorry,exact dat jut for charities )..yes,yes,the guy in the report says differently,but either him or myself is wrong...guess whom?..hahahhahah..check October 16th when it comes and decide.
S
1-How to decide which components to take for prediction:Use 3 of your v3, set each one of them to (MaxPer-MinPeriod):50-10,100-50,200-100.
2-Then select just one cycle per v3,the one with the highest amplitude.
3-Then using,periodicity,amplitude and phase of each one of the 3 cycles,add them into a composite.
4-Length of prediction?..Besides computing the Lyapunov exponent ? rule of thumb,IMO,is to use one third to half the bars of the shortest cycle ..
Now,You may want to believe me or not,and,sincerely,you don`t have any reason,because I won`t explain why,besides telling you that I have been there and done that...
5-SNR:Forget about it..It doesn`t work,you can try as many ways as you want..they don`t work for selecting good cycles.
6-Using ALL the cycles?.....Try it and you will start believing me..IT DOESN`T WORK...prediction length will be less than 1 bar.
7-Using 3 cycles,but just the highest amplitude ones from 10-200?It can work,usually doesn`t,you will end up losing all the short periodicity cycles .
8-You want to use many cycles?It doesn`t work,but it can work if you use squared amplitudes,in any case ,don`t use more than 8 cycles.
Now..the best counsel I can give you..By far,the best way to TRADE(not predict) these cycles is to do as I wrote in number 1 and 2 and then...
9- Just trade on 3 cycles alignment(forget about composites)..I use phases in degrees not in radians,so when the 3 cycles align above 180 degrees,you can expect an upmove,when they align between 0 and 179 degrees you can expect a downmove...in practice,the rule I use is above 180 and below 270(for longs)..and above 0 and below 90(for shorts)...then,once you have the alignment,wait for price confirmation,specially trendline breakouts,before entering the trade,use a target related to the ATR of the timeframe you are trading and use a stop below previous low/above previous high...if the TP is >than the SL then take the trade,if it is not,then pass....Oh,and a small detail,always trade in the direction of the larger timeframe
Regards
S
Simba..
Thank you for your generosity. I think you have saved me many days of fruitless search. I will use the information as well as I can.
There are a couple of things that I should explain to the forum in case anyone wishes to use _v2 or _v3.
First, the phase computation seems to be in contradiction to the MT4 definition of the MathArctan function. I have not tested the phase computation yet, so be careful using it. I simply left it as is in _v1.
Second, although you may select components from various blocks as Simba suggests, there is no need to split the spectrum into several blocks when making the computation. In fact, you may miss an important component at the edge of a block if you do that. The entire periodicity can be computed in a single pass since the block length is independently computed for each period. But, as Simba says, one picture is worth 1000 words. Here is the spectrum computed in one pass, compared with portions computed in separate passes with limited periodicity.
...I scanned 540 h1 bars in h1,m30(1080 m30 bars),m15,m5...for the same range of absolute H1frequencies 1/90 to 1/180..so,I looked for cycles with periodicity 90 to 180 H1 periods(180 to 360 m30 periods,etc,etc)..and I found the same 2 cycles at all timeframes.
The attached document is my set of notes on the variable length Goertzel, and why I prefer it. I would appreciate any feedback.
Linear Mode
