Introducing an 'OpenFoam Results Stability Index' (ORSI) that provides a new way to quantify confidence levels of Simulation Convergence

EDIT: ORSI convergence analysis has been added as a feature to my yPlusHistogram program available here: Introducing yPlusHistogram program with ORSI as a new metric for y+ mapping evaluation and convergence analysis

ORSI M500 = -316.03Nm_0.08%

Here is what that means:

  1. This is an ORSI (OpenFoam Results Stability Index) where the ‘M’ prefix signifies that it is with respect to ‘Moments’ results from a Hijacked OpenFoam Coefficients Results data in the Solver Log. Other valid prefix’s could be L, D, Pfx (pressure force x), Pfy, Pfz, Vfx (viscous force x), Vfy, Vfz, Pmx (pressure moment x), Pmy, Pmz, Vmx (viscous moment x), Vmy, Vmz, Mc (moment coefficient), Lc, Dc.
  2. 500 is the number of convergence iterations over which this ORSI has been calculated.
  3. -316.03Nm is the most likely ‘results value’ for the Moment value over the 500 iterations which we are evaluating. In this case it is ‘Nm’ since this is a Hijacked moment coefficient result and is not dimensionless.
  4. 0.8% is the smallest range, with respect to -316.03 Nm, in which all values for Moment can be contained in over the 500 iterations. Consequentially, -316.03 Nm must reside in that range.

Here is why an ORSI provides a good confidence metric for quantifying the quality of Simulation Convergence.

Lets look at this pretty nasty 2000 iteration Residual plot for a Simulation run on a 7M cell TETforCFD mesh, that has 5 boundary layers:

First impression is, the results will be horrible…

BUT, here are the Hijacked Coefficient results:

Those results, are in fact, MORE stable than any CFD results that I have ever seen in ANY of my simulations!

In fact this is what inspired me to come up with ORSI, I needed to quantify how stable these results actually were…

Some background:

  • About a year ago, @dylan (a former PowerUser here, and someone that I believe should go down in our ‘Hall of Fame’ and be granted perpetual Power User status) chatted with me in a private SimScale PM session (about simulations on a proprietary (at that time) aircraft design of mine). He freely shared a very small percentage of his vast OpenFoam knowledge with me (a lot of which can be found in his old forum posts here). He obtains result accuracies from OpenFoam simulations that are within 1% of experimental (WOW).
    .

With that background, Dylan shared that if you want to have results that will match experimantal results to within 1%, not only do you need to have your simulation properly set-up (that is a whole other topic than ORSI’s), you need to get results that are at least 1% stable over 500 iterations … It is that simple :wink:

So, I began by looking at the last 500 iterations, and by only having the ‘Moment Coefficient’ variable showing on the plot:

Uck, that does not look stable, does it?

But it really is, what we see are oscillating values that are descending from left to right. There are even damped ‘spikes’ of data thrown in there for good measure. I have found that those spikes are generally irrelevant and are caused by the weird residual behavior, at the spike iteration.

This image shows how to calculate ORSI, and it turns out that generally you do not have to actually make the Dark blue and the Red curves to get a very close estimate of the 3 red dot values that are used in the ORSI formula. You would be surprised how accurate that your brain can visualize and interpolate the curves. You generally can extract the 3 values needed for ORSI calculation by ‘eye’, a fingernail on the screen and/or your mouse pointer :wink:

And a quick note as to why I have chosen only to specify ORSI on data from a ‘Force and moment Coefficient’ result… It is my opinion that a ‘Moment Coefficient result’ value, is the value that is the most difficult of all the results values to get stable in a simulation results. Not only does my experience validate that opinion, but I think the fact that a moment coefficient is a scaled combination of all 3 axis of Pressure and Viscous forces could also validate that opinion :wink:

EDIT: One issue that I have found with using ‘Moment Coefficient result’ value as an ORSI that generally represents all the values that an ORSI can be specified for in a sim runs’ results, is that, since a Moment value is always relative to a rotation point, if the rotation point results in a Moment that is close to zero (at the Aerodynamic Center of the geometry), then a Moment ORSI could be almost impossible to specify that has meaning as compared to all the other ORSI values for that sim run. SO, for the moment (pun intended :joy::rofl::joy::rofl:), be careful when the Moment Coefficient Result value is close to zero (for example, use ORSI for other results values to specify ORSI for that sim run). I am looking for a way to ‘normalize’ ORSI values to get around this issue…

SO, a big question in my mind right now is…

If you have very stable output results, how much do the actual ‘values of residuals’ mean, as to whether the value of the stable results are any value as to whether they can be relied on, as being ‘GOOD’ results :question: (I wish Dylan were here :wink: )

After all, I have 0.08% stable values over 500 iterations, but (if you ignore the ‘long grass’ on the residual plot) only ONE residual even gets down to 1E-3, which itself is 1000 times the 1E-6 nirvana …

OK, flame away :wink:

EDIT: ORSI convergence analysis has been added as a feature to my yPlusHistogram program available here: Introducing yPlusHistogram program with ORSI as a new metric for y+ mapping evaluation and convergence analysis

2 Likes

Reserved 1

Reserved 2

Nice one again Dale! Did not have time to read through it yesterday but that’s some very cool stuff that you are doing here! :wink: Will share it with my colleagues from engineering, maybe they have a comment on that!

Great work!

Jousef

1 Like

Very cool indeed, nice work and thanks for contributing here :smiley:

2 Likes

Thanks @Jon_Wilde , imagine what would be possible for me to do if I could download and analyze the 2D data for the result plots…

I would not have to use my ‘eye’ and fingernail, I could write a program to determine ORSI% values, of all pertinent results, automatically. :wink:

It took a while, but I now have an ORSI by moving average program that analyzes 2D data from the SimScale results plots (for Forces, Coefficients and Residuals)…

It is an extension of my yPlusHistogram program which can now be put into ‘Monitor Mode’…

In this mode, the program monitors the directory, in which the programs’ .exe file resides, for incoming .csv files…

It you have the program in your Browser ‘downloads directory’ and it is in monitoring mode… it will scan the directory every second and look at any .csv files that get put in that directory… If the .csv file is a 2D plot data from SimScale, it will analyze the data using a moving average ORSI calculation and present the results in a pop up window…

You can look at the results from any completed simulation run but the beauty of this program is so that you can monitor sim runs that are ‘in progress’ and then you can stop the sim runs manually when the desired ORSI values are reached…

Some sim runs take many hours and even days to run, so if you can stop them easily when your desired results stability is reached, you can save many core hours…

Here is the ORSI by moving average popup window for the sim run of Post #1 of this topic:

It has been a month and this launch of my ‘ORSI by moving average program’ seems to have been lost in the fray.

I know a few of you out there have tried it and perhaps one of you could provide a brief critical review of its possible usefulness to someone other than me.

I have had the pleasure of using the ORSI program created by Dale. It is an excellent tool to verify stability in simulated results though its use of a moving average. By analyzing the results from this program you can extract two very important things.

  • 1st - After the first simulation is run, the average value that is being oscillated upon is found at a specific iteration. This means that using ORSI after a simulation run with a write interval of 2000, the program will reveal that 1% stability was achieved at iteration 1832 for example. This allows you to set a much shorter write interval, at iteration 1832, for succeeding runs and saves many core hours in the process.

  • 2nd - Using this specific write interval we can then match, as closely as possible, the visual results obtained in the solutions field, with the numerical average of results. For example, let’s say the average of the numerical results is 9.87e-5. This value was obtained at iteration 1832 yet the write interval for the visual results that will be shown in the solutions field is at the preset value of 2000. This means that the visual results will not exactly match the numerical average of the results, leading to possible false conclusions based on visual data. Using the ORSI program we can set the exact write interval need, and be confident that both our data sets match and are verified as stable. Very Useful!!!

2 Likes