Flight Testing in CoD

[Viper2000]

Looking at the threads in this forum, it appears to me that a concerted and organised flight test effort will be required to work out exactly what is going on.

It isn't just a matter of posting (for example) a figure showing TAS vs altitude for a Spitfire I and then comparing it with the performance attained during a flight test in the sim.

The fact is that at the moment, we don't know what atmosphere model is used in the sim, we don't know what airframe and engine mod state is being simulated, we don't know whether the aircraft instruments read IAS, CAS or EAS, and we therefore don't know what corrections to apply. Therefore, we could have:

Perfect simulation of the "wrong" aircraft
Imperfect simulation of the "right" aircraft
Non-standard ambient conditions
Some combination of the above, across all components affecting kinematic performance


Because this simulation is considerably more complex than the original IL2, it follows that we're going to need to do an awful lot of work in order to properly understand what is going on so that we can make sensible comments to 1c. For example, the very fact that altimeters now have variable pressure settings implies that barometric pressure is a variable, and may well be a function of both geographical location and time.

I am therefore looking for volunteers to form a team to help me conduct some serious flight testing work.

Effort may usefully be divided into several strands across two departments:

History department
This half of the effort comprises the collection of historical data and its conversion into SI for comparison with data collected from the sim. Conversion work is particularly important because different flight testing organisations used different standards, and therefore it is not necessarily possible to immediately compare their data.

Data reduction standards (historical)
Atmosphere model
Constants & conversion factors
Other corrections
Aircraft performance data collection
Airframe drag estimation
Airframe lift estimation
Handling information
Airframe limits
Transonic behaviour
Failure modes

Engine performance data collection
Engine brake power map
Engine brake SFC map
Exhaust thrust map
Propeller map
Reliability data & failure modes
Conversion of the above to SI under a single set of data reduction assumptions for subsequent comparison with the simulation.


Simulation Department
This half of the effort comprises collecting data from the sim for comparison with the converted historical data collected by the History Department.

Atmosphere/weather testing
Temperature vs Altitude
Pressure vs Altitude
Density vs Altitude
Acceleration due to gravity
Aircraft & engine mod state derivation (in conjunction with history department)
Flight testing
Level speed
Rate of Climb
Turn performance
Cruise performance
Dive performance (including transonic behaviour)
Handling
Exceedances & failure modes
Data reduction & correction


It would obviously be extremely helpful to get authoritative information from 1c about modelling assumptions; as may be seen from the above, a substantial proportion of the effort envisaged at present is associated with deriving those assumptions rather than with testing the aeroplanes themselves.

N.B. The intention of this thread is to assemble a test team and discuss flight test methods & standards, not to get into a flamewar about the performance of aircraft A vs aircraft B.

[Stealth_Eagle]

I'll join the team if needed but I am in North America so I have to wait some.

[Sven]

Big thumbs up for this plan! :)

[Viper2000]

I'll join the team if needed but I am in North America so I have to wait some.

Given the scale of the effort likely to be required it's pretty unlikely that you'll miss the party! :grin:

[Stealth_Eagle]

Given the scale of the effort likely to be required it's pretty unlikely that you'll miss the party! :grin:

I personally want to test bombers and the G. 50. In IL-2 online play, I can dominate late war fighters with it so I'll put that in my comparison. However I don't have TrackIR since it is too expensive.

[Viper2000]

Perhaps it would be a good idea for those people who are interested in this work but cannot yet access the sim to concentrate on collecting historical data?

In order to avoid possible accusations of bias, I suggest that we work exclusively from verifiable historical documents (which may be translated to *.pdf or *.jpeg files for transmission, provided that it's obvious that they are derived from original sources).

At this stage, I would like to point out that a lot of the historical work is going to come down to interpretation, especially as regards handling. Unfortunately, interpretation is emotive.

My intention is to try to keep this thread objective. Past experience strongly suggest to me that this will be almost impossible if we discuss individual aircraft types, because there will always be partisans both "for" and "against" any given type's performance characteristics*.

Therefore, I suggest that we split our work as follows:

This thread should be the "general purpose" flight testing thread
Individual aircraft types should each have 3 threads:
An historical thread, detailing reference material
A flight test thread detailing what the simulation does
A comparison thread detailing what (if anything) we think needs to be changed


This arrangement should act as a sort of fire-break, such that we only get flamewars in a relatively small percentage of our threads.

I hope that in the fullness of time we might get our own sub-forum, possibly with limited posting rights in order to reduce the risk of trolling; naturally any such hope does not constitute an expectation.

*I've never really understood this; if you think that "your" airforce's aeroplanes were "better" then you're naturally implying that its pilots were "worse" and vice versa, since the historical record is essentially fixed...

[Kianoni]

...

The fact is that at the moment, we don't know what atmosphere model is used in the sim, we don't know what airframe and engine mod state is being simulated, we don't know whether the aircraft instruments read IAS, CAS or EAS, and we therefore don't know what corrections to apply.
...
well, this is one of the features that should be looked into in the future. would be nice to get viable information about the planes/weather.

[JG14_Jagr]

You really need to determine what resources you will have to actually measure the data.. using the GUI is not a good idea. Also, certain maps often create issues.. standardize the testng on a particular map..maybe one of the simple online maps...

[Viper2000]

I think that the first thing we need to do is to try to find standard day conditions.

There are quite a lot of possibilities available.

One option would be to use a relatively modern standard like the ISO standard atmosphere or the 1976 US standard atmosphere. This would facilitate compatibility with modern data. However, the ISO standard atmosphere is not directly available for free.

Alternatively, we could attempt to use older NACA standards; doing so would lend an historical flavour to our work, but might complicate comparison with more modern sources. A few possible standards may be found by following the links below. This list is not exhaustive.

1926 NACA standard (http://naca.central.cranfield.ac.uk/reports/1926/naca-report-218.pdf)
1930 simplification (http://naca.central.cranfield.ac.uk/reports/1932/naca-report-376.pdf)
1952 NACA/ICAO standard (http://naca.central.cranfield.ac.uk/reports/1955/naca-report-1235.pdf)
1976 US Standard Atmosphere (http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19770009539_1977009539.pdf)

I suggest that we use the 1976 standard for comparison purposes because this allows us to avoid some otherwise potentially nasty conversions (eg changes to Temperature scales over time (http://www.bipm.org/en/CGPM/db/10/3/)...).

Obviously the next thing that we need to do is to find some way of measuring the atmosphere properties on a variety of available maps so that we can pick the best one for testing.

Does anybody have any suggestions as to how we might best go about this?

[TUCKIE_JG52]

Viper, that's a really nice effort, seriously focused... some time ago I was discussing in other forums about real charts of the planes, that were not corrected to the same atmosfere models...

I fly a Cessna 152 and from my early stages of training, I can see the big influence that ambient contitions have in performances like climb rate.

As ambient conditions I mean specially pressure and temperature. For example, before each takeoff I check the forecast for exact data and I get the exact calculation for the Pressure Altitude. If the pressure altitude of my airfield is below the real altitude of the field, that day the Cessna climbs a lot. It's the most common case in winter, with temperatures below ISA (15º) and sometimes pressures over ISA (1013Mb).

The inverse case is a higher pressure altitude when ambient is very hot and low pressures are registered... to the point that if pressure altitude is a lot higher than real altitude of the airfield, it can be a factor that can lead to an accident.

I see that your initial point of view takes this into account, so I'll follow your results with very interest.

I would collaborate with you if I were not so busy with video editing...

[TheGrunch]

We have ambient temperature gauges in the Ju 88 and He 111 if this helps. Likewise, if we can work out exactly the altitude at an airfield (FMB maybe?) and compare it to the altitude given by the aircraft's gauges we may be able to work out pressures.

[TheGrunch]

The biggest problem really at the moment is the lack of a DeviceLink style interface yet and the inaccuracy of the rollover text on cockpit gauges (seems to be 1 decimal place, not very useful when altitudes are measure in km for example). Although I haven't even looked at the no-cockpit view so that might be better.

[TUCKIE_JG52]

Any data on this subject?

I've been focusing in Bf-109E-3 tests.

By landing beside a beach and setting the altimeter to 0, the indicated pressure in all maps is 993 mbar.

Ambient temperature is about 17ºC, but I must confirm with Ju-88 and He-111 instruments.



I've also looking for which are the german standards, named "Normaltag" ("Normal Day"):
I'm not sure about the data they considerred standard, but it looks like 1018mbar and 16ºC.

In this test there's a correction factor developed, but I can't undestarnd what every value really means:
http://www.kurfurst.org/Performance_tests/109E_V15a/Geschw_109V15a.html#blatt5

If that correction factor is correct, it would mean that with the current pressure and temperature settings for all maps in CoD, we'll never be able to reach the german values of the performance charts like climbing. Even more if it's considered that, as commented in this post, Bf-109 has an overloading problem:
http://forum.1cpublishing.eu/showthread.php?t=20160

Overloaded E-3 + not so favorable meteorological contitions = our graphs will look always under the german values.

We need to apply correction factors to really evaluate if CoD E-3's performances are correct or not.

[Viper2000]

Overloaded E-3 + not so favorable meteorological contitions = our graphs will look always under the german values.

We need to apply correction factors to really evaluate if CoD E-3's performances are correct or not.

Exactly.

I think that we should probably correct all of our performance references to ISO standard atmosphere conditions to produce an overall aircraft performance database, and then convert performance data from that database to whatever the conditions on our chosen test map are.

BTW, formation flying is the most obvious way to overcome the 1 d.p. limitations of the cockpit instruments, at least to some degree...

[Sternjaeger II]

definitely a good effort, but let me just say one thing..

we need to know exactly what parametres are needed in order to provide usable data and not waste any time. It would be worth to try and talk to Luthier about this, so maybe we could be directed in the right way..

Viper has a factual approach to things that is quite the right way to go, but before you jump in this head first try and seek for some support from the guys at Maddox Games.

[Vengeanze]

I don't understand anything you guys are saying here but I make darn good coffee!

[SG1_Lud]

Viper, if you are leading this project I'm in.

Tuckie and me are trying to do the same in a spanish forum, so we can work together.

Congrats for the idea.

Cheyennepilot ( aka Lud)

[kimosabi]

There's no use going by FM testing as detailed as this one yet. The FM is just as immature as the rest of the game is, so in a year a test like this one should be better suited.

[Peril]

Does the engine have the capability to output debug data in digital?

Detailed info like rate of turn, G's, RoC, hp, IAS, TAS, boost, rpm etc. all in digital?? This is detailed data invaluable to defining the errors. I find the hp and boost in digital very useful in defining RAM effects at alts etc. Analogue gauges are just to vague to be really definitive, we don't even know if they are calibrated correctly.

[TUCKIE_JG52]

Exactly.

I think that we should probably correct all of our performance references to ISO standard atmosphere conditions to produce an overall aircraft performance database, and then convert performance data from that database to whatever the conditions on our chosen test map are.

BTW, formation flying is the most obvious way to overcome the 1 d.p. limitations of the cockpit instruments, at least to some degree...

That was my idea too... :)



Yesterday I did pressure and temperature measurements. And got some conclusions that must be cross-checked.

Pressure measuring method:
1. Place a Ju-88 in a beach (Sea Level).
2. Set altimeter to 0.
3. Read pressure in milibars directly from the Kollsman window, in the upper side of the german altimeter.

Temperature measuring method:
1. Place a Ju-88 where you want to test the plane.
2. Read outisde temperature indicator :)

Facts found:

A) I get different values of pressure and temperature in the same place every time I reload the mission. Sometimes they'll repeat, but they tend to change more when a different plane is loaded in the same place. ---> Temperature and pressure are not always the same, there's some grade of randomisation. (This must be confirmed).

B) Ambient temperature is different from water and oil temperature before starting engine.


Clues to test:
-There's the possibility to switch from the cockpit of one plane to another, at least altimeters can be cross cheched in that way; same mission load, planes placed together on a beach, check if all altimeters have the same pressure reading in Kollsman when altimeter set to 0. --> If same readings oK, reload the mission to test randomisation of parameters.

[TUCKIE_JG52]

There's no use going by FM testing as detailed as this one yet. The FM is just as immature as the rest of the game is, so in a year a test like this one should be better suited.

Of course, but there's also no reason to stay with nothing to do, at least we must know in which atmosfere are we flying... to avoid people whining about bugs that aren't bugs... :)

Our learning curve is slower than developers speed of work ;)

[kimosabi]

My learning curve tells me that the mixture issue need to be taken care of, and that the FM needs to provide us with more feedback and certain elements of aircrafts behaviour before a FM test is justified. Otherwise mapping an aircrafts performance envelope will not be correct once the FMs have been changed. They know about those issues and I'm sure they are working on it. I'll say it like this: There's no use taking a Fiat Uno to Nürburgring to test lap times, if the race is classed as a FIA GT. But by all means, if you people need something to do just for the sake of doing it......

[ivo]

Bye see here:
http://www.fourthfightergroup.com/eagles/spit2.html

[Peril]

That's useful data on early Spits, thanks.

[Osprey]

I don't know if this ever started or how serious Viper is but I am willing to help with the testing but ONLY if there is some assurance from the development team that it is being taken seriously.

From what I am seeing wider Europe is undertaking a Public Beta test which is adhoc and incredibly disorganised.

So, if we are given genuine co-operation from Luthier & friends, and that we supply genuine reliable information to them to use, then I'm signed up.

[heloguy]

I suggest that we use the 1976 standard for comparison purposes because this allows us to avoid some otherwise potentially nasty conversions (eg changes to Temperature scales over time (http://www.bipm.org/en/CGPM/db/10/3/)...).

Obviously the next thing that we need to do is to find some way of measuring the atmosphere properties on a variety of available maps so that we can pick the best one for testing.

Does anybody have any suggestions as to how we might best go about this?

I would think that the 1976 standard day wouldn't be the best choice as that's not what testing data from the 30's and 40's was based on. Maybe the earlier two would be the best to use.

As far as ambient conditions, either use the gauges in game and hope they are calibrated correctly, or just e-mail the developers to find out what the stock standard atmospheric conditions are.

If there's a way to adjust atmospheric conditions in the FMB (I would look, but will not have my computer with COD for awhile as I'm in the States), then you could create a mission in the FMB that all testers would be compelled to use that are a part of this project.

Really, the best historical data (if it exists) would be that which has variable test data, such as a curve that represents the difference in performance compared to altitude and temperature on the graph axes in order to adjust for density altitude. If this sim has variable weather as it says, then it will be hard to see if the aircraft performs correctly over a range of temps and pressures if it's only tested on a standard day.

[Viper2000]

I don't know if this ever started or how serious Viper is but I am willing to help with the testing but ONLY if there is some assurance from the development team that it is being taken seriously.

From what I am seeing wider Europe is undertaking a Public Beta test which is adhoc and incredibly disorganised.

So, if we are given genuine co-operation from Luthier & friends, and that we supply genuine reliable information to them to use, then I'm signed up.

I haven't started any work on this yet because, as you point out, we're effectively in a public beta at the moment, and with major code changes taking place so regularly the chances are that a lot of the work we put in would be wasted.

I've also got a PhD to finish.

I would think that the 1976 standard day wouldn't be the best choice as that's not what testing data from the 30's and 40's was based on. Maybe the earlier two would be the best to use.

As far as ambient conditions, either use the gauges in game and hope they are calibrated correctly, or just e-mail the developers to find out what the stock standard atmospheric conditions are.

If there's a way to adjust atmospheric conditions in the FMB (I would look, but will not have my computer with COD for awhile as I'm in the States), then you could create a mission in the FMB that all testers would be compelled to use that are a part of this project.

Really, the best historical data (if it exists) would be that which has variable test data, such as a curve that represents the difference in performance compared to altitude and temperature on the graph axes in order to adjust for density altitude. If this sim has variable weather as it says, then it will be hard to see if the aircraft performs correctly over a range of temps and pressures if it's only tested on a standard day.

Using a 1930s or 1940s atmosphere standard is likely to cause confusion, because all sorts of things were different in those days (eg the definition of the Kelvin).

It's also inherently more likely to produce flame wars because if we pick a NACA atmosphere then people will see American aeroplanes with data which looks like primary source data and potentially German or British aeroplanes with corrected data which disagrees with primary sources. We would then find ourselves having to explain the concept of standard atmospheres and correction factors in the face of vociferous accusations of bias from the large population of trolls that inhabit the forum.

Whatever we do, we're going to end up picking a single standard atmosphere so that we can compare the performance of all the aeroplanes in the sim on the same chart. Apart from anything else, if we don't do this, the chances are the somebody else will do so in a biased way with the intention of forwarding their own agenda, since quite a lot of forum trolls seem more interested in being able to say "my aeroplane is better than yours" than in historical accuracy.

Ideally, I'd use the ISO standard atmosphere, because it's neutral and current. However, I don't think that it's freely available, and that would both interfere with testing and lead to accusations that the process was not transparent.

The 1976 US standard atmosphere is freely available on the internet, and avoids most of the risk of accusations of bias it's post-war*, and it is relatively modern (so we get basically modern SI units, though it uses its own private value of the gas constant, presumably for historical reasons).

*Therefore all of the aeroplanes we test will see correction factors.

[CaptainDoggles]

I've got some of my old university textbooks still, one of them has a standard atmosphere in the appendix; I'll check to see which version it is. The textbook is relatively new so it ought to be a modern ISO atmosphere.

[Skoshi Tiger]

Just done a quick check on the Spitfire II, on the ground at Mansten ( 178 feet) the closest I can get in 1 MB increments is a bout 993mb, at Shoreham (7 feet) on the same map 992 MB.


With the current standard SL pressure at 1013 MB it looks like theres a bit of a Low over the South East of England.

Also, FTIW, from the oil temp it's about 21 Degrees.

Cheers

[CaptainDoggles]

Ideally, I'd use the ISO standard atmosphere, because it's neutral and current.

Just had a look. The appendix in this particular book gives temperature, pressure, density, and speed of sound in SI units and English units. Data are every 2000m up to 70km for SI, and every 5000 feet up to 175000 feet for English units.

Let me know if that's worth anything to you and I'll see what I can do about scanning it.

[Crumpp]

Using a 1930s or 1940s atmosphere standard is likely to cause confusion, because all sorts of things were different in those days

I remember the same conversation a few years ago.

Good Luck. :rolleyes:

[Viper2000]

:)

It's fairly easy to code up the ISO standard atmosphere to 20-30 km in excel. When we get to that stage, we will probably build a test kit of some sort with a various tools for data reduction. Eventually we'll get a load of test data points from different sources and then plot them all onto a single large chart for each set of parameters of interest (e.g. one TAS vs altitude chart, one ROC vs altitude chart etc.).

It would still be useful to have tables from another source for crosschecking, but don't feel under any pressure because:

There's not much point in testing anything until the sim settles down, which I expect will be a couple of patches after the US release.
Obviously until then the sim is in a constant state of flux, so our results would probably be obsolete before we could publish them.
The object of the exercise is to make the sim better. That means that it's only worth producing test results if the devs are likely to have time to read them, which they obviously don't at the moment, and won't until the other game engine related bugs are fixed.
Additionally, a lot of the effort required comes from the fact that at the moment we don't have a lot of information about the assumptions underlying the model. We also don't have testing tools (like the old devicelink autopilot in IL2). In the longer term, when the devs are under less pressure, they might provide us with some helpful information, and indeed somebody might even come up with a new autopilot.
There are therefore very strong arguments for waiting a while before we invest serious effort into this project.
The way to actually attack the problem will be to code up a model and then interrogate it to derive the correct standard values for our exact test points. We don't want to be interpolating from tables if we can possibly avoid it because this leads to error and argument.

We're still probably going to have issues of course, because you'll generally find that standard atmospheres use geopotential altitude (it makes the maths easier; there's a nice discussion in the document which sets out the 1976 US standard atmosphere), whereas I somehow suspect that this sim might just use geometric altitude.

Indicated altitude will almost certainly come out as geopotential, because it's referenced to ambient pressure (if the model doesn't differentiate between geometric and geopotential altitude then the most likely fudge would be to just use the geometric altitude as input to a standard, geopotential atmosphere model, which is a small source of error), but "wonder woman" altitude will probably be geometric (WW alt was effectively radar alt in IL2, and thus geometric, but indicated altitude was true altitude above MSL because the altimeter pressure setting was fixed at QNH for the map - but this was probably also geometric because IL2's model was quite simple and it doesn't make much difference at low altitude anyway).

Converting between geopotential and geometric altitude isn't a problem, but explaining the differences to certain sections of the community could be a pain (it's only a tiny difference at the sort of altitudes we're going to be working at, but if a job's worth doing then it's worth doing properly... and also, if people see a difference they're likely to make accusations of error and/or bias, because that's how the cookie crumbles - spot the jaded realist...).

[CaptainDoggles]

We don't want to be interpolating from tables if we can possibly avoid it because this leads to error and argument.[/list]

I agree, which is why I held off posting it.