Aerosoft Aerosoft Team [Inactive Account] 51558 Posted April 21, 2015 Aerosoft Share Posted April 21, 2015 Yesterday I got a mail from an upset Airbus customer. He tried something and ran into a rather big issue that affected only one model on one simulator. He could not understand why our testers missed such a massive issue, wanted his money back etc. Telling him that FSX is not without bugs and it runs on Windows that also is patched weekly did not matter a lot. But it made me think that it is perhaps a good idea to explain why bugs are not found. It's all about money. Say we change an important parameter in the FADEC (engine management systems). It could affect every stage of the flight but let’s look at just one part, a late missed approach where the systems need to go into a high power mode. Now testing that means setting up an approach and doing the procedure twice (the second time directly after the first without resetting the flight). That takes about 12 minutes. But we got to do that for FSX SP2, FSX:SE and Prepar3D. And for all four models. And for both engine types. And with a low weight and a max landing weight. So already we are up to 3x4x2x2= 48 test and close to 10 solid hours of excruciatingly boring tests. As a shortcut we ignore the same test with different flaps settings and density altitudes, all things that can seriously affect these procedures. And that is just to test one change and only on one procedure. Fact is that testing is virtually impossible on aircraft this complex. The time it takes to do a complete test would exceed the time to develop it. For every iteration of the software you are looking at days and days of testing. And at the same time the newer iterations are being done. You never catch up. So our tests basically consists of a huge amount of flights being done every day by friends and colleagues. Most of the issues will pop up, but some will be missed. Luck plays a large role. The only way to solve that is to have 5 full time testers for the Airbus projects and that just about doubles our costs. And that means there will be no Airbus for sale. Link to comment Share on other sites More sharing options...
barrel_owl 1 Posted April 21, 2015 Share Posted April 21, 2015 Makes full sense. On the other side, passionate simmers may offer their testing for free. Most of them for pure pleasure only. Not everything we do in life is for money (thanks God). One possible solution for the future could be extending your base of beta testers to a larger number of customers, possibly with different simulators and different specs. I don't know if this is feasible, but it could be an idea to be considered. Regards Link to comment Share on other sites More sharing options...
Deputy Sheriffs Tom A320 4915 Posted April 21, 2015 Deputy Sheriffs Share Posted April 21, 2015 Software quality assurance is a complete topic on it's own. There is no way of doing 100% tests. It's simply impossible as Mathijs already explained. There are industries who actually come quite close to such numbers (e.g. space and aeronautics). But that comes with huge costs. The smaller the market is, the less software can be tested. A good number is something between 30-50%. Not more! What are companies doing to ensure the quality of their products anyway? It's a question of how the software is developed, how many people are involved, how early in the development process testers get involved, how easy it is to automate tests, how effectively a company organizes beta test phases, ... But still, as Mathijs said, there are good chances to simply miss bugs/defects. And that's what the customer support is there for and hotfixes. Companies do not provide so many hotfixes after a release of a new version because they were lazy before, but because of the above given reasons. Hotfixes are there to fix a certain problem as quickly as possible. As time is a question here, they do not see much quality assurance. After a certain amount of time such hotfixes are usually gathered together into new versions, so called service packs. Such service packs of course see a lot of quality assurance, but then again, in terms of numbers, 30-50%... I am in software development for more than 20 years now and from what I see Aerosoft is doing an excellent job! They found a good balance between the expenses for development and quality assurance and the market revenue. All angry customers should always keep in mind that they are sitting in front a highly complex software, nowadays with tons of add-ons included, within a niche market, not a mass market. Link to comment Share on other sites More sharing options...
Deputy Sheriffs Tom A320 4915 Posted April 21, 2015 Deputy Sheriffs Share Posted April 21, 2015 You could always get testers to fly in different regions. One flies higher airports, the other flies under high turbulence, one flies with clear skies, some planes empty, some loaded, some fully loaded... You can add any number of tester, but that doesn't necessarily add to the quality of the product or increase the completeness of testing. Just because someone likes to fly and would like to offer his help doesn't make him a good tester. The forum here is the best example. Lets say someone reports a problem, five other say "me too", but none provides the information the developers need to reproduce the problem. Even if they have been asked for several times in the topic. Testing is something else than "just flying" and see if everything is working fine. If things are not working fine a tester needs a good background on the stuff that is needed to get the problem fixed. And to provide all needed information can actually really be time consuming (creating tons of screenshots, logs, detailed explanations, ...) And all of a sudden it is real work and not a hobby anymore. And at that point many people draw back form their previous offer to help. Link to comment Share on other sites More sharing options...
runjen310 14 Posted April 21, 2015 Share Posted April 21, 2015 It is understandable that you are not able to test every situation so I know that new releases means a period of users discovering a lot of issues followed by hotfixes and hopefully at the end a stable release. If I could not accept errors I would simply wait and monitor the forum until it seems like every issue is solved. However I dont like to wait so I jump in and get burned So, yes we understand your situation but you should also consider our feedback as valuable in your development towards a stable product even if we sometimes get really frustrated Link to comment Share on other sites More sharing options...
Aerosoft Aerosoft Team [Inactive Account] 51558 Posted April 21, 2015 Author Aerosoft Share Posted April 21, 2015 It is understandable that you are not able to test every situation so I know that new releases means a period of users discovering a lot of issues followed by hotfixes and hopefully at the end a stable release. If I could not accept errors I would simply wait and monitor the forum until it seems like every issue is solved. However I dont like to wait so I jump in and get burned So, yes we understand your situation but you should also consider our feedback as valuable in your development towards a stable product even if we sometimes get really frustrated Ohh but we most certainly do. You should hear the discussions we have internally about that! Link to comment Share on other sites More sharing options...
Geddy_2112 99 Posted April 21, 2015 Share Posted April 21, 2015 Not to mention if you have 1,000 simmers, you also have 1,000 different system configurations and you have to realize that no two are alike, EVER! So even though you could have 999 people who are absolutely thrilled at there product, it's always the one person who claims the product is %^$&. There's always one who thinks he's entitled, and to make it plain as day, the majority of the current generation has no time to trouble shoot or even read the manual. If it's not perfect out of the box it's garbage. Sure I've had some hiccups but I also spend the time to research it on the almost infinite amount of resources available. I'm absolutely thrilled with this product and can't wait for the CRJ and A330's to fill my hangar. Great Job Aerosoft!!! Link to comment Share on other sites More sharing options...
ShrekFSX 6 Posted April 21, 2015 Share Posted April 21, 2015 Mathijs Willing to beta test any Aerosoft Product on the FSX and FSX SE Free of charge Link to comment Share on other sites More sharing options...
barrel_owl 1 Posted April 21, 2015 Share Posted April 21, 2015 You can add any number of tester, but that doesn't necessarily add to the quality of the product or increase the completeness of testing. Just because someone likes to fly and would like to offer his help doesn't make him a good tester. The forum here is the best example. Lets say someone reports a problem, five other say "me too", but none provides the information the developers need to reproduce the problem. Even if they have been asked for several times in the topic. Testing is something else than "just flying" and see if everything is working fine. If things are not working fine a tester needs a good background on the stuff that is needed to get the problem fixed. And to provide all needed information can actually really be time consuming (creating tons of screenshots, logs, detailed explanations, ...) And all of a sudden it is real work and not a hobby anymore. And at that point many people draw back form their previous offer to help. Good point, but... if you do that anyway after the release, collecting hundreds of bug reports and working hastily to cool down angry payers who can't wait to use their baby, why not also before? And for free? As said, I don't know if this is feasible on Aerosoft's side, but something suggests me it could help pinpoint potential bugs or issues timely and reduce the frustration after the release for certain customers. Link to comment Share on other sites More sharing options...
acra24 83 Posted April 21, 2015 Share Posted April 21, 2015 As a former tester for in-car mobility software under the old ITIL regime I can completely understand why bugs do get through. They always do, even in very large organisations with many testers. Some bugs seem to lie hidden even within the most rigorous test cycle. It's infuriating but simply cannot be avoided. Reassuringly the worst bugs, i.e. those that can cause the most damage or can lead to serious failures are caught early and it's normally the little ones that sneak through, sometimes many times. The more a company invests in testing the more expensive their products become. Do we want to pay more for the bus? Andrew Link to comment Share on other sites More sharing options...
Deputy Sheriffs jrotaetxe 242 Posted April 21, 2015 Deputy Sheriffs Share Posted April 21, 2015 Cannot leave this thread without giving my 5 cents... Testing "things" has been a substantial part of my non hobby life. The "real" life. Those "things" I tested (and still diagnose) are quite important to maintain our every day life without issues, and, believe me, those things can cause billion euros loses in case of malfunction, being their mean cost some thousand(s). Are very sophisticated and flexible to cover almost every case. So we test to the extent we can but there is a factor we cannot replicate. The customer All our customers are (suposedly) highly trained engineers, with experience dealing with "things" and with a broad knowledge about how, when and where are those to be appied. But sometimes there are chains of events no one coud expect. Or "ways" of using them that are completelly imaginative and out of specs. So, being humans, as we are, we should have a little more care with our exigences to other humans about perfection. Regards Link to comment Share on other sites More sharing options...
Deputy Sheriffs mopperle 4162 Posted April 21, 2015 Deputy Sheriffs Share Posted April 21, 2015 Not to mention if you have 1,000 simmers, you also have 1,000 different system configurations and you have to realize that no two are alike, EVER! So even though you could have 999 people who are absolutely thrilled at there product, it's always the one person who claims the product is %^$&. ....... Great Job Aerosoft!!! A very good point. Just to give an example: I spent about about an hour trying to recreate this issue reported here: http://forum.aerosoft.com/index.php?/topic/94159-some-boxes-in-the-cockpit/ I tried all aircraft variants, nearly all liveries, tons of different views and zoom levels, FSX in windowed and fullscreen etc, but it was impossible for me to see this issue. Fortunately our developer got an idea what might cause this and maybe he will have a solution. Link to comment Share on other sites More sharing options...
tombeckett2285 10 Posted April 21, 2015 Share Posted April 21, 2015 The trouble I find is that the level of entitlement that many (mainly younger people may I add) display nowadays is both breathtaking and saddening in equal measure. So many expect something for a little, to deliver way beyond a price point. Or complain bitterly about something that they have absolutely no idea what it took to produce. My favourite however, are those that say 'why are there no failures' or 'this is a bug, it shouldn't do it' and the 'Bus behaves in exactly the manner of the real thing! Unequal engine parameters, incorrectly drawn ND routes, autoflight systems reacting slightly differently to the manual or in a different way that they have before, turbulence or difficult weather affecting aircraft performance and flying characteristics. I could go on and on, but you get my point. I haven't purchased this upgrade from my Extended yet due to personal circumstances, but as soon as I am back with FS in a meaningful way it will be top of my shopping list. AS support always has been first class, and I can't see that changing anytime soon. Cheers fellas. Link to comment Share on other sites More sharing options...
alainneedle1 32 Posted April 21, 2015 Share Posted April 21, 2015 Is it about how much it will cost Aerosoft to have a plane bugs free or is it about how much they can sell it for bugs free.... If I'm offered a ticket from a airline company with a warning or a disclaimer on my ticket that read as followed...."The airplane you are about to board could not be 100% tested for malfunctions that may or may not occur during taxing or in flight because of budget restrain, but if you are welling to pay a lot more for another ticket on another airplane you will be boarding a fully tested airplane"....which ticket do you think I will buy?.....and of course there will be peoples who will go for the cheapest ticket and take a chance.... So a warning or a disclaimer on the retail box or the download site for the plane in question may be the appropriate measure to solve future problems with angry customers, knowing what one buy may dissuade one from protesting. Alain Link to comment Share on other sites More sharing options...
Cheyenne Chief Pilot 829 Posted April 21, 2015 Share Posted April 21, 2015 Alain, planes have been certified despite the fact they had not performed well during flight tests ... ;-) Link to comment Share on other sites More sharing options...
alainneedle1 32 Posted April 21, 2015 Share Posted April 21, 2015 olli, of course my example about real plane can't be apply in real life, can you imagine reading a warning like that on you ticket...LOL If your up for a good reading... http://www.ainonline.com/aviation-news/aviation-international-news/2006-12-18/aircraft-certification-process Alain Link to comment Share on other sites More sharing options...
John Glanville 821 Posted April 22, 2015 Share Posted April 22, 2015 I am sure that certain individuals spend most ( if not all ) of their time actively searching for " bugs " . Never chosen as a Beta tester possibly and have no idea what being a Beta tester entails ( and no , I was not a tester either ) . Having an Operating Manual and actually understanding what it all means are two very different things . If that wasn't the case we would all be Airbus pilots wouldn't we ? If the Airbus ( whichever version ) was as bad as some try to make out , why is it that it DOES work for what must be the " silent " majority . What these " customers " tend to forget is that even when they do complain the mods and developers are here ( nearly 24 / 7 ) to help but I hardly ever see anyone take the time to say " Thank you " for their assistence . It has been the same since before the release of the AirbusX in 2010 , so this has been ongoing for nearly 5 years and Aerosoft still continue to develop this product without the support of ALL of their customers . When it is taking to long to release , people complain , if it released too early , people complain , so they are damned if they do and damned if they don't . Here's a quote from a post I made ( in a now closed topic ) : ... People should all take a step back , take a deep breath , count to one hundred and be thankful that Aerosoft have continued to " develop " the A320/321 to what it has become when you compare this version to the original AirbusX released in 2010 .Maybe it should ( also ) be said that if Aerosoft released a 14ct gold DVD with every single repaint made for the A318 / 319 / 320 / 321 pre installed with monthly ( FREE ) updates for NavDataPro for life and every possible flight plan included , someone would complain that the DVD wasn't 24ct . ... The text in red was added for clarity and not in the original post . John Link to comment Share on other sites More sharing options...
DetCord 1539 Posted April 22, 2015 Share Posted April 22, 2015 John's and Mathijs' posts are right on the money. Bugs slip though, it happens, and as someone that does independent/contracted game development, I can easily attest to that aspect. I understand that people can be irked or even pissed off by certain quirks or bugs. What I don't understand is why some choose to voice their opinions in a negative manner, especially when it is downright disrespectful. That pisses me off. Some of the members here, not all but some, need a reality check with regards to the massive amount of hours and capital that goes into developing addon's for this small, niche genre. To the credit of Aerosoft, they don't abandon projects. Some of the people are around here need to grow up and loose the child-like sense of self entitlement. Link to comment Share on other sites More sharing options...
Aerosoft Aerosoft Team [Inactive Account] 51558 Posted April 22, 2015 Author Aerosoft Share Posted April 22, 2015 Some people have commented that we should just add more testers (and some are upset we did not accept them as beta testers for some reason). Problem is that finding testers that are useful is incredibly difficult. You try finding somebody willing to spend a whole afternoon cruising at different altitudes with different weights and writing down a detailed report. Then do the same thing the next day after we done some tweaks. Believe me, we do try to find them but it's simply very very hard. In fact, 80% of testers either stop reporting or are not very useful. Every new tester needs additional time to manage and handle (and takes away time for development). A testers that is not doing a good job can slow down several developers. Adding more testers is simply not the answer to this problem. Now I know many people want to test because they are experts on the Airbus for example. But we got enough experts, knowledge is not what's needed. It's resources. As I said it all comes back to money. An ideal tester is patient, very good at communication, has a basic knowledge of development, has ample experience with software testing, does know how to use SVN file versioning software, has multiple systems and possibly needs to have some knowledge of the aircraft we simulate. Those are hard to find. We find our testers by asking for them and then checking what they have done on the forums. It's basically the only way we can get an idea on how a person is. Is he helpful to others, did he make solid comments, not an ass etc. We often also ask people who have a high standing on our forum to help with tests. Link to comment Share on other sites More sharing options...
Aerosoft Aerosoft Team [Inactive Account] 51558 Posted April 22, 2015 Author Aerosoft Share Posted April 22, 2015 Is it about how much it will cost Aerosoft to have a plane bugs free or is it about how much they can sell it for bugs free.... If I'm offered a ticket from a airline company with a warning or a disclaimer on my ticket that read as followed...."The airplane you are about to board could not be 100% tested for malfunctions that may or may not occur during taxing or in flight because of budget restrain, but if you are welling to pay a lot more for another ticket on another airplane you will be boarding a fully tested airplane"....which ticket do you think I will buy?.....and of course there will be peoples who will go for the cheapest ticket and take a chance.... So a warning or a disclaimer on the retail box or the download site for the plane in question may be the appropriate measure to solve future problems with angry customers, knowing what one buy may dissuade one from protesting. Alain Being involved with Airbus software and having a lot of discussions with Airbus pilots I can tell you that the software that runs an Airbus is far from bug free, lol..... Link to comment Share on other sites More sharing options...
El Stevo 2 Posted April 22, 2015 Share Posted April 22, 2015 Matthijs, have you ever given crowdsourcing a thought? Link to comment Share on other sites More sharing options...
Deputy Sheriffs Tom A320 4915 Posted April 22, 2015 Deputy Sheriffs Share Posted April 22, 2015 Crowdsourcing is just another word for beta testing. So everything that has been explained above applies here as well. Link to comment Share on other sites More sharing options...
Aerosoft Aerosoft Team [Inactive Account] 51558 Posted April 22, 2015 Author Aerosoft Share Posted April 22, 2015 Matthijs, have you ever given crowdsourcing a thought? You mean having the aircraft made by the community? I know some Operating Systems are made like that so the complexity would not be an issue, but it's not our style. Link to comment Share on other sites More sharing options...
KAPTEJNLN 237 Posted April 22, 2015 Share Posted April 22, 2015 well I can only agree it´s can be hard to track down Bugs, the end user might find and it´s also for airports but aircrafts is often more complex to test in general by some aspects. Because at airports we do not have a manual to follow and such, it´s more have attention to details and whats wrong, and no that´s not fun either as we do not fly that much if thats what you might think. we have a close attention to all things by that airport and installers and such. Then we have two of the biggest problems out there that gives developers headaches, and it´s combinations of add-ons the end user might have. I know it might be the airports that has biggest problems by that.. Then we have the cross platforms, that can be a big problem for all projects. and I have seen that some times in test and it´s a pain you have to do stuff different by several platforms.. so try combining all those combinations platform related stuff, add-on mix in combinations.. The bigest problem by airports is, that when some comes to the forums and say how could we testers miss a related issue in combination to another add-on. Let me say that clear we can only test by the add-ons we purchased or got along the way. We are not given all projects out there in order to test right.. so that´s a problem by it self, on the other hand I am nearly sure, you can´t build a project that would fit all terrain projects out there or what els it might be. I would how ever to give an example of how much time just 1 problem can take. I have been testing airports and just by that, I had a bug that costed me around 60-80 flights of the same route or close to the same, in order to se if I could re-create what happend once. Just to find a clear way to give the rest of the team / developers a way to restore the given situation. already here it had taken several attempts for 1 thing it could be more ofcause.. but just sayeing it takes a lot of time to test.. there are the easy ones ofcause as well, but the complex ones can take quite some time.. as the 60-80 flights was a flyeing time of 30-45 min, have tryeid flights for hours as well.. Hope that explanes a bit of how it´s for a Beta tester. we can all agree bugs are not funny but can be costly to find.. and even if we test from now and until christmas the end user would find a bug.. Best regards LN Link to comment Share on other sites More sharing options...
alainneedle1 32 Posted April 22, 2015 Share Posted April 22, 2015 Being involved with Airbus software and having a lot of discussions with Airbus pilots I can tell you that the software that runs an Airbus is far from bug free, lol..... Of course their software is not perfect, perfect software with perfect plane and pilot would result in no crash what so ever (including perfect passengers) .....ain't the case in this world. My suggestion was only to prevent a buyer to comeback and ask for a refund because of a bug(s) in the plane they purchased from your company, with a disclaimer explaining that the software (plane) is NOT bug(s) free and you as a company are welling to fix it as soon as it is revealed to you (if possible and also written in the disclaimer) one can't ask for a refund knowing what they bought. I clearly understand what is involved in testing and producing a plane for a simulator like FSX, and one can't hope for a perfect and fully functional software (plane) bug(s) free for a decent price, selling 50,000 of them at $200.00 a pop would be another story, ain't the case. Anyway I would give the disclaimer a thought if I was in your shoes. Alain Link to comment Share on other sites More sharing options...
![Aerosoft Team [Inactive Account]](https://forum.aerosoft.com/uploads/monthly_2023_09/Aerosoft_Social_Media_Profilbild_Neu23.thumb.jpg.03721825e6a434ed4af0c8b2b73a5898.jpg)
Recommended Posts
Archived
This topic is now archived and is closed to further replies.