The Hidden Troubles of the F-35 [DefenseNews]

[Corsair1963]

Had a quick look, but not sure if this report has been posted. It contains a wealth of details on availability and operations costs for most aircraft in US service up until FY2018/2019.

PS. Had to zip it due to the 11MB upload limit.

GAO_710794.zip

CBO / GAO Reports aren't worth the paper there printed on............

[spazsinbad]

Lockheed Martin F-35 deficiencies: two fewer in 2020, 871 issues remain
15 Jan 2021 Garrett Reim

"Lockheed Martin managed to reduce the total number of identified problems with its F-35 Lightning II stealth fighter by two in 2020 - though 871 deficiencies remain.

That's according to the Director of Operational Test and Evaluation's (DOT&E's) 2020 annual report to the US Congress, released on 13 January. The DOT&E is the Department of Defense's top weapons tester.

It appears that Lockheed solved more than two deficiencies in 2020; however, it also discovered additional problems, which meant the total number of open issues only fell slightly. The total number of deficiencies solved and discovered in 2020 was not disclosed in the DOT&E report.

"Approximately 100 new deficiency reports were written in 2020 and about as many were resolved and adjudicated," Lockheed says.

The F-35's problems included 10 category 1 deficiencies, three fewer than in 2019. Such problems "may cause death or severe injury; may cause loss or major damage to a weapon system; critically restricts the combat readiness capabilities of the using organisation; or results in a production line stoppage," according to the US Air Force's definition.

The DOT&E did not disclose a list of specific deficiencies. But Lockheed says many of these are categorised as "low priority" or are with the F-35 Joint Program Office (JPO) for resolution.

"There are currently no CAT 1A (risk to life or limb) and 10 open CAT 1B (mission impacts) deficiency reports," says the company. "Nine of these have closure resolution plans, with seven already delivered to the government awaiting action. The other is currently being reviewed by the [Joint Program Office]".

The company claims that "deficiency reports are not always contractual deficiencies".

"In fact, many deficiency reports document opportunities for improvement from pilots and maintainers for consideration above and beyond contractual obligations," says Lockheed. "These are called enhancements and are documented as deficiency reports, because there is no other program or process in place to record this feedback from the test sites."

The DOT&E report notes that issues also continue to emerge with the F-35 modernisation effort called Continuous Capability Development and Delivery (C2D2). It is modelled on a commercial software and hardware development process called agile development. That strategy tries to introduce system updates on a continuous basis, instead of the more traditional "waterfall" approach which dumps many software and hardware changes at once, every several years.

Lockheed was bullish that agile development could solve the F-35's problems, but growing evidence over the last year indicates it is falling far short of expectations.

"The current development process used by the F-35 JPO and Lockheed Martin, that is supposed to provide new capabilities and updates in six-month increments, is not working. It is causing significant delays to planned schedules and results in poor software quality containing deficiencies," says the DOT&E report. "The current C2D2 process has not been able to keep pace with the scheduled additions of new increments of capability."

In fact, the process appears to be making the F-35's problems worse.

"Software changes, intended to introduce new capabilities or fix deficiencies, often introduced stability problems and/ or adversely affected other functionality," says the DOT&E. "Due to these inefficiencies, along with a large amount of planned new capabilities, DOT&E considers the programme's current Revision 15 master schedule to be high risk."

There are also concerns that the Operational Data Integrated Network (ODIN), the replacement for the F- 35's troubled Autonomic Logistics Information System (ALIS), is falling behind. ALIS is a system that manages prognostics, maintenance, supply chain, flight operations and training for the F-35, but is so cumbersome and error-ridden that Lockheed is scrapping it and replacing it with the new cloud-based ODIN.

"Although the programme continues data, software and hardware development for ODIN, an overarching test strategy that includes government and contractor laboratory facilities has yet to be provided," says the DOT&E report. "The schedules for ODIN initial operational capability and final operational capability remain high risk."

ODIN was expected to reach full operational capability in December 2022.

Availability for the F-35 fleet showed "modest improvement" in 2019 and early 2020, says the DOT&E report. However, "the average fleet-wide monthly availability rate for only the US aircraft, for the 12 months ending in September 2020, is below the target value of 65%."

Lockheed has often pointed to higher availability rates for F-35s that are deployed, but the Pentagon's weapons testing office says those gains are fleeting.

"Individual deployed units met or exceeded the 80% mission capable and 70% fully mission capable rate goals intermittently, but were not able to meet these goals on a sustained basis," the DOT&E report says."

Source: https://www.flightglobal.com/fixed-wing ... 69.article

[hornetfinn]

Lockheed Martin F-35 deficiencies: two fewer in 2020, 871 issues remain
15 Jan 2021 Garrett Reim

The DOT&E report notes that issues also continue to emerge with the F-35 modernisation effort called Continuous Capability Development and Delivery (C2D2). It is modelled on a commercial software and hardware development process called agile development. That strategy tries to introduce system updates on a continuous basis, instead of the more traditional "waterfall" approach which dumps many software and hardware changes at once, every several years.

Lockheed was bullish that agile development could solve the F-35's problems, but growing evidence over the last year indicates it is falling far short of expectations.

"The current development process used by the F-35 JPO and Lockheed Martin, that is supposed to provide new capabilities and updates in six-month increments, is not working. It is causing significant delays to planned schedules and results in poor software quality containing deficiencies," says the DOT&E report. "The current C2D2 process has not been able to keep pace with the scheduled additions of new increments of capability."

In fact, the process appears to be making the F-35's problems worse.

"Software changes, intended to introduce new capabilities or fix deficiencies, often introduced stability problems and/ or adversely affected other functionality," says the DOT&E. "Due to these inefficiencies, along with a large amount of planned new capabilities, DOT&E considers the programme's current Revision 15 master schedule to be high risk."

Source: https://www.flightglobal.com/fixed-wing ... 69.article

DOT&E will consider master schedule of a major program high risk no matter what. You will never hear them or other similar organisations to do otherwise.

I think agile development process is still better than doing very big changes every few years. There will always be problems with new software and changes always introduce new problems. It's just impossible to do perfect software without testing it in real world operations and then fixing problems found. If they did big upgrades instead of smaller increments, then figuring out and fixing those problems would be a lot more challenging and take more time. There would be even more new problems and it'd take a lot more time to fix them. And some deficiencies will be there always and people just figure way to go around them. I know that even Classic Hornets (and likely all aircraft in existence) still have some deficiencies that will never be fixed because there is really no need.

[steve2267]

Was the F-35 software systems developed from the beginning with agile development? If not, has agile development had a good track record being implemented mid stream on projects started using another software paradigm? That is, perhaps agile development only really shines when implemented from the very beginning of a project?

Another thought is that the regression testing necessary in a software system as large as the JSF could be a real cast iron b*tch. So even though new software development might work in an agile development environment, perhaps the regression testing is not so agile?

[quicksilver]

“Another thought is that the regression testing necessary in a software system as large as the JSF could be a real cast iron b*tch. So even though new software development might work in an agile development environment, perhaps the regression testing is not so agile?“

Great point.

How many lines of code?

[spazsinbad]

OLD AGILE SOFTWARE DEVELOPMENT story in 2 page PDF from Flight International 10-16 July 2018 attached.

[blindpilot]

I'm going to wade in with some thoughts for what they are worth. Many, if not most, may write this off as an old man ranting about how "the horse drawn wagon carried firewood through the muddy fields better than that there 4X4 truck you got ..." So take it as you will.

We ... programmers in the 60's/70's using sliderules ... got a man to the moon. Boeing with ISO 9001 certified approaches and highly qualified programmers can't get the Starliner into orbit, or keep the Max in the air. Just saying....

My experience (with horse drawn wagons) is that smart people make it work. Average people using "ISO 9000 certified" approaches, can't find the door to get out of the room. And take up the oxygen in the room until a smart programmer fixes it all, over night working with the janitor. Plainly that is my experience with some fairly sophisticated software. I was a Microsoft alpha tester who trained MCE's while never bothering to certify myself. I didn't need to. If they needed something they called me .. I didn't call them, nor "interview" nor "take certification tests" or fill in after action paperwork. I let the previous programmers fill in those squares with worthless paperwork that collected dust in a drawer, while operational folks used cheat sheets on card board.

Real world example. NORAD Cheyenne Mountain was upgrading systems, and the prime contractor couldn't get the $billion system to work, with wonderful testing systems, and rooms full of programmers. That was operationally unacceptable, so we brought in an RCA programmer to work with our MITRE lead and in 2 months they built a system from the ground up (including different hardware) that worked. I will admit that they wrote what we called "write only" code. but with follow on documentation it was maintainable. Smart people get things done. No "system" ever has.

Now some might say we should use the "Elon Musk method." I would say that Elon Musk could use any method from Sunday to Saturday, and it would work. And Boeing Starliner/Max programmers could document and use the "Musk Method," and the Starliner would still not get into orbit, while the Dragon would be at the ISS with SpaceX programmers using the Boeing handbook upside down just for fun and a change of pace.

Now maybe these newfangled automobile thingies are different ... but it was true for me with horse drawn wagons ... AND

Boeing can't get the Starliner into orbit, and HR at SpaceX doesn't even care or look at your diploma, in deciding whether to hire you. Doesn't sound much different to me.

Come back to me when Boeing gets the Starliner to actually work, and even then I'll bet hard cash I can find out who the smart guy was that bailed Boeing out... It won't be "a system approach".

Talent is definitely not appreciated these days.

FWIW MHO,
BP

[madrat]

I always found it funny how the NOC guys would call with their problems so one of us lowly techs could tell them the not so obvious solution. Some guys have certs. Some guys fix things.

[johnwill]

Blindpilot +++

[Gums]

Salute!

I echo "John-boy", BP, and next time in Woodland Park, "AMEN"!

Gums sends..

[ricnunes]

...

Talent is definitely not appreciated these days.

FWIW MHO,
BP

[blindpilot]

Salute!

... next time in Woodland Park, "AMEN"!

Gums sends..

You bet, I'll be here .. I got my vaccine shot 1, and shot 2 in three weeks, so maybe we'll get by these COVID lock downs. There is, it appears, at least one advantage to being "over 70."

BP

[skyward]

Talent is definitely not appreciated these days.

That because it take a talented person to spot another.

Lambeau: You're right, Will. I can't do this proof. But you can, and when it comes to that it's only about... it's just a handful of people in the world who can tell the difference between you and me. But I'm one of them.

[spazsinbad]

I'm happy to admit complete ignorance about the wares of softness or whatever it is called regressively pre & post so wot.

Software Testing - Software everywhere [4 page PDF of article attached]
Sep 2018 David Smith

"With delays and cost increases of aircraft test programs increasingly blamed on software problems, what technology and tactics are available to keep software testing under control?...

...1,000 Software faults identified on the F-35 after more than 25 years of development in January 2018...

...F-35 DELAYS CAUSED BY UNREALISTIC GOALS AND C++
Tucker Taft, director of language research at AdaCore, says that the F-35 Joint Strike Fighter has become too complex because every one of the high number of stakeholders in the project is demanding ambitious requirements. “All the amazing technical qualities can be in conflict and they keep changing their minds.”

The F-35 has taken more than two decades to develop and has been plagued by huge time and cost overruns. The lifetime costs stand at an estimated US$1.5tn, partly because of the enormous price tag for software development and testing. As recently as January 2018 the Pentagon was forced to admit that there are still close to 1,000 software faults on the jet, but won’t say precisely what they are.

“The software development keeps getting the blame, but the whole project management can be seen as at fault,” Taft says. “The program has goals that are almost impossible to reach. The lesson is to put a stake in the ground and say we will build it this way and stick to it.”

Over-ambition may bear some of the blame, but it is undeniable that software development has also contributed. Taft believes that writing the software in C++ has also caused the overruns. “When you factor in the cost of debugging later on, it’s worth doing a little more training to use a language that’s less problematic, such as Ada,” he says...."

Source: AEROSPACE TESTING INTERNATIONAL SEPTEMBER 2018

[zhangmdev]

It is hard to believe an aviation project wihout using Ada. They probably use a lot of languages.

Impossible to tell how many lines of code are used in F-35. One guess is about 24 million LOC. Which is a very low number considering how complex it is. The Google Chromium, the web browser, has more lines than that. Building that thing takes hours.

https://www.openhub.net/p/chrome/analys ... es_summary

Some fancy cars these days could be running on more than 100 million LOC. That is how incredibly bloated modern software has become. A lot of people are working on different modules, running on different architectures, each relies on some midware, which is maintained by another bunch of people. Testing and ensuring everying works is going to be increasingly harder, when junk code accumulated over the years and nobody cares or dares to cleanup. Those old days of Apollo and Space Shuttle, when software was developed by a very small number of people for a very specific system, when stuff was done in thousands of lines of code, are never coming back.