What a Performance! – The CAA exam with a 95% failure rate

Following the introduction of a new Question Bank (QB), between January and April 2021 the UK ATPL theoretical knowledge exam ‘Flight Performance and Planning – Performance (A)’ dropped from a previous 88% average pass rate to just 5%, marking a new low for the subject and indicating that all is not well with the current UK exam system for commercial pilot cadets.

Prior to Brexit, the UK CAA had ‘acquired’ ATPL exam database ECQB2020 from the EU Aviation Safety Agency (EASA) and made it its new QB for UK ATPL theoretical knowledge (TK) exams, adopted by a number of schools from January this year. Schools specialising in TK training were quick to point out that many of the questions in the QB were not fit for purpose, being badly constructed, written in poor English and examining areas outside of the established Learning Objectives of the ATPL syllabus. It was estimated that between 5-10% of the QB was unfit for purpose and a further significant percentage covered subject matter not currently taught as part of the ATPL LOs.

During June, a new UK flight training industry association was formed – the Pilot Licensing Advisory Group (PLAG). PLAG is made up of senior representatives from UK Approved Training Organisations (ATOs) and it is currently challenging UK Government on what it says has been a rushed and badly managed exit from the EASA regulatory system.

Concerning ECQB2020, PLAG states that the issues with the database were well known early on and that EASA, acknowledging this, was quick to react and have already put the QB through two revisions. The CAA’s response, meanwhile, is that nothing can be done because they have neither the capacity nor capability to review the database. PLAG advises that the reality is that pass rates will improve in the UK as ATOs become familiar with the questions being asked and candidates use online databases of feedback questions, but clearly neither of these remedies address the fundamental problem of certain exam questions being unfit for purpose.

Of equal concern is a perception within industry that by using these databases the UK is in breach of EU copyright. PLAG advises that EASA’s stated position is that the ownership of the exam databases rests collectively with EASA and its Member States and that they will prosecute breaches of their copyright. The CAA’s position, meanwhile, says PLAG, is that they own some of the historic content of the EASA databases and are therefore entitled to use them in their entirety, even without an agreement in place between the two parties. The UK’s legal position appears precarious, and industry insiders are pointing to a lack of willingness by EASA Member States to challenge the situation, rather than the CAA being in a strong legal position, as being the only reason that the UK hasn’t yet been taken to court.

“This is wrong in law and exposes the UK flight training industry to the possibility that EASA may enforce their rights and prevent the CAA from using their exam question databases. The CAA has no back-up plan. If EASA successfully pursued an injunction it would be a very public failure of post-Brexit Britain that the Government might wish to avoid,” says PLAG.

According to Alex Whittingham, MD of TK specialist company Bristol Groundschool, there is only one ATPL TK expert currently employed at the CAA and so the Authority is clearly unable to produce its own QB yet, or even enact changes to the current one, meaning that cadets will need to rely on schools becoming familiar with the ‘gotchas’ in the current QB and feeding them onto them. As to what happens when/if the International Civil Aviation Organisation (ICAO) makes any substantial changes to the ATPL learning requirements, necessitating a rewrite of the QB, the jury remains out. Indeed, Alex Whittingham confirms that there are already instances of this, whereby schools are having to teach cadets out of date subject matter in order for them to pass out of date exam questions.

There are also wider concerns within industry about how the CAA using its commercial arm, CAA International (CAAi), to deliver pilot e-Exams. CAAi has existed for years, but has only relatively recently become a Social Enterprise. Social enterprises are businesses which still aim to make a profit but then use that profit to enact positive social change, reinvesting or donating it. Well known examples include The Big Issue and The Eden Project. To be a social enterprise, businesses must reinvest the majority of their profits, be autonomous of the state, and be accountable and transparent.

The last time CAAi issued any dividends (which were issued to the CAA) was just before it became a social enterprise, and since then, industry insiders have told FTN that its profits are being passed onto the CAA in the form of loans, leading to queries over its profit reinvestment strategy as well as questions about its transparency. There are also questions being asked about CAAi’s autonomy, due to allegations that it doesn’t actually employ any staff but instead uses fulltime CAA employees to do its work. This symbiotic relationship between the CAA and CAAi has led to members of the flight training industry questioning the propriety of the CAA effectively acting as both regulator and profit-making centre, ultimately leading to the question of whether CAAi can justify its position as a social enterprise.

Further muddying the water is how both the CAA and CAAi are awarding contracts to industry, which in the case of the e-Exams platform appears to have been done without public tender. Questioning the CAA on its contract tender process, FTN was told: In 2008, and subsequently in 2011, we ran an open competitive process for the procurement of an e-examinations system to modernise our licensing work. Regrettably this process was unsuccessful. A revised approach was therefore taken and CAAi, the wholly owned technical assistance arm of the CAA, formed an innovation partnership with a leading global e-assessment provider. 

 “The partnership developed the supplier’s existing systems to meet our needs, while also enabling CAAi to develop and deliver regulatory e-tools for other national aviation authorities, helping to meet our aim to assist the raising of global aviation standards. The transition from CAA paper-based to e-examinations was funded by CAAi and the technology has proved extremely successful in delivering licensing examinations.

 “Through the innovation partnership, CAAi, the CAA and the supplier configured the [ATPL] e-examinations platform to include the Private Pilot Licence (PPL). This was a natural evolution of the existing system, tailored to meet the specific needs of PPL examinations without incurring undue cost for GA.     

 “CAAi’s role in PPL e-exams also includes supplier management and coordination and the provision of subject matter expertise to assist technical question bank queries and content management.”

FTN asked for further clarification on how the CAA/CAAi advertises its contract tenders, given that a number of flight training businesses had told us that they would have liked to have entered bids for the e-Exams contract but were unaware of its existence. At the time of going to print the CAA had yet to respond.

Returning to the new UK ATPL question bank, FTN has been contacted by a number of students who have struggled through the new exams, in many cases having to undergo resits. They are concerned that they have effectively been made guinea pigs for the new exams and that their future career prospects have been impacted as a result. One cadet, who spoke with FTN under the condition of anonymity, said that his ultimate goal has always been to join British Airways, but that this has effectively been taken from him due to the fact that the UK flag carrier insists on applicants having achieved first time passes in their exams, and he was one of the 95% that failed his first attempt at the now notorious Flight Planning and Performance exam. While future cadets will likely be assisted through the gradual development of a feedback question bank, as schools gain more knowledge of the new QB, these early adopters appear to be out of luck. Having approached the CAA over the issue, the cadet was informed that due to the relatively low uptake of the new QB to date the current failure rates can’t be defined yet as being representative and so they would not review his situation.

 

 

 

 

 

 

Author: FTN Editor

Share This News On