The spending watchdog found customers waited on average nearly 23 minutes to get through to an adviser in the first 11 months of 2023-24, up from just five minutes five years before.
The findings come after HMRC was forced to reverse its decision to close its self-assessment telephone helpline for half of the year after a public outcry.
It found many customer calls are caused by HMRC itself for reasons including delays and customers chasing progress.
The NAO suggested that taxpayers were being "let down".