The state's Ombudsman found 21 past and present MPs breached parliamentary guidelines by wrongly approving $388,000 be spent on campaign staff, and Labor has repaid the cash.
But speaking on the Nine Network on Friday, federal opposition infrastructure minister Mr Albanese played down the findings.
"Let's get some reality in here, (is) ALP staffers worked for the election of ALP members that they worked for," Mr Albanese said.
"It seems to me from a distance this is an enormous beat-up."
He also said Premier Daniel Andrews "is doing a fantastic job".
"He is ahead as preferred Premier on both primary and two-party- preferred basis. He is a great premier," Mr Albanese said.
However, federal minister Christopher Pyne rebutted the claim of a beat up, saying what happened in Victoria was not just about staffers of Labor MPs working for the re-election of the Andrews Government.
"It was actually people being paid full-time to work as campaign organisers in other elections across Victoria," Mr Pyne said.
"The person who did the inquiry, said it is very, very serious. It is very serious. Daniel Andrews and his government have a lot of questions to answer. This issue is not going away."
Mr Andrews and his government are now facing the prospect of further inquiries over the scandal.
The Greens want the matter sent to the privileges committee, for still-serving MPs named in the report to be fined, for Premier Daniel Andrews to apologise to parliament, and for Labor to pay a $100,000 fine and cover the costs of the Ombudsman.
The coalition opposition wants to establish a select committee to look further into the rort.
Mr Andrews fanned away criticism of his government on Thursday, insisting his colleagues acted in good faith and no heads would roll.
The government has vowed to implement all six of the Ombudsman's recommendations.