Housing minister Robert Jenrick says the government will take “all the steps that are necessary” following the arrest.
A Conservative MP arrested over allegations of rape will not be suspended from the party while police investigate.
A spokesperson for the Conservative whips’ office said: “These are serious allegations and it is right that they are investigated fully. The whip has not been suspended.
“This decision will be reviewed once the police investigation has been concluded.”
It comes after shadow safeguarding minister Jess Phillips said it was “shocking” the party had not decided to withdraw the whip.
The accused former minister was arrested and taken to an east London police station on Saturday morning and released on bail later that day.
He has been accused by a young woman who used to work in parliament, Sky News understands.
“In any other organisation, were this police investigation to be going on, somebody would be suspended while the investigation was taking place,” Ms Phillips told Times Radio.
“I find it shocking this morning in the news that the Conservative Party has decided not to withdraw the whip in this case.”
She said it was “only right” the whip was withdrawn and pointed to the conviction earlier this week of former Tory MP Charlie Elphicke, who was allowed to maintain the whip while being investigated for sexual assault.
An MP who has the whip withdrawn is effectively expelled from their party, but keeps their seat and must sit as an independent until they have the whip restored.
Housing minister Robert Jenrick said the government will “take all the steps that are necessary” following the arrest.
“This is a very serious allegation and it’s right that it’s treated as such,” he said.
“We need to be a political party that ensures women are safe and feel comfortable at all times and that’s exactly what we are doing.”
Liberal Democrat MP Layla Moran told Sky News: “Given the seriousness of these allegations and that a suspension, in a workplace or anywhere else, is actually a neutral act… I would urge (the Conservative whips’ office) to think again on that decision.
“But more broadly I think we need to acknowledge the bravery of those who have come forward to make these allegations, to do that against an MP, especially if they are staff, it’s very very difficult.”
The woman’s complaint was reported to a fellow MP, Sky News understands.
It is claimed some allegations were raised with Jacob Rees-Mogg, the leader of the House of Commons, and chief whip Mark Spencer.
Mr Spencer had reportedly not known the “magnitude” of the allegations, and it is understood he told the woman to make a formal complaint to parliamentary authorities, who would investigate.
The woman later spoke to police.
A police statement said: “The Metropolitan Police Service received allegations relating to four separate incidents involving allegations of sexual offences and assault.
“These offences are alleged to have occurred at addresses in Westminster, Lambeth and Hackney between July 2019 and January 2020.
“The Met has launched an investigation into the allegations.
“A man in his 50s was arrested on Saturday 1 August on suspicion of rape.”
In a statement, the Conservative Party said it takes “all allegations of this nature extremely seriously” but refused to comment further as the matter is now being investigated by police.
A statement from Mr Spencer said: “The chief whip takes all allegations of harassment and abuse extremely seriously and has strongly encouraged anybody who has approached him to contact the appropriate authorities, including parliament’s independent complaints and grievance scheme, which can formally carry out independent and confidential investigations.”