HS2 does not offer good value for money and risks "short-term changes" in northern England, a group of colleagues warned.
A report from the House of Lords Economic Committee also said it was "far from convinced" that the new high-speed rail system would be built within the GBP 55.7 billion budget.
The project should not be carried out without a new assessment of costs and benefits.
The government said it "basically disagreed" with the report.
The committee, which includes former Chancellor Lord Darling and Lord Lamont, said that more than GBP 4 billion has already been spent on the first phase of HS2 to be carried out between London and Birmingham.
However, the program has placed too much emphasis on shortening travel times and not enough on the economic impact on the regions.
- Minister promises no "betrayal" to HS2
- High-speed train "benefits Scotland"
The first phase of the project has "brought little benefit" to northern cities, even though they most needed better rail infrastructure.
In the second phase, when the journey times between Leeds and Sheffield were to be shortened, they were threatened with never going further because of expenditure overruns.
"The northern sections of High Speed 2 should not be sacrificed to offset the excess spending on the southern sections of the railroad," said Lord Forsyth, chairman of the committee.
The report called for the Northern Powerhouse Rail system – a separate system linking cities in the region – to be completed with HS2.
Currently, work on this system is only starting in the mid-2020s.
It also called on the government to control the cost of HS2, warning that "nobody knows" what the final cost will be.
It suggested reducing the speed of HS2 or placing its terminal in West London instead of Euston saving money.
The report is the latest in a series of warnings to the railways about their finances and environmental impact.
HS2 is said to run up to 18 trains per hour at a top speed of 225 mph. Phase 1 should begin in 2026.
A spokeswoman for the Ministry of Transport said: "By 2020, the government will invest a record £ 13bn in transport in the North, and we have a clear plan for linking the Midlands and North through HS2 and Northern Powerhouse Rail – the full benefits of which can only be supplied on the back of HS2.
"This is not either / or, we are sure we want both.
"HS2 will provide additional rail capacity that will greatly enhance connectivity and provide opportunities for economic growth – with benefits of around £ 92 billion – for people and businesses in the North."