David Einhorn's Greenlight Capital crushed last quarter by betting big on gold, and the hedge fund boss said the metal's big rally isn't done yet.

Einhorn said that he sees gold continuing to rise even after a record-setting run so far in 2025, but he also said he'd be concerned if the price rose significantly higher.

"I'd be really happy if it went to $3,500 or $3,800; I'd be really unhappy if it went to $30,000 or $50,000," the billionaire investor told CNBC on the sidelines of the Sohn Investment Conference in New York.

Bullion briefly peaked at $3,500 per ounce in April, a move many have tied to tariff-linked inflation concerns. Gold is historically considered the premier hedge against runaway price growth, which could justify the metal's 22% surge so far this year.

But even as prices have eased to a one-month low amid softer inflation data, Einhorn sees gold continuing to rally for other reasons.

"Gold is not about inflation. Gold is about the confidence in the fiscal policy and the monetary policy," he said, suggesting that the government has become aggressive on both fronts, altogether contributing to a deficit policymakers are largely ignoring.

In his view, gold's appreciation reflects disappointment in the efforts to slim the $1.9 trillion federal budget deficit.

Einhorn pointed to the Department of Government Efficiency, an agency that initially promised to slash $2 trillion in federal spending.

"A few months have gone by — It's like $150 billion, maybe," Einhorn said. "That's enough to cover next year's defense funding spending increase; that's going to get eaten up really, really fast."

The same goes for tariffs, which the administration touted as a massive boost to government revenue. But Einhorn said the new duties appear set to bring in around $100 billion.

Fiscal concerns will also continue to grow with the new tax policy, with Trump expected to extend his 2017 tax cuts. The bill unveiled by Congress this week is expected to add trillions to the deficit over 10 years.

"We're not really concerned about the deficit. There's a bipartisan agreement to do nothing about the deficit until we actually get to the crisis," Einhorn summarized.

If this continues to propel gold higher, that should continue to boost Greenlight's portfolio. The hedge fund beat the S&P 500 with an 8.2% gain in the first quarter, previously noting that gold's 19% advance made it the fund's "biggest winner."

However, doubt has risen as to whether the precious metal can keep climbing this year. ING expects gold to average $3,128 per ounce through 2025, citing that some tailwinds are losing momentum.

Get the latest Gold price here.