Make lowering pull-based

[cjgillot]

Based on #90451
Part of #88186

The current lowering code visits all the item-likes in the AST in order, and lowers them one by one.
This PR changes it to index the AST and then proceed to lowering on-demand. This is closer to the logic of query-based lowering.

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 30c685c9a863546ffe3be12e65ec08fa38c15acb with merge 40908995bf1a9271ebb46383e84e7fb7238b5971...

[bors]

☀️ Try build successful - checks-actions
Build commit: 40908995bf1a9271ebb46383e84e7fb7238b5971 (40908995bf1a9271ebb46383e84e7fb7238b5971)

[rust-timer]

Queued 40908995bf1a9271ebb46383e84e7fb7238b5971 with parent 55ccbd0, future comparison URL.

[rust-timer]

Finished benchmarking commit (40908995bf1a9271ebb46383e84e7fb7238b5971): comparison url.

Summary: This change led to moderate relevant mixed results 🤷 in compiler performance.

• Small improvement in instruction counts (up to -1.4% on full builds of coercions)
• Moderate regression in instruction counts (up to 2.1% on incr-full builds of ctfe-stress-4)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

[cjgillot]

r? @michaelwoerister

[michaelwoerister]

Thanks, @cjgillot! I hope to get to this next week.

[bors]

☔ The latest upstream changes (presumably #90145) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #90564) made this pull request unmergeable. Please resolve the merge conflicts.

[michaelwoerister]

Marking this as blocked on #90451 for now.

[bors]

☔ The latest upstream changes (presumably #94584) made this pull request unmergeable. Please resolve the merge conflicts.

[michaelwoerister]

Do you think that some of the performance will be recovered once the whole refactoring is done?
Overall, the performance hit of this PR doesn't look very big. IMO, it's not a blocking issue.

[cjgillot]

Do you think that some of the performance will be recovered once the whole refactoring is done?

I don't know, though I hope so. For now, #88186 is a perf regression (up to 4% on primary benchmarks, up to 11% with stress benchmarks). If we want to skip parts of lowering altogether, we will need in addition:

• to hash the AST and part of ResolverOutputs -> the perf tradeoff is not obvious;
• avoid many accesses to HIR (for instance, Refactor HIR item-like traversal #95004 may help to get us closer).

[michaelwoerister]

OK, as stated before, I think the performance hit is acceptable as it should not be noticeable in any of the real-world benchmarks. You can r=me after rebasing. Thanks for all the hard work you are putting into this!

[cjgillot]

@bors r=michaelwoerister

[bors]

📌 Commit 6b099db has been approved by michaelwoerister

[bors]

⌛ Testing commit 6b099db with merge bd1a869...

[bors]

☀️ Test successful - checks-actions
Approved by: michaelwoerister
Pushing bd1a869 to master...

[rust-timer]

Finished benchmarking commit (bd1a869): comparison url.

Summary: This benchmark run shows 35 relevant regressions 😿 to instruction counts.

• Arithmetic mean of relevant regressions: 0.6%
• Largest regression in instruction counts: 1.7% on full builds of unused-warnings check

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression

[rylev]

Marking this as triaged since a justification was already given here.

@rustbot label +perf-regression-triaged