Work list for R 4.4/Bioconductor 3.19 builds #49778

mfansler · 2024-08-03T10:16:11Z

Preparation for R 4.4/Bioconductor 3.19

It looks like we'll cross the r-essentials milestone¹ for building R 4.4 on Conda Forge this weekend. This seems like a good point to start thinking about the transition of Bioconductor packages here to v3.19. To expedite this, it would be useful to identify required dependencies that need work to migrate. This helps those of us working on Conda Forge to focus our efforts.

This list should now be comprehensive thanks to a helpful script from @aliciaaevans. However, feel free comment or check things off if you see they have completed. Updated info is available on the Conda Forge Status page.

Problematic R 4.4 Migrations

📢 Updated 2 September 2024 using missingCranPackages.py script.

Awaiting migration of 62 packages

In PR (20 packages)

NB: This is where the most help is needed!

Bot Error (4 packages)

Licenses have been packaged for these as of 9 Sept. and should start migration within ~24 hours.

r-rebus.unicode: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:
r-rebus.datetimes: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:
r-rebus.numbers: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:
r-logr: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:

Not Solvable (7 packages)

r-spdep: not solvable (bot CI job) @ main r-sf * cannot be installed because there are no viable options
r-magick: not solvable (bot CI job) @ main No candidates were found for imagemagick *.
r-rmpi: not solvable (bot CI job) @ main No candidates were found for openmpi 5.*.
r-transformr: not solvable (bot CI job) @ main r-sf * cannot be installed because there are no viable options
r-protolite: not solvable (bot CI job) @ main r-sf * cannot be installed because there are no viable options
r-gstat: not solvable (bot CI job) @ main r-sf >=0.7_2 cannot be installed because there are no viable options
r-tkrplot: not solvable (bot CI job) @ main No candidates were found for tcl *.

Awaiting Parents (31 packages)

"How can I help?"

If you would like to help, you can either send PRs with fixes or make comments on the Conda Forge feedstocks about possible solutions.

Generally the three status categories correspond to different types of troubleshooting.

"In PR" - feedstocks with this are usually having compilation problems (mostly C++); typically the issue is not about compilation per se but rather getting linking working.
"Not solvable" - feedstocks with this usually indicate something about the packages they depend on is wrong; identifying those specific packages is the first step and then figuring out how to fix those upstream
"Bot error" - feedstocks with this usually mean something is substandard about the recipe itself and the migrator is crashing on it; anecdotally, this frequently involves license issues, such as not including an explicit license_file: entry in the meta.yaml

The "In PR" is often the most difficult and requires some knowledge of how code compilation is done in R; the "Not solvable" involves sleuthing around; and "Bot error" is more like review work - read the recipe and compare it one that works.

_{[1] The Conda Forge r-essentials package is a metapackage that represents common workflows used in R, such as Shiny, tidyverse, and Jupyter kernel support.}

The text was updated successfully, but these errors were encountered:

mfansler · 2024-08-03T10:18:46Z

CC: @daler just a heads up

aliciaaevans · 2024-08-14T20:53:26Z

conda-forge/r-clusterr-feedstock#25 seems like maybe the Travis build just needs to be re-triggered.

mfansler · 2024-08-16T07:21:24Z

conda-forge/r-clusterr-feedstock#25 seems like maybe the Travis build just needs to be re-triggered.

Travis CI is failing for all linux-ppc64le jobs. There are some notes in the Status Issue about how to convert to either cross-compiling or emulation on Azure to get around this.

danielnachun · 2024-09-03T23:43:28Z

I took a pass through all the "In PR" failures (rerendering a handful whose logs had expired) and posted a comment for each one either extracting the error, or when possible proposing a fix. I will try to push fixes for the ones where I had a possible solution.

@mfansler One quick question - what do you think about just adding -pthread to PKG_LIBS for the handful of packages failing due to this? I think there only 3 affected by this in the above list:
conda-forge/r-pdftools-feedstock#48
conda-forge/r-strawr-feedstock#4
conda-forge/r-rmixmod-feedstock#17

I agree with the your suggestion in conda-forge/r-base-feedstock#325 to add -pthread to the Makeconf for Windows so that we don't have to keep adding this manually, but I'm not sure we need to wait for that be done for just 3 packages.

mfansler · 2024-09-04T07:29:05Z

@danielnachun that's wonderful - thanks for spending time with this! ❤️

Unfortunately, I'm short on bandwidth today, but will try having a look as soon as I can.

RE: -pthread - yes, we can just manually override to expedite getting to Bioconductor 3.19.

danielnachun · 2024-09-06T17:46:53Z

The discussion in conda-forge/r-rcppalgos-feedstock#16 prompted me to check something I should have looked into sooner - among the packages that are only failing on Windows, which ones have never had a successful Windows build? It turns out these 6 packages have not, and they are some of the weirder or more difficult failures:

So as not to block the R 4.4 migration, I think we should consider skipping the Windows builds on these for now as they don't have obvious solutions and Windows users were never able to use them anyway.

danielnachun · 2024-09-06T18:06:04Z

@mfansler for packages that don't have a PR because of bot errors or unsolvable dependencies, is it sufficient just to rerender the feedstock with the necessary manual fixes? Or does the migrator make other manual changes (aside from re-enabling Windows if it was skipped)?

Also does the migrator automatically open PRs for packages that were previously blocked by a missing parent? Or does a PR need to be made to https://github.com/conda-forge/conda-forge-pinning-feedstock?

mfansler · 2024-09-09T07:43:14Z

@danielnachun so far, I've adopted the practice that it is better, when possible, to fix the recipe without manually forcing a R 4.4/UCRT migration, so that the bot can start working normally. Specifically, to your question: Yes, the migrator will eventually notice fixed recipes and (AFAIK) will retry solving failed recipes every ~24 h.

When we manually migrate it, the problem often doesn't go away (e.g., downstream dependencies may encounter same issue) and technical debt accrues. However, there are exceptions that are unavoidable, so here are the different cases I see:

bot error - all of these I've fixed to now have been driven by substandard recipes, typically involving the lack of a proper license file. Rerendering the recipe without adding R 4.4 migration, but fixing the substandard aspect will unlock the recipe and should make it so that recipe just works in future migrations.
not solvable + CI error - some recipes come out as not solvable because one of their requirements made it through PR, but had a CI error that resulted in not uploading the build artifact. This is a matter of tracking down the problematic dependency.
not solvable + missing platform - another scenario is that the recipe's dependency couldn't build a particular platform. Sometimes this can be an error (e.g., there was a selector on skip: true that the bot didn't recognize) and we should try to fix it; in other cases, it can really be intrinsic, and I think this case is what you're interested.

Manual Migration

For the case 3, manual migration is done by adding in the migration file (see example commit) and making some changes to the meta.yaml:

delete definition and uses of {{ native }} Jinja variable
bump build number
delete merge_build_host: true
remove any skips
add any stdlib if missing
if pkg-config is used, remove the {{ posix }} prefix from it

Then a rerender needs to be done.

I believe the bot will recognize that the package is now available, and downstream dependencies should start flowing. However, that will not be the case if you are still skipping platforms - the downstream dependencies will be not solvable.

mfansler · 2024-09-10T10:21:37Z

@danielnachun FYI, any new PRs for R packages appear blocked at the moment: conda-forge/conda-forge-pinning-feedstock#6401

danielnachun · 2024-09-10T23:45:53Z

I was going to ask why things weren't restarting but that makes sense. Hopefully this gets figured out soon!

When we manually migrate it, the problem often doesn't go away (e.g., downstream dependencies may encounter same issue) and technical debt accrues.

This an important point for skipping Windows - if nothing depends on the package, then it doesn't propagate elsewhere. So we should put the most effort on fixing packages with dependents. Of the 6 I posted above, only r-magick and r-ttfpt1 seem to have a significant number of dependents, and we've now fixed r-ttfpt1 on Windows anyway. Getting imagemagick built on Windows may be a challenge but seems like it would resolve some issues

One other question - if a package is noarch: generic, does it still get count as being blocked by missing parents if one its dependencies is available for Linux but not Windows? I ask because noarch: generic packages are only built on Linux. Of course the package wouldn't be installable on Windows but that ideally shouldn't prevent its migration to a newer R.

mfansler · 2024-09-11T04:40:41Z

@danielnachun correct, noarch builds won't be blocked by a parent that fails on win-64. The solver will only try a linux-64 solve.

For the 6 feedstocks you highlighted, I've fixed all but r-rcppalgos (merged with win-64 skipped) and r-primme (just haven't gotten to yet, but seems doable). The r-magick in particular I was able to use the vendored imagemagick rather than trying to get Conda Forge building that for win-64. For r-rmpi, I've addressed the not solvable aspect, but we will still need to adjust further once the migrator actually sends a PR.

Thanks for all the new reviews! 🤩 I may not have time today, but I will eventually get to them.

danielnachun · 2024-10-01T00:17:16Z

@mfansler the xorg packages that were holding back the migration have been fixed and quite a few of the "Awaiting parents" package have now been migrated or just need to be merged! I've also made comments on the handful of new PRs which have failed now that their dependences are available.

From what I can see, there are two big blockers that will fix a lot of the remaining "Awaiting parents"/"Not solvable":

r-locfit: we can discuss this more in the feedstock but it seems like the issue is just that some Makevars variables aren't being set correctly for ARM64 and PPC64LE Linux. While the ultimate fix is probably in r-base, if we can fix it for now in the feedstock with environment variables or patching that might be an acceptable solution to move the migration forward.
r-sf/r-raster/r-terra: in the discussion in Windows builds conda-forge/r-terra-feedstock#97 it seems using r_clang could get these packages working on Windows again. There are also many other "Awaiting parents" in the larger R 4.4 migration that would probably be unblocked if we could get this working.

mfansler · 2024-10-22T07:23:53Z

FYI, I'm going to merge what I can of the outstanding PRs with Windows skipped for now. This will also mean we'll have to do some manual migration PRs on some of the depending packages, since otherwise the migrator will consider them "Not Solvable".

danielnachun · 2024-10-22T18:42:50Z

This is a good call. From what I can see there were only 4 packages that had to be merged with Windows skipped:

r-cairo: 2 dependents, neither of which have their own dependents
r-copula: 2 dependents, neither of which have their own dependents
r-primme: no dependents
~~- r-straw: no dependents~~

So the impact of this on the migration overall is small.

I think the only really big impact right now on migrations is the lack of an r-terra build on Windows. I was able to fix r-sf, but I've hit a pretty hard wall on r-terra in conda-forge/r-terra-feedstock#102 where the package builds but fails to load. So we'll have to do a few more manual migrations to skip native dependents of r-terra and r-raster until we can figure out r-terra. For Bioconductor it seems the last manual migration is just r-gstat - there will be many others not needed for Bioconductor.

The only other package blocking migrations are:

r-locfit - we know why this is failing as noted above, we just have to decide on the best solution
r-tkrplot - I think we need to change the dependency on m2w64-tk on Windows to just tk

Assuming the dependents of r-locfit don't run into any serious issues then we'll be done with this huge effort!

mfansler · 2024-10-23T08:56:50Z

@danielnachun I think the snag with r-gstat is r-lwgeom, not r-terra. I think that hasn't been retried yet since you got r-sf built. Do you want to give that a try? (possible r_clang use)

I got r-locfit done and hopefully its dependents have smooth migrations. 🤞

I'll try to have a look a r-tkrplot later - such recipes with extra {{ native }}* dependencies often need manual migration.

danielnachun · 2024-10-24T01:26:46Z

@danielnachun I think the snag with r-gstat is r-lwgeom, not r-terra. I think that hasn't been retried yet since you got r-sf built. Do you want to give that a try? (possible r_clang use)

I think we're all set for r-lwgeom now - see conda-forge/r-lwgeom-feedstock#63 when you have a chance.

I got r-locfit done and hopefully its dependents have smooth migrations. 🤞

Sadly we hit a new snag here as we're discussing over there but hopefully we'll straighten it out soon!

I'll try to have a look a r-tkrplot later - such recipes with extra {{ native }}* dependencies often need manual migration.

That's very helpful to know that {{ native }}* dependencies may need manual migration, as I identified some other packages not needed for the Bioconductor migration that also have similar issues with unsolvable dependencies which I will look into manually migrating now as well.

mfansler · 2024-10-25T14:39:03Z

@dpryan79 FYI, this is just about complete. I believe you've handled the bulk Bioconductor migrations in the past.

aliciaaevans · 2024-10-29T12:51:27Z

@mfansler Thanks for all your hard work and for keeping us updated. I did the last Bioconductor update and will probably end up doing this one unless someone else on the core team decides to do it before I get to it in a couple of weeks.

danielnachun · 2024-10-29T16:57:56Z

We just need to get conda-forge/r-robcompositions-feedstock#20 merged and then hopefully the migrator will update https://github.com/conda-forge/r-mvoutlier-feedstock smoothly (it's noarch: generic so it's less likely to have issues).

Maybe once those are done we can close this issue as completed and @aliciaaevans can open a new tracking issue for the actual migration? Please feel free to assign me to that issue as I'd like to help get the transition done as quickly as possible!

danielnachun · 2024-10-29T22:43:06Z

The last 2 feedstocks are now updated, so we can probably close this as completed now.

aliciaaevans · 2024-10-30T13:08:53Z

@danielnachun Thanks! Sounds good to me. Starting the initial Bioconductor update on Bulk is a 1-person job, but if/when packages fail, I could use some help investigating or fixing the ones with a lot of dependencies. I'll keep you posted.

danielnachun · 2024-10-31T07:02:12Z

@aliciaaevans please free free to just @ me on failing recipes and I'll try to take a look.

Do you think it's feasible to default to start enabling Apple Silicon builds for this migration? I recently joined the conda-forge R team so I can try to get PRs to enable Apple Silicon builds on packages that are missing that platform.

bgruening · 2024-11-03T09:28:27Z

I have started a bulk-bioc branch at #51889 please have a look and fell free to push to that branch directly. If you are happy with it, please let us know and we can merge it into bulk and start building packages.

mfansler added the help wanted label Aug 3, 2024

aliciaaevans mentioned this issue Aug 20, 2024

feat: Prereqs for Bioconductor script #50209

Merged

danielnachun mentioned this issue Nov 3, 2024

Update BioConductor packages to BioC 3.20 #51889

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work list for R 4.4/Bioconductor 3.19 builds #49778

Work list for R 4.4/Bioconductor 3.19 builds #49778

mfansler commented Aug 3, 2024 •

edited

Loading

mfansler commented Aug 3, 2024

aliciaaevans commented Aug 14, 2024

mfansler commented Aug 16, 2024

danielnachun commented Sep 3, 2024

mfansler commented Sep 4, 2024

danielnachun commented Sep 6, 2024

danielnachun commented Sep 6, 2024

mfansler commented Sep 9, 2024 •

edited

Loading

mfansler commented Sep 10, 2024

danielnachun commented Sep 10, 2024

mfansler commented Sep 11, 2024

danielnachun commented Oct 1, 2024 •

edited

Loading

mfansler commented Oct 22, 2024

danielnachun commented Oct 22, 2024 •

edited

Loading

mfansler commented Oct 23, 2024

danielnachun commented Oct 24, 2024

mfansler commented Oct 25, 2024

aliciaaevans commented Oct 29, 2024

danielnachun commented Oct 29, 2024

danielnachun commented Oct 29, 2024

aliciaaevans commented Oct 30, 2024

danielnachun commented Oct 31, 2024

bgruening commented Nov 3, 2024

Work list for R 4.4/Bioconductor 3.19 builds #49778

Work list for R 4.4/Bioconductor 3.19 builds #49778

Comments

mfansler commented Aug 3, 2024 • edited Loading

Preparation for R 4.4/Bioconductor 3.19

Problematic R 4.4 Migrations

Awaiting migration of 62 packages

In PR (20 packages)

Bot Error (4 packages)

Not Solvable (7 packages)

Awaiting Parents (31 packages)

"How can I help?"

mfansler commented Aug 3, 2024

aliciaaevans commented Aug 14, 2024

mfansler commented Aug 16, 2024

danielnachun commented Sep 3, 2024

mfansler commented Sep 4, 2024

danielnachun commented Sep 6, 2024

danielnachun commented Sep 6, 2024

mfansler commented Sep 9, 2024 • edited Loading

Manual Migration

mfansler commented Sep 10, 2024

danielnachun commented Sep 10, 2024

mfansler commented Sep 11, 2024

danielnachun commented Oct 1, 2024 • edited Loading

mfansler commented Oct 22, 2024

danielnachun commented Oct 22, 2024 • edited Loading

mfansler commented Oct 23, 2024

danielnachun commented Oct 24, 2024

mfansler commented Oct 25, 2024

aliciaaevans commented Oct 29, 2024

danielnachun commented Oct 29, 2024

danielnachun commented Oct 29, 2024

aliciaaevans commented Oct 30, 2024

danielnachun commented Oct 31, 2024

bgruening commented Nov 3, 2024

mfansler commented Aug 3, 2024 •

edited

Loading

mfansler commented Sep 9, 2024 •

edited

Loading

danielnachun commented Oct 1, 2024 •

edited

Loading

danielnachun commented Oct 22, 2024 •

edited

Loading