My thoughts on changes to this.

[rkent]

I've been thinking for awhile about proposing some major changes to rosindex. Here are some not-fully-developed thoughts that I have had. I would appreciate any comments from others on directions.

I've always found rosindex hard to use for a variety of reasons. Some general issues:

• Too much extraneous content.

The front page has all of these links outside of rosindex. Once you start using it, the core functionality (showing packages, repos, and system dependencies) is behind a drop-down menu, while plenty of external or less-important items are one click.

• Too many lines of listing without filters.

The "packages" page is 39 pages long, with no filtering available to narrow the focus.

• The search pages have limited information (no description, for example), no sorting nor filtering, and also show way too many pages of information typically.

---

I don't know what the stated purpose of rosindex is, but it seems to me the main function should be package discovery. A secondary purpose today is also clearly to give package details.

What would I change?

• Revise the pages (particularly the home) to focus more on capabilities of rosindex, and less on links to external resources.
• Combine the existing duplication of listings of Packages and System Dependencies (that is, the search output duplicates the general listing) into a single type.
• Limit the default display of packages and resources to the currently selected (via rosdistro cookie) rosdistro.
• Add javascript-driven sort and filters to the listings, with a filter interface. (See https://lowcostminipcs.com/ for an example). Use lunr search mostly for the text parts (like descriptions).
• Finish implementing tags, and curate an initial assignment of tags to existing packages.
• (If feasible) reorganize into a single-page web app. On initial load, just load the json details for the current rosdistro, lazy-load the rest. (If we don't do this, we might be forced to constantly reload the json package, repo, and system dependency info that will be used to construct the listings).
• Remove some of the package details that are not primarily focused on interactions with other packages, and refer users to rosdoc2 instead. (Not sure how this works with ROS 1 packages though). Maybe show rosdoc2 output itself in an iframe within rosindex.
• Add descriptions to system dependencies from, for example, https://packages.debian.org/stable/allpackages?format=txt.gz

This can be done incrementally from the present site, but my experience with rosdoc2 is that this is going to be review limited. But it might be worth considering having a development site with looser review rules, planning to switch to a revised rosindex as a major update rather than incrementally.

Thoughts?

[tfoote]

There's definitely a lot there we could improve on. And finding a way to developer more quickly would be good. I need to get to sleep, but wanted to say that this is on my queue to reply in more detail.

[rkent]

Here's an update.

1. " having a development site". As my development branch gets farther and farther ahead of the release branch, it is getting increasingly untenable to use a single test build for both PRs and development demo. So I have put together a second nightly build of rosindex. https://index.rosdabbler.com has a build of my submitted PRs, while https://devindex.rosdabbler.com has a demo of my latest development work.

2. "descriptions to system dependencies". This is done for Debian, awaiting final review. To get additional descriptions (for example of Pip) might require more extensive web scraping than is practical in a daily build.

3. "Add ... sort to the listings". At http://devindex.rosdabbler.com/new_packages/ I have a demo of a revised package list, using an existing js library Tabulator for the table display. Some observations.

• data source: Previously, I revised the search data generation to shard by distro rather than in numbered pages. This data table reads in a single file with the packages data (but not the index) for a single distro. The distro switch buttons are active (though there are bugs), lazily loading the data JSON for any newly selected distro. My brief experiments show this is incredibly fast. Sorting by all columns is now available by default, and virtually instantaneous. You can also scroll the entirelist without resorting to pagination buttons. IMHO this is clearly the right path forward.

• "duplication of listings". The next step is to merge search functionality, possible with some filtering UI, into this. The intention is that this list would replace both the current static list, as well as the searched packages. I would also use the same approach for repos and dependencies, so they would all look and work the same.

• Cross-distro search. With the large number of distros currently included in this, loading all of the search info for all distros is a lot of data(== slow). I don't really see the point of this expensive cross-distro search, so at least initially I would only enable search over a single distro. If an outcry arises demanding it, perhaps cross-distro search could be added later.

• "Single-page webapp". I am currently leaning away from this. The point would be to store expensive json downloads for an entire session, but if they are fast anyway (after limiting to a single distro). I'm not sure that is needed.

• "Revise the pages." Current devindex has an initial attempt to limit display material above the tables, but not enough. The height of the static top needs to get smaller to leave more room for the displayed data.

• Tags. The existing tag implementation in rosindex is quite limited. The data.json files tried to expand the existing tags using a parsing of the package name, which IMHO is not effective. For this to be effective is going to be a lot of work. Possible some auto tags ("has_messages", "included_in_default_ros_packages"). But ultimately to make tags effective would probably require some initial curation of tags per package, which would be a huge undertaking.

[tfoote]

" having a development site". As my development branch gets farther and farther ahead of the release branch, it is getting increasingly untenable to use a single test build for both PRs and development demo. So I have put together a second nightly build of rosindex. https://index.rosdabbler.com/ has a build of my submitted PRs, while https://devindex.rosdabbler.com/ has a demo of my latest development work.

Thanks being able to see the effects of the PRs is great for faster reviews.

"descriptions to system dependencies". This is done for Debian, awaiting final review. To get additional descriptions (for example of Pip) might require more extensive web scraping than is practical in a daily build.

Lets follow up on that one in the other thread. There's a level that we need to make the standardization and decisions at the rosdistro maintenance level before we put it all into documentation.

"Add ... sort to the listings". At http://devindex.rosdabbler.com/new_packages/ I have a demo of a revised package list, using an existing js library Tabulator for the table display. Some observations.

This is great! I actually just filed #425 and #426 before getting to this issue because it was bugging me so much when I was testing the site for the JS changes.

The speed is great! The data density is much better.

Avoiding the duplication is great.

We don't need the cross-distro work with the tabs. In my other issue I think that adding the number of rows available will help people know to look in other distros if they are not getting what they want.

Single-page webapp

Yeah, if it's fast enough to reload that's fine. And if the json is static it should get cached by the browser and or CDN/proxies.

Revise the pages

I"m not as worried about the space above. For me the problem is that the bottom doesn't flow down.

Also in usability, the right scroll bar doesn't drag properly for me. I could scroll wheel and page up and down, but not drag the scrollbar to jump to the top.

And the distro column is now redundant with the tab filter at the top so could make more space.

Tags

The existing tags are mostly I think for rendering flags in the package display versus for user consumption. But potentially exposing some of them might make sense. We could then add filters to the search as well based on the tags.

[rkent]

Yes, this is far from the final form. It is mainly meant to be a trial of Tabulator to see if this is a viable approach. So far I am impressed. These development builds will have lots of glitches for awhile.

The height of the visible rows is a parameter that is set when creating the table. I could show a lot more rows, but there are advantages to being able to see all displayed rows on a single desktop computer screen. Not sure what the right tradeoff is here. But the fixed height is needed for the speed, as Tabulator uses a virtual DOM and only displays the rows that are visible. You could add some logic to adjust the height depending on the size of the available screen, but it is never going to respond smoothly to that like it would if you could rely on the browser to resize in CSS.

Right scroll works fine for me, not sure about your issues. Tabulator is a large, complex project so we might have to live with whatever limitations or bugs it has if we use it.

I'm currently working on merging a filtering functionality into the display, then I will also add "filtering" via the Lunr search, to have the unified display I mentioned above. Yes tags and distro are currently useless on this screen. But tags I believe are an important feature long-term. I keep distro partly because distro switch is buggy with this, and until there is a clear decision to limit this display to a single distro.

[tfoote]

If we have a filterable display that is quick, then the package list can just be the search without any search string. And the filtering for tags or distro are also just more advanced search functions too. So we can really collapse those capabilities fully potentially.

I think that this experience is better than our current pagination approach so lets see where it can go.

[rkent]

I integrated search into https://devindex.rosdabbler.com

It's amazing how much more effective search is with just searching over a single distro. That really narrows the search, and gets rid of the problem of zillions of tf2 results, all in different distros.

I was expecting to have to deal with several hundred entries per search, but it seems like I am getting several tens at most. Those are not difficult to visually search with the scrollable list. So I am questioning the value of the filtering separate from search that I was planning to add. Perhaps a simple text search is going to be enough?

So far, this has been a conceptual demo. There are many UI glitches, but the search functionality seems to be working well, and a major improvement over the current experience. I think I will work on solving some of the UI issues (like getting the proper table height) and see if we can land this.

[tfoote]

Yeah, that search is definitely working well. Snappy and easy to get to what you want already. I think that the filtering is a nice to have, but it's not at the top of my priority list.

For the top bar I know that you were looking to compress it. And I agree with making the package listing, repo listing, and rosdep listings always visible. But maybe it would look less compressed to keep them in the blue menu bar at the top (where they used to be a drop down) instead of the larger clickable boxes that take up a lot of space.

[rkent]

The design for the package listing, repo listing, and rosdep listings was copied from the previous version. I also find that ugly. But they need to stand out somehow from the external links which currently are the upper page items.

After playing some with Tabulator layout, I think that the overall design of the web page needs to change to properly show it responsively. Basically Tabulator needs a fixed height web design to properly enable its virtual dom (or at least a preset height on the table), which I believe is important for proper performance.

If the design is going to change completely, it might be time to also do two other changes at the same time:

• Update bootstrap to 5
• Stop using JQuery.

I'm going to experiment with doing that just on the package list page initially rather than tackle the whole site. Not sure if that will be feasible or not yet.

Also it would be time to think through not only responsiveness (effect of screen size) but also accessibility. I'm not really a web designer, so I am learning as I go how to do this. If there are people more suitable for the task feel free to recommend yourself or others, but barring that I will muddle through somehow.

[tfoote]

I'm very much muddling through web design too.

Upgrading bootstrap is fine. And JQuery use or not is an implementation detail not an architectural decision. Though it's a pretty low level tool that is usually pretty useful.

Focusing the testing in one place sounds like a good idea. I look forward to seeing what you can do with Tabulator.

[rkent]

Update:

1. I've abandoned the hope of updating Bootstrap for now, the changes are just too large. If we are stuck on Bootstrap 3, then we also still need jQuery. But I think I will try to use standard DOM requests instead of jQuery when possible in new code.

2. Yesterday I worked on the header. I've condensed 3 rows into a single row, and also made it responsive at phone screen levels. This is currently available at https://devindex.rosdabbler.com. I use dropdowns instead of the usual height expansions for responsiveness, as the goal is to switch to a fixed-height layout for the list pages, so that all of the scroll is in Tabulator and not also in the web layout.

[tfoote]

Too bad about the bootstrap changes. I agree with not trying to do the bigger chagnes.

The new layout looks reasonable. One thing I noticed is that it took ~40 seconds for me to load the package index content when I did a force reload one of the times. Maybe that was the dev hosting spinning up. As I can't reproduce it now.

[rkent]

Latest update, available at https://devindex.rosdabbler.com/, was focused on getting the package list page to only scroll in the Tabulator section. This now WFM on desktop. On mobile (FF on a Pixel 6) it scrolls in the browser for the height of the URL bar. I believe though this is a feature of the mobile browser, so I am not concerned. Earlier @tfoote reported scrolling problems, hopefully those are now gone.

The next step is to change what fields are actually shown in the listing, searched, and filtered (should that make sense to do.)

Concerning "it took ~40 seconds for me to load the package index " there was a build running on the machine at the time. The web server is currently serving directly from the _site directory, so during builds it might not only be slow, but non-functional. Usually they only run in the night, but sometimes I trigger them manually when I am working on stuff.

[rkent]

The site at https://devindex.rosdabbler.com/ is now fully completed. Last chance for comments before I start submitting PRs to land this. Major changes since the last comment here:

• the new package list page was renamed to search_packages.
• the displayed columns changed significantly. Gone are tabs and distro, new are existing search fields Authors and Maintainers, plus a new field "org" which is the parent organization of the repos's respository.
• the filtering capability is removed
• search fields are now maintained in the page URL as well as the using the search UI.

I am not aware of any issues, so please report any glitches. In the version presented, the old "Search" pages can still be accessed, but the only way to reach them is from a search bar on the home page. My expectation is this will be completely removed along with the old packages list code when the system deps page is made to be searchable. There is no direct link now to the old packages list, but it can still be accessed for now with /packages (though the intention is to remove it completely).

The full landing can be done in a few PRs, roughly:

• change package search sharding from page-count to distro shards.
• change the header and footer to the condensed versions
• install the new package search page but do not point package list to it.
• change the package list link to point to the new page.

[rkent]

In case anyone follows this, there was an important experimental update today.

I created a repo https://github.com/rkent/ros_webdata that uses github actions to keep updated a few items that are slow to update, but also slow to change. This includes 1) descriptions for pip dependencies, and 2) package download counts from osuosl.org. The intention is that this repo would be transferred to some ros organization if this idea has merit.

https://devindex.rosdabbler.com now include both of these, that is pip descriptions as well as package download counts. I have no doubt that pip descriptions will eventually be added in some form or another. The package download counts could be somewhat more controversial, but I find it useful to know what are the popular ROS packages.