Flatpak – verification and nightly

General Discussion
1

Wasn’t sure if a new thread was better or to bump the old thread, so feel free to merge this if you like!

Flatpak is seeing wider and wider adoption in the Linux world and is in my opinion very much the future direction of most 3rd-party software distribution. Several distros (e.g. Fedora and openSUSE) are experimenting with immutable versions where Flatpak is the primary way to install apps. So even though the Avo Flatpak currently sees around 20 installs a day, eventually the way most people will be discovering and getting Avogadro will be through Flathub, so @kevinsmia1939’s work here is very valuable. :slight_smile:

Its increasing popularity plus my Flatpak evangelism mean I have a couple of ideas, if that’s ok, two simple and one more complex…

  1. Get the Avogadro Flatpak verified status

Flathub has recently introduced verification so that users can tell whether apps have been uploaded by the original developers or by the community. And it really doesn’t sound like a huge amount of work!

As with other app stores, a verified tick has a significant impression on users, giving a feeling of professionalism and security due to trust in the source.

  1. Add a “Get on Flathub” link to the main page of two.avogadro.cc

Mostly to raise awareness that the Flatpak exists. It can link to the Flathub page, from which users can do a click-to-install in Discover, GNOME Software etc.

  1. A nightly build as a Flatpak? :smiley:

The nightly AppImage works pretty well at providing bug fixes and latest features in a distro-agnostic way. Flatpaks have two extra key advantages over AppImages. Firstly, their handling of dependencies is better and more consistent across distros, as they use Flatpak runtimes, whereas AppImages still use lots of system packages. Secondly, they have an update mechanism, and can even be set to auto-update.

Could we think about providing a nightly build as a Flatpak in a hosted repo? For me at least, this would be the ideal way to get up-to-date Avogadro.

Just like with any other package manager, Flatpak can be set up with external repos. The Flathub docs indicate you can host in a GitHub repo. And they provide an example of how to do automatic builds with GitHub Actions.

I saw in the old thread that @ghutchis did a fair bit of work looking at automating Flatpak builds, but I don’t know if it got anywhere. The way @kevinsmia1939 does everything seems absolutely perfect for the main releases, so I’m not suggesting that that needs to change at all!

2

I remember talking with @kevinsmia1939 - but can’t find the conversation. (Probably on GitHub comments.)

It shouldn’t be too hard - I’ve just been pushing more on the code side than the deployment side recently. (e.g., with OpenSSL in the AppImage and Windows)

1 Like
3

I’ll look into this.

4

Of course, this is fairly low priority and is more for after 1.99 is out the door. I just thought I’d post about it while I remembered, after stumbling over the manifest example.

5

Do you have some info on how to create new Flatpak branch?

6

Hi @kevinsmia1939, thanks for all your work! :smiley:

What do you mean by a new branch?

My suggestion is to package the nightly build as a flatpak, but not to publish it to Flathub as you do with the main releases. So nothing needs to change in your Flathub repo.

Instead it can simply be hosted on the GitHub repo, like the other nightly binaries, and then people who are keen to have the nightly version rather than the latest release can add our repo manually.

The example in the docs indicates that all users then have to do at the end is enter along the lines of flatpak install --user https://gitlab.com/your_user_name/repo_name/-/raw/branch/app_name.flatpakref via the command line.

7

The main thing there would be to itemize the list of actions / commands needed to build a flatpak so we can add it as a GitHub action.

8

Ok, I can have a look at that in more detail.

Incidentally, I saw that KDE is currently in the process of updating their apps to follow Flathub’s new Quality guidelines | Flathub Documentation, which they say makes it more likely that apps are featured more prominently on flathub.com. Just little things like having changelogs, a high-resolution icon, a “secondary colour” and so on.

Maybe I could help with doing similar for Avogadro, if people were happy with that? I could also then look at the nightly thing once I have a better understanding of packaging for Flathub.

@kevinsmia1939, @ghutchis, how would you both feel about me being added as a maintainer of the Flathub manifest repo? The standard procedure for this is to open an issue, and @ghutchis should get added as a maintainer in any case.

But I really really don’t want to trample on your toes at all here @kevinsmia1939! You’ve put significant effort into getting Avogadro on Flathub. I don’t want to push in unwelcomely, just to collaborate :slight_smile:

1 Like
9

No problem at all. Welcome.

10

As an update to my original post, for anyone who might come here in future:

  1. Verification is a work in progress and is part of a wider process of sprucing up the flatpak to meet the Flathub quality guidelines

  2. A link to Avogadro’s page on Flathub was added to Install — Avogadro 1.99.0 documentation a while ago now

  3. Nightly builds are not appropriate for hosting on Flathub according to their documentation, partly as they don’t want unstable software, and partly because they don’t want the continuous integration/testing/building infrastructure to be overtaxed. They do offer a flathub-beta repository though specifically for beta releases of apps.

As such, I have spun up a beta branch that is basically ready to go. It’s live and uploaded, so it can already be tested, and there will soon be a page in the Install section of the site with an explanation of how to set it up.

The idea is to regularly update the beta branch to be on pretty current code, but not to have it be literally nightly. So it achieves the same end without being a problem for the Flathub people.

I think with that we don’t need to implement a nightly build any more, but I would still like to implement a GitHub Actions workflow for a flatpak so it can be part of the tests. It’s possible to set it up to produce a flatpak “bundle” which can be downloaded and installed for testing. I’ll take a look at that in the coming weeks.

11

However, the new beta branch has a completely different way of building the flatpak in comparison to the current stable branch, for various reasons, and while almost everything is working, there are a few outstanding issues I’ve found that I could do with help on.

For reference, the new manifest can be seen here. It now pretty closely matches the upstream workflows for Qt 5 and Qt 6.

The issues I am seeing are:

  1. When the flatpak is run, but also whenever you try to use the force field framework to optimize the geometry, the console prints the following error. Note that the old Open Babel optimize command works fine, it’s just the FFF one. Note also that a native Qt 6 build doesn’t do this, and neither does the stable flatpak. The AppImage reports a different error (Cannot open parameter file)
*** Open Babel Error  in LoadAllPlugins  
Unable to find OpenBabel plugins. Try setting the BABEL_LIBDIR environment variable.

  1. The plugin downloader can be opened, and is populated (unlike when we had SSL issues before, when the dialog would be empty), but nothing happens when you click download.

    • If you install the stable branch, you can download a plugin as normal. The plugin is then loaded correctly by the beta branch (both branches of the app are considered to be the same app and therefore share the same sandbox), and avo_xtb works perfectly. So clearly Python in general works, as do plugins. But the downloader doesn’t, and neither does the FFF.

  2. Despite having cloned avogadro-i18n, there are no languages other than the system language available. The old flatpak manifest uses the -DENABLE_TRANSLATIONS=ON CMake flag for avogadrolibs, but this surely can’t be the issue, as the upstream builds don’t use this flag.

Any insights appreciated.

12

I think several people have reported the BABEL_LIBDIR piece. So that’s not limited to the flatpak. I suspect that part of the code isn’t setting the Open Babel environment properly.

As far as the downloader, I’d check where it’s trying to download the file / uncompress it. It may be trying to save it into the internal flatpak directories, which will definitely fail.

13

Fair enough. Just weird that it doesn’t happen on all current builds, only some.

Why would it fail? And what do you mean by the internal directories?

Looking at avogadro/qtplugins/plugindownloader/downloaderwidget.cpp:

...
DownloaderWidget::DownloaderWidget(QWidget* parent)
  : QDialog(parent), m_ui(new Ui::DownloaderWidget)
{
  m_filePath =
    QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation);
  m_NetworkAccessManager = new QNetworkAccessManager(this);
...

So the target used is QStandardPaths::AppLocalDataLocation which should respect whatever $XDG_DATA_HOME is set to. In the flatpak sandbox, this is one of the limited number of locations that is actually writeable by the flatpak.

And this seems to be the case, the other non-flatpak formats install plugins to ~/.local/share/OpenChemistry/Avogadro/ while the current stable flatpak installs plugins to ~/.var/app/org.openchemistry.Avogadro2/data/OpenChemistry/Avogadro, as it should.

Also, going back to the beta flatpak, once avo_xtb is installed, but xtb itself isn’t, I can use the plugin’s Get xtb command to download and install xtb and it does it successfully, using the correct directory (i.e. within ~/.var/app/org.openchemistry.Avogadro2/data). The plugin also uses $XDG_DATA_HOME to inform the install location. So the detection of the environment variable seems to be fine.

As a correction, something does happen: when I click download, the right-hand box goes from showing the plugin’s README to showing nothing but a blank white space.

Anyway I’ll do some testing using a barebones command script and see what I can work out.

14

That implies it’s starting to download. I’d suggest adding some qDebug() statements to the code to see where it’s failing:

1 Like
15

Ok so for starters it looks like an issue when building against Qt 6 – my native build can’t download plugins either. I’ll keep digging.

By the way, am I right in thinking that the nightly builds are done in Release mode? Debug mode is something that you only use when testing locally?

16

Yes, exactly. The nightly builds are, essentially continual release builds. There might be some slight benefit to making them RelWithDebInfo so there’s at least some debugging symbols, but :man_shrugging:

I think the main thing would be to use some qDebug() calls to make sure it’s getting a valid network reply, and then to see at what stage unzipPlugin() stops. If it helps, I can make a branch with a bunch of useful debugging statements.

17

Ok then, I’ll stick to that for the beta flatpak too then.

You already have a lot to do, and I think might for once be within my C++ capabilities :slight_smile:

Fixing whatever the cause is something different entirely, of course, so I’ll likely be back asking for help on that :sweat_smile:

I built a version of the beta flatpak with Qt 5 and it works fine, so it’s not the flatpak’s fault after all.

The other two issues (Open Babel and other localizations) remain though. The Open Babel seems like possibly a more general problem, but might you have any insight on why other languages aren’t showing up? Where should I be looking for the localization files?

18

Here’s where the app looks for translations:

It looks a bit earlier for an environment variable, but here’s where it checks a few locations for the qm files.

1 Like
19

Right then.

I found the issue for the missing multilingual options – a typo in the manifest so that avogadro-i18n was cloned to avogadro-i8n. :upside_down_face: So that was an easy fix. Tick.

I also successfully tracked down the problem with the plugin downloader. The code assumed that the request to the GitHub URL would always return a redirect status and ignored any other non-error result, not allowing for a potential 200 status. In Qt 6 though, in some cases redirection is done for you by default. I fixed it and opened a PR. Tick.

So getting pretty close to the beta flatpak being on-par with the nightly AppImage. And there’s no onus on the user to have openssl 1.1 on their system.

1 Like
20

Just so there’s a written record of this:

The CMake in the KDE SDK until 6.8 had a patch that changed the directory that libraries get installed into during Flatpak builds, but from 6.8 the usual CMake variable set by flatpak-builder is deferred to, see here for some details. That variable doesn’t seem to be picked up by the Avogadro build, so this caused an issue for our Flatpak as the default Avogadro build process puts the libraries it builds into both prefix/lib and prefix/lib64. The solution that’s now been implemented is to merge the contents of the two into /app/lib during the post-install part of the Flatpak manifest, while also creating a symlink to /app/lib at /app/lib64 to ensure that plugins and stuff can be found appropriately.

1 Like