Full and privileged xtb integration?

[matterhorn103]

@ghutchis

I mean, from my perspective, you want to cover most use cases with separate scripts. It’s the 80/20 or 90/10 rule … most usage will be “optimize this,” “give me some quick orbitals,” and “show me the vibrations.” Maybe add in “can I see a quick MD run?”

Adding a generic “run” command is somewhat useful, but more likely for people who already know what they want to do.

Yeah I get this, and you’re right that most people will never need the generic Run.... My main motivation was just to avoid people being hamstrung by the extent of my implementation, that they don’t have to wait for me to add explicit options for solvation before they can run a calculation with solvation, for example. This way they can always run whatever xtb can run, even if the output won’t be automatically displayed in Avogadro.

Equally, should the command line usage of xtb ever change, even if everything else breaks, free specification of the CLI command will always still work.

@ghutchis

although toolTips are also supported.

Oh nice, how? Doesn’t seem to be mentioned in the docs. Just with "toolTip": string? Does this work for all the widgets?

@ghutchis

One other possibility would be an editable combo menu (eg., "type": "stringList") that offers a few examples, but allows free-form text entry?

Hmm, maybe that could work. Though those sorts of fields tend to suffer from it not being at all obvious that the text can be edited even after selecting.

@ghutchis

Or more like:

instructions: Type a command for xtb .. see https://xtb-docs.readthedocs.io/

Inspired by this I thought I’d simply put the url in the text field label. It actually looks adequate (for now) but the link is neither clickable (hard to implement) nor selectable with the cursor so copy+paste is not possible.

@matterhorn103

Maybe an alternative would be a QButton that can be set to run another command script like docs.py, which is what is run by clicking xtb Help but that seems unnecessarily complex.

Having thought about it more I actually think this would be very nice. It would be the easiest to implement on the Avogadro side as all the QButton needs is text and the name of a Python file to execute in the exact same way as when menu options are selected, no? While on the scripting side, it would really make a lot of interesting functionality possible.

Regardless, there is a menu option for xtb Help which I think is fairly obvious and opens the docs in the browser. So there is no immediate need for anything else, I’m just spitballing really.

[matterhorn103]

I found another use case for a simple QLabel, by the way – it would be nice to alert the user to the fact that changes made in the plugin’s Configure... dialog won’t diffuse to the other dialogs like Run... or Conformers... until Avogadro restarts. The only other way to do that at the moment would be with a popup using the functionality provided by "message", but I would like to avoid an extra popup appearing every single time the user changes something.

[matterhorn103]

@matterhorn103

Couple of problems for the frequency calculation are:

1. openbabel seems to convert the Gaussian format frequencies file produced by xtb incorrectly – all the atoms are present in the coords, all the frequencies are there, but atom 1 gets dropped from all the entries "eigenvectors" section. I’ve attached the results of doing a frequency calculation on acetone: output.out and g98.out from xtb and the freq.cjson generated by openbabel. If you load the cjson, you’ll see the strong mode at 1700 cm^-1 is not moving quite the atoms one would expect…

I’ve identified the problem in the openbabel code and opened a bug report for it.

[ghutchis]

@matterhorn103

I’ve identified the problem in the openbabel code and opened a bug report for it.

Yes, I think it’s also this one by @arugula_jones Animation Jitter Freq. Modes · Issue #1445 · OpenChemistry/avogadrolibs · GitHub

[matterhorn103]

Having looked at the log file there, I’m not sure it is. The problem is stemming from openbabel skipping too many lines if a line for Ramen activity is present (see GitHub issue). @arugula_jones’s log file doesn’t have a Ramen Activ line, so it parses correctly, and converting it with obabel gives a .cjson with the correct number of displacement floats for each vibration.

[ghutchis]

Okay, I fixed your bug: Fix parsing "Gaussian" vibration output from xtb program by ghutchis · Pull Request #2653 · openbabel/openbabel · GitHub

There’s a bunch of Open Babel work, and I don’t know when I’ll get to all of that.

I also fixed the other bug - the problem was that optimizations show a trajectory which wasn’t being cleared properly for the vibration animations.

[matterhorn103]

This is amazing and I am very grateful for such a fast fix! The vibrations now look like they should

It seems now also the Raman activity is written to the cjson output. Unfortunately this looks to have a new bug: the Raman intensities are simply a direct copy of the IR intensities. They should all be zero for all the examples I uploaded. Can you reproduce this?

Since Raman output has been removed permanently from xtb due to its inaccuracy, the results will always be 0, so I’ll change the plugin code anyway to make sure the Raman intensities get removed from the .cjson before passing it back. It would be misleading to show the user all 0s when some of the frequencies are Raman-active.

[ghutchis]

@matterhorn103

They should all be zero for all the examples I uploaded. Can you reproduce this?

Yeah, coding at night clearly leads to

That’s fixed. As a bonus, actual Gaussian calculations for Raman should actually work now.

[matterhorn103]

Not far off having the MD implementation done! Is there a reason the float fields created using "userOptions" are limited to a max value of 99.99? I need the temperature in K…

Here is an example of the output for acetone from a 1 ps MD simulation. The results should be viewable with the animation tool, right? Because it’s not working for me, either by passing back the results with the script or by directly loading the resulting cjson. Would you mind taking a quick look at it and seeing what the issue is?

result.cjson.txt (956.5 KB)

[ghutchis]

@matterhorn103

Not far off having the MD implementation done! Is there a reason the float fields created using "userOptions" are limited to a max value of 99.99? I need the temperature in K…

Did you set maximum (and minimum) for that field? There’s also suffix if you want to add “K” to the field.

The problem with your result file is that the 3dSets are arrays of strings:

      "3dSets": [
        [
          "-5.83383888405905",
          "0.08166487209162",
          "0.01438853881845",

not

      "3dSets": [
        [
          -5.83383888405905,
          0.08166487209162,
          0.01438853881845,

Not sure how you’re handling them, but should be an easy fix.

[matterhorn103]

Obviously where was syntax highlighting when I needed it…

@ghutchis

Did you set maximum (and minimum) for that field? There’s also suffix if you want to add “K” to the field.

I set minimum for several float fields to make sure they are positive values but never maximum. It seems setting the maximum value to something higher lets it work. And yes, I had seen, the suffix thing works very nicely, thanks.

[ghutchis]

@matterhorn103

Inspired by this I thought I’d simply put the url in the text field label. It actually looks adequate (for now) but the link is neither clickable (hard to implement) nor selectable with the cursor so copy+paste is not possible.

Okay, there’s now a “text” type for adding a text field. I’m wondering if QTextEdit is a better choice (e.g., it includes support for Markdown) … and just make the default non-editable.

[matterhorn103]

Nice, thanks! Would Markdown allow for a link and/or selection then copy/paste of the text?

Also currently the text item receives a colon on the end like with all the other labels, which is not really desirable for a textbox. Or should I have used something other than "label" to put in the text?

[ghutchis]

@matterhorn103

Also currently the text item receives a colon on the end like with all the other labels, which is not really desirable for a textbox. Or should I have used something other than "label" to put in the text?

label is always going to be the part of the form on the left. (Hmm, I should check for right-to-left languages)

default is always going to be the element on the right of the labels.

You want to add something like:

{
    "help" : {
       "type": "text",
       "label": "Help",
       "default" : "Whatever you want to say"
    }
}

[matterhorn103]

So the new ability to order elements in the menus and dialogs is very nice, thank you Makes the xtb plugin easier to use and much slicker looking.

Unfortunately the relevant PR #1503 at the same time introduced a slightly odd bug whereby if the first item in a dialog (set by "order") is a field of type "string", the value set by "default" is wiped and the text box is displayed as empty. Drop down "stringList"s also get reset to the value with index 0 if they are the top object. If the type is "text" then the contents are simply not displayed i.e. presumably they are replaced with an empty string. This only happens to the first object.

[matterhorn103]

I feel like the avo_xtb plugin now does a pretty decent job at providing the basic functionality that you suggested desirable for an xtb interface i.e. the things in your list on the GitHub issue.

Did you have any further thoughts on how, or more importantly, whether to integrate xtb any deeper into Avogadro and make it effectively part of the software in the same way as Open Babel?

Based on your previous comments I imagine your preferred solution would be to keep it as a plugin, am I right?

This of course doesn’t necessarily preclude having the plugin, even simply in its current state, included as part of the default Avogadro installation. I don’t know if that is desirable to you?

An issue was the question of whether we should be bundling xtb with Avo, or relying on users to manually download it, or to bundle it with the plugin. The latter either requires a script that executes automatically, or xtb to be a dependency that the plugin manager downloads once that functionality is there. That would only work for conda environments though.

I have coded a fourth alternative that is possibly a fairly elegant solution. If an xtb binary wasn’t found on launch, two things are different. Firstly, the various calculation commands are not shown, to tip off the user that something is missing. Secondly, there is a new menu option for “Get xtb…”.

This option installs the binaries from the Grimme repos, after first bringing up a dialog stating that clicking OK will download xtb (and crest on Linux), stating the URLs it will use, and offering the option to customize the install location (which defaults to the plugin directory, or ~/avo_xtb depending on permissions).

I figured this allows a fairly seamless experience for users who don’t already have xtb.

I haven’t pushed that to the main branch of the plugin yet, as I wanted to check if you think it’s a good solution. You can have a look at the code in this branch. Not sure how you feel about plugins downloading things from the web. Or do you have better/more far-reaching ideas than this one anyway?

If you were keen to promote xtb, a similar approach could be adopted to encourage use/increase visibility of the availability of the plugin: Avogadro could show a menu option for semi-empirical calcs/xtb, even if the avo_xtb plugin isn’t installed, with only one option in the sub-menu: “Get the xtb plugin…”, that instructs the plugin manager to download the plugin.

[ghutchis]

@matterhorn103

This of course doesn’t necessarily preclude having the plugin, even simply in its current state, included as part of the default Avogadro installation. I don’t know if that is desirable to you?

I think the plugin is great in its current state, and will be more useful once the code for installing requirements.txt / handling environments is in place.

What I’ve noticed in previous releases is that users get very confused by scripts. For example, I included partial charge scripts for GFN2 and antechamber AM1BCC and of course few people have those installed.

As part of the new forcefield framework, any warnings from scripts are similarly confusing.

So I think my answer is “any Python script that requires an additional non-standard Python package shouldn’t be installed by default.”

I’m happy to discuss how to integrate xtb into packaged builds like AppImage / Windows / Mac. The latter one will take a bit of work, but I have some ideas. I think that’s a better way to go vs. “Get xtb…” (And yes, having it fully integrated is better than as a plugin … just that the plugin is easier to finish.)

Beyond that… I’m just busy. So please don’t take silence as some indication.

[matterhorn103]

@ghutchis

So I think my answer is “any Python script that requires an additional non-standard Python package shouldn’t be installed by default.”

Reasonable.

@ghutchis

(And yes, having it fully integrated is better than as a plugin … just that the plugin is easier to finish.)

Ok, great to hear. Inclusion in Avo by default (at least as you say for packaged builds) could be very cool.

@ghutchis

I think that’s a better way to go vs. “Get xtb…”

Fair. For the time until then I will add that feature to the plugin though, unless you have objections to plugins being able to download stuff from URLs.

@ghutchis

Beyond that… I’m just busy. So please don’t take silence as some indication.

I absolutely don’t. I often wonder how on earth you fit professorial duties in on top of Avogadro maintenance and support. And besides, I ought to be busy myself. I’m sorry that you see a post or issue or pull request from me most days, I’m aware it might be a little relentless

[ghutchis]

@matterhorn103

For the time until then I will add that feature to the plugin though, unless you have objections to plugins being able to download stuff from URLs.

I’m guessing the approach is to open a URL. That’s fine. I think we want to be very careful about allowing plugins to install things behind the scenes.

@matterhorn103

I’m sorry that you see a post or issue or pull request from me most days

No, it’s great. Good ideas, discussion, bug reports, etc. all make it better - that’s the whole idea of open source science. Be relentless.

[matterhorn103]

@ghutchis

I’m happy to discuss how to integrate xtb into packaged builds like AppImage / Windows / Mac. The latter one will take a bit of work, but I have some ideas. I think that’s a better way to go vs. “Get xtb…” (And yes, having it fully integrated is better than as a plugin … just that the plugin is easier to finish.)

Given the concerns over the inflationary effects of Python on Avogadro’s footprint, I wonder if the fact that xtb + crest are 87 + 28 MiB (on my system) affects how keen you are to bundle them…