# Full and privileged xtb integration? **URL:** \ **Category:** Development\ **Created:** [November 7, 2023, 7:58pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953 "2023-11-07T19:58:31Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [November 29, 2023, 11:47am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/41 "2023-11-29T11:47:09Z")
> [@ghutchis](#): > > I mean, from my perspective, you want to cover most use cases with separate scripts. It’s the 80/20 or 90/10 rule … most usage will be “optimize this,” “give me some quick orbitals,” and “show me the vibrations.” Maybe add in “can I see a quick MD run?” > > Adding a generic “run” command is somewhat useful, but more likely for people who already know what they want to do. Yeah I get this, and you’re right that most people will never need the generic `Run...`. My main motivation was just to avoid people being hamstrung by the extent of my implementation, that they don’t have to wait for me to add explicit options for solvation before they can run a calculation with solvation, for example. This way they can always run whatever `xtb` can run, even if the output won’t be automatically displayed in Avogadro. Equally, should the command line usage of `xtb` ever change, even if everything else breaks, free specification of the CLI command will always still work. > [@ghutchis](#): > > although toolTips are also supported. Oh nice, how? Doesn’t seem to be mentioned in the docs. Just with `"toolTip": string`? Does this work for all the widgets? > [@ghutchis](#): > > One other possibility would be an editable combo menu (eg., `"type": "stringList"`) that offers a few examples, but allows free-form text entry? Hmm, maybe that could work. Though those sorts of fields tend to suffer from it not being at all obvious that the text can be edited even after selecting. > [@ghutchis](#): > > Or more like: > > `instructions: Type a command for xtb .. see https://xtb-docs.readthedocs.io/ ` Inspired by this I thought I’d simply put the url in the text field label. It actually looks adequate (for now) but the link is neither clickable (hard to implement) nor selectable with the cursor so copy+paste is not possible. > [@matterhorn103](#): > > Maybe an alternative would be a QButton that can be set to run another command script like `docs.py`, which is what is run by clicking `xtb Help` but that seems unnecessarily complex. Having thought about it more I actually think this would be very nice. It would be the easiest to implement on the Avogadro side as all the `QButton` needs is text and the name of a Python file to execute in the exact same way as when menu options are selected, no? While on the scripting side, it would really make a lot of interesting functionality possible. Regardless, there is a menu option for `xtb Help` which I think is fairly obvious and opens the docs in the browser. So there is no immediate need for anything else, I’m just spitballing really. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [November 29, 2023, 12:36pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/42 "2023-11-29T12:36:23Z")
I found another use case for a simple QLabel, by the way – it would be nice to alert the user to the fact that changes made in the plugin’s `Configure...` dialog won’t diffuse to the other dialogs like `Run...` or `Conformers...` until Avogadro restarts. The only other way to do that at the moment would be with a popup using the functionality provided by `"message"`, but I would like to avoid an extra popup appearing every single time the user changes something. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 1, 2023, 12:34am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/43 "2023-12-01T00:34:11Z")
> [@matterhorn103](#): > > Couple of problems for the frequency calculation are: > > 1. `openbabel` seems to convert the Gaussian format frequencies file produced by `xtb` incorrectly – all the atoms are present in the coords, all the frequencies are there, but atom 1 gets dropped from all the entries `"eigenvectors"` section. I’ve attached the results of doing a frequency calculation on acetone: `output.out` and `g98.out` from `xtb` and the `freq.cjson` generated by `openbabel`. If you load the `cjson`, you’ll see the strong mode at 1700 cm^-1 is not moving quite the atoms one would expect… I’ve identified the problem in the openbabel code and opened a [bug report](https://github.com/openbabel/openbabel/issues/2652) for it. ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 1, 2023, 12:52am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/44 "2023-12-01T00:52:09Z")
> [@matterhorn103](#): > > I’ve identified the problem in the openbabel code and opened a [bug report](https://github.com/openbabel/openbabel/issues/2652) for it. Yes, I think it’s also this one by @arugula_jones [Animation Jitter Freq. Modes · Issue #1445 · OpenChemistry/avogadrolibs · GitHub](https://github.com/OpenChemistry/avogadrolibs/issues/1445) ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 1, 2023, 1:19am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/45 "2023-12-01T01:19:10Z")
Having looked at the log file there, I’m not sure it is. The problem is stemming from openbabel skipping too many lines if a line for Ramen activity is present (see GitHub issue). @arugula_jones’s log file doesn’t have a `Ramen Activ` line, so it parses correctly, and converting it with `obabel` gives a `.cjson` with the correct number of displacement floats for each vibration. ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 1, 2023, 3:26am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/46 "2023-12-01T03:26:05Z")
Okay, I fixed your bug: [Fix parsing "Gaussian" vibration output from xtb program by ghutchis · Pull Request #2653 · openbabel/openbabel · GitHub](https://github.com/openbabel/openbabel/pull/2653) There’s a bunch of Open Babel work, and I don’t know when I’ll get to all of that. I also fixed the other bug - the problem was that optimizations show a trajectory which wasn’t being cleared properly for the vibration animations. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 1, 2023, 11:46am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/47 "2023-12-01T11:46:24Z")
This is amazing and I am very grateful for such a fast fix! The vibrations now look like they should 🙂 It seems now also the Raman activity is written to the cjson output. Unfortunately this looks to have a new bug: the Raman intensities are simply a direct copy of the IR intensities. They should all be zero for all the examples I uploaded. Can you reproduce this? Since Raman output has been removed permanently from `xtb` due to its inaccuracy, the results will always be 0, so I’ll change the plugin code anyway to make sure the Raman intensities get removed from the `.cjson` before passing it back. It would be misleading to show the user all 0s when some of the frequencies are Raman-active. ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 1, 2023, 12:33pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/48 "2023-12-01T12:33:41Z")
> [@matterhorn103](#): > > They should all be zero for all the examples I uploaded. Can you reproduce this? Yeah, coding at night clearly leads to 🪲 That’s fixed. As a bonus, actual Gaussian calculations for Raman should actually work now. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 2, 2023, 6:05pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/49 "2023-12-02T18:05:13Z")
Not far off having the MD implementation done! Is there a reason the float fields created using `"userOptions"` are limited to a max value of 99.99? I need the temperature in K… Here is an example of the output for acetone from a 1 ps MD simulation. The results should be viewable with the animation tool, right? Because it’s not working for me, either by passing back the results with the script or by directly loading the resulting cjson. Would you mind taking a quick look at it and seeing what the issue is? [result.cjson.txt](https://discuss.avogadro.cc/uploads/short-url/p5Em263yTPvXZ7VuVelktz2tOXJ.txt) (956.5 KB) ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 2, 2023, 7:17pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/50 "2023-12-02T19:17:20Z")
> [@matterhorn103](#): > > Not far off having the MD implementation done! Is there a reason the float fields created using `"userOptions"` are limited to a max value of 99.99? I need the temperature in K… Did you set `maximum` (and `minimum`) for that field? There’s also `suffix` if you want to add “K” to the field. The problem with your result file is that the `3dSets` are arrays of strings: ```auto "3dSets": [ [ "-5.83383888405905", "0.08166487209162", "0.01438853881845", ``` not ```auto "3dSets": [ [ -5.83383888405905, 0.08166487209162, 0.01438853881845, ``` Not sure how you’re handling them, but should be an easy fix. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 2, 2023, 8:08pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/51 "2023-12-02T20:08:43Z")
Obviously 🤦‍♂️ where was syntax highlighting when I needed it… > [@ghutchis](#): > > Did you set `maximum` (and `minimum`) for that field? There’s also `suffix` if you want to add “K” to the field. I set `minimum` for several float fields to make sure they are positive values but never `maximum`. It seems setting the maximum value to something higher lets it work. And yes, I had seen, the `suffix` thing works very nicely, thanks. ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 2, 2023, 9:31pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/52 "2023-12-02T21:31:08Z")
> [@matterhorn103](#): > > Inspired by this I thought I’d simply put the url in the text field label. It actually looks adequate (for now) but the link is neither clickable (hard to implement) nor selectable with the cursor so copy+paste is not possible. Okay, there’s now a “text” type for adding a text field. I’m wondering if QTextEdit is a better choice (e.g., it includes support for Markdown) … and just make the default non-editable. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 3, 2023, 10:26am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/53 "2023-12-03T10:26:27Z")
Nice, thanks! Would Markdown allow for a link and/or selection then copy/paste of the text? Also currently the text item receives a colon on the end like with all the other labels, which is not really desirable for a textbox. Or should I have used something other than `"label"` to put in the text? ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 3, 2023, 2:59pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/54 "2023-12-03T14:59:12Z")
> [@matterhorn103](#): > > Also currently the text item receives a colon on the end like with all the other labels, which is not really desirable for a textbox. Or should I have used something other than `"label"` to put in the text? `label` is always going to be the part of the form on the left. (Hmm, I should check for right-to-left languages) `default` is always going to be the element on the right of the labels. You want to add something like: ```auto { "help" : { "type": "text", "label": "Help", "default" : "Whatever you want to say" } } ``` ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 7, 2023, 11:24am UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/55 "2023-12-07T11:24:57Z")
So the new ability to order elements in the menus and dialogs is very nice, thank you 🙂 Makes the xtb plugin easier to use and much slicker looking. Unfortunately the relevant PR [#1503](https://github.com/OpenChemistry/avogadrolibs/pull/1503) at the same time introduced a slightly odd bug whereby if the first item in a dialog (set by `"order"`) is a field of type `"string"`, the value set by `"default"` is wiped and the text box is displayed as empty. Drop down `"stringList"`s also get reset to the value with index 0 if they are the top object. If the type is `"text"` then the contents are simply not displayed i.e. presumably they are replaced with an empty string. This only happens to the first object. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 12, 2023, 2:41pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/56 "2023-12-12T14:41:17Z")
I feel like the `avo_xtb` plugin now does a pretty decent job at providing the basic functionality that you suggested desirable for an xtb interface i.e. the things in your list on the [GitHub issue](https://github.com/OpenChemistry/avogadrolibs/issues/1447). Did you have any further thoughts on how, or more importantly, whether to _integrate_ xtb any deeper into Avogadro and make it effectively part of the software in the same way as Open Babel? Based on your previous comments I imagine your preferred solution would be to keep it as a plugin, am I right? This of course doesn’t necessarily preclude having the plugin, even simply in its current state, included as part of the default Avogadro installation. I don’t know if that is desirable to you? An issue was the question of whether we should be bundling `xtb` with Avo, or relying on users to manually download it, or to bundle it with the plugin. The latter either requires a script that executes automatically, or `xtb` to be a dependency that the plugin manager downloads once that functionality is there. That would only work for `conda` environments though. I have coded a fourth alternative that is possibly a fairly elegant solution. If an `xtb` binary wasn’t found on launch, two things are different. Firstly, the various calculation commands are not shown, to tip off the user that something is missing. Secondly, there is a new menu option for “Get xtb…”. This option installs the binaries from the Grimme repos, after first bringing up a dialog stating that clicking OK will download xtb (and crest on Linux), stating the URLs it will use, and offering the option to customize the install location (which defaults to the plugin directory, or ~/avo\_xtb depending on permissions). I figured this allows a fairly seamless experience for users who don’t already have xtb. I haven’t pushed that to the main branch of the plugin yet, as I wanted to check if you think it’s a good solution. You can have a look at the code in [this branch](https://github.com/matterhorn103/avo_xtb/tree/install-xtb). Not sure how you feel about plugins downloading things from the web. Or do you have better/more far-reaching ideas than this one anyway? If you were keen to promote xtb, a similar approach could be adopted to encourage use/increase visibility of the availability of the plugin: Avogadro could show a menu option for semi-empirical calcs/xtb, even if the `avo_xtb` plugin isn’t installed, with only one option in the sub-menu: “Get the xtb plugin…”, that instructs the plugin manager to download the plugin. ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 12, 2023, 3:15pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/57 "2023-12-12T15:15:08Z")
> [@matterhorn103](#): > > This of course doesn’t necessarily preclude having the plugin, even simply in its current state, included as part of the default Avogadro installation. I don’t know if that is desirable to you? I think the plugin is great in its current state, and will be more useful once the code for installing `requirements.txt` / handling environments is in place. What I’ve noticed in previous releases is that users get very confused by scripts. For example, I included partial charge scripts for GFN2 and `antechamber` AM1BCC and of course few people have those installed. As part of the new forcefield framework, any warnings from scripts are similarly confusing. So I think my answer is “any Python script that requires an additional non-standard Python package shouldn’t be installed by default.” I’m happy to discuss how to integrate `xtb` into packaged builds like AppImage / Windows / Mac. The latter one will take a bit of work, but I have some ideas. I think that’s a better way to go vs. “Get xtb…” (And yes, having it fully integrated is better than as a plugin … just that the plugin is easier to finish.) Beyond that… I’m just busy. So please don’t take silence as some indication. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 12, 2023, 3:32pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/58 "2023-12-12T15:32:48Z")
> [@ghutchis](#): > > So I think my answer is “any Python script that requires an additional non-standard Python package shouldn’t be installed by default.” Reasonable. > [@ghutchis](#): > > (And yes, having it fully integrated is better than as a plugin … just that the plugin is easier to finish.) Ok, great to hear. Inclusion in Avo by default (at least as you say for packaged builds) could be very cool. > [@ghutchis](#): > > I think that’s a better way to go vs. “Get xtb…” Fair. For the time until then I will add that feature to the plugin though, unless you have objections to plugins being able to download stuff from URLs. > [@ghutchis](#): > > Beyond that… I’m just busy. So please don’t take silence as some indication. I absolutely don’t. I often wonder how on earth you fit professorial duties in on top of Avogadro maintenance and support. And besides, I ought to be busy myself. I’m sorry that you see a post or issue or pull request from me most days, I’m aware it might be a little relentless 😶 ---
**Author:** ![ghutchis](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/ghutchis/32/456_2.png) [@ghutchis](https://discuss.avogadro.cc/u/ghutchis)\ **Post date:** [December 12, 2023, 4:14pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/59 "2023-12-12T16:14:21Z")
> [@matterhorn103](#): > > For the time until then I will add that feature to the plugin though, unless you have objections to plugins being able to download stuff from URLs. I’m guessing the approach is to open a URL. That’s fine. I think we want to be very careful about allowing plugins to install things behind the scenes. > [@matterhorn103](#): > > I’m sorry that you see a post or issue or pull request from me most days No, it’s great. Good ideas, discussion, bug reports, etc. all make it better - that’s the whole idea of open source science. Be relentless. ---
**Author:** ![matterhorn103](https://discuss.avogadro.cc/user_avatar/discuss.avogadro.cc/matterhorn103/32/1751_2.png) [@matterhorn103](https://discuss.avogadro.cc/u/matterhorn103)\ **Post date:** [December 13, 2023, 11:53pm UTC](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953/60 "2023-12-13T23:53:41Z")
> [@ghutchis](#): > > I’m happy to discuss how to integrate `xtb` into packaged builds like AppImage / Windows / Mac. The latter one will take a bit of work, but I have some ideas. I think that’s a better way to go vs. “Get xtb…” (And yes, having it fully integrated is better than as a plugin … just that the plugin is easier to finish.) Given the [concerns](https://discuss.avogadro.cc/t/managing-python/5098) over the inflationary effects of Python on Avogadro’s footprint, I wonder if the fact that `xtb` + `crest` are 87 + 28 MiB (on my system) affects how keen you are to bundle them… [Previous page](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953.md?page=2) [Next page](https://discuss.avogadro.cc/t/full-and-privileged-xtb-integration/4953.md?page=4)