Running xtb via Avogadro on mac - Nothing happens

Plugins
1

Hi,

I am trying to play with the xtb plugin for Avogadro 1.99.0 on my Mac (M1, 14.5). After installing the plugin and manually downloading xtb via homebrew I saw the Semi-Empirical QM (xtb) option appear under Extensions. However, there were no options to calculate something and Avogadron would forget the path to the xtb binary that I manually set at Extensions>Semi-Empirical QM (xtb)>Configure everytime I closed that menu.

I fixed those problem by manually fixing the path to the xtb binary by editing the config.json file and adding:
ā€œxtb_binā€ : ā€œ/opt/homebrew/bin/xtbā€
Now it remembers the path to the binary and I can see. the energy, optimize, etc. options under Semi-Empirical QM (xtb).

When I then load a molecule and click optimise nothing happens, however. Looking at the easyxtb/calcs/last directory the only thing is that an input file is created there.

The easyxtb/log.log file has the following content (where I editted the path before printing it here):
easyxtb.geometry:205: Instantiating a geometry from a cjson
main:39: avo_xtb is requesting a geometry optimization
easyxtb.calculate:132: New Calculation created for runtype opt with xtb
easyxtb.calculate:211: Calculation of runtype opt has been asked to run
easyxtb.calculate:212: The calculation will be run in the following directory: /Users/XXXX/Library/Application Support/easyxtb/calcs/last
easyxtb.calculate:226: Saving input geometry to /Users/XXXX/Library/Application Support/easyxtb/calcs/last/input.xyz
easyxtb.geometry:107: Saving the geometry as an xyz file to /Users/XXXX/Library/Application Support/easyxtb/calcs/last/input.xyz
easyxtb.geometry:58: Generating an xyz, as a list of lines, for the geometry
easyxtb.calculate:233: Working directory changed to /Users/XXXX/Library/Application Support/easyxtb/calcs/last
easyxtb.calculate:254: Calculation will be run with the command: ā€¦/Cellar/xtb/6.7.1/bin/xtb --gfn 2 -p 6 --opt normal ā€“ /Users/XXX/Library/Application Support/easyxtb/calcs/last/input.xyz
easyxtb.calculate:257: Running calculation in new subprocessā€¦
Suggesting that something should have been happening but isnā€™t. Itā€™s odd how it claims xtb is running but no files are produced not even ones containing errors

Runnig the same input file with the same options directly with the xtb binary from the command line works without problems so the problem is not the binary.

Thanks,

Martijn

1 Like
2

Note added in proof, I am wondering if this issue has something to do with the separate forcefield optimisation issue noted by avervit (Faulty geometry optimisation in the new build) as for themac excecutable also only the LJ forcefield results in the moving of atoms. With all other forcefield, when used directy via the Optimize Geometry option instead of via openbabel, the same happens as for xtb, exacly nothing.

3

No, that would be entirely different things.

Are you using the 1.99 release or a nightly build? If the former, Iā€™d strongly suggest trying a nightly build since weā€™re about to release 1.100.

Iā€™d leave it to @matterhorn103 - I guess my initial question is about the path, since ā€¦/Library/Application Support/ has a space, but Iā€™ll see if I can reproduce this with my MacBook.

4

Hi!

Firstly, to confirm - this is v0.8.1 of the plugin, right?

Pretty sure this is the correct answer. xtb would then indeed not produce any output files because it would immediately fail with a parsing error. Obviously paths with spaces should be quoted. Iā€™ll see if I can sort that asap.

5

This was the 1.99 release combined with what I assume is the 0.8.1 version of the plugin (I downloaded the plugin last night). I can try the daily build later.

6

Thanks. I assume it is the 0.8.1 version of the plugin, I downloaded the plugin last night and if I would redownload now it would download 0.8.1 but I donā€™t know where you see the version of a plugin thatā€™s already installed.

Yes, thereā€™s indeed a space in the path to the calculation directory. I have tried inserting a backslash after Application in the path to escape the space like you would do in the shell but that does not only not work, it actually means that the options under Semi-Empirical QM (xtb) dissapear. For what it is worth without doing anything the input.xyz file is created in the correct place and the ā€œGo to Calculations Filesā€ menu option opens the calculations directory, suggesting part of the plugin at least is handeling the path with space correctly.

7

Yeah ok, it will be then :slight_smile:

There isnā€™t a general way for all plugins but for reference (and for anyone who might read this later) the last entry in the pluginā€™s menu (the menu under Semi-Empirical QM (xtb)) is About avo_xtb which brings up a dialog showing the plugin version and the versions of the various other bits of software that the plugin relies on.

Yeah, assuming you inserted the backslash in the JSON file, this wonā€™t work because backslash is used for escape characters like \t in JSON and \ isnā€™t a valid escape character, so trying to read it causes an error. But wasnā€™t a bad idea to try it, thanks.

The commands that require xtb are only shown when an xtb installation can be located, and similarly for the crest commands. So the commands being absent is generally an indication that the xtb binary hasnā€™t been found, for whatever reason ā€“ usually because thereā€™s none installed, but in this case because the user-specified path isnā€™t valid (because of the backslashes).

Yeah, in general the path handling is correct because itā€™s all handled internally by the pathlib module so itā€™s robust and cross-platform. But thatā€™s reassuring to know, thanks. The point where it breaks down is when xtb is evoked on the command line.

In reality it seems that when using Pythonā€™s subprocess the command is not executed as a single string but passed to the process execution API of the OS as a list of arguments, so an argument containing a space shouldnā€™t need to be escaped in order to be interpreted as a single argument rather than two. I need to look deeper into this.

8

The plugin doesnā€™t currently specifically check any locations for the binary other than under itā€™s own folder i.e. ~/Library/Application Support/easyxtb/bin on Mac. If thereā€™s nothing there, it checks the PATH. I guess /opt/homebrew/bin isnā€™t in your PATH i.e. you canā€™t just type xtb on the command line but instead need to use the full path to the program?

9

I found part of the problem and hopefully a temporary workaround for @martijnzw

Since Iā€™m constantly building, I usually launch from the terminal, which of course has my PATH set (e.g. for conda). Everything was working, but I tried /opt/homebrew/bin/xtb in case thereā€™s some minor difference.

Thereā€™s currently a bug in the configure dialog. When I set the path, I get:
"Error: value must be object for key 'xtb_bin'."

I also get:
("Parse error at offset 1: 'illegal value'\nRaw JSON:\n\nTraceback (most recent call last):\n File \"/Users/xxxxxx/Library/Application Support/OpenChemistry/Avogadro/commands/matterhorn103-avo_xtb-06be567/avo_xtb/config.py\", line 248, in <module>\n update_config()\nTypeError: update_config() missing 1 required positional argument: 'avo_input'\n")

If I run a copy from /Applications I canā€™t set the xtb path at all ā€“ it wonā€™t save between runs. (Itā€™s non-trivial to set the PATH for a Mac app.)

As for the workaroundā€¦ @martijnzw - can you try launching Avogadro from the Terminal? For example, just run /Applications/Avogadro2.app/Contents/MacOS/Avogadro2 from a command prompt and youā€™ll see all sorts of console debugging statementsā€¦

On the plus side, it should pick up the xtb from your PATH.

10

Thanks. Some replies:

-) Homebrew adds xtb to the path. I can run xtb from the command line by just typing xtb. No need to provide the whole path.

-) Running xtb in the ā€¦/calcs/last directory when starting from there works fine and without problems.

-) Running xtb in the ā€¦/calcs/last directory when starting from somwhere else and hence having to provide a path the directory, i.e. doing xtb --gfn 2 -p 6 --opt normal ā€“ /Users/XXXX/Library/Application Support/easyxtb/calcs/last/input.xyz, does not work. xtb dies with a segmentation fault, similar as what it would do when there was no structure provided to xtb.

-) Running xtb in the ā€¦/calcs/last directory when starting from somwhere else and hence having to provide a path the directory but escaping the space with a slah, i.e. doing xtb --gfn 2 -p 6 --opt normal ā€“ /Users/XXXX/Library/Application\ Support/easyxtb/calcs/last/input.xyz, does work. xtb runs as normal.

1 Like
11

Thanks Geoff. These are the debug errors that appear after I click on Optimize in the Semi-Empirical QM (xtb) menu:

(ā€œParse error at offset 1: ā€˜illegal valueā€™\nRaw JSON:\n\nTraceback (most recent call last):\n File "/Users/martijn/Library/Application Support/OpenChemistry/Avogadro/commands/avo_xtb/avo_xtb/opt.py", line 104, in \n output = opt(avo_input)\n ^^^^^^^^^^^^^^\n File "/Users/martijn/Library/Application Support/OpenChemistry/Avogadro/commands/avo_xtb/avo_xtb/opt.py", line 40, in opt\n opt_geom, calc = easyxtb.calculate.optimize(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/martijn/Library/Application Support/OpenChemistry/Avogadro/commands/avo_xtb/easyxtb/src/easyxtb/calculate.py", line 449, in optimize\n calc.run()\n File "/Users/martijn/Library/Application Support/OpenChemistry/Avogadro/commands/avo_xtb/easyxtb/src/easyxtb/calculate.py", line 258, in run\n subproc = subprocess.run(command, capture_output=True, encoding="utf-8")\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 548, in run\n with Popen(*popenargs, **kwargs) as process:\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 1026, in init\n self._execute_child(args, executable, preexec_fn, close_fds,\n File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 1955, in _execute_child\n raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: ā€˜ā€¦/Cellar/xtb/6.7.1/bin/xtbā€™\nā€)

When I set the path by editting the config.json file then it remembers it. It shows correctly when I go the configure option of the Semi-Empirical QM (xtb) menu. However, even though the explicit path is there correctly itā€™s not being used when trying to run xtb. Instead it tries to use the ā€¦/Cellar/xtb/6.7.1/bin/xtb version that does not exist and where I do not know where that comes from. Starting Avogadro from the command line seems to makes no difference.

It seems to me there might be two issues: (i) an issue with the path to the xtb excutable where the path set or obtained from the environment is ignored, and (ii) an issue arising from the space in the path of the default calculation directory.

12

Thanks for the info, I could also reproduce the issue on Linux.

Edit: Well I could, now itā€™s suddenly working fineā€¦

13

The path you are providing manually (/opt/homebrew/bin/xtb) will be a symlink to ā€¦/Cellar/xtb/6.7.1/bin/xtb, since this seems to be how Homebrew behaves, see here. The plugin resolves symlinks to absolute paths.

14

Well, I was thinking space because ~/Library/Application Support/ā€¦ on Mac is famous for tripping up Linux tools because of the space.

provide a path the directory, i.e. doing xtb --gfn 2 -p 6 --opt normal ā€“ /Users/XXXX/Library/Application Support/easyxtb/calcs/last/input.xyz

Right. When youā€™re running via the terminal, you do need to provide the path in quotes or escape the space. As @matterhorn103 indicated, the Python script handles this already.

So on the command-line, youā€™d want

xtb --gfn 2 -p 6 --opt normal -- '/Users/XXXX/Library/Application Support/easyxtb/calcs/last/input.xyz' # with the quotes

As far as the homebrew pathā€¦ This is actually how brew works to accommodate multiple versions. It installs things into /opt/homebrew/Cellar/package/version/ and then uses symlinks as needed.

15

Ah yes, Iā€™ve found the origin of that. Itā€™s an easy fix, so itā€™s sorted.

1 Like
16

@martijnzw Ok so starting with the latest Avogadro and installing 0.8.1 of the plugin from fresh and after having deleted <user data>/easyxtb so that my config and everything is completely reset:

  1. I created a directory in my home folder called ā€œtest folderā€ i.e. /home/matt/test folder
  2. I then created a ā€œcalcsā€ folder within that
  3. I put an xtb (6.7.1) binary into the folder (within the xtb-dist folder you get when you download xtb from GitHub)
  4. As configuring things via the configuration menu is currently broken by the bug mentioned above, I edited <user data>/easyxtb/config.json manually to look like this:
{
  "solvent": null,
  "method": 2,
  "opt_lvl": "normal",
  "n_proc": null,
  "calcs_dir": "/home/xxx/test folder/calcs",
  "xtb_bin": "/home/xxx/test folder/xtb-dist/bin/xtb",
  "version": "0.8.1"
}

The result of this is that both the path of the xtb binary and the path of the directory used to run calculations in (and therefore where the calculation files are) have spaces in them, just like yours.

After restarting Avogadro, everything works fine despite the spaces, and calculations work as expected. So essentially I canā€™t reproduce your bug, which is puzzling.

The xtb output file contains the section (~ line 100)

           -------------------------------------------------
          |                Calculation Setup                |
           -------------------------------------------------

          program call               : /home/xxx/test folder/xtb-dist/bin/xtb --gfn 2 -p 9 --opt normal -- /home/xxx/test folder/calcs/last/input.xyz
          hostname                   :
          coordinate file            : /home/xxx/test folder/calcs/last/input.xyz
          omp threads                :                    12

so xtb doesnā€™t actually seem to mind the space.

The plugin/easyxtb log file contains:

easyxtb.geometry:205: Instantiating a geometry from a cjson
__main__:39: avo_xtb is requesting a geometry optimization
easyxtb.calculate:132: New Calculation created for runtype opt with xtb
easyxtb.calculate:211: Calculation of runtype opt has been asked to run
easyxtb.calculate:212: The calculation will be run in the following directory: /home/xxx/test folder/calcs/last
easyxtb.calculate:226: Saving input geometry to /home/xxx/test folder/calcs/last/input.xyz
easyxtb.geometry:107: Saving the geometry as an xyz file to /home/xxx/test folder/calcs/last/input.xyz
easyxtb.geometry:58: Generating an xyz, as a list of lines, for the geometry
easyxtb.calculate:233: Working directory changed to /home/xxx/test folder/calcs/last
easyxtb.calculate:254: Calculation will be run with the command: /home/xxx/test folder/xtb-dist/bin/xtb --gfn 2 -p 9 --opt normal -- /home/xxx/test folder/calcs/last/input.xyz
easyxtb.calculate:257: Running calculation in new subprocess...
easyxtb.calculate:259: ...calculation complete.
...

Can you try to specify both calcs_dir and xtb_bin manually in config.json and see if the problem persists?

As a final thought: what version of Python do you have?

17

I donā€™t understand why these ellipsis characters ā€¦ are coming from?

18

Weā€™re just skipping some parts of the path (e.g., usernames) or in this case /opt/homebrew

19

Thanks for all your help. I got it to work but I donā€™t understand why. I simply replaced ā€œ/opt/homebrew/bin/xtbā€ in config.json with ā€œ/opt/homebrew/Cellar/xtb/6.7.1/bin/xtbā€ and now it worksā€¦ The former is a symbolic link to the latter so no clue what the difference is but Iā€™m not going to look a gifted horse in the mouth.

Is it correct btw that when choosing optimisation the final energy does not get reported back in Avogadro? If you run energy thereā€™s pop up window with the energy but not so in the case of optimisation I think (you can obviously run a single point on the optimised geometry).

1 Like
20

I know you are!

But I suspect that @martijnzw didnā€™t mean to put an ellipsis, but because they are posting output as normal text rather than in code blocks, that the forum is turning .. into ā€¦

This would be important because it would mean the plugin is trying to work with relative paths, which it shouldnā€™t be.

@martijnzw Iā€™d appreciate it if you could save and run the following Python script for me:

from pathlib import Path
from shutil import which

bin_path = Path(which("xtb"))
print(bin_path.__repr__())
print(f"{bin_path =}")
if bin_path.is_symlink():
    resolved = bin_path.readlink()
    print(resolved.__repr__())
    print(f"{resolved =}")

If you can then post the output as a code block please thatā€™d be great ā€“ to do that surround the text with backtick ` characters e.g.

```
../some/path
```

gets formatted as

../some/path

instead of

ā€¦/some/path