Pixi and Plugin Environments

Development
1

I’ve been talking a bit with @matterhorn103 about the best way to go forward with plugins:

  • installing Python as needed (e.g., Windows, Mac)
  • installing dependencies for plugins
  • handling both conda-forge and pip
  • ideally handling things in a user-friendly way
  • ideally language agnostic (e.g., works for Julia, R, etc.)

And of course ideally separate environments for different plugins.

I think the best option is pixi - Getting Started - Pixi by prefix.dev

This requires a bit of work for plugin developers (e.g. using a pyproject.toml file): Python - Pixi by prefix.dev

Concerns? Thoughts? Other ideas? Better solutions?

2

One question for Windows users. The webpage suggests it can be installed:

powershell -ExecutionPolicy ByPass -c "irm -useb https://pixi.sh/install.ps1 | iex"

Does this work on all versions of Windows? Assuming the user agrees, can I just run this as a Windows terminal command or is PowerShell something different?

The alternative suggestion is:
winget install prefix-dev.pixi

Again, is winget on all Windows computers? Should that be preferred if winget is available?

3

For migrating from requirements.txt

pixi add --pypi $(cat requirements.txt)

For migrating from conda:

pixi init --import environment.yml

4

You won’t be surprised that I’ve been thinking about this quite a bit for quite a while. I have a fair few ideas on how we can achieve the stated goals going forwards.

I personally still think it is a mistake to try to support dependencies sourced from conda. But if you are determined that we need to allow plugin authors to use dependencies from conda-forge, then I agree, pixi is our best bet.

I think this is important. The current way in which Avogadro interacts with plugins/scripts consists of calling the command line interface with arguments, and this could be used to run a binary/executable plugin in exactly the same way, but currently Avogadro is hard-coded to run everything with Python.

5

I would also like it if we took this final pre-2.0 opportunity to somewhat redesign the plugin API. One thing I am very convinced about, if I may present my proposal, is that Python plugins of the “package” variety (as opposed to individual scripts) should be structured and written so that they are completely normal proper Python packages. They can then be installed by Avogadro into the respective virtual environment.

An entry point would be defined for the package in pyproject.toml, allowing the package to be run from within the environment as if it was an executable.

I think all of Avogadro’s interaction with a plugin package should in future go via that entry point. This means instead of executing individual scripts directly, Avogadro runs the package with appropriate commands, subcommands, and options.

This would have a host of advantages, among them:

  1. It is possible to be much more flexible in the way a plugin is structured internally. I speak from experience when I say that the current system is quite restrictive.

  2. The “package” is no longer just a collection of scripts; this allows it to function internally in a more normal way and internal imports work more sensibly.

  3. There no longer needs to be the whole argument parsing boilerplate for every runnable command.

  4. It would allow us to get away from the reliance on separate folders for different plugin types. Each plugin can tell Avogadro the nature of its various commands on request. Each plugin can contain commands and energies and input generators and so on.

  5. It probably eliminates the need to maintain plugin.json.

  6. Development and testing is easier for the plugin authors.

  7. In theory the plugin could be published to PyPI.

  8. The same API can be used for executable binary plugins.

Taking a plugin called foo as an example, Avogadro would then call it with:

foo info  # Get info on what the plugin offers
foo [category] [options...] [item] [input]  # General form
foo command --menu-path opt  # Get menu path for opt
foo command opt <input>  # Run opt (remove the need for --run-command flag)
foo inputgenerator --print-options orca  # Get options for the ORCA generator
foo inputgenerator orca <input>  # Generate using orca generator (no need for --generate-input flag any more)

Which means what Avogadro would actually run if pixi were adopted would be e.g. pixi run foo command opt <input>.

If Avogadro detected that the plugin was a Python package it’d be run with pixi, otherwise it would just try to run a binary with the same name as the folder.

Plugins adopting this interface would be contained in <user data>/OpenChemistry/Avogadro/plugins/ rather than commands, inputGenerators etc. as currently.

Obviously the exact details are up for discussion, but I would definitely like Python plugins to be proper Python packages.

How to handle scripts would then be something else we’d have to consider, but the way it currently works is honestly not bad. And I don’t think it’s unreasonable to restrict support for dependencies to “package”-style plugins and let simple scripts continue to work.

Thoughts? :slight_smile:

6

I’m certainly open to a different API / approach. What you describe sounds good for more sophisticated packages (e.g., your xtb plugin or perhaps my group’s qupkake model).

For the input generators, for example, we wanted a super-simple script ideally with no dependencies that anyone could write or edit.

So for the commands, we largely adopted that same approach, although of course it’s a bit trickier since most commands want to integrate other dependencies (e.g., xtb or a ML model, etc.)

I think I mentioned elsewhere that it would be great to have “combined” plugins, since some package dependencies offer a bunch of features (e.g., rdkit offers force fields and charges and commands, etc.)

The best way is often to hash out an example and revise along the way.

Dunno. Certainly most of the energy scripts and probably most of the commands will want at least some dependencies. I’d guess a lot of charge scripts too.

Thus, my inclination would be to separate dependency mechanisms from this.

There are just far, far too many chemistry packages that expect conda and a lot of existing conda users.

  • openff
  • antechamber / ambertools
  • tblite / dftbplus
    …

Anyway, I think we’re largely in agreement - if you want to take a stab at an example project / prototype, we can refine.

7

Oh and I forgot to mention that it would also allow plugins to depend on each other, if we permit it.

Ok then, this is fair enough, but I think we should keep simple scripts simple – which to me means that they should retain their self-contained nature and not need to be supported by metadata files.

Happily around a year ago the Python community standardized on a way to specify dependencies within a script using inline script metadata, so I suggest we take advantage of that existing standardized mechanism.

Unfortunately, pixi doesn’t support this, as it’s only really geared towards a project-based or environment-based workflow. This is one thing I like about uv – it makes a great effort to be suitable for all the various workflows people have. Conda ecosystem support is low on their agenda though.

I don’t see that as a huge problem though – parsing the metadata block is fairly straightforward with the regex-based reference implementation and Avogadro can make sure the environment has the packages the script needs.

I suggest that we use a [tool.avogadro] table, both in script metadata for scripts and in pyproject.toml for packages, to store any metadata we need to have access to without executing any code.

In scripts, we can also support putting pixi’s table in our table and then pass it wholesale to pixi.

Adapting the example from the above links, the top of an energy script for Avogadro could then look like this for PyPI dependencies:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "numpy>=2.2.0",
# ]
# 
# [tool.avogadro]
# script-type = "energy"
# ///

import numpy as np
...

or like this for conda ones:

# /// script
# requires-python = ">=3.11"
# 
# [tool.avogadro]
# script-type = "energy"
# 
# [tool.avogadro.pixi]
# channels = ["conda-forge"]
#
# [tool.avogadro.pixi.dependencies]
# numpy = ">=2.2.0"
# ///

import numpy as np
...

In this way we could in theory have all the scripts in a single directory, just like now, and the scripts can be run with pixi exec --spec <dependencies> <script> in isolated – but cached – environments.

If you like the sort of direction I’m thinking about, I’d be very happy to.

8

The Windows Terminal is just a terminal emulator, by the way, so there’s no such thing as a Windows Terminal command.

The two shells on Windows are cmd.exe and PowerShell, and both can be found on all Windows 10 and 11 machines. There is a weird distinction between “Windows PowerShell” (up to v5) and “PowerShell” (currently v7) and I believe Windows only comes with the former by default, we’d have to check the command works with both.

winget is great and preferable, since it almost certainly doesn’t need admin rights (whereas that -ExecutionPolicy ByPass scares me), and is baked in to Windows 10 and 11 (seemingly from 2020 onwards).

9

Great. Yes, “bypass execution policy” seems like a bad idea, so I’d prefer winget

I’d certainly like to see some prototypes. If you’re willing to work on that, I’ll work on some separate things.

1 Like
10

They’re functionally identical, the newer powershell is slightly nicer to use since it remembers common commands.

The -ExecutionPolicy ByPass just allows you to run commands from the internet. It may need admin rights, but I am too untrusting of a user to attempt it. In my experience with winget, it just works exactly how you want it to, and is the preferable alternative.

1 Like
11

Alternatively, is it worth including pixi in the Mac and Windows releases? That is, grabbing the latest release of pixi in the GitHub builds and adding them to the installers?

That way, if the user doesn’t have python installed, we don’t have to first ask to install pixi - we can just ask about installing python and do it for the user (e.g., either global or to an avogenerators environment)

2 Likes