Abstract

Review of Nixmac as an advanced Nix user but casual macOS user.

The goal is to explore whether Nixmac in its current form can be valuable for users that are experienced in Nix.

Process

I bought a refurbished Macbook Air M2. I used the opportunity to use Nixmac during the setting up of the new machine. This allowed me to go through a full onboarding process from scratch without cruft on my computer affecting it (prior Nix installations), etc.

I used a nix-darwin configuration that I was already using on a previous machine.
The configuration lives on this flake, machine tyrell0. Commits from May 3rd are all part of this migration effort. https://github.com/arximboldi/dotfiles/tree/master/nix/os
In installed the download from the the http://nixmac.com website from 3rd of May.
The machine is running macOS 15 (I know, not the latest, but in my recent experience macOS gets worse with every version so I stay on the oldest version possible).

Results

Overall, I had the impression that this could be a very useful tool for the intended use-case. There were however too many issues with it and I would judge the software not ready for release.

Positive remarks

Before we dive into the issues, I would like to highlight a few things that I did like about it.

The concept is sound: I kept wanting to use it actually in the process of setting up the machine, as there are lots of things to install and tweak and Nixmac did indeed feel like it could be a nice way to achieve this.
I was very positively suprised that it was no problem to use my existing configuration setup. At the beginning I feared that it would only work with a git checkout in ~/.darwin following a rigid structure (e.g. flake.nix) at root. I had the formentioned repo checked out in ~/dotfiles and it was no problem chosing nix/os/flake.nix as the path to the flake, which contains the configurations for all my machines, both nix-darwin and NixOS. Nixmac allowed me to chose tyrell2 as the machine configuration without problems.
Overall Nixmac was good at dealing with the changes done manually, it would detect changes properly and update the UI, and allow to revert to the last manual after evolutions. I never lost any manual edits due to playing with Nixmac. This is particularly important for the advanced Nix user (who may edit things manually often) and it was again something pleasing.
I was pleased to notice that it includes a feature that detects configuration changes done in MacOS directly, and suggests to integrate them into the Nix configuration. While I did have problems with the execution of the feature, I think that this is conceptually super cool, and it can be a major selling point for the experienced Nix users. It is probably the major differentiating feature, when talking about advanced Nix users, against something like Claude Code. I would suggest investing more on this.

Evolution issues

The main issue I had with the software is that the AI aspect of it wasn't very good, to such a degree that I rarely could achieve what I was intending using the tool, whereas Claude would give me satisfactory configurations with similar prompts. To systematize the issues:

First, the tool is very slow. The evolution process is so slow, most of the time it just took me less time to go to claude.com, ask for the configuration, and copy paste in my configuration.
But more importantly, most of the time it didn't even produce a result at all. A couple of times it succeeded after I increased "Max iterations" in the settings, but most of the time it didn't succeed even with very large number of iterations (>100, which would take forever anyway).
But other times it would produce wrong results. For example it would produce a configuration that builds, but does not have the intended effect. Or it would change the formatting in very annoying ways.
It doesn't add comments to the Nix code explaining the configuration changes. Claude tends to do this which I think is better.

Example failing prompts

I tried lots of prompts. Here are some of the failing prompts, but many others failed. I didn't keep so much track of the successful ones but you can see them in the Git history for the most part.

Install Slack, Telegram and WhatsApp Result: fail after 18 iterations. Succeeded after I increased the number of iterations to 50, but...
Install <any package> This would create this very annoying issues where the list of installed packages (in https://github.com/arximboldi/dotfiles/blob/master/nix/os/tyrell0/darwin-configuration.nix#L7) would be collapsed into a single line, and the commented out lines removed. This prompt, which could be one of the most useful ones, was rendered useless because of this and I had to just edit the file manually every time.
Make bash the default terminal on the terminal app: No result.
Use the existing devel.nix module from the other NixOS configurations. Make sure to make conditional those parts thay may only work on NixOS. I thought this was a very good use-case for an existing Nix user, I wanted it to integrate some of the configurations I use on Linux into Mac. It failed with no result.
Can you make inconsolata the font in Iterm. No result.
Unbind Ctrl+Space at OS level (I think it does input language or source switching) so that it is passed down to the underlying application. This is important for Emacs, that is currently not receiving Ctrl+Space. Only produced a result when using the deep model, but the result was wrong in subtle ways. Claude produced a good result.

In cases where the configuration was perhaps not working as intended or insatisfactory, "Refinements" sound like a nice feature to iterate with the AI to make the changes work.

However, it was useless, mostly because it seemed completely unaware of the changes the evolution had just proposed, and seemed to just take the current state at face value.

For example, if the evolution had deleted somme commented out lines and I told it: "restore the commented out lines" it would reply something like: "I’ve double‑checked the tyrell0/darwin-configuration.nix file: all of the original commented‑out lines are still present, and the only changes made were the addition of telegram-desktop and whatsapp-for-mac to environment.systemPackages. Any future edits will keep the existing comments intact."

It also seemed unaware of what the original prompt was, for example: I asked it to set up for Syncthing and it installed it as a package. Then I ask "can you enable it as a service via home-manager" and it replies: I don't know which service you mean.

Onboarding issues

I installed Nixmac first thing on the computer. This allowed to me to get a sense of what a new user to Nix would experience, which may be a critical workflow in the future.

It tried to install git first; it which was easy and worked. This was nice.
It would then show an error .darwin is not a git repository or something like that. Clicking on something brought up the Nix installer.
This was the Determinate Systems Nix installer. Sadly the installer failed.
Because of this, Nixmac was stuck showing the .darwin is not a git repository error.

My concerns with this:

When Nix is not installed, it should show a more appropriate error than .darwin is not a git repository, which was misleading.
I doubt the value of using the Determinate Systems Nix. Actually this left a broken installation. It was painful process to finally get a clean Nix installation working.
I prefer the official Nix installer from https://nixos.org, but I understand that this is a bit more low-level / advanced.
For now perhaps, it would be better to not try to auto-install Nix or nix-darwin, but ask the user to do it themselves and link to the various options available.

Other bugs

On error messages there is a "Report error" button that open a form. When clicking "Send report" I would get "Failed to send".
Often building a configuration would fail with a notification: "Bash was prevented from modifying apps on your Mac". I had to press "Allow" and enable "bash" in a settings menu. This would have to be done again after restarting the computer. To be fair, I had a similar issue when running darwin-rebuild from the terminal. It would be nice to somehow have a better workflow for enabling this authorization.
Often the commit message would often be left empty, with a misleading placeholder that said "Loading...".
The "Build" process leaves a result symlink in the git repository, that it commits in the Save page if you're not careful. Eventually I added it to .gitignore but it should just not be there as it's not technically necessary in my opinion.
Once I made a partial commit manually (from the terminal) while Nixmac is in the "(3) Save" or perhaps "(2) Review" page, Nixmac would go back to the Describe page discarding the changes that it had suggested before. This happened when modifying .gitignore to fix the issue above.
When the app is launched on reboot when enabling "Reopen windows when loggin back in", it would reopen in a broken state that said "no nix-darwin configuration found in this folder". This could be fixed by restarting Nixmac.
Pressing "Edit" on changes would fail with: failed to resolve path nix/os/tyrell0/darwin-configuration.nix (os error 2) The file does exist though (inside my git repo).
When the computer goes to sleep mid-evolution and then you come back it seems to fail with "Something went wrong connecting to the AI provider. Please check your connection and try again." Ideally the evolution process should prevent going to sleep so this doesn't happen.

UI issues

The UI contains a transparent background without blur. This means it's hard to read the UI when hovering on top of another window with lots of text.
When editing numbers in the preferences I could not delete the whole text to introduce my own number.
Console view at the bottom seemed broken when opening it during the evolution phase.
Describe -> Refine -> Build -> Undo. Bubble shows response to last refinement in the "describe" window.
When using the scroll gesture the whole page bounces. The page should not be scrollable.
Text can be selected on the page.
Multi-line errors are not shown properly on build failed.

Feature requests

For an Nix user "Show Diff" should be the default in the "Review" page. Perhaps there should be a setting in the preferences to enable this.
"Discard" button on blocks in diff
The "Describe" page shows "8 untracked customizations" which seems to allow adding to the Nix configuration stuff that you've changed elsewhere. This is a killer feature! However:
- It only allows adding all the configurations at once. It must allow selecting them individually for it to be really useful.
- I'm not sure the detection of unapplied changes was 100% correct...
- Most of the time "Applying changes" would get stuck..
When making a configuration change that did homebrew.enable = true the Build failed with: "Activation failed (exit code 1): 0:888: execution error: [1;31merror: Using the homebrew module requires homebrew installed, aborting activation[0m Homebrew doesn't seem to be installed. Please install homebrew separately. You can install homebrew using the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" (2)" It would be nice if Nixmac would suggest and execute the Homebrew installation itself.

Nixmac creates Git tags of the form: nixmac-commit-fcb47e15 polluting tags. I understand this is to be able to more easily identify what has been comitted by Nixmac. Not sure what the right solution is.

Nixmac review