amling search program principles discussion / brain dump

amling · Post by **amling** » March 17th, 2026, 3:27 am

I've added --pre-partials to most normal search mains which captures (and allows changing) steps that run and display pattern bits before each expansion (these are the "unique", "first", "thinnest", and "random" partials). "--help" can show the default values, which differ by search mode.

I did a bunch of work to dust off wrap_hax, fixing all the disabled --filters options and wildly improving performance in ways that the one-bit-at-a-time changes unlocked. I've also sketched reworking it to operate via MDSE V2 (the "bit tree" stuff) and some very provisional benchmarking is suggesting it'll be maybe ~8-9x slower and use maybe ~4x less memory. Considering my most recent run of wrap_hax has been running for months, buried ~3.5 TB into swap and going nowhere on my fanciest machine (and hasn't so much as printed another log line for over 3 weeks now), I'm hopeful this might be useful.

We could also at this point pursue MDSE V2 for recentering or even a hypothetical MDSE V3 removing jcols entirely (presumably sketching for fixed board first similar to how MDSE V2 was done). Both of those are much, much larger undertakings which is the real reason I went after wrap_hax first.

amling · Post by **amling** » March 18th, 2026, 6:27 pm

I've sketched a rework of LLSSS's parsing of commandline string arguments into their complex in-memory counterparts. This includes a small expression language and a great deal of fun code to write, but for most users I suspect it would be little actual change if shipped.

With my local changes, places that take parts (things like "--ends") all now evaluate the expression language instead but they understand singletons in place of lists and parts with no arguments don't require parentheses to "call" and thus have unchanged syntax.

This means simple things like "--ends bg" work unchanged, but if you want arguments ("--ends 'pd_lite(2)'") or multiple values ("--ends '[bg, odd]'") it gets a little more complex.

The most likely visible changes to users would be...

(*) Fixed search bg magic grid takes an argument and so now "@bg:12" must be instead "@bg(12)". I think this is the only semi-common place requiring arguments or multiple values. I guess something like "--ends odd,even" having to be "--ends '[odd, even]'" is a distant second.

(*) Help is a little wonkier. Previously "--ends borked" would get help, but now it gets an expression language error about not knowing what "borked" is. I added "help" calls so "--ends help" (or even "--ends '[bg, help]'") will get help displayed. I suppose we could dump help on expression language errors, but it seems like it might be a little overzealous (like maybe not appropriate for simple syntax mistakes).

I'm hoping that going forward this would improve things for trying to write parts that take more complex data. As a quick example skew_gutter wants a list of integers but the existing framework is sort of limiting and can't do it. Now instead of "--ends skew_gutter:-1,skew_gutter:1" we could make "--ends 'skew_gutter([-1, 1])'") work.

If I had had this on day one I might have handled input grids differently and instead of burying special ones behind "@" ("magic grids"), it'd all be domain language and you'd have to write something like "file('1.in')" to load a file. This change I guess could still be made but it would obviously be quite disruptive and I suspect a little tedious.

In a more distant future we might even allow expression language steps and be able to implement something like custom state checkpointing logic or custom halting logic directly inline.

I will think on it some more, but my inclination is to ship it. I've pushed it to codeberg as 20260318-dl-15 in case anyone wants to poke around first.

amling · Post by **amling** » March 23rd, 2026, 3:19 am

I've sketched some very rough step-level integration with the custom expression language. For now it's hung on an extra "--extras" phase that goes at the end, although I'm imagining that we'd allow custom steps in any phase ("pre-partials", "filters", "partials", "ends", "halts", although "halts" would likely go away if this ships). As for some examples...

As it is right now you can checkpoint, both with custom conditions and custom path, e.g. here saving a new state file every w_pos. It sort of seems off by one since "Completed" logs the w_pos from when the round started and "$e.w_pos" gets the w_pos at the moment the custom step runs. Here "cb()" means a step with a custom expression language callback. Its single argument is a lambda which gets one argument which is the "environment" object available to such steps. "$e.w_pos" returns the integer w_pos and "$e.save_state(...)" checkpoints the state to a provided path. "str()" is a quick hack I sketched just for this to format an integer to a string. TBD adding something built in to the expression language itself for that.