I wrote an bash enumerator because I was sick of xargs

being consistent in a sea of inconsistency is a good thing, no matter how hard you try to paint otherwise. the solution have to start at some point. But the rest, yeah, scary.

Some of these are wrong, as standard shell globs don't traverse directories. For instance the first example should be:

    ls **/*.sh

But yeah, even in POSIX it feels redundant when we already have find, seq, grep and whatnot.

The main reason to use xargs is to split up argument lists before they exceed the limit.

On a second thought I think that the effort in trying to find a shell uncomfortability problem and trying to solve it with a "wrapped workaround" still represents positive effort and should be supported and fostered without jumping directly in 90's like rtfm mode.

especially in the case of a younger profile, learning and putting dev effort, even if vibed or redundant, its still effort and a learning activity!

https://github.com/wallach-game

so I support this even if the result and problem solving can be less significant

> not strictly necessary to use -1 as I believe all common ls detect !isatty(stdout)

I remember, though from the dim an distant past so it could be a long-fixed bug, this not working in at least one circumstance. I've explicitly included -1 in scripted calls to ls since. Of course we are breaking the best practice rule of not trying to parse the output of ls, so problems are not unexpected…

As a generally thing I like to include directives like this in scripted calls even if they happen by default anyway, because is makes my intent clear: I expect the output to being in simple single-column format and the rest will break if it isn't. I even sometimes go as far as specifying --sort=name if nothing else in the pipeline is going to enforce that.

> as long as your xargs has -0 support

I don't think I've encountered an xargs that doesn't have this support for a long time, though I don't work with embedded stuff so maybe there are cut-down versions out there still in active use for space reasons.

The problem I've hit numerous times is wanting to do something between “find -print0” and “xargs -0” and that something not supporting NUL as the item delimiter.

No, there are two problems here:

1. Things have developed over time. Bash and other shells weren't always this way. If you have ever touched different shells, awk, sed and so on and then touch perl you see how it is basically just the glue between the other tools put into one language.

2. The majority of scripts is written in sh or bash. So these shells won't go away anytime soon and any newcomer will be confronted with them. So it's best to know all the edge cases before you do stuff you don't want to do. And yes, you could always install another shell. But that brings another dependency and opens up another can of worms. In professional environments it's not really an option to have the next exotic language.

It's like having a tmux.conf on your local machine that configures tmux exactly the way you want it. Nice to have but the moment you touch any of the other billions of systems out there that run the default configuration, you might be lost because you never learned the default and only use your modified version.

Given the alternative proposed by OP was more of the same but with layers on top, saying "how about just using xargs and sh anyway" is a simplification -- learn 2 tiny things vs start depending on some new tool whose only purpose is to let you avoid learning those 2 tiny things by adding template language, etc., on top? I'll learn the 2.

As for widening the scope to other tools, I've written about that on here before, how I have lots of powershell experience, some nushell, etc. I haven't become a convert (definitely not on Linux, of course occasionally on Windows).

Happy to respond to any input or anything, but you should reply directly to people if you're going to criticize them -- very poor form to blanket criticize while also taking an uncharitable read.

As others have pointed out throughout the thread, the reason we defend the skills is that there’s a high likelihood that these tools will always be available on any machine; PowerShell and nushell, not so much.

Maybe it is out of habit, but I never managed to get into PowerShell, in fact, I am not at easy with the Microsoft way of doing things, with few exceptions. Too much UNIX I guess. Nushell seems to be based on the PowerShell philosophy of using structured data and not text, not my thing.

I really like the UNIX way of using text I/O, it has it flaws but it works for me. But that being said, I still hate bash and all its family. It has so many footguns it is an entire armory at this point, mostly related to spaces and escaping.

Something Perl-like could be a saner replacement. It is already a bit shell-like, it doesn't struggle with escaping the way bash does, and it has very powerful text processing abilities that go well with traditional UNIX tools.

You could also say that English is an awful language, because the spelling and the pronunciation have diverged too far. But it's the international lingua franca in most contexts, because it's the international lingua franca in most contexts.

People use Bash as the default shell for scripting, because people use Bash as the default shell for scripting. If you want to replace it, you should pick a winner and discourage the use of alternatives, especially when they are better than Bash. So don't say "use something more sane, like PowerShell or nushell". Say something like "use PowerShell, or Bash if you really have to for legacy purposes, but never use nushell for any reason" instead.

This is a sleeper option, but 'xonsh' as a Python superset is a very pleasant experience, especially when you just don't feel like learning another bespoke language for the complex stuff.

I once had to scrape the output out of something hacked together by a grad student 10 years before me, scrape yet another hacked together application, do some fancy numerics, and then plonk it into a database. I did it without tears!

Bash is for whippersnappers, us old gits use POSIX sh

I'm sure it's essentially a familiarity thing, but PowerShell and nushell is way too much typing for me.

Switched to Nushell and I am not looking back. I don't see any major reason why we should keep dragging Bash into the twenty first century. Nushell is the first time I feel like I can write complex systems operations in a shell without having to spend either a ton of time in the docs or being at the mercy of an LLM. It is godsend.

Yes, Bash is filled with footguns and is awful due to that. However, it's still incredibly useful as it's everywhere and has a far greater lifespan than almost anything else. You can write scripts in PowerShell or nushell, but then find that twenty years later they're no longer usable or you find a twenty year old machine that won't have PowerShell/nushell installed.

It's not so much about defending arcane scripting skills, but that Bash functions as a lowest common denominator and is useful because of that. If you want something that works reliably over decades, then it's best not to go for an "improved" shell as it may not still be around.

I like to think of Bash script writing as the opposite of riding a bike - you have to relearn it almost every time you write a script.

I was the kind of shell guru whose teeth itched when they saw someone doing `grep | awk`. Then I had to try and bring Windows into a Mac+Linux CI system under Jenkins groovy files that were full of `"""sh` fragments, shell scriptlets wrapping python, etc, etc, and I don't bat (I don't groovy either).

Option 1: Learn to bat and try to translate. Hmm. No. Just no Option 2: ?

Pwsh core had just come out. My immediate thought was that it would be great material for an anti-MS-ragging blog post, but then a line leapt out at me from one article I was glossing over: "... POSIX Terminal Shell Spec ...".

I still wanted that anti-MS-ragging blog material, so I decided to try and use Pwsh as a Rosetta stone until I got to a point I could convert to a real language.

But things just began to click for me. It was like going from Perl to Python - suddenly everything is an object and you can interact with everything* that way.

There's no need for grep or awk or sed in pwsh, because the output of a shell command is an object -- a string (or []byte). It has methods.

(netstat -an).replace("192.168.86.", "10.0.100.")

6 years later, pwsh is the default shell on my Mac, Ubuntu boxes, lab vms, ... everything but my docker containers unless I'm feeling feisty.

Yes, Bash has no place in modern software engineering.

The only reasons people still use it are historical and laziness.

If it was invented today, professionals would cringe at it.

Bash while loops are pretty readable and the above would be a nice to iterate over lines if it wasn't for that gnarly pipe, which is a common source of errors in this construct. Remember that a pipe starts a new shell. So:

  grep stuff file.txt | while read key value ; do [ "$key" = "target" ] && found="$value" ; done

  while read key value ; do [ "$key" = "target" ] && found="$value" ; done < <(grep stuff file.txt)

While FTW! I just wondered whether it could be simplified because I get a bit tired of typing out my own belt and braces version of it:

  { # code that generates one item/filename per line of output }  | { while read ITEM; do # do something with "$ITEM"; done; }

  enum() {
    local source=$1; shift; local action=$1; shift  
    { eval "$source" ; } | { while read ITEM; do eval "$action"; done; }
  }

  $ enum "find /tmp" "echo found: \$ITEM"

  found: /tmp/.XIM-unix
  found: /tmp/.ICE-unix

Depending on the actual command, this can be far slower and less efficient than xargs. You're creating a separate process for each invocation of bar when a lot of commands will take many targets for a single invocation.

Try this with find and grep vs xargs. There's a big difference.

I use "| while read" as well, because it works well in a pipeline and handles embedded spaces. It doesn't handle embedded newlines, but in practice, real files have embedded spaces, while embedded newlines only happen in test cases and exploits. (You can do actual NUL-delimited reads with `-d ''`, but for a quick command-line operation that's generally not necessary, and if you're going to be that careful you probably also need `-r`.)

Yes, I think the author is just showing their ignorance, not actually doing anything about that ignorance, and re-inventing the wheel:

    $ find /path -name "*.pdf" -print0 | xargs -0 -I {} echo "Processing: {}" # handles paths properly, obviates the need to write anything new whatsoever

I just use while’s default variable name, $REPLY most of the time. But `while` is my preferred tool for the xargs problem in most cases.

To support your recommendation and redress the strawman the article postulates, the post's author could have replaced:

  find . -name '*.log' | xargs rm

  find . -name '*.log' -print0 | xargs -0 rm

No dependency on GNU required anymore, POSIX 2024 supports it: https://blog.toast.cafe/posix2024-xcu#the-null-option

For tools that don't support -0, you can add the NULs yourself without much fuss. It's a one-liner in awk/perl/sed. Very handy.

I have used echo as a sort of poor mans equivalent for a safe check of a pipeline, removing the echo when I felt the rest of the pipeline was working correctly.

  shell stuff | xargs -n 1 -I % echo real command and % args

  script

  script | sh

Oh man, parallel is awful. I have to rant about this because I literally tried it again today morning.

Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way.

Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted another command that was supposed to work, and thought surely this would be straightforward. Boy was I wrong. I got some manifesto about academic citations and plagiarism, which confused the hell out of me. After I wasted time trying to figure out how to turn off that nonsense, the app just hung there trying to figure out how long its command line can be? Literally doing nothing? What the hell? I killed it but then my terminal didn't close because every time I did this apparently some perl command was spawned in the background blocked on nothing. Why the hell was perl even relevant? Nothing I wrote used Perl. Just run the darn commands I asked in parallel, is that so hard?

Every time I’ve used Gnu Parallel on a new system it required accepting a Eula, which is annoying. I stick to xargs -P99 these days and am happier.

Utility software which has non-essential different first-run behavior is hostile to users.

isn't parallel that tool that dumps some kind of begging message into your output each time you invoke it?

I love GNU Parallel too. I love how it's free software and how I can patch out the obnoxious citation notice.

Parallel is ok but rush is nicer and much faster.

https://github.com/shenwei356/rush

Same thought I had, and that is where I stopped reading. All of these looping/enumerating techniques, including the weirdness with null-terminating strings to get around the shell's inherent treatment of spacing, are muscle memory for anyone that has spent more than 27 minutes in the shell at any point in the last 30 years. Yes it's weird because it's old. Deal with it. Why make yet another tool with yet another set of quirks?

This is why I tend to stick to POSIX shell scripting these days, avoiding even bash. The restriction does make some things too painful, though. Before the latest POSIX revision, certain workflows were comically hard to do (unless you combine sh with... M4). The 2024 revision injects some much-needed sanity, but I am wondering, will random computers have shells compliant with that revision?

https://imgur.com/a/K9TPYgf

> There is a world of difference between "here's a problem that I'm really concerned with and poured all my expertise to solve it" and "I told Claude to fix it for me and now I'm gonna abandon it as soon as I'm done with the HN advertising"

https://en.wikipedia.org/wiki/False_dilemma

Xargs is fine, but openbsd has a -J option and every time I read the man page to figure out how to use it I read the -I and -J options and my brain glazes over.

https://man.openbsd.org/xargs

Other brain glazing obsd wierdness is it's two argument cd command, a cryptid I am unable to wrap my head around.

https://man.openbsd.org/ksh#cd~2

And for most of those commands add a dash dash so that nothing with a dash prefix turns into options.

why would you ever pipe find into xargs instead of calling -exec?

    find -exec the '{}' iterated command ';'

Shells imo are legacy on legacy, I like bash and zsh, they’re nonsensical but they work. Powershell is undoubtedly saner, but I hate it.

I think the whole concept could be rewritten from the ground up to be modern, but reaching critical mass and mvp is too damn difficult, and the majority of mac/Linux users probably don’t want it.

Related: I wish there were a portable way for xargs to treat newlines as delimeters but not other spaces. Since 99.9% of the time that's what you want. (GNU's has this with -d/--delimiter, but BSD's, and thus macOS's, doesn't.)

--null/-0 is portable and helps, but it means the input has to be NUL-delimited, which is... rare. Sure, find has -print0 but (a) I wish I didn't need to do that, and (b) sometimes it's nice to just pipe `ls` output into xargs, without a `... | while read i; do echo -ne "${i}\0"; done | ...` kludge in the pipeline.

Because spaces in filenames is just common enough to be something that will bite you eventually, but newlines in filenames are a straight evil that you can basically always say is the filename's fault.

I thought -- was typically a separator for passing a list of arguments directly to some subcommand

Was exactly my thought: why not to use null termination?

Looks like a case where reading man page would have spared writing another copycat utility.

Yeah, and I invariably add -r to xargs, so as not to execute the command unless results come through the pipeline.

If one is working with whole lines of text, setting the delimiter to newline is often desirable:

xargs -d \\n

Try not to do that in an uncontrolled file system. Filenames can and will have newlines, including trailing newlines, because evil people like myself inject them everywhere in our local user folders to keep sysadmins on their toes. Use the (now, finally, POSIX) -0/-print0 options for all file parsing which produces the correct behavior on all UNIX systems.

Or, you use `xargs -0` for null termination instead of white space termination. `find` conveniently supports `-print0` that will use null character as separator.

Another trick that took me years to learn is to use `xargs -I` to split the results into "items"

For example

`ps aux | grep process-name | grep -v grep | awk '{ print $2 }' | xargs -Ieach kill -9 each`

This is the way. Works very well.

That won't save you from weird file names, but null termination will.

Hm. Underrated comment

missed the point. reason to use xargs or parallel are generally two: list of arguments is too long, or list of arguments that is kept in memory would take too much

for example

    for a in `find / ` ; do echo $a ; done

along with: I love standards- Theres soany to choose from!

I'm glad you here to save us. What would we do without you?

Have you seen parallel?