Linux/UNIX: Argument list too long error for rm, cp, mv commands in a shell error

I‘m a new Linux user. We have a backup server and most our developers sftp or ftp to upload files. I just need to backup of last four weeks. If I try to list or remove directories I get an error cannot remove directory. I’m getting “Argument list too long” message. How do I solve this problem when executing commands in the bash/ksh/csh/tcsh/zsh shell?

Each command under Linux and UNIX-like systems accepts a parameter commonly known as command arguments (or args). For example, the command ‘cd /etc’ has one command line arguments, namely, /etc. Some command can accept more than two arguments and act on supplied args. For example, consider the following cp command:
cp /etc/file1 /etc/file2 /etc/file3 /mnt/pen[donotprint][/donotprint]

The cp command has total four command line arguments. The shell can hold a maximum of 131072 bytes for command line arguments. If you try to pass more than that number you will greeted with an error that read as follows:

Argument list too long

Remember each character requires one byte of storage.

Finding out maximum length of the command line argument

The maximum length of the command line argument is limited to *128k* within the Linux kernel. To find your limits, type:

getconf ARG_MAX

Sample outputs:

2621440

The limit affects the execve() kernel function, which is utilized by all other exec() functions (execl, execlp, execle, etc).

Option #1: How to get rid of “Argument list too long error” problem while using rm, ls, cp, mv or any other shell commands

The best way to deal with this problem is combination of the find and xargs commands. The syntax is:

find /dir/to/search -type f -name 'file-pattern' -print0 | xargs -0 -n 10 command-name
find /dir/to/search -type f -iname "*.EXT" -print0 | xargs -0 command-name
find /dir/to/search -type f -iname "*.EXT" -command

Just list files when you get “Argument list too long” error

Use any one of the following syntax to list only files

find /dir/to/search -type f -iname "*.EXT" -ls
find /nas01/data/ -type f -iname "*.doc" -ls

OR the following portable syntax:

find /nas01/data/ -name '*.doc' -exec ls {} +
## GNU/Find ##
find /nas01/data/ -type f -name '*.doc' -exec ls -l {} +

To avoid recursion use the following GNU find option:

find /nas01/data/ -name '*.doc' -maxdepth 1 -exec ls -l {} +

Just delete files when “Argument list too long” error

The syntax is as follows:

# GNU/find and BSD/find may not work on older *nix
find /dir/to/search -type f -iname "*.EXT" -delete
find ~/Downloads/ -type f -iname "*.avi" -delete
find ~/Projects/ -type f -iname "*.pdf" -delete
## use xargs and rm command ##
find ~/Downloads/ -name '*.avi' -print0 | xargs -0 -n 10 rm -f

More portable syntax:

find ~/Projects/ -name '*.pdf' -exec rm -vf {} +

To avoid recursion use the following GNU find option:

## GNU find and may work with bsd find too ##
find ~/Projects/ -name '*.pdf' -maxdepth 1 -exec rm -f {} +
find ~/Downloads/ -type f -iname '*.avi' -print0 | xargs -0 -n 10 rm -f

You can also use GNU parallel command:

## man parallel for more ##
find ~/Downloads/ -type f -iname '*.avi' -print0 | parallel -X -0 rm -vf

Where,

  • -name "*.pdf" : Search pattern.
  • -type f : Only match files.
  • -print 0 : Avoid nasty surprises when dealing with weird file names with blank spaces, ‘, “, , tab, and newline characters. The -print0 primary in conjunction with xargs -0 as an effective alternative to dealing with weird filenames.
  • -maxdepth 1 : Descend at most one directory levels below the command line arguments i.e avoid recursion.
  • -exec rm -f {} + OR -exec ls {} + : Run rm or ls commands on as many as files possible.

Option #2: Iterate with the shell to avoid arg list too long error

The syntax is as follows and it is very slow and may not work at all:

for f in *; do command "$i"; done     
 
## delete all pdf files ##
for f in *.pdf; do /bin/rm -rf "$f"; done
 
############################################################################
### Delete all *.mp4 and *.avi files using pure Bash solution using arrays
### Based on http://mywiki.wooledge.org/BashFAQ/095 
############################################################################
 files=($HOME/Downloads/*.avi $HOME/Downloads/*.mp4)
 for ((i=0; i<${#files[*]}; i+=100)); do
   /bin/rm -vf "${files[@]:i:100}" 
 done

Command summary to to avoid the limit in a shell

## deals with weird filenames too ##
find /dir/to/search/ -name "file-pattern-search" -exec command {} + 
 
## find and xargs based solution ##
find /dir/to/search/ -name "file-pattern-search" -print0 | xargs -0 command
 
## For gnu parallel fans ##
find /dir/to/search/ -name 'file-pattern-search' -print0 | parallel -X -0 command
 
## older and not recommended solution; but kept here for the historical reasons only ##
for i in *; do command "$i"; done

See man pages for more info: mv(1)

Posted by: SXI ADMIN

The author is the creator of SXI LLC and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.