This is episode four in my mini-series about shiny new features in the upcoming curl 8.10.0 release.

One of the most commonly used curl command line options is the dash capital O (-O) which also is known as dash dash remote-name (--remote-name) in its long form.

This option tells curl to create a local file using the name from the filename part of the provided URL when downloading. I.e. when you tell curl

curl -O https://example.com/file.html

This command line conveniently creates a local file called file.html in which it saves the downloaded data.

The -O option has been supported with this functionality since curl first shipped, in March 1998. An important point here is that it picks the name from the URL so that a user can tell what filename it creates. No surprises. The remote server is not involved in naming it.

What about no filename scenarios?

URLs do not necessarily need to have filename parts. Like these examples:

http://example.com/
http://example.com/path/
http://example.com/one/two/?id=12345

Since there are no filename parts in these URLs, they used to cause curl to refuse to operate with -O and instead return error. curl could not create a local filename to use:

$ curl -O http://example.com/
curl: Remote filename has no length
curl: (23) Failed writing received data to disk/application

Trying harder

Starting in curl 8.10.0, curl works a little harder to come up with a filename to store the download in when -O is used. While there is no filename part in the URL, the user did ask curl to download the URL to a local file so it now tries a few extra steps:

1. Use the filename part from the URL if there is one, like before.
2. If there is no filename but there is a path provided in the URL, extract the right-most directory name from the URL and use as filename.
3. If there is neither a filename nor a path in the URL, curl uses a default, fixed, filename as a final backup: curl_response. This name intentionally has no extension because curl has no idea what data that will come and using an extension could mislead users into believing it says something about the type of content.

Several people have insisted that index.html would be better and sensible default file name. I cannot agree with that, since it might just as well be an image or a tarball of your favorite open source project. I think naming such a file index.html would be more misleading than simply sticking to the neutral curl_response.

Let me give you a little table showing what filenames that will be used with curl -O and a given set of URLs:

Find out which name

You can use curl’s -w, –write-out option and its %{filename_effective} variable to learn exactly which name that was used.

Prefer another name?

There is always the -o (lowercase o) option that lets you specify whatever filename you like. You do not have to let curl pick the filename for you.

Clobber or not

curl will by default overwrite, clobber if you will, any previously existing file using the same name. If you rather curl took a more careful approach, consider using –no-clobber in your command lines. It makes curl pick an alternative filename if the chosen one already exists when curl is about to download data into a local file.

This is episode three in my mini-series of posts describing news in the coming curl 8.10.0 release. Part one was more help, part two verbose, verbose and verbosest.

This new command line option in curl 8.10.0 is a simple one that has been requested by users repeatedly over the years so I figure it was about time we actually provide it.

If the target file already exists on disk, skip downloading it.

It is exactly as simple as that. No date check, no size check, no checking if the file is even what you want it to be. If the target file is present and exists that is a signal enough that the file should not be downloaded; to skip the transfer.

A real-world command line using this feature could then look like this:

curl --skip-existing --output local/dir/file https://example.com

Or if instead -O is used, it still works the same:

curl --skip-existing -O https://example.com/me.jpg

Easy, right? See the manpage.

Broken files can also be present

To avoid a previous broken download remainder to linger around and cause future transfers to get skipped, remember that curl also has a –remove-on-errror option.

Ships

In curl 8.10.0, on September 11, 2024.

Image

From a movie with a suitable if even perhaps subtle reference.

A key feature for a tool like curl is its ability to help the user diagnose command lines and operations that do not work the way the user intended them to.

When I do XYZ, why does it not work?

The command line option -v and its longer version --verbose have been supported by curl since day one for this purpose. A boolean flag that when used shows what is going on by outputting extra information from the execution.

I need to emphasize the boolean part here. Up until curl 8.10.0, this option was a plain boolean. You either did not get verbose output or you got it. There was no levels or ways to increase or decrease the amount of information shown. Just a binary one or zero. On or off.

But starting in 8.10.0 the story is different.

The world

Meanwhile, there is a universe of additional command line tools out there. Many other tools also offers a -v option for outputting verbose tracing information. In many other tools, the -v is not a boolean but instead you might get additional output if you add more vs. -vv shows a little more, -vvv even more etc.

Users are fairly trained on this. To the extent that we often get to see users use -vvv etc on curl command lines in bug reports etc. The curl command line parser accepts more of them fine (any amount really), but repeating them just enables the boolean again and again to no extra effect.

When we asked users for their favorite command line option in the annual curl user survey in May 2024, a noticeable amount of respondents said -vv or -vvv. Even though they do nothing extra than -v. I think this shows to which extent people are trained to and are used to having these options for command line tools.

Make curl do what you think it did

Therefore.

In curl 8.10.0, coming on September 11, 2024, we introduce support for -vv, -vvv and -vvvv. One, two, three or four vs. (Maybe we add more in the future.)

If you write the v-letters consecutive next to each other like that, you increase the logging level; the amount of verbose output curl shows. It might then possibly do something in the style that many users already expected it to do.

The extra logging it can do in 8.10.0 is actually nothing new, what is new is that you can get to it by simply using -vv and friends. The old style of getting such extra verbose tracing is to instead use a selected combination of –trace-time, –trace-ascii and –trace-config.

Backwards compatibility

In curl we care deeply about backwards compatibility and not breaking users existing scripts and use cases when we ship new versions. This change is perhaps on the border of possibly doing this, so we have tried to tread as gently as we can to make sure that risk is slim.

This is why doing something like curl -v -v will not increase the level, because a user might have one of the switches in ~/.curlrc, another one on the command line and a third one in a custom file read with curl’s -K option etc. We want the extra output level to be explicitly asked for.

Using a single -v after a -vv or -vvv resets the level back to the original lowest-but-enabled for the same reason. The --no-verbose option also still works, even though the option is not strictly a boolean anymore and curl normally otherwise only supports the --no- prefix for boolean command line options.

Credits

Stefan Eissing wrote the pull-request that landed this change.