HTTrack Only Downloads index.html? Causes and Fixes

When HTTrack only downloads index.html, the crawl did not crash. It fetched the start page, found no links it was allowed or able to follow, and stopped. The usual reasons, roughly from most to least common, are these. The start address redirects to https or www, so every link looks like a different site. robots.txt or a user-agent block stops the crawler. The links are built by JavaScript. Or depth, directory and scan-rule settings are too tight. The file hts-log.txt in your project folder usually tells you which one applies, and each cause has a specific fix.

Why does HTTrack only download index.html? The quick diagnosis

First, make sure you are looking in the right place. HTTrack writes its own index.html into the project folder. That file is a project overview page, not the site. The mirrored site lives in a subfolder named after the host, such as example.com/ or www.example.com/. If that host folder contains a single index.html and nothing else, you have the problem this article is about.

Next, open that saved index.html in a text editor rather than a browser. What you find narrows things down quickly:

Read hts-log.txt first: what each message points to

hts-log.txt sits in the root of the project folder. When only the homepage is saved, the log is short, so read all of it. Exact wording varies between HTTrack versions, but these are the lines that matter:

• "File has moved from http://example.com/ to https://www.example.com/" (or another 301/302 notice): the start URL redirected. See Cause 1.
• "Note: due to example.com remote robots.txt rules, links beginning with these path will be forbidden": robots.txt is in play. This only matters if the list includes / or the paths you need. See Cause 2.
• "Error: Forbidden (403)", or 429/503 errors: the server is refusing the crawler. See Cause 2 and the HTTrack 403 Forbidden guide.
• "No data seems to have been transferred during this session": either the connection was refused or your filters excluded everything, including the start page.
• A clean log with no errors and only one or two files: HTTrack found nothing to follow. That points to JavaScript links (Cause 3) or restrictive settings (Cause 4).

If the log is too terse, run again with more detail. On the command line, add -z for extra log information or -Z for debug output. In WinHTTrack, raise the log level under Set options > Log, Index, Cache.

Cause 1: the start URL redirects to https or www, so every link looks external

This is the cause the old forum threads usually miss. By default HTTrack stays on the same address, meaning the exact host you typed. Suppose you enter http://example.com and the server redirects to https://www.example.com/. Depending on the version and settings, HTTrack either saves only the redirect stub, or saves the homepage and then treats every link to www.example.com as external and skips it. Either way you end up with one file.

The fix:

1. Open the site in a normal browser and copy the final address from the address bar, including https://, www. if present, and the trailing slash. Use exactly that as the start URL.
2. Add a scan rule that allows both host variants: +*example.com/*. In WinHTTrack this goes under Set options > Scan Rules.
3. Optionally widen the travel mode from "stay on the same address" to "stay on the same domain" (Experts Only > Travel mode, or -d on the command line). This lets HTTrack follow both example.com and www.example.com.

A command-line version:

httrack "https://www.example.com/" -O "./example-mirror" "+*example.com/*" -d -v

Cause 2: robots.txt, nofollow and user-agent blocking

robots.txt and meta robots

HTTrack obeys robots.txt and meta robots tags by default (-s2). If robots.txt disallows / for all crawlers, or names HTTrack specifically, the crawl ends at the start page. The same happens when the homepage carries <meta name="robots" content="nofollow">. To turn this off, go to Set options > Spider and choose "no robots.txt rules", or use -s0.

Should you turn it off? robots.txt states the site owner's wishes; it is not a technical barrier. On your own site, ignoring it is fine. On someone else's, think about whether you have permission and what you will do with the copy. Whether ignoring robots.txt or copying content has legal consequences depends on your country, the site's terms and your purpose. This article is not legal advice, so ask a lawyer if the stakes are real. Whatever you decide, crawl politely. Limit connections with -c2 and the transfer rate with something like -A100000 (bytes per second).

User-agent and bot blocking

HTTrack's default user-agent string contains the word "HTTrack", and many firewalls and hosting providers block it. The saved index.html is then an error or challenge page with no real links. Set a browser-like identity under Set options > Browser ID, or on the command line:

-F "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"

If you still get 403s, or a "checking your browser" page, the site is using bot protection that HTTrack cannot pass. The HTTrack 403 article covers what you can still try.

Cause 3: links built by JavaScript or hidden behind "load more" pagination

HTTrack reads HTML and tries to spot URLs inside scripts, but it does not run JavaScript. On sites built with React, Vue or Angular, or on sites with menus rendered by scripts, onclick navigation or content fetched from an API, the HTML that arrives contains almost no <a href> links. HTTrack has nothing to follow.

To test this, press Ctrl+U (View Source) on the homepage and search for href=. The browser's Inspect panel shows the page after scripts have run, so it is the wrong place to look. If View Source has no links to subpages, no conventional crawler will find them either. Why those copies often come out blank is covered in Download JavaScript Website Offline.

Why HTTrack gets the category pages but not the articles inside them

A related pattern: the menu and category pages download, but most articles are missing. Usually page one of each category lists ten posts and the rest appear through a "load more" button or infinite scroll, which HTTrack never clicks. Ways around it:

• Look for an HTML fallback. Many CMSs still answer /page/2/ or ?page=2 even when the theme hides those links. Add those URLs as extra start addresses.
• Use the sitemap. Open /sitemap.xml, copy the article URLs into a text file, and give HTTrack the list. Use the URL list field in WinHTTrack or %L urls.txt on the command line.
• Check your filters. A rule like -*?* that excludes query strings also excludes ?page=2.
• Check the directory. If the articles live at /posts/ but you started at /blog/, see the start-directory point below.

Cause 4: depth, scan rules and the "Store HTML files" setting

If the HTML is normal, the links are there and the log is clean, the cause is almost always a setting:

• Mirror depth. The default depth is effectively unlimited, but a depth of 1 (Set options > Limits > Maximum mirroring depth, or -r1) saves only the start page. Clear the field or set a sensible number such as -r20.
• Start directory. By default HTTrack only goes down from the start URL's folder. Starting at https://example.com/blog/post.html means nothing outside /blog/ is followed. Start from the root, or allow upward travel with -B (up and down) under Experts Only > Global travel mode.
• Scan rules. A broad exclusion such as -* or -*.html without matching + rules blocks every subpage. Remove leftovers from earlier projects.
• Primary scan rule. This is the fix buried in old forum threads. Under Experts Only > Primary Scan Rule, make sure HTML is actually stored: choose "Store all files (default)" or "Store html files", never "Just scan" or "Store non html files". On the command line that is -p3 (all files) or -p1 (HTML only).
• Stale cache. If you chose "Continue interrupted download" or "Update existing download", HTTrack reuses hts-cache and may skip pages it thinks it already has. Delete the hts-cache folder or start a fresh project.

Putting the fixes together, a fairly permissive command looks like this. Only include -s0 if ignoring robots.txt is appropriate:

httrack "https://www.example.com/" -O "./example-mirror" "+*example.com/*" -d -r20 -p3 -s0 -F "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" -c2 -v

If pages now arrive but look broken, that is a separate problem, covered in Downloaded Website Missing Images and CSS?

Does wget --mirror have the same only-index.html problem?

Yes, for the same reasons. A typical mirror command is:

wget --mirror --page-requisites --convert-links --adjust-extension --no-parent https://www.example.com/

The equivalents of the HTTrack fixes:

• Redirects: start from the final URL. To allow both hosts, add --span-hosts --domains=example.com,www.example.com.
• robots.txt: wget obeys it by default. -e robots=off disables that, with the same considerations as above.
• User-agent: --user-agent="Mozilla/5.0 ...".
• Start directory: --no-parent keeps wget below the start folder. Drop it, or start from the root.
• Depth: --mirror already implies unlimited depth.
• JavaScript: wget does not execute it either, so JavaScript navigation defeats both tools equally.

For a broader comparison of the two command-line tools, see wget vs HTTrack vs Online Website Downloader.

When an online downloader is the simpler fix, and when nothing helps

If your problem turned out to be configuration (depth, directories, scan rules, a stale cache), an online tool removes that layer. With Website Downloader you enter the address, watch the page count rise live, and get the site as one ZIP file that opens offline in any browser. There is nothing to install and no option tabs to get wrong. The free tier needs no account and covers up to 100 subpages. A one-time Single Site download for 7.99 EUR covers up to 5000 subpages. The other options are on the pricing page, and the trade-offs against HTTrack are in HTTrack Alternative: Copy Websites Online.

Switching tools will not help in these cases:

• Pages behind a login are not downloaded by any of these approaches.
• JavaScript-only sites can produce an archive that contains the page shell but not the content, whichever crawler you use.
• Sites that refuse automated downloads with a 403 keep refusing. Website Downloader reports that refusal instead of handing you an empty archive, but it cannot get around it.
• Server features such as search, forms, carts and comments never work in an offline copy.

If the site you are trying to save is already offline, crawling it is not an option at all. Restoring it from the Internet Archive with the Wayback Machine downloader may still work.

FAQ

Why does HTTrack finish in a few seconds with only index.html?

A crawl that finishes in seconds found no links it was allowed to follow after the first page. The most common reason is a redirect from http to https or from example.com to www.example.com, which makes every link look external. Check hts-log.txt for a "moved" notice, then restart from the exact final URL with a +*example.com/* scan rule.

How do I make HTTrack follow links to subpages?

Start from the site root using the final redirected address. Leave the mirroring depth empty or set it high. Allow both host variants with a scan rule, and make sure the primary scan rule stores HTML files. If View Source shows no <a href> links, the navigation is built by JavaScript and HTTrack cannot follow it.

How do I stop HTTrack treating www and non-www or http and https as different sites?

Add the scan rule +*example.com/*, which leaves out the protocol and the www prefix and so matches every variant. Alternatively, switch the travel mode to "stay on the same domain" with -d. Starting from the canonical address the browser ends up on avoids most of the problem in the first place.

Does HTTrack ignore robots.txt, and should I turn that off?

No. HTTrack follows robots.txt and meta robots tags by default. You can disable this under Spider options or with -s0. For your own site that is fine. For other sites, consider permission and purpose, keep the crawl slow, and remember that the legal position depends on your country and situation.