1 Star 1 Fork 0

Jun Peng / wget2

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
gnulib @ 475eac6
Loading...
README
LGPL-3.0

Build status Fuzzing Status Coverage status Coverage status Coverity Scan Build Status

Solaris OpenCSW Build Status Solaris amd64 Build Status Solaris i386 Build Status Solaris Sparc Build Status Solaris SparcV9

GNU Wget2 - Introduction

GNU Wget2 is the successor of GNU Wget, a file and recursive website downloader.

Designed and written from scratch it wraps around libwget, that provides the basic functions needed by a web client.

Wget2 works multi-threaded and uses many features to allow fast operation.

In many cases Wget2 downloads much faster than Wget1.x due to HTTP2, HTTP compression, parallel connections and use of If-Modified-Since HTTP header.

GNU Wget2 is licensed under GPLv3+.

Libwget is licensed under LGPLv3+.

Features

A non-exhaustive list of features

  • Support for HTTP/1.1 and HTTP/2.0 protocol
  • brotli decompression support (Accept-Encoding: br)
  • zstandard decompression support, RFC8478 (Accept-Encoding: zstd)
  • lzip decompression support, (Accept-Encoding: lzip)
  • HPKP - HTTP Public Key Pinning (RFC7469) with persistent database
  • TCP Fast Open for plain text and for HTTPS
  • TLS Session Resumption including persistent session data cache
  • TLS False Start (with GnuTLS >= 3.5.0)
  • HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining
  • OCSP stapling + OCSP server querying as a fallback (experimental, needs GnuTLS >= 3.3.11)
  • Use libpsl for cookie domain checking (using Public Suffix List)
  • Support link conversion (-k/--convert-links and -K/--backup-converted)
  • Support for RFC 6266 compliant Content-Disposition
  • RFC 6797 HSTS (HTTP Strict Transport Security)
  • Support for bzip2 Content-Encoding / Accept-Encoding compression type
  • New Year 2014 gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type
  • Character encoding of input files may be specified despite from local and remote encoding (--input-encoding)
  • Support scanning RSS 2.0 feeds from local files (--force-rss -i <filename>)
  • Support scanning RSS 2.0 feeds.
  • Support scanning Atom 1.0 feeds from local files (--force-atom -i <filename>)
  • Support scanning Atom 1.0 feeds.
  • Support scanning URLs from local Sitemap XML file (--force-sitemap -i <filename>)
  • Support scanning sitemap files given in robots.txt (Sitemap XML, gzipped Sitemap XML, plain text) including sitemap index files.
  • Support arbitrary number of proxies for parallel downloads
  • Multithreaded download of single files (option --chunk-size)
  • Internationalized Domain Names in Applications (compile-selectable IDNA2008 or IDNA2003)
  • ICEcast / SHOUTcast support via library (see examples/getstream.c)
  • respect /robots.txt "Robot Exclusion Standard" and <META name="robots" ...>
  • new option --secure-protocol=PFS to have TLS only plus forcing Perfect Forward Secrecy (PFS)
  • IDN support for international domains
  • autotools support
  • proxy support
  • cookies (session/non-session), detection of supercookies via Mozilla Public Suffix List (use the new option --cookie-suffixes <filename>, better: put it into ~/.wgetrc)
  • recursive download of websites with or without spanning hosts
  • download of single web pages / resources
  • zlib/gzip compressed HTTP/HTTPS downloads (gzip, deflate)
  • number of parallel download threads is adjustable
  • include directive for config files (wildcards allowed)
  • support for keep-alive connections
  • included CSS, HTML, XML parser needed for recursive downloads
  • gettext support
  • HTTPS via libgnutls (and basic WolfSSL support)
  • support for Metalink RFC 6249 (Metalink/HTTP: Mirrors and Hashes)
  • support for Metalink RFC 5854 (Metalink Download Description Format / .meta4 files)
  • support for Metalink 3
  • Metalink checksumming via libgnutls
  • DNS lookup cache
  • IPv4 and IPv6 support
  • built and tested on Linux, OSX, OpenBSD, FreeBSD, Solaris, Windows

Links

Online Docs

Mailing List

Bug Tracker

Development

Code Coverage

Fuzz Code Coverage

Build Requirements

The following packages are needed to build the software

  • autotools (autoconf, autogen, automake, autopoint, libtool)
  • python (recommended for faster bootstrap)
  • rsync
  • tar
  • makeinfo (part of texinfo)
  • pkg-config >= 0.28 (recommended)
  • doxygen (for creating the documentation)
  • pandoc (for creating the wget2 man page)
  • gettext >= 0.18.2
  • libiconv (needed for IRI and IDN support)
  • libz >= 1.2.3 (the distribution may call the package zlib*, eg. zlib1g on Debian)
  • liblzma >= 5.1.1alpha (optional, if you want HTTP lzma decompression)
  • libbz2 >= 1.0.6 (optional, if you want HTTP bzip2 decompression)
  • libbrotlidec/libbrotli >= 1.0.0 (optional, if you want HTTP brotli decompression)
  • libzstd >= 1.3.0 (optional, if you want HTTP zstd decompression)
  • libgnutls (3.3, 3.5 or 3.6)
  • libidn2 >= 0.14 (libidn >= 1.25 if you don't have libidn2)
  • flex >= 2.5.35
  • libpsl >= 0.5.0
  • libnghttp2 >= 1.3.0 (optional, if you want HTTP/2 support)
  • libmicrohttpd >= 0.9.51 (optional, if you want to run the test suite)
  • lzip (optional, if you want to build distribution tarballs)
  • lcov (optional, for coverage reports)
  • libgpgme >= 0.4.2 (optional, for automatic signature verification)
  • libpcre | libpcre2 (optional, for filtering by PCRE|PCRE2 regex)
  • libhsts (optional, to support HSTS preload lists)
  • libwolfssl (optional, to support WolfSSL instead of GnuTLS)

The versions are recommended, but older versions may also work.

Downloading and building from tarball

	wget https://gnuwget.gitlab.io/wget2/wget2-latest.tar.gz
	tar xf wget2-latest.tar.gz
	cd wget2-*
	./configure
	make
	make check
	sudo make install

Building from git

Download project and prepare sources with

	git clone https://gitlab.com/gnuwget/wget2.git
	cd wget2
	./bootstrap
	# on shell failure try 'bash ./bootstrap'

Build Wget2 with

	./configure
	make

In Haiku build Wget2 with

    setarch x86
    ./configure --prefix=/boot/home/config/non-packaged
    rm /boot/home/config/non-packaged/wget2 && mv /boot/home/config/non-packaged/wget2_noinstall /boot/home/config/non-packaged/wget2

Test the functionality

	make check

Install Wget2 and libwget

	sudo make install (or su -c "make install")

License

Copyright (C) 2015-2020 Free Software Foundation, Inc.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.

GNU LESSER GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/> Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. This version of the GNU Lesser General Public License incorporates the terms and conditions of version 3 of the GNU General Public License, supplemented by the additional permissions listed below. 0. Additional Definitions. As used herein, "this License" refers to version 3 of the GNU Lesser General Public License, and the "GNU GPL" refers to version 3 of the GNU General Public License. "The Library" refers to a covered work governed by this License, other than an Application or a Combined Work as defined below. An "Application" is any work that makes use of an interface provided by the Library, but which is not otherwise based on the Library. Defining a subclass of a class defined by the Library is deemed a mode of using an interface provided by the Library. A "Combined Work" is a work produced by combining or linking an Application with the Library. The particular version of the Library with which the Combined Work was made is also called the "Linked Version". The "Minimal Corresponding Source" for a Combined Work means the Corresponding Source for the Combined Work, excluding any source code for portions of the Combined Work that, considered in isolation, are based on the Application, and not on the Linked Version. The "Corresponding Application Code" for a Combined Work means the object code and/or source code for the Application, including any data and utility programs needed for reproducing the Combined Work from the Application, but excluding the System Libraries of the Combined Work. 1. Exception to Section 3 of the GNU GPL. You may convey a covered work under sections 3 and 4 of this License without being bound by section 3 of the GNU GPL. 2. Conveying Modified Versions. If you modify a copy of the Library, and, in your modifications, a facility refers to a function or data to be supplied by an Application that uses the facility (other than as an argument passed when the facility is invoked), then you may convey a copy of the modified version: a) under this License, provided that you make a good faith effort to ensure that, in the event an Application does not supply the function or data, the facility still operates, and performs whatever part of its purpose remains meaningful, or b) under the GNU GPL, with none of the additional permissions of this License applicable to that copy. 3. Object Code Incorporating Material from Library Header Files. The object code form of an Application may incorporate material from a header file that is part of the Library. You may convey such object code under terms of your choice, provided that, if the incorporated material is not limited to numerical parameters, data structure layouts and accessors, or small macros, inline functions and templates (ten or fewer lines in length), you do both of the following: a) Give prominent notice with each copy of the object code that the Library is used in it and that the Library and its use are covered by this License. b) Accompany the object code with a copy of the GNU GPL and this license document. 4. Combined Works. You may convey a Combined Work under terms of your choice that, taken together, effectively do not restrict modification of the portions of the Library contained in the Combined Work and reverse engineering for debugging such modifications, if you also do each of the following: a) Give prominent notice with each copy of the Combined Work that the Library is used in it and that the Library and its use are covered by this License. b) Accompany the Combined Work with a copy of the GNU GPL and this license document. c) For a Combined Work that displays copyright notices during execution, include the copyright notice for the Library among these notices, as well as a reference directing the user to the copies of the GNU GPL and this license document. d) Do one of the following: 0) Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work, in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source. 1) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (a) uses at run time a copy of the Library already present on the user's computer system, and (b) will operate properly with a modified version of the Library that is interface-compatible with the Linked Version. e) Provide Installation Information, but only if you would otherwise be required to provide such information under section 6 of the GNU GPL, and only to the extent that such information is necessary to install and execute a modified version of the Combined Work produced by recombining or relinking the Application with a modified version of the Linked Version. (If you use option 4d0, the Installation Information must accompany the Minimal Corresponding Source and Corresponding Application Code. If you use option 4d1, you must provide the Installation Information in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.) 5. Combined Libraries. You may place library facilities that are a work based on the Library side by side in a single library together with other library facilities that are not Applications and are not covered by this License, and convey such a combined library under terms of your choice, if you do both of the following: a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities, conveyed under the terms of this License. b) Give prominent notice with the combined library that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. 6. Revised Versions of the GNU Lesser General Public License. The Free Software Foundation may publish revised and/or new versions of the GNU Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Library as you received it specifies that a certain numbered version of the GNU Lesser General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that published version or of any later version published by the Free Software Foundation. If the Library as you received it does not specify a version number of the GNU Lesser General Public License, you may choose any version of the GNU Lesser General Public License ever published by the Free Software Foundation. If the Library as you received it specifies that a proxy can decide whether future versions of the GNU Lesser General Public License shall apply, that proxy's public statement of acceptance of any version is permanent authorization for you to choose that version for the Library.

简介

Clone of the GNU Wget2 repository for collaboration via GitLab GNU Wget2是对流行的wget下载实用程序的重写。GNU Wget2包含了libwget库,同时现在是多线程的,并支持其他功能,以提供比当前wget版本更好的性能。 GNU Wget2是GNU Wget的继承者,GNU Wget是一个文件和递归网站下载器。 NU Wget2通过多线程、正确支持HTTP2连接、处理HTTP压缩特性、处理并行连接、考虑到If-Modified-Since HTTP标头和其他特性,提供了更快的性能。Wget2的下载速度比Wget1.x快得多。 它是从头开始设计和编写的,它包含了libwget,它提供了Web客户端所需的基... 展开 收起
C
LGPL-3.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
C
1
https://gitee.com/penjun71/wget2.git
git@gitee.com:penjun71/wget2.git
penjun71
wget2
wget2
master

搜索帮助