efa97bdcf1
Move write_xattr to utils.py
...
There are some other places that use xattr functions. It's better to
move it to a common place so that others can use it.
2016-09-30 00:28:32 +08:00
3e4185c396
[utils] Use native french month names
2016-09-14 23:59:38 +07:00
f6717dec8a
[utils] Improve month_by_name and add tests
2016-09-14 23:59:38 +07:00
a942d6cb48
[utils,franceinter] Add french months' names and fix extraction
...
Update of the "FranceInter" radio extractor : webpages HTML structure
had changed, the extractor didn't work. So I updated this extractor to
get the mp3 URL and all details.
2016-09-14 23:59:38 +07:00
c2b2c7e138
[utils] Add quicktime to mimetype2ext
2016-09-02 23:50:42 +07:00
6562d34a8c
[utils] Improve mimetype2ext
2016-09-02 22:57:48 +07:00
073ac1225f
[utils] add ac-3 to the list of audio codecs in parse_codecs
2016-08-25 08:49:44 +01:00
70852b47ca
[utils] Recognize units with full names in parse_filename
...
Reference: https://en.wikipedia.org/wiki/Template:Quantities_of_bytes
2016-08-20 00:17:26 +08:00
e4659b4547
[utils] Correct octal/hexadecimal number detection in js_to_json
2016-08-19 20:37:17 +08:00
13585d7682
[utils] Recognize lowercase units in parse_filesize
2016-08-18 23:32:00 +07:00
98e698f1ff
[external/curl] respect more downloader options and display progress
2016-08-12 12:30:02 +01:00
81c13222c6
[utils] Recognize more formats in unified_timestamp
...
Used in CtsNews
2016-08-10 11:37:23 +08:00
a8795327ca
[utils] Add support TV Parental Guidelines ratings in parse_age_limit
2016-08-07 20:45:18 +07:00
d3f8e038fe
[utils] Add decode_png for openload ( #9706 )
2016-08-07 02:42:58 +08:00
7dc2a74e0a
[utils] Fix unified_timestamp for formats parsed by parsedate_tz()
2016-08-05 11:41:55 +08:00
f164b97123
[utils] Add another f4m mimetype to mimetype2ext
2016-07-23 16:48:59 +07:00
e910fe2fe4
[brightcove] skip ism manifests
2016-07-14 14:13:57 +01:00
0b68de3cc1
Merge pull request #8876 from remitamine/html5_media
...
[extractor/common] add helper method to extract html5 media entries
2016-07-10 23:40:45 +08:00
84c237fb8a
[utils] Add get_element_by_class
...
For #9950
2016-07-06 20:02:52 +08:00
b4173f1551
[utils] add mimetypes to determine manifest ext(m3u8, f4m, mpd)
2016-07-06 09:06:28 +01:00
81953d1ae5
[kaltura] add support videos stored on custom kaltura servers( closes #5557 )
2016-07-04 17:59:58 +01:00
95cf60e826
[utils] Add PUTRequest
2016-07-03 02:21:32 +07:00
6b03e1e25d
[HRTi] Implement extractor for Croatian Radiotelevision
2016-07-03 02:20:41 +07:00
4f3c5e0627
[utils] add helper function for parsing codecs
2016-06-26 14:03:58 +01:00
1143535d76
[utils] Add urshift()
...
Used in IqiyiIE and LeIE
2016-06-26 15:16:49 +08:00
b72b44318c
[utils] Add strip_or_none
2016-06-25 23:19:18 +07:00
46f59e89ea
[utils] Add unified_timestamp
2016-06-25 23:19:18 +07:00
e154c65128
[downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader
2016-06-19 01:01:40 +01:00
47212f7bcb
[utils] Don't transform numbers not starting with a zero
...
Fix test_Viidea and maybe others
2016-06-16 11:00:54 +08:00
329ca3bef6
[utils] Add try_get
...
To reduce boilerplate when accessing JSON
2016-06-12 06:05:34 +07:00
15d106787e
[utils] Change Firefox 44 to 47
...
See commit title.
2016-06-11 05:36:31 -04:00
55b2f099c0
[utils] Decode HTML5 entities
...
Used in test_Vporn_1. Also related to #9270
2016-06-10 15:11:55 +08:00
6c33d24b46
[utils] Add audio/mpeg to mimetype2ext()
...
Used in WDR live radios (#6147 )
2016-06-09 12:58:24 +08:00
c88270271e
Added sanitization support for Hungarian letters Ő and Ű
2016-06-02 11:51:48 +02:00
9a4aec8b7e
[utils] Use bytes-like objects as header values on Python 2
2016-06-02 15:00:49 +08:00
0ea590076f
[utils] Always decode Location header
...
escape_url is broken for bytes-like objects
2016-06-02 15:00:49 +08:00
293c255688
[utils] Remove debugging codes
2016-05-26 22:54:16 +08:00
5950cb1d6d
[utils] Support a new form of date
...
Found in dw.com (#9475 )
2016-05-26 22:44:00 +08:00
c6b9cf05e1
[utils] Do not fail on unknown date formats in unified_strdate
2016-05-22 08:28:41 +06:00
46bc9b7d7c
[utils] Allow None in remove_{start,end}
2016-05-19 04:31:30 +06:00
cdd94c2eae
[utils] Check for None values in SOCKS proxy
...
Originally reported at
https://github.com/rg3/youtube-dl/pull/9287#issuecomment-219617864
2016-05-17 14:38:15 +08:00
79298173c5
[utils] Fix getheader in urlhandle_detect_ext
...
Fixes #7049 , related to #9440
2016-05-15 15:34:50 +08:00
cda6d47aad
[utils] Simplify integer conversion in js_to_json
2016-05-14 23:41:57 +06:00
89ac4a19e6
[utils] Process non-base 10 integers in js_to_json
2016-05-14 20:39:58 +06:00
bd1e484448
[utils] js_to_json: various improvements
...
now JS object literals like { /* " */ 0: ",]\xaa<\/p>", } will be correctly converted to JSON.
2016-05-14 20:12:39 +06:00
7581bfc958
[utils] Unquote crendentials passed to SOCKS proxies
...
Fixes #9450
2016-05-13 00:27:25 +08:00
778a1ccca7
[utils] Add Œ and œ found in French to ACCENT_CHARS
...
Fixes #9463
2016-05-12 19:48:48 +08:00
702ccf2dc0
[compat] Rename shlex_quote and remove unused subprocess_check_output
2016-05-10 16:00:21 +08:00
edaa23f822
[compat] Rename struct_(un)pack to compat_struct_(un)pack
2016-05-10 14:51:38 +08:00
d5ae6bb501
[utils] Add rationale for register_socks_protocols
2016-05-10 14:51:38 +08:00
51fb4995a5
[utils] Register SOCKS protocols in urllib and support SOCKS4A
2016-05-10 14:51:38 +08:00
71aff18809
[socks] Support SOCKS proxies
2016-05-10 14:51:38 +08:00
dab0daeeb0
[utils,compat] Move struct_pack and struct_unpack to compat.py
2016-05-10 14:51:38 +08:00
abc97b5eda
[utils] Allow empty attribute values in get_element_by_attribute ( Closes #9415 )
2016-05-06 22:07:30 +06:00
c587cbb793
improved performance by extracting accented chars to top level
2016-05-03 10:40:30 +10:00
79a2e94e79
Instead of replacing accented characters with an underscore when sanitizing file names in restricted mode, replace them with their non-accented equivalents fixes #9347
2016-05-02 13:21:39 +10:00
eb9ee19422
[utils] Allow None mimetypes in mimetype2ext
2016-04-25 00:03:12 +06:00
b6c0d4f431
Merge pull request #9110 from remitamine/parse_duration
...
[utils] imporove parse_duration to handle more formats
2016-04-21 22:53:16 +07:00
acaff49575
[utils] imporove parse_duration to handle more formats
2016-04-21 16:34:54 +01:00
cacd996662
[utils] Don't touch URLs if not necessary
...
Fix test_Generic_15 (Google redirect)
2016-04-09 19:27:54 +08:00
5bf28d7864
[utils] dfxp2srt: add additional namespace
...
Used by the ZDF subtitles (#9081 ).
2016-04-04 20:46:35 +02:00
15d260ebaa
[utils] Use update_Request in http_request
2016-03-31 22:55:49 +06:00
ed0291d153
[utils] Add update_Request
2016-03-31 22:55:01 +06:00
17bcc626bf
[utils] Extract sanitize_url routine
2016-03-26 19:33:57 +06:00
15707c7e02
[compat] Add compat_urllib_parse_urlencode and eliminate encode_dict
...
encode_dict functionality has been improved and moved directly into compat_urllib_parse_urlencode
All occurrences of compat_urllib_parse.urlencode throughout the codebase have been replaced by compat_urllib_parse_urlencode
Closes #8974
2016-03-26 01:46:57 +06:00
622d19160b
[utils] Clarify Python versions affected by buggy struct module
2016-03-24 18:06:15 +08:00
efbed08dc2
[utils] Encode hostnames before passing to urllib
...
With IDN (Internationalized Domain Name) and a proxy, non-ascii URLs
are passed down to urllib/urllib2, causing UnicodeEncodeError
Fixes #8890
2016-03-23 22:24:52 +08:00
782b1b5bd1
[utils] lookup_unit_table: Match word boundary instead of end of string
2016-03-19 11:44:49 +01:00
09fc33198a
utils: lookup_unit_table: Use a stricter regex
...
In parse_count multiple units start with the same letter, so it would match different units depending on the order they were sorted when iterating over them.
2016-03-18 19:23:06 +01:00
810c10baa1
[utils] Use compat_xpath
2016-03-18 02:52:23 +06:00
c5229f3926
[utils] PEP 8
2016-03-16 21:50:04 +06:00
83548824c2
Merge pull request #8092 from bpfoley/twitter-thumbnail
...
[utils] Add extract_attributes for extracting html tag attributes
2016-03-16 13:16:27 +01:00
2f7ae819ac
[utils] PEP 8
2016-03-13 17:23:08 +06:00
fb47597b09
[bbc] Generalize unit table lookup and add parse_count
2016-03-13 16:27:20 +06:00
25cb05bda9
[utils] Remove codec2ext
...
This function is orignally used for determining file extensions of DASH
formats. Now in DASH, ext is determined by mime_type. See #8766 for more
information.
2016-03-11 23:51:42 +08:00
6d210f2090
[utils] Add more codecs to codec2ext
...
BBC uses avc3. Here's an example (thanks to @remitamine for this example)
http://rdmedia.bbc.co.uk/dash/ondemand/bbb/2/client_manifest-common_init.mpd
See also https://trac.ffmpeg.org/ticket/5217
2016-03-06 17:57:48 +08:00
19a17d4623
[utils] Add codec2ext
2016-03-05 18:18:28 +08:00
3233a68fbb
[utils] update_url_query: Encode the strings in the query dict
...
The test case with {'test': '第二行тест'} was failing on python 2 (the non-ascii characters were replaced with '?').
2016-03-04 22:18:40 +01:00
1255733945
Merge pull request #8739 from remitamine/update_url_params
...
[utils] add update_url_query function to create or update query string params
2016-03-03 19:24:04 +01:00
38f9ef31dc
[utils] add update_url_query function
2016-03-03 18:34:52 +01:00
0cae023b24
Merge branch 'jython-support'
...
Closes #8302
2016-03-03 18:49:32 +08:00
8ee239e921
[utils] Jython support - handle filenames correctly
...
Now test:youtube downloads
2016-03-03 18:47:54 +08:00
8bb56eeeea
[utils] Add extract_attributes for extracting html tag attributes
...
This is much more robust than just using regexps, and handles all
the common scenarios, such as empty/no values, repeated attributes,
entity decoding, mixed case names, and the different possible value
quoting schemes.
2016-03-03 10:11:37 +00:00
e07237f640
[utils] remove check for val from find_xpath_attr
2016-03-02 21:40:21 +01:00
5eb6bdced4
[utils] Multiple changes to base_n()
...
1. Renamed to encode_base_n()
2. Allow tables longer than 62 characters
3. Raise ValueError instead of AssertionError for invalid input data
4. Return the first character in the table instead of '0' for number 0
5. Add tests
2016-02-27 03:22:52 +08:00
680079be39
[utils] Relaxing regex in decode_packed_codes for vidzi
2016-02-26 15:13:03 +08:00
f52354a889
[utils] Move codes for handling eval() from iqiyi.py
2016-02-26 14:58:29 +08:00
59f898b7a7
[utils] Merge base_n functions
2016-02-26 14:37:20 +08:00
481888294d
[utils] Add base36 for use in Vidzi
2016-02-26 14:26:26 +08:00
81bdc8fdf6
[utils] Move base62 to utils
2016-02-26 14:26:26 +08:00
f160785c5c
[utils] Remove AM/PM from unified_strdate patterns
2016-02-25 00:52:49 +06:00
b95dc034ca
[utils] Implement cache for OnDemandPagedList
2016-02-23 13:11:20 +08:00
cafcf657a4
add more subtitles mime types to mimetype2ext and fix the platform subtitle extraction
2016-02-20 22:02:03 +01:00
c1c05c67ea
[utils] Jython support - disable setproctitle() until ctypes is complete
2016-02-21 03:32:03 +08:00
399a76e67b
[utils] Jython support: tolerate missing fcntl module
2016-02-21 03:32:03 +08:00
765ac263db
[utils] mimetype2ext: return 'm4a' for 'audio/mp4' ( fixes #8620 )
...
The youtube extractor was using 'mp4' for them, therefore filters like 'bestaudio[ext=m4a]' stopped working (94278f7202
2016-02-20 19:55:10 +01:00
5bc880b988
[utils] Add OHDave's RSA encryption function
2016-02-20 19:54:58 +08:00
611c1dd96e
[refactor] Single quotes consistency
2016-02-14 15:37:17 +06:00
d800609c62
[refactor] Do not specify redundant None as second argument in dict.get()
2016-02-14 14:25:04 +06:00
9c7b38981c
[utils] Bump Firefox version in User-Agent
...
Old version number causes Youtube not to serve some formats in ytplayer.config
2016-02-11 23:12:30 +06:00
8411229bd5
[utils] Allow dot in strip_jsonp
2016-02-07 19:47:09 +06:00
86296ad2cd
[utils] Add ability to control skipping false values in dict_get
2016-02-07 08:13:04 +06:00
cbecc9b903
[utils] Add dict_get convenience method
2016-02-07 06:12:53 +06:00
87de7069b9
[utils] dfxp2srt: make TTMLPElementParser inherit from object
...
For consistency between python 2 and 3.
2016-02-02 22:30:13 +01:00
2b14cb566f
[utils] fix dfxp2srt text extraction( fixes #8055 )
2016-01-28 12:38:34 +01:00
a0d8d704df
[utils] Reorder items in mimetype2ext alphabetically
2016-01-25 01:01:15 +08:00
f6861ec96f
[utils] Add more items to mimetype2ext ( #8293 )
...
These are used in Youtube formats
2016-01-25 00:58:53 +08:00
6ec6cb4e95
Revert "fix typos"
...
This reverts commit 36a0e46c39
2016-01-10 19:27:22 +01:00
36a0e46c39
fix typos
2016-01-10 17:55:41 +01:00
dfb1b1468c
Fix typos
...
Closes #8200 .
2016-01-10 17:24:28 +01:00
a7aaa39863
[utils] Extract known extensions for reuse
2016-01-04 01:08:34 +06:00
c047270c02
[utils] Remove Content-encoding from headers after decompression
...
With cn_verification_proxy, our http_response() is called twice, one from
PerRequestProxyHandler.proxy_open() and another from normal
YoutubeDL.urlopen(). As a result, for proxies honoring Accept-Encoding, the
following bug occurs:
$ youtube-dl -vs --cn-verification-proxy https://secure.uku.im:993 "test:letv"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-vs', '--cn-verification-proxy', 'https://secure.uku.im:993 ', 'test:letv']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.12.23
[debug] Git HEAD: 97f18fahttp://www.letv.com/ptv/vplay/22005890.html
[Letv] 22005890: Downloading webpage
[Letv] 22005890: Downloading playJson data
ERROR: Unable to download JSON metadata: Not a gzipped file (b'{"') (caused by OSError('Not a gzipped file (b\'{"\')',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/extractor/common.py", line 330, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1886, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.5/urllib/request.py", line 471, in open
response = meth(req, response)
File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/utils.py", line 773, in http_response
raise original_ioerror
File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/utils.py", line 761, in http_response
uncompressed = io.BytesIO(gz.read())
File "/usr/lib/python3.5/gzip.py", line 274, in read
return self._buffer.read(size)
File "/usr/lib/python3.5/gzip.py", line 461, in read
if not self._read_gzip_header():
File "/usr/lib/python3.5/gzip.py", line 409, in _read_gzip_header
raise OSError('Not a gzipped file (%r)' % magic)
2015-12-28 01:09:18 +08:00
9b9c5355e4
Rename error_to_str to error_to_compat_str
2015-12-20 07:00:39 +06:00
8e60dc7526
[utils] Add encode_compat_str
2015-12-20 06:26:26 +06:00
fdae235858
[utils] Add error_to_str
2015-12-20 05:26:47 +06:00
db2fe38b55
[utils] Support alternative timestamp format in TTML
...
Fixes #7608
2015-12-19 19:29:51 +08:00
d631d5f9f2
[utils] Fix TTML conversion
...
Tolerate invalid timestamps (closes #7909 )
2015-12-19 18:21:42 +08:00
31b2051e21
[utils] Add remove_quotes
2015-12-14 21:30:58 +06:00
992fc9d6e1
[utils] Refactor handle_youtubedl_headers for future extension
2015-11-29 12:58:29 +08:00
0424ec307b
[utils] Correct docstring of YoutubeDLHandler
2015-11-29 12:46:04 +08:00
87f0e62d94
[utils] Separate codes for handling Youtubedl-* headers
2015-11-29 12:42:50 +08:00
67dda51722
Rename compat_urllib_request_Request to sanitized_Request and move to utils
2015-11-23 21:55:15 +06:00
9cb9a5df77
[utils] Check ext with trailing slash against the list of known extensions
2015-11-22 17:27:13 +06:00
3e12bc583a
[utils] Improve determine_ext ( Closes #7593 )
2015-11-22 06:29:39 +06:00
7e1f5447e7
[utils] Improve encode_dict
2015-11-21 20:46:33 +06:00
7a3f0c00ad
[utils] Style
2015-11-16 20:24:09 +06:00
7aefc49c40
[utils] Skip invalid/non HTML entities ( Closes #7518 )
2015-11-16 20:20:16 +06:00
6a75040278
[utils] unified_strdate: Return None if the date format can't be recognized ( fixes #7340 )
...
This issue was introduced with ae12bc3ebb
2015-11-02 14:08:38 +01:00
c90d16cf36
[utils:sanitize_path] Disallow trailing whitespace in path segment ( Closes #7332 )
2015-11-02 04:26:20 +06:00
30eecc6a04
Merge pull request #7296 from jaimeMF/xml_attrib_unicode
...
Use a wrapper around xml.etree.ElementTree.fromstring in python 2.x (…
2015-10-31 18:15:21 +00:00
ae12bc3ebb
[utils] Make unified_strdate always return unicode string
2015-10-31 23:07:37 +06:00
578c074575
[utils] Support list of xpath in xpath_element
2015-10-31 22:39:44 +06:00
52c3a6e49d
[utils] Improve parse_iso8601
2015-10-28 21:40:22 +06:00
f78546272c
[compat] compat_etree_fromstring: also decode the text attribute
...
Deletes parse_xml from utils, because it also does it.
2015-10-26 16:41:24 +01:00
36e6f62cd0
Use a wrapper around xml.etree.ElementTree.fromstring in python 2.x ( #7178 )
...
Attributes aren't unicode objects, so they couldn't be directly used in info_dict fields (for example '--write-description' doesn't work with bytes).
2015-10-25 20:13:16 +01:00
d01949dc89
[utils:js_to_json] Fix bad escape in double quoted strings
2015-10-20 23:09:51 +06:00
1e399778ee
[letv] Fix extraction
...
Using data URIs for passing the decrypted M3U8 manifest, which is
supported by ffmpeg only.
2015-10-18 13:42:57 +08:00
af98f8ff37
[utils] Return default on fail in int_or_none
2015-10-14 22:37:03 +06:00
caf80631f0
[utils] Do not fail in float_or_none on non-numeric data
2015-10-14 22:36:37 +06:00
1812afb7b3
[utils] Do not fail in int_or_none on non-numeric data ( Closes #7175 )
2015-10-14 22:35:01 +06:00
5a1a2e9454
[utils] Fix kwargs on old python 2 ( Closes #6905 )
2015-09-20 21:08:29 +06:00
e28034c5ac
[utils] Comment cookie processing until result from travis and some more testing
2015-09-06 08:16:39 +06:00
266e466ee4
[utils] Simplify cookie processor
2015-09-06 07:53:11 +06:00
1639282434
[utils] Add encode_dict
2015-09-06 07:22:20 +06:00
ad72917274
[utils] Add issue URL in comment for #6457
2015-09-06 06:23:44 +06:00
a6420bf50c
[utils] Add cookie processor for cookie correction ( Closes #6769 )
2015-09-06 06:20:48 +06:00
66e289bab4
[utils] Generalize cli option converters
2015-09-05 03:05:11 +06:00
8e636da499
[utils] Improve xpath_text
2015-09-05 00:34:49 +06:00
5d2354f177
[utils] Relax attribute key assert
2015-09-04 23:57:27 +06:00
a41fb80ce1
[utils] Add xpath_element and xpath_attr
2015-09-04 23:56:45 +06:00
Yen Chi Hsuan
Sergey M․
renalid
Remita Amine
Aleksandar Topuzovic
Paul Henning
bzc6p
felix
Adam Thalhammer
Jaime Marquínez Ferrándiz
Brian Foley
Jakub Wilk