Skip to content

Python:urllib

Example

다음은 Request를 사용하여 PUT 요청을 수행하는 예입니다: 

import urllib.request
DATA = b'some data'
req = urllib.request.Request(url='http://localhost:8080', data=DATA, method='PUT')
with urllib.request.urlopen(req) as f:
    pass
print(f.status)
print(f.reason)
print(u.getcode())
print(u.headers)
print(u.read())

POST 로 josn 데이터 요청 예제: 

req = Request("http://localhost:48888/live", headers={"Content-Type": "application/json"}, method="POST")
data = b'{"live": [], "history": []}'
with urlopen(req, data) as f
    f.read()
    # ...

application/x-www-form-urlencoded

문자열 인자(application/x-www-form-urlencoded MIME)는 urllib.parse.parse_qsl 또는 urllib.parse.parse_qs를 사용하여 파싱할 수 있다.

쿼리 문자(URL 인코딩)로 변환하고 싶을 경우 urllib.parse.urlencode를 사용하면 된다.

urlencode 의 결과로 반환 받는 ParseResult 객체는 urllib.parse.urlunparse 를 사용하면 원본 URL 텍스트로 반환받을 수 있다.

Troubleshooting

CERTIFICATE_VERIFY_FAILED

Python에서 https접속시 아래와 같은 에러가 발생할 수 있다. 

/usr/local/c2core/lib/python3.7/site-packages/torch/hub.py:429: UserWarning: torch.hub._download_url_to_file has been renamed to            torch.hub.download_url_to_file to be a public API,            _download_url_to_file will be removed in after 1.3 release
  _download_url_to_file will be removed in after 1.3 release')
Traceback (most recent call last):
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/usr/local/c2core/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/local/c2core/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/local/c2core/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "webcam.py", line 80, in <module>
    main()
  File "webcam.py", line 64, in main
    min_image_size=args.min_image_size,
  File "/home/aaa/Project/_c2core/script/python/maskrcnn_benchmark/demo/predictor.py", line 117, in __init__
    _ = checkpointer.load(cfg.MODEL.WEIGHT)
  File "/usr/local/c2core/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/utils/checkpoint.py", line 61, in load
    checkpoint = self._load_file(f)
  File "/usr/local/c2core/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/utils/checkpoint.py", line 129, in _load_file
    cached_f = cache_url(f)
  File "/usr/local/c2core/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/utils/model_zoo.py", line 55, in cache_url
    _download_url_to_file(url, cached_file, hash_prefix, progress=progress)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/hub.py", line 430, in _download_url_to_file
    download_url_to_file(url, dst, hash_prefix, progress)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/hub.py", line 384, in download_url_to_file
    u = urlopen(url)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

개정된 PEP 467에 따라 모든 https 통신은 필요한 인증서와 호스트명을 기본으로 체크하도록 되어 있어서 그렇다. 영향을 받는 라이브러리는 urllib, urllib2, http, httplib 이다.

인증서를 설치하면 된다. 

$ pip install certifi

코드로 직접 해결하고 싶다면 ssl._create_unverified_context()를 urllib.request.urlopen의 context 파라미터로 넘겨주면 된다: 

context = ssl._create_unverified_context()
response = urllib.request.urlopen(requests, data=data.encode('utf-8'), context=context)

또는 기본 컨텍스트를 아래와 같이 변경하면 된다. 

import ssl
try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context

curl이 없을 경우 대체 방법 

python -c "import urllib.request as r; exit(0 if r.urlopen('http://www.python.org/').status == 201 else 1)"

See also

Favorite site