Python help, specifically with urllib2

Commissario
Joined
16 Oct 2002
Posts
2,646
Location
In the radio shack
Hi,

I use a Raspberry Pi as an internet watchdog but I'm not entirely convinced I have the code right.

This is supposed to check for the presence of a web site, and wait for up to four minutes for a response.

However, I don't think the timeout bit is working correctly. I think that if it can't access the site it's trying to then it errors straight away.

Here's the full script I'm using. I've part pinched it from the internet and tweaked it a bit.

Code:
import urllib2
import RPi.GPIO as GPIO
import time

RELAY = 21 #RELAY PIN, physical pin 40
ALIVE = 20 #LED PIN to flash LED on live internet, physical pin 38
CHECK_EVERY = 10 #Seconds

GPIO.setmode(GPIO.BCM)
GPIO.setup(RELAY, GPIO.OUT)
GPIO.setup(ALIVE, GPIO.OUT)

# Force both outputs low
GPIO.output(RELAY, GPIO.LOW)
GPIO.output(ALIVE, GPIO.LOW)

def flash_ok():
    GPIO.output(ALIVE, GPIO.HIGH)
        time.sleep(0.2)
        GPIO.output(ALIVE, GPIO.LOW)


def internet_on():
    try:
        response=urllib2.urlopen('https://www.website.com', timeout=240)
          flash_ok()
        return True
    except urllib2.URLError as err: pass
    return False

def reboot_router():
    print "Rebooting Router"
    GPIO.output(RELAY, GPIO.HIGH)
    time.sleep(10) #Energise relay for ten seconds
    GPIO.output(RELAY, GPIO.LOW)
    time.sleep(240) #Wait for four minutes for router to reboot

while True:
    if not internet_on():
        reboot_router()
    time.sleep(CHECK_EVERY)

GPIO.cleanup()

This is the part of the code in question.

Code:
def internet_on():
   try:
        response=urllib2.urlopen('https://www.website.com', timeout=240)
          flash_ok()
        return True
    except urllib2.URLError as err: pass
    return False
Have I got something wrong here, specifically in the timeout bit?

I'm (clearly) not a Python programmer, just someone who finds code and attempts to tweak it!

Thanks for any advice/suggestions.
 
Man of Honour
Joined
13 Oct 2006
Posts
90,805
Don't really have my head in python at the moment but you may need a conditional check against socket.timeout as part of that function to get what you want.
 
Caporegime
Joined
18 Oct 2002
Posts
32,615
Code:
def internet_on():
   try:
        response=urllib2.urlopen('https://www.website.com', timeout=240)
          flash_ok()
        return True
    except urllib2.URLError as err: pass
    return False
Have I got something wrong here, specifically in the timeout bit?

I'm (clearly) not a Python programmer, just someone who finds code and attempts to tweak it!

Thanks for any advice/suggestions.
Instead of pass-ing the exception why not print it out and then you can see if there is an actual exception being thrown.


You can easily test the behavior of your code. Just give it a broken URL and it will fail, I doubt it will wait 4 minutes before the timeout is raised and instead will raise some URLError exception to which you are oblivious.
 
Commissario
OP
Joined
16 Oct 2002
Posts
2,646
Location
In the radio shack
Good suggestion, thanks.

I've added two lines just to give me an idea of what's happening.

Code:
def internet_on():
    try:
        response=urllib2.urlopen('https://www.unavailable.com', timeout=240)
        print "It's working"
        flash_ok()
        return True
    except urllib2.URLError as err: pass
    print "it's broken"
    return False

And yes, it errors after just a few seconds so the timeout isn't doing what it's supposed to.

The suggestion above about needing a conditional check against socket.timeout may be a good one but I really wouldn't know where to start when it comes to adding that.
 
Caporegime
Joined
18 Oct 2002
Posts
32,615
I was more thinking:

Code:
def internet_on():
    try:
        response=urllib2.urlopen('https://www.unavailable.com', timeout=240)
        print "It's working"
        flash_ok()
        return True
    except urllib2.URLError as err: 
        print "it's broken,url error, reason = ", err
   exception urllib.error.HTTPErroras err:
       print "it's broken, http error, reason = ", err
   return False


But I fully expect if there is some http error then it simply wont wait. For example:

Code:
response = urllib2.urlopen('http://a_bad_URL.org/',timeout=240)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

The timeout makes no difference because it isn't blocking on an HTTP connection
 
Commissario
OP
Joined
16 Oct 2002
Posts
2,646
Location
In the radio shack
Right, I tried the first code you've added there, I had to make some changes to actually make it work (all part of the learning process, that's good). It looks like this now:

Code:
def internet_on():
    try:
        response=urllib2.urlopen('https://www.udfsdf5sdg.com', timeout=240)
        print "It's working"
        flash_ok()
        return True
    except urllib2.URLError as err:
        print "It's broken, url error, reason = ", err
    except urllib2.HTTPError as err:
       print "It's broken, http error, reason = ", err
    return False


And you're right, the timeout still does nothing, I get the following after just a few seconds:
Code:
It's broken, url error, reason =  <urlopen error [Errno -2] Name or service not known>

I'm really not sure what that last code segment you've posted is all about.
 
Associate
Joined
20 Jan 2013
Posts
140
Is that failing on the hostname which happens before the URL open? In other words, the hostname has to resolve first. Try putting a duff IP address in instead to test the timeout option.
 
Soldato
Joined
18 Oct 2002
Posts
4,152
Location
West Lancashire
Reading this I was sure I'd had a similar issue. Just searched my code and found this:

Code:
        # Set timeout to 30s to stop bad connections ruining my life
        socket.setdefaulttimeout(30)

This is using httplib2 so YMMV with urlib2.

If you can switch to requests you can use the timeout=x parameter.

I'll test it out now and report back...
 
Soldato
Joined
18 Oct 2002
Posts
4,152
Location
West Lancashire
Code:
import urllib2
import socket
# import RPi.GPIO as GPIO
from time import gmtime, strftime, sleep


RELAY = 21  # RELAY PIN, physical pin 40
ALIVE = 20  # LED PIN to flash LED on live internet, physical pin 38
CHECK_EVERY = 10  # Seconds

# GPIO.setmode(GPIO.BCM)
# GPIO.setup(RELAY, GPIO.OUT)
# GPIO.setup(ALIVE, GPIO.OUT)
#
# # Force both outputs low
# GPIO.output(RELAY, GPIO.LOW)
# GPIO.output(ALIVE, GPIO.LOW)


# def flash_ok():
#     GPIO.output(ALIVE, GPIO.HIGH)
#     time.sleep(0.2)
#     GPIO.output(ALIVE, GPIO.LOW)


def internet_on():
    # This is where the magic happens! ;)
    socket.setdefaulttimeout(3)
    print "{}: Internet On".format(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
    try:
        response = urllib2.urlopen('http://www.fakeresponse.com/api/?data={"hello":"World"}&sleep=5')
        print "{}: It's working".format(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
        # flash_ok()
        return True
    except urllib2.URLError as err:
        print "It's broken, url error, reason = {}".format(err)
    except urllib2.HTTPError as err:
        print "It's broken, http error, reason = {}".format(err)
    except socket.error as err:
        print "It's broken, socket error, reason = {}".format(err)
    return False


def reboot_router():
    print "Rebooting Router"
    # GPIO.output(RELAY, GPIO.HIGH)
    # time.sleep(10)  # Energise relay for ten seconds
    # GPIO.output(RELAY, GPIO.LOW)
    # time.sleep(240)  # Wait for four minutes for router to reboot


while True:
    if not internet_on():
        reboot_router()
    sleep(CHECK_EVERY)

# GPIO.cleanup()

It's basically working by adding the "socket.setdefaulttimeout(3)" line.

If you run this code it will fail at socket timeout because the url waits 5 secs to respond but the socket timeout is 3 secs

Make the sockettimeout 10 at it will work as expected.

The rest of the changes are to add timestamps to the print statements and "fix" some coding style ;)

Oh and since I don't have the rpi libs installed I had to comment out all the references to them.
 
Last edited:
Commissario
OP
Joined
16 Oct 2002
Posts
2,646
Location
In the radio shack
That's good, yes I can see that and I can actually see how it's working, thanks.

The web page you're calling doesn't complete the load for five seconds - Providing I increase the setdefaulttimeout to higher than that then I should be OK. If I set it to 180 then it'll wait for three minutes before it errors.

Is that a legit URL to carry on using or is that likely to change? Would using something like google.co.uk be more reliable?

So simply replace

Code:
        response = urllib2.urlopen('http://www.fakeresponse.com/api/?data={"hello":"World"}&sleep=5')

With
Code:
        response = urllib2.urlopen('https://www.google.co.uk')

Looks like I'm nearly there :)
 
Back
Top Bottom