Proxy support

[GoogleCodeExporter]

Specto should support http Proxy. Ideally, the gnome proxy configuration
should be used so it works transparently for the user.

Original issue reported on code.google.com by [email protected] on 7 Apr 2008 at 10:34

[GoogleCodeExporter]

It would make sense, though I don't know how to implement it or when I will get
around to it (especially since I'm unfamiliar with testing proxies), so if 
someone
wants to give it a go, go!

Original comment by [email protected] on 7 Apr 2008 at 1:16

• Changed state: Accepted
• Added labels: Priority-Low, Type-Enhancement
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

As an example, look at the patch that added Proxy support to Conseil :
http://conseil.fritalk.com/attachment/ticket/68/protocol_launchpadweb.py.diff

Original comment by [email protected] on 7 Apr 2008 at 1:25

[GoogleCodeExporter]

Nekohayo, I've tried and it's very simple to add proxy support for a given 
watch.

Example : in watch_web_static.py add :


import gconf


and, just before the urllib2 request, line 107, add :

client = gconf.client_get_default() 
http_proxy = "http://%s:%s" % (client.get_string("/system/http_proxy/host"),
client.get_int("/system/http_proxy/port")) 
https_proxy = "https://%s:%s" % (client.get_string("/system/proxy/secure_host"),
client.get_int("/system/proxy/secure_port")) 
proxies = {"http": http_proxy, "https": https_proxy} 


urllib2 automatically use the proxies variable. I've tested it and it works for 
me
and it should still work for people without proxies.

I'm not sure about the best way to handle this in Specto (should it be global 
or per
watch ?) so I let you make a correct patch. You can send me a patch against the 
0.2.2
sources if you want me to test it.

Original comment by [email protected] on 7 Apr 2008 at 1:55

[GoogleCodeExporter]

hmm, any idea if proxies would need to work with 
mail/whatever_accesses_the_Internet
watches too?

Original comment by [email protected] on 7 Apr 2008 at 2:11

• Added labels: Patch

[GoogleCodeExporter]

As long as it uses http/https/ftp, yes. 
For whatever reason, I've never seen any IMAP/POP3 proxies. Strange, I never 
realized.

A simple implementation like this would be a first step and enough to be 
released.

Anyway, it should be noted that a real proxy support should also take care of 
proxy's
exception (intranet website,etc). I'm not sure that it's automatically 
supported by
the code pasted above but it should not be that hard.

Original comment by [email protected] on 7 Apr 2008 at 2:17

[GoogleCodeExporter]

Original comment by [email protected] on 25 Apr 2008 at 8:08

• Added labels: Milestone-Release0.3

[GoogleCodeExporter]

I added this in watch.py but i am not sure if it will work for other watches 
than the
web watch, because the call from urllib2 is not done in the watch class but in
another class....so should i make it a global variable so it will be available 
in all
classes?

Original comment by [email protected] on 13 May 2008 at 6:46

• Changed state: Started

[GoogleCodeExporter]

Fixed in rev 103.

Please test!

Original comment by [email protected] on 14 Sep 2008 at 3:32

• Changed state: Fixed

[GoogleCodeExporter]

Please note that this is only in Wout's branch (not yet in main):
https://code.launchpad.net/~woutc/specto/specto-woutc

Original comment by [email protected] on 14 Sep 2008 at 8:15

• Added labels: DevBranch-Wout

[GoogleCodeExporter]

I think that this issue should not be marked as "fixed". Unfortunatly, I'm 
unable to
test wout's branch for a very simple reason : bzr does not have good proxy 
support so
I cannot bzr get it !

Original comment by [email protected] on 15 Oct 2008 at 3:19

[GoogleCodeExporter]

Attached you will find my current specto development snapshot :)

unpack it and run ./launch.sh to start it

enjoy

Original comment by [email protected] on 15 Oct 2008 at 5:53

Attachments:

• specto-woutc.tar.gz

[GoogleCodeExporter]

Thanks for the tarball :-)

Testing it. So far, it seems that the proxy is working well (which is a really 
good
news :-) ) with one exception : it doesn't handle exceptions as configured in 
Gnome.
Meaning that it will not work with intranet website.

Original comment by [email protected] on 16 Oct 2008 at 8:00

[GoogleCodeExporter]

I m happy that it is almost working for you :)
Indeed exceptions are not handled, but i can have a look at it... although the 
current implementation is too dynamic to handle it.

Original comment by [email protected] on 16 Oct 2008 at 8:10

[GoogleCodeExporter]

I said "it was working" because it was checking the website with no error. But 
now,
it appears that it didn't detect any change. As a test, I'm checking the 
slashdot
page since this morning : specto doesn't see any change (which is obviously not 
true)...

Sad :-(

Original comment by [email protected] on 16 Oct 2008 at 9:11

[GoogleCodeExporter]

ok, this requires some debugging:

could you put this in spectlib/plugins/watch_web_static.py between line nr 122 
and 
123:
print self.page_source

this prints the source from the webpage to the terminal.
Copy it and send it to me (my username + "@gmail.com")

Sorry but i don't use proxies so i could not test this decently.

Original comment by [email protected] on 16 Oct 2008 at 9:17

[GoogleCodeExporter]

printing the content print the source of the page so all seems to be fine.

I attach the result because there's nothing confidential (only 3 websites :
taskcoach, slashdot, linuxfr.org/journal )

Original comment by [email protected] on 16 Oct 2008 at 9:40

Attachments:

• web.txt

[GoogleCodeExporter]

very strange, indeed the correct website is fetched (so proxy support works)
now add after line nr 123
print self.diff

this will show the difference between the cached source and the new source, so 
you 
can see if something actually changed or not.

It is just an idea, but maybe the page is fetched from the proxies cache? so no 
changes are visible?

Original comment by [email protected] on 16 Oct 2008 at 9:49

[GoogleCodeExporter]

It seems that the diff is correct. It output exactly the change between the 
current
page and the page like it was last time I launched specto.

Is your tarball working for you for those website ?

Original comment by [email protected] on 16 Oct 2008 at 9:57

[GoogleCodeExporter]

I can not test it right now, but this is how it checks if the website was 
updated:
f self.cached and self.diff and (self.filesize_difference  >= 
float(self.error_margin)*100) and (self.filesize_difference != 0.0)

self.cached = not first check
self.diff = is filled (as you could see when you printed it)
self.filesize_difference = difference in size between previous and new (correct 
i 
guess)

so maybe your error margin is too high?

or try to change on line 171 this:  and self.diff
to: and self.diff <> ""

Hope this helps and thank you for testing!

Original comment by [email protected] on 16 Oct 2008 at 10:04

[GoogleCodeExporter]

After some debugging, it appears that the error is in the *100 factor. My 
treshold
was at 0.9% which seems to be quite low. But after looking at the number, it 
really
looks like this "100" factor is not needed at all.

Putting this factor at "1" seems to have a perfect behaviour.

Original comment by [email protected] on 16 Oct 2008 at 12:31

[GoogleCodeExporter]

hmm strange :) 
But thank you for finding the bug!
I will also test it when I'm home, but it is possible that the percentage is 
already 
converted when I rewrote specto.

Original comment by [email protected] on 16 Oct 2008 at 12:34

[GoogleCodeExporter]

mmm, no there's definitly something else because the rss watch suffer of the 
same
problem and that, when changing the "100" factor, the popup display a change of 
121%.

Original comment by [email protected] on 16 Oct 2008 at 12:47

[GoogleCodeExporter]

The rss watch uses the same logic, so if one is broken, also is the other...

Maybe you should look at this line:
self.filesize_difference = (fabs(int(self.new_filesize) - 
int(self.old_filesize)) / 
int(self.old_filesize))*100

it should give the difference in filesize as a percentage and is also used in 
the 
balloon.
so print it and see what happens :)

Original comment by [email protected] on 16 Oct 2008 at 12:54

[GoogleCodeExporter]

OK, this feature doesn't work, and if I'm not mistaken on this, then it means 
that
it's a major security hole.

Two test cases to prove that proxies don't seem to be taken into account:
1- go in the gnome proxy settings and point to some fake host (in my 
screenshot, I
point to "whatever" as a socks proxy). In such a case, if specto really takes 
proxies
into account, it should fail with errors like "can't resolve the host" and such
(epiphany does that). In my case, I saw all my mail and website watches refresh
perfectly, thus bypassing my SSH tunnel proxy. This is bad.
2- if one makes an SSH tunnel as a proxy using the following command: "ssh -v 
-N -D
8000 the-remote-host", then whenever an application uses this proxy, we should 
see
network channels opening in the terminal... In Specto's case, this doesn't 
happen (it
does in Pidgin/Epiphany's case).

Am I missing something fundamental here? This basically looks to me like Specto
ignoring my gnome proxy settings, and thus bypassing any security I may have 
set up
over untrusted network connections.

Original comment by [email protected] on 6 Mar 2009 at 8:08

• Changed state: Started
• Added labels: DevBranch-All, Priority-Critical, Security, Type-Defect
• Removed labels: DevBranch-Wout, Patch, Priority-Low, Type-Enhancement

Attachments:

• [whatever proxy.png](https://storage.googleapis.com/google-code-attachments/specto/issue-135/comment-24/whatever proxy.png)

[GoogleCodeExporter]

Took a look at the code, it seems we only have http and https proxy support, no 
ftp
and Socks proxy support. I don't know if urllib2.ProxyHandler would handle that.
Maybe we could look into python-libproxy too.

But there's still the problem that if I put a fake host as an http or https 
proxy,
specto still refreshes as if nothing bad was going on (it /should/ be failing,
afaik). Pretty suspicious. Further investigation welcome.

Original comment by [email protected] on 7 Mar 2009 at 1:26

• Changed state: New

[GoogleCodeExporter]

Too complex according to Wout, so the full proxy support will have to wait.

Original comment by [email protected] on 3 Apr 2009 at 4:16

• Added labels: Milestone-Release0.4
• Removed labels: Milestone-Release0.3

[GoogleCodeExporter]

Original comment by [email protected] on 3 Oct 2009 at 3:02

• Changed state: Accepted

[GoogleCodeExporter]

http://socksipy.sourceforge.net/ is what we are looking for. We only need to 
use that 
module in Specto's web_proxy.py.

Original comment by [email protected] on 7 Oct 2009 at 2:29

[GoogleCodeExporter]

http://en.opensuse.org/GNOME/Proxy_configuration sheds some light on the mess 
that are 
gconf keys for proxy support in GNOME.

Original comment by [email protected] on 7 Oct 2009 at 5:58

[GoogleCodeExporter]

I forgot there was the start of an implementation here: 
https://code.launchpad.net/~kiddo/specto/socks-proxy

But, for some reason, it breaks (watch plugins don't load).

Original comment by [email protected] on 13 Nov 2010 at 3:46