My personal notes on all things technical and factual....

Clayton's Tech Bits

Search this site:

Custom Search

Sun, 14 Feb 2010

/Security/circumvention: Downloading Files When You Are Being Blocked

I live in China, and I have to deal with file downloads being blocked all the time. Particularly of the Canadian and US news podcasts I am fond of watching. For the record, it is hard to tell whether the Chinese are censoring, or whether the providers are trying to save bandwidth by blocking all of China. I think there is a little bit of both going on. So here is how I deal with it.....

A lot of things are not blocked, and for that I use Miro[1]. Miro has a lot of nice features for video podcatching. In my often bandwidth-starved situation, chief among them is that Miro is pretty good at resuming interrupted downloads, even after an over-night shutdown.

For blocked podcasts, I have an encrypted SSH tunnel setup from my desktop to one of my servers in the USA. Setup ssh, autossh, and proxychains per these posts[2]. With my tunnel setup, I use a second piece of podcatching software called gpodder[3], which I start in a terminal like this:

proxychains gpodder&

to force all gpodder traffic through my encrypted tunnel. Then use gpodder per normal to download blocked podcasts. However....

In China, there are often extended periods of time when the powers-that-be use the Great Firewall to interfere with these kind of downloads. Sometimes there are repeated network interruptions that cause large files to fail before there download completes. Sometimes this seems to be combined with bandwidth throttling, where each network connection I make is limited to 5 kb/s of download, which makes the download VERY long, and even more prone to interruptions.

For these particularly difficult situations, I just use gpodder to pull down the list of files from the podcast feeds. Then I use a Firefox plugin called "SQLite Manager"[4] to open gpodder's datebase in ~/.config/gpodder/database.sqlite. In the table called "episodes" can be found a record for each podcast that has been pulled down from the feed. From that record I can extract the actual URL of the file for the podcast, so that I can then download it with wget[5], which is an extremely robust command-line file downloader.

Then I can start the download in a terminal like this:

proxychains wget http://www.url.com/path/to/file/filename.something

Continuing a partially download podcast is as simple as:

proxychains wget -c http://www.url.com/path/to/file/filename.something

(Note the "-c".) It is also worth trying the above wget line first without proxychains, as sometimes only the feed URL is blocked, but the server where the actual files reside is not blocked.

[1] http://www.getmiro.com
[2] http://blog.langex.net/index.cgi/Admin/SSH-Proxy/ [3] http://gpodder.org/
[4] https://addons.mozilla.org/en-US/firefox/addon/5817
[5] http://www.gnu.org/software/wget/

posted at: 12:14 | path: /Security/circumvention | permanent link to this entry

Home

Contact

Resumé / C.V.

Links

Categories:

Archives:

Subscribe XML RSS Feed

Sun, 14 Feb 2010