WordRider Home
Tor plugin fails with BBC downloads
May 21, 2014 03:05AM
Hello to all smiling smiley
First, an apology in advance regarding the length of this post;
I'll try to be as detailed as possible and will provide some additional
knowledge regarding the BBC iplayer (tv+radio) streams, with the aim
of better helping the devs (namely tong2shot, who himself said that
he's not too familiar with BBC...) and maybe shed some more light
in other posters' queries...
I, on the other hand, have been following BBC iPlayer since way back
in 2008, in its infancy days...Am also a user of the specialized tool
get_iplayer (used to provide guidance in its mailing list, too), but I do also
like FRD for its simplicity winking smiley.

As mentioned in some of my previous posts, my physical location is non-UK;
up until last Sunday (18/05/2014), FRD 0.9u3, tor.frp 1.0.0, geoip 1.0.0 &
latest bbc.frp 1.1.9 worked in perfect unison together, so I was able to download
(without using a UK proxy) tv files as well as the UK-only HQ radio (audio) files...
1) Regarding the video-on-demand files, the SD ones (pixel height <=480p) are being
distributed by the Akamai (ak) & Limelight (ll) CDNs.
ak CDNs are very reliable, but they are geo-blocked at FMS level,
meaning they are available to UK users (or non-UK ones who use UK VPNs),
but off-limits to the rest; FRD can't connect to those ones when using the
Tor Network or a specific UK proxy.
ll CDNs used to also be very reliable and the only ones who worked for overseas
users, since they're not blocked at server level - these are the ones FRD connects to
when used with Tor or a UK proxy. Sadly, especially after the introduction of the
new version of TV iplayer, there has been a change implemented on those
(possibly deliberate, to fence off international users, or just a technical glitch, maybe?),
that makes them exhibit a very erratic behaviour, also discussed by tong2shot here.
However, with some perseverance, I was always able to download from those via Tor.
FWIW, this Limelight CDN issue is not unique to FRD: Even in the case where I configure Firefox
to use a UK http proxy and try to play a BBC iplayer TV programme, I have to reload the
page several times for the show to start playing in the browser (via a ll CDN).

HD (720p) TV files are likewise served through Akamai (again, geo-blocked at FMS level)
and Level3 (not geo-blocked) CDNs; Level3 servers are very reliable, FRD could easily
connect to those via Tor plugin/ UK proxy.

2) Regarding audio-on-demand radio files, these are, almost exclusively, served through Limelight CDNs.
If in the UK, there exist two quality variants:
a. "Normal Bandwidth" (HQ) one, which, for the majority of the National Stations, is encoded with
AAC LC @128kbpsABR. The notable exception here is Radio 3 programmes, which are offered at an elevated
bitrate of 320kbpsABR.
b. "Lower Bandwidth" (LQ) one, which is for all National Stations encoded with HE-AACv2 @48kbpsABR.
Judging by the nominal bitrate, LQ files are roughly 1/3 the size of the HQ ones, but in the case of Radio 3 only
3/20 of the HQ size...
For overseas users, all radio files are offered at LQ; the specs of the audio files are the same to their UK LQ counterparts,
but there exist separate international Limelight rtmp streams for non-UK IPs.

So, to recap, on last Sunday afternoon, if I tried to fetch the 480p version of Click 17/05/2014 by using FRD's inate tor plugin,
I could grab it from the ll CDN on the 4th to 6th attempt, whereas if I wanted the 720p version of it,
this could be easily grabbed at first attempt from the l3 CDN.
And if I wanted to save BBC Radio 1's The Official Chart , I would end up with a ~ 170MB fle (= UK HQ version).

Late in the evening of Sunday came the prompt to upgrade FRD to 0.9u4; I did so without a problem,
keeping 0.9u3 version's settings...

In the morning of Monday (19/05/2014), the "ll CDN issue" had become even more pronounced -
I had to retry 8 -10 times to begin the download of a TV programme, but the radio downloads still worked OK,
producing the HQ audio files.

But in the afternoon, there was a totally new situation:
1) Every TV programme tried failed to download, with the ominous message "ERROR - Cannot download from service"
shown on the "Est. Time" column of FRD, immediately after the first download attempt.
2) Radio downloads did work, but now they only produced the LQ versions; e.g. the radio show linked to previously would yield a 65 MB only file...
It was as if FRD was using my real non-UK IP to connect to iplayer servers, resulting in a denial for TV content and in the int'l LQ radio version!

Initially I blamed the malfunction to the upgrade, but I did a downgrade to version 0.9u3
(from a kept backup, just in case...) only to discover that the error persisted angry smiley

I then generated a debug log with v.0.9u4 while trying to download Click@480p (attached, b044trs9_tor_fail.log);
I took a brief look at it & it appears there is some problem with the tor plugin:

INFO: Connection: default, country: gr
INFO: Initializing proxy for 'gb'
INFO: Starting Orchid (version: 1.0.0)
INFO: Loading cached network information from disk
INFO: Loading certificates
INFO: Cannot build circuits because we don't have enough directory information
INFO: Loading consensus
INFO: Loading microdescriptor cache
INFO: loading state file
INFO: =====Saving updated/added files into the database (1) =====
SEVERE: Status code je 500
INFO: Setting state to ERROR

To test my theory, I then configured FRD to use a UK http proxy (79.143.xx.xxx:8080,
not showing the full IP because the proxy will get abused...), and lo & behold
the video file quickly started to download (see attachment b044trs9_UKproxy_success.jpg).

The same goes for audio downloads - my default connection with the (malfunctioning)
tor plugin produces the LQ file (see attachment b043jk4j_tor_LQ.jpg), while using a UK
http proxy in FRD (which overrides the tor plugin) fetches the HQ file (attachment b043jk4j_UKproxy_HQ.jpg).

Unfortunately, tor.frp can't be disabled and bbc.frp is hooked to the former, so for a non-UK user like me to regain
complete BBC iplayer functions would require the use of UK proxies in FRD (or the use of a UK VPN with
default rtmp port 1935 open - these are very few and usually paid for... - despite my pleas, the devs do not want
to rewrite rtmp.frp so as to try the alternate rtmp ports 443,80 when 1935 is closed; that would be ideal for using
several FREE VPNs which always have port 1935 blocked...).

If p.vanderwal isn't located inside the UK, then maybe he is in the same boat as I...

I can't tell how the tor.frp issue I described could affect UK users; in theory, their original UK IP should prevail
over the tor plugin and so they could still download stuff (with the known Limelight CDNs caveat).

Thanks for your time in reading this...
Attachments:
open | download - b044trs9_tor_fail.log (10.5 KB)
open | download - b044trs9_UKproxy_success.jpg (46.2 KB)
open | download - b043jk4j_tor_LQ.jpg (46 KB)
open | download - b043jk4j_UKproxy_HQ.jpg (46.4 KB)
Re: Tor plugin fails with BBC downloads
May 21, 2014 02:50PM
TVjunkie Wrote:
-------------------------------------------------------
> It was as if FRD was using my real non-UK IP to
> connect to iplayer servers, resulting in a denial
> for TV content and in the int'l LQ radio version!
> ......
> I took a brief look at it & it appears there is
> some problem with the tor plugin:
>
>
> INFO: Connection: default, country: gr
> INFO: Initializing proxy for 'gb'
> INFO: Starting Orchid (version: 1.0.0)
> INFO: Loading cached network information from
> disk
> INFO: Loading certificates
> INFO: Cannot build circuits because we don't have
> enough directory information
> INFO: Loading consensus
> INFO: Loading microdescriptor cache
> INFO: loading state file
> INFO: =====Saving updated/added files into the
> database (1) =====
> SEVERE: Status code je 500
> INFO: Setting state to ERROR

Actually FRD still uses Tor, if you are not using UK ip address. The Tor plugin is working just fine. The problem is that tor exit node ip address produced by tor network cannot be identified as UK ip address, even if we select UK for exit node ( 'gb' as ISO country code). The same behaviour also reproduce in Tor Browser bundle, with the same config. That means when the plugin requests media selector for TV programme via Tor with 'gb' as exit node, it gets HTTP error 500, with content mentioning about geolocation error. I also tried to open BBC TV programme with Tor Browser bundle with the same config, it showed "BBC iPlayer TV programmes are available to play in the UK only". Not only that, both FRD Tor plugin and Tor Browser bundle with same config produce exactly the same exit node ip address, when I tested them. Maybe exit nodes for 'gb' are so limited, I'm not sure. In plugin v1.1.10 you'll get more meaningful error message for this kind of behaviour.


> Unfortunately, tor.frp can't be disabled and
> bbc.frp is hooked to the former, so for a non-UK
> user like me to regain
> complete BBC iplayer functions would require the
> use of UK proxies in FRD (or the use of a UK VPN
> with
> default rtmp port 1935 open - these are very few
> and usually paid for... - despite my pleas, the
> devs do not want
> to rewrite rtmp.frp so as to try the alternate
> rtmp ports 443,80 when 1935 is closed; that would
> be ideal for using
> several FREE VPNs which always have port 1935
> blocked...).

Plugin v1.1.10 now supports RTMP port selection. You can choose port number 1935 or 80 or 443.




> I can't tell how the tor.frp issue I described
> could affect UK users; in theory, their original
> UK IP should prevail
> over the tor plugin and so they could still
> download stuff (with the known Limelight CDNs
> caveat).
>
> Thanks for your time in reading this...

UK users don't use Tor. Also the new plugin prefers akamai for UK users.
Thanks for the detail info. It helps me a lot.
Re: Tor plugin fails with BBC downloads
May 21, 2014 07:40PM
Plugin v1.1.10: problem with download some files (0 Bytes) example file [www.bbc.co.uk] Download via UK proxy (old plugin working OK).

File [www.bbc.co.uk] HD 720p download direct (polish IP).



Edited 1 time(s). Last edit at 05/21/2014 07:44PM by rancor.
Re: Tor plugin fails with BBC downloads
May 21, 2014 09:45PM
Do you mean the first link is not working, and the second link is working? Or both links are not working?

For me both links are working just fine, also via UK proxy.

Please run FRD in debug mode, then test working link and not working link, then attach app.log. See "Troubleshooting" section of this page on how to enable debug mode.
Re: Tor plugin fails with BBC downloads
May 22, 2014 03:49AM
rancor wrote (I took the liberty of doing some rewording):
-------------------------------------------------------
> Plugin v1.1.10: problem with downloading some files (0 Bytes)
> via UK proxy (old plugin working OK)
> Example file:
> http://www.bbc.co.uk/iplayer/episode/b01cnyff/escape-to-the-country-series-12-50-cotswolds

tong2shot Wrote:
-------------------------------------------------------
> Do you mean the first link is not working, and the second link is working?
> Or both links are not working?

I can easily reproduce rancor's issue from my Greek original IP.
If I set no UK proxy within FRD (0.9u4) and attempt to download his 1st
sample link, then I quickly get the

Quote
ERROR - This video is not available in your area

message; as you have explained, this is because

Quote
tor exit node ip address produced by tor network cannot be identified as UK ip address, even if we select UK for exit node

in other words, the sad news is that the Beeb have somehow blocked the Tor Network as a way of IP spoofing...sad smiley
Hulu did it, so I guess they can, too! angry smiley

Now, if I configure FRD to use a UK http proxy (a tested & working one),
then the bbc plugin thinks I am in the UK and according to what you've said

Quote
Also the new plugin (v1.1.10) prefers akamai for UK users

tries to download the video file from the Akamai CDN;
but for reasons I have explained in detail in the first post of this thread,
ak rtmp streams are not downloadable by FRD by simply using a UK proxy...

Quote
they are geo-blocked at FMS level, meaning they are available to UK users (or non-UK ones who use UK VPNs), but off-limits to the rest;
FRD can't connect to those ones when using the Tor Network or a specific UK proxy.


Hence the 0 bytes download I (& rancor) experience

All the above is very well illustrated in the attached debug log, see file "bbc1110_UKproxy_app.log"

On the other hand, previous version of the plugin (1.1.9) gave precedence to the Limelight CDNs
over Akamai; Limelight is downloadable by FRD with UK proxy (albeit after a small number
of retries...)

Quote
... and the only ones who worked for overseas users, since they're not blocked at server level
- these are the ones FRD connects to when used with Tor or a UK proxy.


hence the "(old plugin working OK)" remark by rancor.

I have a backup copy of bbc.frp 1.1.9 and temporarily reverted to it and tested it with FRD 0.9u4.
Using the same UK proxy as before in the case of v1.1.10, now the file soon begins to download
from the ll CDN.
Look at the attached screengrab (file "b01cnyff_UKproxy_bbcfrp119.jpg")
& also at the attached (partial) debug log (file "bbc119_UKproxy_app.log").

Unfortunately, I cannot write Java code but if I may offer my humble opinion,
the way to go is for the BBC plugin to replicate the downloading behaviour of get_iplayer:
First attempt to connect and dump from the ak CDNs - this will get the UK inhabitants
(& the users of UK VPNs) covered; if this first attempt produces no video file
(see lines
code: NetConnection.Connect.Rejected], [STRING(0x02) description: [ AccessManager.Reject ] : Access denied!]]]]
&
WARNING: video duration is null
inside the log)
then it should fall back to the ll CDNs - this should cater for the non-UK FRD users who connect through a UK proxy.
The current implementation of selecting either only ak CDN (v1.1.10) or ll CDN (v1.1.9)
cannot meet the needs of both sets of FRD users...

> For me both links are working just fine, also via UK proxy.

If you don't mind my asking, on which territory are you testing the BBC plugin?
Because if you actually reside within the UK, setting simply a UK proxy inside FRD's prefs
is not enough to replicate FRD's behaviour for an overseas user - rtmp traffic from the ak
CDN will pass successfully via your UK ISP, so it would (falsely) appear that the bbc plugin
1.1.10 succeeds when used with a UK proxy...

Many greetings cool smiley

PS: Huge thanks for the implementation of rtmp-port selection thumbs up smiley
Attachments:
open | download - bbc1110_UKproxy_app.log (18.3 KB)
open | download - b01cnyff_UKproxy_bbcfrp119.jpg (47 KB)
open | download - bbc119_UKproxy_app.log (17.7 KB)
Re: Tor plugin fails with BBC downloads
May 22, 2014 10:32AM
TVjunkie Wrote:
-------------------------------------------------------
> Unfortunately, I cannot write Java code but if I
> may offer my humble opinion,
> the way to go is for the BBC plugin to replicate
> the downloading behaviour of get_iplayer:
> First attempt to connect and dump from the ak CDNs
> - this will get the UK inhabitants
> (& the users of UK VPNs) covered; if this first
> attempt produces no video file
> (see lines
> code: NetConnection.Connect.Rejected], : Access
> denied!]]]]
> &
> WARNING: video duration is null
> inside the log)
> then it should fall back to the ll CDNs - this
> should cater for the non-UK FRD users who connect
> through a UK proxy.
> The current implementation of selecting either
> only ak CDN (v1.1.10) or ll CDN (v1.1.9)
> cannot meet the needs of both sets of FRD
> users...

How about adding CDN (Akamai, Limelight, Level3) selection in the plugin option instead? If the preferred CDN doesn't exist in the streams, then we'll apply this rule : Level3 > Limelight > Akamai.

Thanks again for the details smiling smiley



Edited 1 time(s). Last edit at 05/22/2014 10:32AM by tong2shot.
Re: Tor plugin fails with BBC downloads
May 24, 2014 09:50PM
BBC plugin v1.1.11 now supports CDN selection.
Re: Tor plugin fails with BBC downloads
May 24, 2014 09:58PM
rancor Wrote:
-------------------------------------------------------
> Plugin v1.1.10: problem with download some files
> (0 Bytes) example file
> [www.bbc.co.uk]
> pe-to-the-country-series-12-50-cotswolds Download
> via UK proxy (old plugin working OK).
>
> File
> [www.bbc.co.uk]
> hs-chelsea-flower-show-2014-episode-7 HD 720p
> download direct (polish IP).

Use plugin v1.1.11 then choose limelight or level3 as the preferred CDN in the plugin option.
Re: Tor plugin fails with BBC downloads
May 25, 2014 01:53AM
tong2shot wrote:
-------------------------------------------------------

> How about adding CDN (Akamai, Limelight, Level3) selection in the plugin option instead?
> If the preferred CDN doesn't exist in the streams, then we'll apply this rule : Level3 > Limelight > Akamai.

> BBC plugin v1.1.11 now supports CDN selection.

> Use plugin v1.1.11 then choose limelight or level3 as the preferred CDN in the plugin option.

Hi smiling smiley

I've just updated directly from bbc.frp 1.1.9 to 1.1.11 and conducted some quick checks...

1) It's been more than 5 days now that the tor plugin isn't producing the desired effect for BBC iPlayer;
it appears they have blocked the Tor Network. Yesterday I ran the same test you did, i.e. I installed the
latest Tor Browser Bundle for Windows (v3.6.1) and modified Tor's torrc file (by adding these lines:
ExitNodes {GB}
StrictExitNodes 1
NumEntryGuards 6
KeepalivePeriod 60
NewCircuitPeriod 15
)

to configure a UK exit node; it did give me this UK IP: 37.130.227.133,
but sadly it is treated as non-UK by iplayer's geolocation mechanism angry smiley
(this same IP address does still work with ITVplayer, though - thank God for that -
so the itv.frp is still good to go without the need for a UK proxy).
I am not very optimistic about things returning to the way they were, so
maybe in the next release of the plugin, bbc.frp could be unhooked from tor.frp
or (which seems better) you could include a "use Tor plugin" selection in the options
(much like the option "use tunlr" that existed back in those glorious days of the hulu plugin...)
for when/if it begins to (hopefully) work again...confused smiley

2) Video file sample link: Click 17/05/2014

This TV show is available both @ 720p (CDNs: Ak, L3) and @ 480p (CDNs: Ak, Ll)

I first set a working UK proxy within FRD's connections, then inside bbc.frp's options I chose 720p /
1935 (default) / Limelight.
The HD version (415 MiB ) began quickly to download from L3 CDN thumbs up
I then opted to download the 480p SD version; connection to the Limelight CDNs is still
a trial-and-error thing these days moody smiley, but upon my fifth attempt I managed to get it saving on disk (261 MiB ) thumbs up.

3) I finally tried saving an audio file, The Official Chart Update 21/05/2014 - Ll CDN,
and by using a UK proxy it indeed fetched the HQ file (28.7 MiB ) thumbs up, while with a direct connection
it yielded the LQ variant (10.9 MiB ).

So, from the end of an overseas user employing a UK http proxy,
SO FAR SO GOOD! smileys with beer

Your constant efforts have been highly appreciated!
Much obliged, really!
KES
Re: Tor plugin fails with BBC downloads
May 29, 2014 02:07AM
Please help those of us that are not so savvy as you.

EXACTLY how to make bbc.co.uk programs downloadable. Clear simple easy to follow instructions with examples of working proxies and port numbers etc.

[www.bbc.co.uk]
Sorry, only registered users may post in this forum.

Click here to login