Tor plugin fails with BBC downloads

Re: Tor plugin fails with BBC downloads

tong2shot — Tue, 03 Jun 2014 22:52:49 +0000

KES Wrote:
-------------------------------------------------------
> Will try but not hopeful as configure of proxy
> still a mess to me.
>
> Still no work. tried many proxies from
> [www.xroxy.com]
> =&country=GB&latency=&reliability=&sort=reliabilit

Define no work.

> with differing RTMP ports
> and CDN and cannot get anything that works.

Quote
As @TVjunkie said some proxies can't connect to Akamai CDN, so if you choose Akamai and you get 0 byte flv, that indicates that the proxy can't connect to Akamai, so you have to choose the other (Limelight or Level3). At this moment, Limelight is unstable, so you may have to try several times before the file can be downloaded.

How do you know which stream, which CDN get chosen eventually? Start FRD in debug mode, try to download the file, then open app.log. Search for : "Selected stream", CDN is mentioned as 'supplier'.

How to start FRD in debug mode : see "Troubleshooting" section of this page.

Where is app.log? Help -> Browse to log file.

> Extremly boring....if others can get it to work
> why can't they give the FULL details of how.

They already gave the FULL details.

> And if according to plugin settings if TOR is not
> used when in UK or using UK proxy then does tor
> work when you're not? because it doesn't seem to.

It was mentioned several times :

Quote
tor exit node ip address produced by tor network cannot be identified as UK ip address

in short at this moment Tor is not working for BBC plugin. We put the option, so that if in the future Tor is working, you can just enable it without having to use proxy.

Re: Tor plugin fails with BBC downloads

KES — Sun, 01 Jun 2014 04:15:52 +0000

Will try but not hopeful as configure of proxy still a mess to me.

Still no work. tried many proxies from [www.xroxy.com] with differing RTMP ports and CDN and cannot get anything that works.

Extremly boring....if others can get it to work why can't they give the FULL details of how.

And if according to plugin settings if TOR is not used when in UK or using UK proxy then does tor work when you're not? because it doesn't seem to.

Re: Tor plugin fails with BBC downloads

tong2shot — Sun, 01 Jun 2014 01:49:00 +0000

TVjunkie Wrote:
-------------------------------------------------------
> .....
> I know this is a very small bug (possibly windows
> only?), could it be fixed, please?
> (I could not figure out for more than 15min why my
> player wasn't showing the subs...:S)

Thanks for the report.

Fixed in plugin v1.1.12, also there is option to disable Tor.

Re: Tor plugin fails with BBC downloads

TVjunkie — Fri, 30 May 2014 01:43:31 +0000

Hello :-)

OS: WinVista SP2 x86
JRE 7u60
FRD 0.9u4
bbc.frp 1.1.11
(its settings are:
480p
1935 (default)
Limelight
"Download subtitles" is ticked)
UK http proxy configured in FRD.

Sample TV file:

BBC Radio 1's Big Weekend - 2014: Calvin Harris & Example

The video file does download (eventually) from the Limelight CDN,
but as its file name contains the "wild" character "&",

BBC Radio 1's Big Weekend - 2014 - Calvin Harris & Example.flv

the corresponding subtitle file (.srt) does not have a matching file name;

BBC Radio 1's Big Weekend - 2014 - Calvin Harris &_amp; Example.srt

(I have included an underscore between "&" and "amp;", else the forum software 
wouldn't show it correctly...)

I know this is a very small bug (possibly windows only?), could it be fixed, please?
(I could not figure out for more than 15min why my player wasn't showing the subs...:S)

Many regards

Re: Tor plugin fails with BBC downloads

tong2shot — Thu, 29 May 2014 22:35:35 +0000

KES Wrote:
-------------------------------------------------------
> Please help those of us that are not so savvy as
> you.
>
> EXACTLY how to make bbc.co.uk programs
> downloadable. Clear simple easy to follow
> instructions with examples of working proxies and
> port numbers etc.
>
> [www.bbc.co.uk]
> /Rockets_Island_Series_2_The_Mermaids_Song/

For example of working proxies you can google/bing/yahoo (I'm being indifferent here :-) ) it.

Once you have one, you can configure FRD to use the proxy (Options->Preferences->Connections->Configure proxy..).

Now to configure BBC plugin (Options->Preferences->Plugins->Settings), select bbc.co.uk, then press "Options" button. Choose your preferred quality level. You can leave RTMP port with the default value (1935), unless the proxy that you are using blocks port 1935.

As @TVjunkie said some proxies can't connect to Akamai CDN, so if you choose Akamai and you get 0 byte flv, that indicates that the proxy can't connect to Akamai, so you have to choose the other (Limelight or Level3). At this moment, Limelight is unstable, so you may have to try several times before the file can be downloaded.

For the details, you can read @TVjunkie posts, it's very informative.

Re: Tor plugin fails with BBC downloads

KES — Thu, 29 May 2014 02:07:44 +0000

Please help those of us that are not so savvy as you.

EXACTLY how to make bbc.co.uk programs downloadable. Clear simple easy to follow instructions with examples of working proxies and port numbers etc.

[www.bbc.co.uk]

Re: Tor plugin fails with BBC downloads

TVjunkie — Sun, 25 May 2014 01:53:40 +0000

tong2shot wrote:
-------------------------------------------------------

> How about adding CDN (Akamai, Limelight, Level3) selection in the plugin option instead?
> If the preferred CDN doesn't exist in the streams, then we'll apply this rule : Level3 > Limelight > Akamai.

> BBC plugin v1.1.11 now supports CDN selection.

> Use plugin v1.1.11 then choose limelight or level3 as the preferred CDN in the plugin option.

Hi :-)

I've just updated directly from bbc.frp 1.1.9 to 1.1.11 and conducted some quick checks...

1) It's been more than 5 days now that the tor plugin isn't producing the desired effect for BBC iPlayer;
it appears they have blocked the Tor Network. Yesterday I ran the same test you did, i.e. I installed the
latest Tor Browser Bundle for Windows (v3.6.1) and modified Tor's torrc file (by adding these lines:

ExitNodes {GB}
StrictExitNodes 1
NumEntryGuards 6
KeepalivePeriod 60
NewCircuitPeriod 15

)

to configure a UK exit node; it did give me this UK IP: 37.130.227.133,
but sadly it is treated as non-UK by iplayer's geolocation mechanism :X
(this same IP address does still work with ITVplayer, though - thank God for that -
so the itv.frp is still good to go without the need for a UK proxy).
I am not very optimistic about things returning to the way they were, so
maybe in the next release of the plugin, bbc.frp could be unhooked from tor.frp
or (which seems better) you could include a "use Tor plugin" selection in the options
(much like the option "use tunlr" that existed back in those glorious days of the hulu plugin...)
for when/if it begins to (hopefully) work again...:S

2) Video file sample link: Click 17/05/2014

This TV show is available both @ 720p (CDNs: Ak, L3) and @ 480p (CDNs: Ak, Ll)

I first set a working UK proxy within FRD's connections, then inside bbc.frp's options I chose 720p /
1935 (default) / Limelight.
The HD version (415 MiB ) began quickly to download from L3 CDN (tu)
I then opted to download the 480p SD version; connection to the Limelight CDNs is still
a trial-and-error thing these days :?, but upon my fifth attempt I managed to get it saving on disk (261 MiB ) (tu).

3) I finally tried saving an audio file, The Official Chart Update 21/05/2014 - Ll CDN,
and by using a UK proxy it indeed fetched the HQ file (28.7 MiB ) (tu), while with a direct connection
it yielded the LQ variant (10.9 MiB ).

So, from the end of an overseas user employing a UK http proxy,
SO FAR SO GOOD! :)-D

Your constant efforts have been highly appreciated!
Much obliged, really!

Re: Tor plugin fails with BBC downloads

tong2shot — Sat, 24 May 2014 21:58:34 +0000

rancor Wrote:
-------------------------------------------------------
> Plugin v1.1.10: problem with download some files
> (0 Bytes) example file
> [www.bbc.co.uk]
> pe-to-the-country-series-12-50-cotswolds Download
> via UK proxy (old plugin working OK).
>
> File
> [www.bbc.co.uk]
> hs-chelsea-flower-show-2014-episode-7 HD 720p
> download direct (polish IP).

Use plugin v1.1.11 then choose limelight or level3 as the preferred CDN in the plugin option.

Re: Tor plugin fails with BBC downloads

tong2shot — Sat, 24 May 2014 21:50:00 +0000

BBC plugin v1.1.11 now supports CDN selection.

Re: Tor plugin fails with BBC downloads

tong2shot — Thu, 22 May 2014 10:32:04 +0000

TVjunkie Wrote:
-------------------------------------------------------
> Unfortunately, I cannot write Java code but if I
> may offer my humble opinion,
> the way to go is for the BBC plugin to replicate
> the downloading behaviour of get_iplayer:
> First attempt to connect and dump from the ak CDNs
> - this will get the UK inhabitants
> (& the users of UK VPNs) covered; if this first
> attempt produces no video file
> (see lines
> code: NetConnection.Connect.Rejected], : Access
> denied!]]]]
> &
> WARNING: video duration is null
> inside the log)
> then it should fall back to the ll CDNs - this
> should cater for the non-UK FRD users who connect
> through a UK proxy.
> The current implementation of selecting either
> only ak CDN (v1.1.10) or ll CDN (v1.1.9)
> cannot meet the needs of both sets of FRD
> users...

How about adding CDN (Akamai, Limelight, Level3) selection in the plugin option instead? If the preferred CDN doesn't exist in the streams, then we'll apply this rule : Level3 > Limelight > Akamai.

Thanks again for the details :-)

Re: Tor plugin fails with BBC downloads

TVjunkie — Thu, 22 May 2014 03:49:48 +0000

rancor wrote (I took the liberty of doing some rewording):
-------------------------------------------------------
> Plugin v1.1.10: problem with downloading some files (0 Bytes)
> via UK proxy (old plugin working OK)
> Example file:
> http://www.bbc.co.uk/iplayer/episode/b01cnyff/escape-to-the-country-series-12-50-cotswolds

tong2shot Wrote:
-------------------------------------------------------
> Do you mean the first link is not working, and the second link is working?
> Or both links are not working?

I can easily reproduce rancor's issue from my Greek original IP.
If I set no UK proxy within FRD (0.9u4) and attempt to download his 1st
sample link, then I quickly get the

Quote
ERROR - This video is not available in your area

message; as you have explained, this is because

Quote
tor exit node ip address produced by tor network cannot be identified as UK ip address, even if we select UK for exit node

in other words, the sad news is that the Beeb have somehow blocked the Tor Network as a way of IP spoofing...:-(
Hulu did it, so I guess they can, too! :X

Now, if I configure FRD to use a UK http proxy (a tested & working one),
then the bbc plugin thinks I am in the UK and according to what you've said

Quote
Also the new plugin (v1.1.10) prefers akamai for UK users

tries to download the video file from the Akamai CDN;
but for reasons I have explained in detail in the first post of this thread,
ak rtmp streams are not downloadable by FRD by simply using a UK proxy...

Quote
they are geo-blocked at FMS level, meaning they are available to UK users (or non-UK ones who use UK VPNs), but off-limits to the rest;
FRD can't connect to those ones when using the Tor Network or a specific UK proxy.

Hence the 0 bytes download I (& rancor) experience

All the above is very well illustrated in the attached debug log, see file "bbc1110_UKproxy_app.log"

On the other hand, previous version of the plugin (1.1.9) gave precedence to the Limelight CDNs
over Akamai; Limelight is downloadable by FRD with UK proxy (albeit after a small number
of retries...)

Quote
... and the only ones who worked for overseas users, since they're not blocked at server level
- these are the ones FRD connects to when used with Tor or a UK proxy.

hence the "(old plugin working OK)" remark by rancor.

I have a backup copy of bbc.frp 1.1.9 and temporarily reverted to it and tested it with FRD 0.9u4.
Using the same UK proxy as before in the case of v1.1.10, now the file soon begins to download
from the ll CDN.
Look at the attached screengrab (file "b01cnyff_UKproxy_bbcfrp119.jpg")
& also at the attached (partial) debug log (file "bbc119_UKproxy_app.log").

Unfortunately, I cannot write Java code but if I may offer my humble opinion,
the way to go is for the BBC plugin to replicate the downloading behaviour of get_iplayer:
First attempt to connect and dump from the ak CDNs - this will get the UK inhabitants
(& the users of UK VPNs) covered; if this first attempt produces no video file
(see lines
code: NetConnection.Connect.Rejected], [STRING(0x02) description: [ AccessManager.Reject ] : Access denied!]]]]
&
WARNING: video duration is null
inside the log)
then it should fall back to the ll CDNs - this should cater for the non-UK FRD users who connect through a UK proxy.
The current implementation of selecting either only ak CDN (v1.1.10) or ll CDN (v1.1.9)
cannot meet the needs of both sets of FRD users...

> For me both links are working just fine, also via UK proxy.

If you don't mind my asking, on which territory are you testing the BBC plugin?
Because if you actually reside within the UK, setting simply a UK proxy inside FRD's prefs
is not enough to replicate FRD's behaviour for an overseas user - rtmp traffic from the ak
CDN will pass successfully via your UK ISP, so it would (falsely) appear that the bbc plugin
1.1.10 succeeds when used with a UK proxy...

Many greetings B)

PS: Huge thanks for the implementation of rtmp-port selection >:D<

Re: Tor plugin fails with BBC downloads

tong2shot — Wed, 21 May 2014 21:45:00 +0000

Do you mean the first link is not working, and the second link is working? Or both links are not working?

For me both links are working just fine, also via UK proxy.

Please run FRD in debug mode, then test working link and not working link, then attach app.log. See "Troubleshooting" section of this page on how to enable debug mode.

Re: Tor plugin fails with BBC downloads

rancor — Wed, 21 May 2014 19:40:21 +0000

Plugin v1.1.10: problem with download some files (0 Bytes) example file [www.bbc.co.uk] Download via UK proxy (old plugin working OK).

File [www.bbc.co.uk] HD 720p download direct (polish IP).

Re: Tor plugin fails with BBC downloads

tong2shot — Wed, 21 May 2014 14:50:42 +0000

TVjunkie Wrote:
-------------------------------------------------------
> It was as if FRD was using my real non-UK IP to
> connect to iplayer servers, resulting in a denial
> for TV content and in the int'l LQ radio version!
> ......
> I took a brief look at it & it appears there is
> some problem with the tor plugin:
>
>
> INFO: Connection: default, country: gr
> INFO: Initializing proxy for 'gb'
> INFO: Starting Orchid (version: 1.0.0)
> INFO: Loading cached network information from
> disk
> INFO: Loading certificates
> INFO: Cannot build circuits because we don't have
> enough directory information
> INFO: Loading consensus
> INFO: Loading microdescriptor cache
> INFO: loading state file
> INFO: =====Saving updated/added files into the
> database (1) =====
> SEVERE: Status code je 500
> INFO: Setting state to ERROR

Actually FRD still uses Tor, if you are not using UK ip address. The Tor plugin is working just fine. The problem is that tor exit node ip address produced by tor network cannot be identified as UK ip address, even if we select UK for exit node ( 'gb' as ISO country code). The same behaviour also reproduce in Tor Browser bundle, with the same config. That means when the plugin requests media selector for TV programme via Tor with 'gb' as exit node, it gets HTTP error 500, with content mentioning about geolocation error. I also tried to open BBC TV programme with Tor Browser bundle with the same config, it showed "BBC iPlayer TV programmes are available to play in the UK only". Not only that, both FRD Tor plugin and Tor Browser bundle with same config produce exactly the same exit node ip address, when I tested them. Maybe exit nodes for 'gb' are so limited, I'm not sure. In plugin v1.1.10 you'll get more meaningful error message for this kind of behaviour.

> Unfortunately, tor.frp can't be disabled and
> bbc.frp is hooked to the former, so for a non-UK
> user like me to regain
> complete BBC iplayer functions would require the
> use of UK proxies in FRD (or the use of a UK VPN
> with
> default rtmp port 1935 open - these are very few
> and usually paid for... - despite my pleas, the
> devs do not want
> to rewrite rtmp.frp so as to try the alternate
> rtmp ports 443,80 when 1935 is closed; that would
> be ideal for using
> several FREE VPNs which always have port 1935
> blocked...).

Plugin v1.1.10 now supports RTMP port selection. You can choose port number 1935 or 80 or 443.

> I can't tell how the tor.frp issue I described
> could affect UK users; in theory, their original
> UK IP should prevail
> over the tor plugin and so they could still
> download stuff (with the known Limelight CDNs
> caveat).
>
> Thanks for your time in reading this...

UK users don't use Tor. Also the new plugin prefers akamai for UK users.
Thanks for the detail info. It helps me a lot.

Tor plugin fails with BBC downloads

TVjunkie — Wed, 21 May 2014 03:05:40 +0000

Hello to all :-)
First, an apology in advance regarding the length of this post;
I'll try to be as detailed as possible and will provide some additional
knowledge regarding the BBC iplayer (tv+radio) streams, with the aim
of better helping the devs (namely tong2shot, who himself said that
he's not too familiar with BBC...) and maybe shed some more light
in other posters' queries...
I, on the other hand, have been following BBC iPlayer since way back
in 2008, in its infancy days...Am also a user of the specialized tool
get_iplayer (used to provide guidance in its mailing list, too), but I do also
like FRD for its simplicity ;-).

As mentioned in some of my previous posts, my physical location is non-UK;
up until last Sunday (18/05/2014), FRD 0.9u3, tor.frp 1.0.0, geoip 1.0.0 &
latest bbc.frp 1.1.9 worked in perfect unison together, so I was able to download
(without using a UK proxy) tv files as well as the UK-only HQ radio (audio) files...
1) Regarding the video-on-demand files, the SD ones (pixel height <=480p) are being
distributed by the Akamai (ak) & Limelight (ll) CDNs.
ak CDNs are very reliable, but they are geo-blocked at FMS level,
meaning they are available to UK users (or non-UK ones who use UK VPNs),
but off-limits to the rest; FRD can't connect to those ones when using the
Tor Network or a specific UK proxy.
ll CDNs used to also be very reliable and the only ones who worked for overseas
users, since they're not blocked at server level - these are the ones FRD connects to
when used with Tor or a UK proxy. Sadly, especially after the introduction of the
new version of TV iplayer, there has been a change implemented on those
(possibly deliberate, to fence off international users, or just a technical glitch, maybe?),
that makes them exhibit a very erratic behaviour, also discussed by tong2shot here.
However, with some perseverance, I was always able to download from those via Tor.
FWIW, this Limelight CDN issue is not unique to FRD: Even in the case where I configure Firefox
to use a UK http proxy and try to play a BBC iplayer TV programme, I have to reload the
page several times for the show to start playing in the browser (via a ll CDN).

HD (720p) TV files are likewise served through Akamai (again, geo-blocked at FMS level)
and Level3 (not geo-blocked) CDNs; Level3 servers are very reliable, FRD could easily
connect to those via Tor plugin/ UK proxy.

2) Regarding audio-on-demand radio files, these are, almost exclusively, served through Limelight CDNs.
If in the UK, there exist two quality variants:
a. "Normal Bandwidth" (HQ) one, which, for the majority of the National Stations, is encoded with
AAC LC @128kbpsABR. The notable exception here is Radio 3 programmes, which are offered at an elevated
bitrate of 320kbpsABR.
b. "Lower Bandwidth" (LQ) one, which is for all National Stations encoded with HE-AACv2 @48kbpsABR.
Judging by the nominal bitrate, LQ files are roughly 1/3 the size of the HQ ones, but in the case of Radio 3 only
3/20 of the HQ size...
For overseas users, all radio files are offered at LQ; the specs of the audio files are the same to their UK LQ counterparts,
but there exist separate international Limelight rtmp streams for non-UK IPs.

So, to recap, on last Sunday afternoon, if I tried to fetch the 480p version of Click 17/05/2014 by using FRD's inate tor plugin,
I could grab it from the ll CDN on the 4th to 6th attempt, whereas if I wanted the 720p version of it,
this could be easily grabbed at first attempt from the l3 CDN.
And if I wanted to save BBC Radio 1's The Official Chart , I would end up with a ~ 170MB fle (= UK HQ version).

Late in the evening of Sunday came the prompt to upgrade FRD to 0.9u4; I did so without a problem,
keeping 0.9u3 version's settings...

In the morning of Monday (19/05/2014), the "ll CDN issue" had become even more pronounced -
I had to retry 8 -10 times to begin the download of a TV programme, but the radio downloads still worked OK,
producing the HQ audio files.

But in the afternoon, there was a totally new situation:
1) Every TV programme tried failed to download, with the ominous message "ERROR - Cannot download from service"
shown on the "Est. Time" column of FRD, immediately after the first download attempt.
2) Radio downloads did work, but now they only produced the LQ versions; e.g. the radio show linked to previously would yield a 65 MB only file...
It was as if FRD was using my real non-UK IP to connect to iplayer servers, resulting in a denial for TV content and in the int'l LQ radio version!

Initially I blamed the malfunction to the upgrade, but I did a downgrade to version 0.9u3
(from a kept backup, just in case...) only to discover that the error persisted :X

I then generated a debug log with v.0.9u4 while trying to download Click@480p (attached, b044trs9_tor_fail.log);
I took a brief look at it & it appears there is some problem with the tor plugin:

INFO: Connection: default, country: gr
INFO: Initializing proxy for 'gb'
INFO: Starting Orchid (version: 1.0.0)
INFO: Loading cached network information from disk
INFO: Loading certificates
INFO: Cannot build circuits because we don't have enough directory information
INFO: Loading consensus
INFO: Loading microdescriptor cache
INFO: loading state file
INFO: =====Saving updated/added files into the database (1) =====
SEVERE: Status code je 500
INFO: Setting state to ERROR

To test my theory, I then configured FRD to use a UK http proxy (79.143.xx.xxx:8080,
not showing the full IP because the proxy will get abused...), and lo & behold
the video file quickly started to download (see attachment b044trs9_UKproxy_success.jpg).

The same goes for audio downloads - my default connection with the (malfunctioning)
tor plugin produces the LQ file (see attachment b043jk4j_tor_LQ.jpg), while using a UK
http proxy in FRD (which overrides the tor plugin) fetches the HQ file (attachment b043jk4j_UKproxy_HQ.jpg).

Unfortunately, tor.frp can't be disabled and bbc.frp is hooked to the former, so for a non-UK user like me to regain
complete BBC iplayer functions would require the use of UK proxies in FRD (or the use of a UK VPN with
default rtmp port 1935 open - these are very few and usually paid for... - despite my pleas, the devs do not want
to rewrite rtmp.frp so as to try the alternate rtmp ports 443,80 when 1935 is closed; that would be ideal for using
several FREE VPNs which always have port 1935 blocked...).

If p.vanderwal isn't located inside the UK, then maybe he is in the same boat as I...

I can't tell how the tor.frp issue I described could affect UK users; in theory, their original UK IP should prevail
over the tor plugin and so they could still download stuff (with the known Limelight CDNs caveat).

Thanks for your time in reading this...