DivShare.com plugin; Can't make it myself so going to help you

Hi,

I use a filesharing service called DivShare [www.divshare.com], but there isn't a plugin for that in FRD.

DivShare threats the user a a clicking monkey and throws a popup and lots of ads, that's why FRD would be perfect for it.

As I have no experience with Java, and I'd rather not download and install JDK, some Java IDE, and download the whole FRD source repository just to make a single plugin, I decided to help somebody make it.

I fully understand if nobody has time or is willing to make it, but the topic is always here for you if you have free time at some point. :)

OK, let's get started... The shareable links look like this:

[www.divshare.com]

There are seven X's which are always numbers, then a dash, and three Y's which can be anything alphanumerical.

Examples:

[www.divshare.com]
[www.divshare.com]

File name source code:

Language: HTML
<div class="file_name"><img src="/images/icons/files_16/file_generic.gif" valign="absmiddle"> FileName </div>

(note the spaces around it)

File size source code:

Language: HTML
<b>File Size:</b> SizeInMegs <span class="tiny">MB</span><br />

There's a link to the second download page in the first one, formatted like this:

[www.divshare.com]

The source code around that link looks like this:

Language: HTML
<a href="/i/8976227-560" class="" style="font-size:11pt;">Download This File</a>

The user can directly go to the "i" URL to bypass one page. Maybe this can somehow be utilized?

To get to the third page, one must either find the small link in the corner named "Skip Ad >" or wait the 15 second timer.

Obviously, finding small text is easy enough for a computer, and the source code around it is:

Language: HTML
<a href="/download/launch/8976227-560">Skip Ad ></a>

The third page's URL is formatted as following:

[www.divshare.com]

The user must have visited the "i" page before going to this one, or it will just redirect back to the first page.

Then the final download link is:

[sNN.divshare.com]

Where NN is the server number (which varies from file to file). Source code around the link:

Language: HTML
<div class="download_message">Your download will start momentarily. If it doesn't, <a href="http://s22.divshare.com/launch.php?f=8976227&s=560">click here</a> to try again.</div>

DivShare changes the download pages a bit if it's a music or video file, those might produce some problems, for example:

[www.divshare.com]

Not sure if this is going to help much... but my $0.02.

I'll be happy to provide more info if necessary.

Well, I am in a good mood and I hate "you are a clicking monkey" websites.
So here is a final code:

Language: Java
package cz.vity.freerapid.plugins.services.divshare;
 
import cz.vity.freerapid.plugins.exceptions.ErrorDuringDownloadingException;
import cz.vity.freerapid.plugins.exceptions.PluginImplementationException;
import cz.vity.freerapid.plugins.exceptions.ServiceConnectionProblemException;
import cz.vity.freerapid.plugins.exceptions.URLNotAvailableAnymoreException;
import cz.vity.freerapid.plugins.webclient.AbstractRunner;
import cz.vity.freerapid.plugins.webclient.FileState;
import cz.vity.freerapid.plugins.webclient.utils.PlugUtils;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.methods.GetMethod;
 
import java.util.logging.Logger;
import java.util.regex.Matcher;
 
/**
 * Class which contains main code
 *
 * @author Vity
 */
class DivshareFileRunner extends AbstractRunner {
    private final static Logger logger = Logger.getLogger(DivshareFileRunner.class.getName());
 
 
    @Override
    public void runCheck() throws Exception { //this method validates file
        super.runCheck();
        final GetMethod getMethod = getGetMethod(fileURL);//make first request
        if (makeRedirectedRequest(getMethod)) {
            checkProblems();
            checkNameAndSize(getContentAsString());//ok let's extract file name and size from the page
        } else
            throw new PluginImplementationException();
    }
 
    private void checkNameAndSize(String content) throws ErrorDuringDownloadingException {
        if (!content.contains("Download Original"))
            PlugUtils.checkName(httpFile, content, ".gif\" valign=\"absmiddle\">", "</div>");
        final Matcher matcher = getMatcherAgainstContent("<b>File Size:</b>(.+?)<span class=\"tiny\">(.+?)</span><br />");
        if (matcher.find()) {
            final long size = PlugUtils.getFileSizeFromString(matcher.group(1) + " " + matcher.group(2));
            httpFile.setFileSize(size);
            httpFile.setFileState(FileState.CHECKED_AND_EXISTING);
        } else throw new PluginImplementationException("File size not found");
    }
 
    @Override
    public void run() throws Exception {
        super.run();
        logger.info("Starting download in TASK " + fileURL);
        final GetMethod method = getGetMethod(fileURL); //create GET request
        if (makeRedirectedRequest(method)) { //we make the main request
            final String contentAsString = getContentAsString();//check for response
            checkProblems();//check problems
            checkNameAndSize(contentAsString);//extract file name and size from the page
            final HttpMethod httpMethod;
            if (!contentAsString.contains("Download Original"))
                httpMethod = getMethodBuilder().setReferer(fileURL).setActionFromAHrefWhereATagContains("Download This File").toHttpMethod();
            else
                httpMethod = getMethodBuilder().setReferer(fileURL).setActionFromAHrefWhereATagContains("Download Original").toHttpMethod();
            if (makeRedirectedRequest(httpMethod)) {
                checkProblems();
                HttpMethod hmethod = getMethodBuilder().setReferer(fileURL).setActionFromAHrefWhereATagContains("Skip Ad >").toHttpMethod();
                downloadTask.sleep(15);
                if (makeRedirectedRequest(hmethod)) {
                    checkProblems();
                    hmethod = getMethodBuilder().setActionFromAHrefWhereATagContains("click here").toHttpMethod();
                    //here is the download link extraction
                    if (!tryDownloadAndSaveFile(hmethod)) {
                        checkProblems();//if downloading failed
                        logger.warning(getContentAsString());//log the info
                        throw new PluginImplementationException();//some unknown problem
                    }
                } else throw new PluginImplementationException("Some waiting problem");
            } else throw new PluginImplementationException("Download this file link problem");
 
        } else {
            checkProblems();
            throw new ServiceConnectionProblemException();
        }
    }
 
    private void checkProblems() throws ErrorDuringDownloadingException {
        final String contentAsString = getContentAsString();
        if (contentAsString.contains("Sorry, we couldn't find this file")) {
            throw new URLNotAvailableAnymoreException("File not found"); //let to know user in FRD
        }
    }
 
    @Override
    protected String getBaseURL() {
        return "http://divshare.com";
    }

You can study it and you can see that's not so difficult as it may look like, but it requires some time. It's almost shorter than your post .

-------------------------------------





Edited 2 time(s). Last edit at 11/01/2009 02:39PM by Vity.

Thanks a lot, Vity!

Tested it and it works perfectly. Only needs a few final tweaks.

You can download more than one file at a time. Have yet to test where the limit is, but it's definitely more than 1.

You can skip the waiting time by clicking the "Skip Ad >" link, could you implement that?

There were some filename issues when downloading MP3 files, it only found the filename after starting the actual download. So it had the number code instead of the real filename when queued, even after verification. Regular, non-music files work just fine.

ntoskrnl Wrote:
-------------------------------------------------------
> Thanks a lot, Vity!
>
> Tested it and it works perfectly. Only needs a few
> final tweaks.
>
> You can download more than one file at a time.
> Have yet to test where the limit is, but it's
> definitely more than 1.

So tell me how many .

> You can skip the waiting time by clicking the
> "Skip Ad >" link, could you implement that?
>
Are you sure with that?

> There were some filename issues when downloading
> MP3 files, it only found the filename after
> starting the actual download. So it had the number
> code instead of the real filename when queued,
> even after verification. Regular, non-music files
> work just fine.

I know - that's the condition in check method. The file name is not shown on the first page, but it's OK, because final file name is earned from file HTTP header.

-------------------------------------





Edited 1 time(s). Last edit at 11/01/2009 03:32PM by Vity.

Vity Wrote:
-------------------------------------------------------
> So tell me how many .

After some extensive testing I managed to download four files simultaneously, but even after finishing/stopping them all I experienced the same issue as with them running. I didn't get any proper results.

Sent them email asking about it, I'll post the results here when they respond.

> Are you sure with that? [Skipping the timer]

Yes. Works for me every time.

> I know - that's the condition in check method. The
> file name is not shown on the first page, but it's
> OK, because final file name is earned from file
> HTTP header.

OK I understand, missed that.

I did some research, and it seems that the issue here is the fact that my test file was in a folder.

This is a music file not in a folder: [www.divshare.com]
This is a music file in a folder: [www.divshare.com]
This is a data file not in a folder: [www.divshare.com]
This is a data file in a folder: [www.divshare.com]

The only problems are music/video files in folders. DivShare seems to create a playlist.

There's still one more issue: Picture folders. Example: [www.divshare.com] It doesn't work with the current version, throws "File name not found".

Oh, there's STILL two more things... DivShare has per-uploader monthly download limits. You should propably program some handling code for that. Can't find what the error page looks like, but if I ever get it, I'll post it here.

Secondly, "This file is not available to free users in China and Southeast Asia.". You might have to check for that aswell.

Updated.
4 files limit + the all test links works now...

-------------------------------------

Thanks!

The limit isn't 4 files at a time... Still waiting for response from their support.

I am fairly certain that they don't actually limit it at all.

4 is good enough to download many files fast and easily with FRD...

-------------------------------------

Indeed. It's fine as it's now.

If I get the "monthly per-account bandwidth exceeded" error (which seems very likely, already at 68% of the allowed amount), I'll post it here aswell so you can get it 100% right (not sure if the error page contains the words "download limit").

Next time you will be able to fix it alone . Obviously, you know where to find sources .

-------------------------------------