Skip to main content

Fetcing ajax based content of the website in JAVA

No replies
asadse
Offline
Joined: 2013-01-24
Points: 0

Hi,

I'm research student. Currently I want to do research on Yahoo news. I found that there are lot of comments on yahoo news. 10 comments come when we open some news, when we click more button, ajax request fire and 10 more comments appear on the screen an so on.

What I want on my first step is that I want to write a code that simply fetch news and all the comment and put it into the database.

Now I have following code that give me html of the news.

String news= "http://news.yahoo.com/yemen-says-regional-al-qaedas-deputy-head-dead-184541137.html";

URL urlObject = new URL( news);
URLConnection con = urlObject.openConnection();
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");

Scanner scanner = new Scanner(con.getInputStream());
scanner.useDelimiter("\\Z");
String content = scanner.next();

Now in actual when i click on the "more" 10 comments added on the page, but how can I achieve this on my code because I donot have link to click. If I parse the more button link and the again fetch the code through recursion then same html will give me again.

Any idea please ?????