Built a web crawler , C/C++ Programming

Assignment Help:

To develop a web crawler such that when given a base URL, it will traverse the entire web tree and then build an index of keywords and what URL link they appear on.   The web crawler is a server-side program where it can begin called the seed.  As the crawler visits the URL, it identifies all the hyperlinks in the page (eg: detecting the " < a href ..." links) and adds them to the list of URLs to visit, called the crawl frontier.   URLs from the frontier are then visited by your server-side program to look for a few keywords (eg: raffles, award, alumni etc) initially on this website and stored their corresponding URL links where these keywords can be found in a file.  

Below is a sample of the data stored in the Server for subsequent quick search.

Another interactive client and server socket programs using C++ can be developed to allow clients to query with the search keywords of the website and obtain a server's reply of a list of URLs where the keyword is found.  The server program developed must be able to handle multiple queries and hence be able to search through its data file for a correct response.  The server program should also allow continuous enquires until the customer enters quit.  

The communication between the client and server machine can be any bidirectional interactive protocol.  An example is to use socket programming where network endpoints (IP address and port number) are represented as sockets.

 

When creating the server application, you must follow these steps:

  • Create a new socket by typing: socket().
  • Bind an address (IP address and port number) to the socket by typing: bind. This step identifies the server so that the client knows where to go.
  • Listen for new connection requests on the socket by typing: listen().
  • Accept new connections by typing: accept().

Often, the servicing of a request on behalf of a client may take a considerable length of time. It would be more efficient in such a case to accept and deal with new connections while a request is being processed. The most common way of doing this is for the server to fork a new copy of it after accepting the new connection.

The "responsing" Server listens on a port and waits for client's request. Based on the Client's questions, the Server will response appropriately by looking up a data file stored at the server's end.  


Related Discussions:- Built a web crawler

Define bitwise left shift and right shift operators, Define Bitwise Left Sh...

Define Bitwise Left Shift and Right Shift Operators: >? The bitwise shift operators shift their first operand right (>>) or left ( // Illustration of the bitwise right shi

Define a procedure called make-avl-tree, This question deals with AVL trees...

This question deals with AVL trees. The representation to be used is similar to the bank account object discussed in class. (a) Define a procedure called make-avl-tree which mak

Assignment, write a function rise to the power to find the exponent of a nu...

write a function rise to the power to find the exponent of a number whether function is called

What happens if you make call "delete this;"?, The code contains two built-...

The code contains two built-in pitfalls. First one, if it executes in a member function for an static, extern or automatic object, the program will possibly crash as soon as the de

How to define a derived class, How to define a derived class ? A singly...

How to define a derived class ? A singly inherited derived class id defined by writing : The keyword class. The name of the derived class . A single colon (:).

Program of sorting algorithms , The program sorting.cpp contains a main f...

The program sorting.cpp contains a main function for testing the operation of several sort algorithms over various data sizes and data set organisations. The program understands

Algorithms, write an algorithm for multiplication of two sparse matrices us...

write an algorithm for multiplication of two sparse matrices using linked lists

Nonlinear least squares minimization, Estimation of the yield curve using n...

Estimation of the yield curve using nonlinear least squares minimization: The last part of this assignment asks you to construct the Nelson Siegel yield curve from observed bond pr

C program to convert time in 24 hour format to 12., Aim: To implement a pr...

Aim: To implement a program to convert time in 24 hour format to 12 hour format. Code:                       #include #include #include class time24 {

Program for simple 4-function calculator, Most first graders know that nine...

Most first graders know that nine hundred and ninety nine plus one is one thousand, but C++ doesn't! Sure, a computer can easily compute 999 + 1 and get 1000. But reading and writi

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd