Socket Mode designer explained - Magic Submitter...In terms of Magic Submitter to emulate GET...

1

Socket Mode designer explained Socket Mode designer explained ......................................................................................................................... 1

1 Introduction.................................................................................................................................................. 1

2 Principles of creating socket based scripts................................................................................................... 1

3 Test Socket Mode button ............................................................................................................................. 2

4 Additional tools............................................................................................................................................ 2

5 Typical Socket Based Scripts....................................................................................................................... 2

6 How web browser works ............................................................................................................................. 3

7 The Browser's Request................................................................................................................................. 3

8 The Server's Response ................................................................................................................................. 4

9 Clicking on a Hyperlink............................................................................................................................... 5

10 HTML Form............................................................................................................................................. 6

11 HTTP Debugger..................................................................................................................................... 10

12 Magic Submitter API ............................................................................................................................. 12

1 Introduction This document intends to explain basic concept of socket based scripts creating.

It is first document version and will be updated based on your feedback and questions you can ask here

http://www.magicsubmitter.com/msforum/viewforum.php?f=145

2 Principles of creating socket based scripts First of all you have to understand how to create web browser based scripts as the best way is first create

browser based script and then covert it to socket mode.

1. Create webbrowser based script. When you create it you have to omit using of Click Button or Click

Element operations if possible as they does not work in socket mode. You have try to use Submit

Form operation instead

2. Convert webbrowser script to socket based by replace all Click Button or Click Element operation by

submit form operation. Also it is very good to determine Register, Login, Submit pages real urls and

use GoTo page operation to directly navigate that pages instead of using a few Click Link operations

3. Test Socket based script using Test Socket Mode button

4. If everything is ok change script to work in socket mode

2

3 Test Socket Mode button When you click this button MS will try to run script in socket mode and also will show HTML output of

communication with server

It shows

1. output in HTML format

2. output in text format

3. last request data

4. last response header

4 Additional tools To successfully create socket based script you will need the following tools

1. Fiddler2 http://www.fiddler2.com/fiddler2 - the tool to debug http protocol data. See HTTP Debugger

section

2. Firebug Firefox Addon http://getfirebug.com/ - The most popular and powerful web development tool

5 Typical Socket Based Scripts The typical socket based register script is as follow

1. Goto register page

2. Select Register form

3. Fill form values

4. Solve captcha

5. Fill captcha value

6. Submit Register Form

7. Validate if submit was successful

3

The typical socket based submit script

1. Goto login page

2. Select login form

3. Fill form values

4. Submit login Form

5. Validate if login was successful

6. Goto Submit page

7. select Submit form

8. Fill form values

9. Solve captcha if needed

10. Fill captcha value if needed

11. Submit Submit form

12. Validate if submit was successful

13. Save Link

6 How web browser works Fist of all we have to understand how webbrowser communicates with sites as we have to emulate all this

communication to be able to submit to our sites in socket mode

7 The Browser's Request

Your browser first takes in a URL and parses it. In this example, the browser is given the following URL:

http://url.org/signup/

4

The browser interprets the URL as follows:

http://

In the first part of the URL, you told the browser to use HTTP, the Hypertext Transfer Protocol.

hypothetical.ora.com

In the next part, you told the browser to contact a computer over the network with the hostname of

hypothetical.ora.com.

/signup/

Anything after the hostname is regarded as a document path. In this example, the document path is

/signup/.

So the browser connects to url.org using the HTTP protocol. Since no port was specified, it assumes port 80,

the default port for HTTP. The message that the browser sends to the server at port 80 is:

GET /signup/ HTTP/1.1 Host: url.org User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729) Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Referer: http://url.org/ Cookie: PHPSESSID=f0ffd67965a25926594f81f4b5734029

Let's look at what these lines are saying:

2. The first line of this request (GET / HTTP/1.1) requests a document at / from the server. HTTP/1.1 is

given as the version of the HTTP protocol that the browser uses.

3. The second line tells the server what the client thinks the server's hostname is.

4. In the third line, beginning with the string User-Agent, the client identifies itself as Mozilla (Firefox)

version 5.0, running on Windows NT.

5. The fourth line tells the server what kind of documents are accepted by the browser.

6. The fifth line The referer field tells the server where the user came from, which is very useful for

logging and keeping track of who links to ones pages.

7. The seven line is what cookies data are send to the server

Together, these 7 lines constitute a request. Lines 2 through 6 are request headers.

8 The Server's Response

Given a request like the one previously shown, the server looks for the file associated with "/signup/ " and

returns it to the browser, preceding it with some "header information":

HTTP/1.1 200 OK Date: Sun, 11 Dec 2011 16:38:14 GMT Server: Apache Content-Type: text/html; charset=utf-8 Content-Length: 4498 Set-Cookie: PHPSESSID=f0ffd67965a25926594f81f4b5734029; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Connection: close

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> <head> <title>URL.ORGanizer</title> …………… …………… </html>

If you look at this response, you'll see that it begins with a series of lines that specify information about the

document and about the server itself. Then after a blank line, it returns the document. The series of lines

5

before the first blank line is called the response header, and the part after the first blank line is called the body

or entity, or entity-body. Let's look at the header information:

1. The first line, HTTP/1.1 200 OK, tells the client what version of the HTTP protocol the server uses.

But more importantly, it says that the document has been found and is going to be transmitted.

2. The second line indicates the current date on the server. The time is expressed in Greenwich Mean

Time (GMT).

3. The third line tells the client what kind of software the server is running. In this case, the server is

Apache

4. The fourth line (Content-type) tells the browser the type of the document. In this case, it is HTML.

5. The fifth line tells the client how many bytes are in the entity body that follows the headers. In this

case, the entity body is 4498 bytes long.

6. The sixth line specifies which cookies client(webbrowser) should remember and send during next

request

After all that, a blank line and the document text follow.

In terms of Magic Submitter to emulate GET request we have to use simply Goto Page operation

and specifying url

With the following settings

What are ErrorCssClass I will explain later

9 Clicking on a Hyperlink When you click on a hyperlink, the client and server go through something similar to what happened when we visited http://url.org/signup/

For example, when you click on the hyperlink from the previous example, the browser looks at its associated

HTML:

6

<a href="http://url.org/signup/">Register</a>

From there, it knows that the next location to retrieve is /signup/. The browser then sends the following to

url.org:


The server responds with:

Date: Sun, 11 Dec 2011 16:38:14 GMT Server: Apache Content-Type: text/html; charset=utf-8 Content-Length: 4498 Set-Cookie: PHPSESSID=f0ffd67965a25926594f81f4b5734029; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Connection: close

[HTML data]

And the browser displays the new HTML page on the user's screen.

In the Magic Submitter for this you have to use ClickLink operation specifying part of url or

that url anchor Register in our case

10 HTML Form

You've probably seen fill-out forms on the Web, in which you enter information into your browser and submit

the form. Common uses for forms are registration, login and submit

When you fill out a form, the browser needs to send that information to the server, along with the name of the

program/page needed to process it. Let's direct our browser to url.org Register Page

When you add the following operation MS will do rest for you sending appropriate request to the server


The server responds with headers and content of signup page:

HTTP/1.1 200 OK Date: Sun, 11 Dec 2011 16:38:14 GMT Server: Apache Set-Cookie: PHPSESSID=f0ffd67965a25926594f81f4b5734029; path=/

7

Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 4498 Connection: close Content-Type: text/html; charset=utf-8

<form method="post" action="http://url.org/signup/"> <input name="query" value="" type="hidden"> <input id="username" name="username" size="20" class="required" onkeyup="isAvailable(this, '')" type="text"> <input id="password" name="password" size="20" class="required" type="password"> <input id="email" name="email" size="40" class="required" type="text"> <script type="text/javascript" src="http://www.google.com/recaptcha/api/challenge?k=6LesgMUSAAAAACkxGFkFcBeCLcXKp3oUSas32fXS"></script> <input autocomplete="off" name="recaptcha_response_field" id="recaptcha_response_field" autocorrect="off" autocapitalize="off" type="text"> <input name="submitted" value="Register" type="submit"> </form>

The formatted document is shown in Figure

Let's fill out the form and submit it

After hitting the Done button, the browser connects to url.org at port 80, as specified with the <FORM> tag in

the HTML:

<form method="post" action="http://url.org/signup/">

The browser then sends:

POST /signup/ HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729) Accept: text/html,application/xhtml+xml,application/xml; q="0.9,*/*"; q="0.8" Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Pragma: no-cache Accept-Charset: ISO-8859-1,utf-8;q="0.7,*"; q="0.7" Connection: keep-alive Content-Type: application/x-www-form-urlencoded Host: url.org Referer: http://url.org/signup/ Content-Length: 341 Cookie: PHPSESSID=f0ffd67965a25926594f81f4b5734029

8

query=&username=jam1vqj4ki&password=s0Kh2L4G8&email=jam1vqj4ki%40hotmail%2Ecom&submitted=Register&recaptcha_challenge_field=03AHJ%5FVusqQTGPG3hLLnxHedzq%2DKnRgPZv4ExPmNWrW%5FjvJ3DOOVdPBNWfS21cuuXnlY4GFAZKhqG268x40NGDlYiosjsMQ6BhlI1rDlhvf5IImHcObaWMWt9W6eZjm1UFN9g0p%5FfeebjIV%5FV%5F34Xn5Jxn8V5k0Uxx2A&recaptcha_response_field=vbuite+original

In the previous example retrieving the initial page at url.org , we showed a series of lines that the browser

output and called it a request header. Calling it a header might not have made any sense at the time, since

there was no content being sent with it--if you're just requesting a document, you don't have to tell the server

anything else. But since in this instance we have to tell the server what the user typed into the form, we have

to use a "body" portion of the message to convey that information. So there are a few new things to note in

this example:

• Instead of GET, the browser started the transaction with the string POST. GET and POST are two

types of request methods recognized by HTTP. The most important thing that POST tells the server is

that there is a body (or "entity") portion of the message to follow.

The browser used the POST method because it was specified in the <FORM> tag:

<form method="post" action="http://url.org/signup/">

• The browser included an extra line specifying a Content-type. This wasn't necessary in the previous

example because no content was being sent with the request. The Content-type line tells the server

what sort of data is coming so it can determine how best to handle it. In this case, it tells the server that

the data to be sent is going to be encoded using the application/x-www-form-urlencoded format. This

format specifies how to encode special characters, and how to send multiple variables and values in

forms.

• The browser included another line specifying a Content-length. Similarly, this wasn't necessary

earlier because there was no content to the entity body. But there is in this example; it tells the server

how much data to retrieve. In this case, the Content-length is 341 bytes.

After a blank line, the entity-body is issued, reading query=&username=jam1vqj4ki&password=s0Kh2L4G8&email=jam1vqj4ki%40hotmail%2Ecom&submitted=Register&recaptcha_challenge_field=03AHJ%5FVusqQTGPG3hLLnxHedzq%2DKnRgPZv4ExPmNWrW%5FjvJ3DOOVdPBNWfS21cuuXnlY4GFAZKhqG268x40NGDlYiosjsMQ6BhlI1rDlhvf5IImHcObaWMWt9W6eZjm1UFN9g0p%5FfeebjIV%5FV%5F34Xn5Jxn8V5k0Uxx2A&recaptcha_response_field=vbui

te+original . (Notice that this string is exactly 341 characters, as specified in the Content-length line.)

Where did this query=&username=jam1vqj4ki&password=s0Kh2L4G8&email=jam1vqj4ki%40hotmail%2Ecom&submitted=Register&recaptcha_challenge_field=03AHJ%5FVusqQTGPG3hLLnxHedzq%2DKnRgPZv4ExPmNWrW%5FjvJ3DOOVdPBNWfS21cuuXnlY4GFAZKhqG268x40NGDlYiosjsMQ6BhlI1rDlhvf5IImHcObaWMWt9W6eZjm1UFN9g0p%5FfeebjIV%5FV%5F34Xn5Jxn8V5k0Uxx2A&recaptcha_response_field=vbui

te+original line come from? In the HTML of the form, the input field was specified with the following lines:

<input id="username" name="username" size="20" class="required" onkeyup="isAvailable(this, '')" type="text"> <input id="password" name="password" size="20" class="required" type="password"> <input id="email" name="email" size="40" class="required" type="text"> <script type="text/javascript" src="http://www.google.com/recaptcha/api/challenge?k=6LesgMUSAAAAACkxGFkFcBeCLcXKp3oUSas32fXS"></script> <input autocomplete="off" name="recaptcha_response_field" id="recaptcha_response_field" autocorrect="off" autocapitalize="off" type="text"> <input name="query" value="" type="hidden"> <input name="submitted" value="Register" type="submit">

The NAME="username" and VALUE="jam1vqj4ki" part of the first <INPUT> tag was encoded as

"username=jam1vqj4ki". The NAME="password" part of the second <INPUT> tag specifies a variable name to

use for whatever text is supplied in that field. We filled password in that field with the words " s0Kh2L4G8" . All

is the same for rest fields

Thus, for the form data entered by the user, the browser sends:

9

query=&username=jam1vqj4ki&password=s0Kh2L4G8&email=jam1vqj4ki%40hotmail%2Ecom&submitted=Register&recaptcha_challenge_field=03AHJ%5FVusqQTGPG3hLLnxHedzq%2DKnRgPZv4ExPmNWrW%5FjvJ3DOOVdPBNWfS21cuuXnlY4GFAZKhqG268x40NGDlYiosjsMQ6BhlI1rDlhvf5IImHcObaWMWt9W6eZjm1UFN9g0p%5FfeebjIV%5FV%5F34Xn5Jxn8V5k0Uxx2A&recaptcha_response_field=vbuite+original

to specify the variable and value pairs used in the form. Two or more variable/value pairs are separated with

an ampersand (&).Certain characters with special meaning are translated into a commonly understood format.

At this point, the server processes the request by forwarding this information on to the program/page. The

program then returns some data, and the server passes it back to the client as follows:

HTTP/1.1 200 Date: Sun, 11 Dec 2011 16:38:32 GMT Server: Apache Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 4682 Connection: close Content-Type: text/html; charset=utf-8

[HTML data]

To emulate the following process using Magic Submitter designer you have to create the following script

Select Form operation helps Magic Submitter to understand what data will be send to

server. You have to use Select Form operation obligatory as there are could be hidden inputs in the form that critically

needed to send to the server like in our example we have

<input name="query" value="" type="hidden"> as you see this input is marked as hidden you don’t see this input in

webrowser but that input/param should be send to server obligatory

Field operation indicates what data we are going so send to server. In pour case it populates Username

Actually it is input which we see on the page

10

And usually you can use Paintbrush

To population this kind of operation to your command

The same principles you have to use with password, email, recaptcha_response_field fields

Solve Captcha works absolutely the same way as in not socket/browser mode so you have to use Paintbrush to add that

operation

Send Form it is the most important operation here is it actually send POST request with

form data to the server.

The form field is defined as follows in our example <form method="post" action="http://url.org/signup/">

You have to specify whole or part of form action attribute like http://url.org/signup/ or /signup/, index of the form

on page like [formindex]:1 as FormName parameter

Note: Sometime it is difficult to define what is action value in that case you have to specify concrete url where POST

request should be send and mark that operation obligatory false

Important: we can’t use Click Button operation in socket mode at all you have to use Submit Form operation

instead all the time

11 HTTP Debugger

Very often you will get the situation when your script will look correct but it will not work. It is because that Magic

Submitter sends not all the data needed to the server. To solve this I recommend to use free Fiddler2 software

http://www.fiddler2.com/fiddler2/ it will help to analyze you all requests that magic submitter sends to server and

compare then which requests are send by firefox webbrowser. How to work with Fiddler you can find on it help page.

11

Actually we have to analyze GET and POST requests which send MS and Firefox and compare them. They should be

identical. We can ignore all request related to get images , css,scripts etc. We have to pay attention on critical

information. It is usually get request of pages and all POST requests you can hide images requests in Rules menu of

Fiddler

Fiddler will show you the following info

To use Fiddler with MS you have to set proxy to 127.0.0.1:8888

if script not works check

1. If MS sends request to the same page POST /signup/ HTTP/1.1

2. If MS sends the same Referer header value. You can set Referer using

processor.SetReferrer("http://www.metacafe.com/submit/"); Execute operation

3. If Firefox send some additional headers and MS does not. You can add additional headers using

processor.SetHeader("X-Requested-With","XMLHttpRequest"); Execute operation

4. check if the same cookies are send. If not try to define what Get request sets that cookies in header Set-

Cookie: PHPSESSID=f0ffd67965a25926594f81f4b5734029; path=/ then goto that page to set that

cookies mark that GoTo operation as not obligatory. There could be also cases when cookies are set by

javascripts that are on the page. In this case you have to define how that cookies are created (algorithm

or it is just simple value) and set then using AddCookie Execute operation like processor.AddCookie("url.org","language","en");

Note: you don’t need to send cookies that are set by google analytics etc that cookies looks like Cookie: __utma=216699474.1915995197.1323615317.1323615317.1323615317.1; __utmb=216699474; __utmc=216699474; __utmz=216699474.1323615317.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none)

5. Check if MS and Firefox send the same parameters and values if not you can add additional using

Field operation and set correct value for it

12

6. If you will see that MS send some parameters but webbrowser doesn’t then use Field operation in

following way

12 Magic Submitter API There will be some sites where you will need to use Magic Submitter API to make sure that you send correct

GET and POST requests

To use Magic Submitter API you have to have basic C# knowledge

The API description with examples you can find here

http://www.magicsubmitter.com/help/api/Magic%20Submitter%20API.chm

Download and Save this file using Firefox only

When you download help file and try to open in and will see message like this

Then uncheck Always ask before opening checkbox

P.S Please leave your questions here

http://www.magicsubmitter.com/msforum/viewforum.php?f=145

Socket Mode designer explained - Magic Submitter...In terms of Magic Submitter to emulate GET...

Documents

Transcript of Socket Mode designer explained - Magic Submitter...In terms of Magic Submitter to emulate GET...