Figuring out how to call a CGI script?

Networking & Multiplayer
#1

Hello,
I’m currently working on an application as a side project, and I’m almost done… except for one thing. My program needs to call a CGI script by the POST method. Trouble is, I don’t know what to send to the server. There are many possible parameters to be sent and I don’t know what the server is expecting. I can get a generic response, something to the effect of “the server has no idea what you’re trying to do”, so I know that the program works, I just need to send the right data to the server.
Possible thoughts:
Is there a Firefox extension that would allow me to “capture” all traffic in and out of my computer? Or a Mac application that would do the same thing?

#2

This should help: http://www.xyzws.com/Javafaq/how-to-use-httpurlconnection-post-data-to-web-server/139

#3

Thanks CaptainJester… but I can already connect to the server. Sorry for not being clear on that…
What I need is specific to the page I’m trying to get my program to interact with. Each page has its own way of handling data specific to its needs, and I want to figure out what the page is looking for. Is there a way of diving into the source of the page to get this?
To clarify, check out this page: http://www.webdeveloper.com/java/java_meets_cgi.html … what I am looking for is what the server needs in the form_data field of this program.

#4

Sounds like you want to know what an HTTP POST request looks like?

As for what exactly you need to send, that is up to the page. Since the code accepting the POST data is server side, there is no way for you to see its code. Use something like ProxyTrace or the FireFox plugin FireBug to see what is being posted when you access the page the normal way through your browser.

#5

If those Firefox extensions don’t work out for you, WireShark will. It’s not a lot of fun to use though.

#6

Thank you so incredibly much for the FireBug tip. I was very easily able to view the raw HTTP request.

I pasted the request into my program and got the following response:


HTTP/1.1 400 Bad Request
Date: Mon, 19 Apr 2010 18:00:00 GMT
Server: Apache/1.3.41 (Unix)
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>400 Bad Request</TITLE>
</HEAD><BODY>
<H1>Bad Request</H1>
Your browser sent a request that this server could not understand.<P>
Request header field is missing colon separator.<P>
<PRE>
POST /cgi-bin/multidb.cgi HTTP/1.0</PRE>
<P>
<HR>
<ADDRESS>Apache/1.3.41 Server at frwebgate5.access.gpo.gov Port 80</ADDRESS>
</BODY></HTML>

I literally copied it verbatim and added \r\n where necessary. How come it works for my browser but not for my program?


	protected void load() throws Exception {
		Socket sock = new Socket("frwebgate.access.gpo.gov", 80);
		String tempData;
		DataOutputStream sendData = new DataOutputStream(sock
				.getOutputStream());
		DataInputStream readData = new DataInputStream(sock
				.getInputStream());
		System.out.println("===========SENDING REQUEST==========");
		sendData.writeBytes("POST /cgi-bin/multidb.cgi HTTP/1.0\r\n");
		sendData.writeBytes("Content-Type: application/x-www-form-urlencoded\r\n");
		sendData.writeBytes("Content-Length: 1523\r\n");
		sendData.writeBytes("POST /cgi-bin/multidb.cgi HTTP/1.0\r\n");
		sendData.writeBytes("WAISdbName=2010_register+Federal+Register%2C+Volume+75+%282010%29&WAISdbName=2009_register+Federal+Register%2C+Volume+74+%282009%29&WAISdbName=2008_register+Federal+Register%2C+Volume+73+%282008%29&WAISdbName=2007_register+Federal+Register%2C+Volume+72+%282007%29&WAISdbName=2006_register+Federal+Register%2C+Volume+71+%282006%29&WAISdbName=2005_register+Federal+Register%2C+Volume+70+%282005%29&WAISdbName=2004_register+Federal+Register%2C+Volume+69+%282004%29&WAISdbName=2003_register+Federal+Register%2C+Volume+68+%282003%29&WAISdbName=2002_register+Federal+Register%2C+Volume+67+%282002%29&WAISdbName=2001_register+Federal+Register%2C+Volume+66+%282001%29&WAISdbName=2000_register+Federal+Register%2C+Volume+65+%282000%29&WAISdbName=1999_register+Federal+Register%2C+Volume+64+%281999%29&WAISdbName=1998_register+Federal+Register%2C+Volume+63+%281998%29&WAISdbName=1997_register+Federal+Register%2C+Volume+62+%281997%29&WAISdbName=1996_register+Federal+Register%2C+Volume+61+%281996%29&WAISdbName=1995_register+Federal+Register%2C+Volume+60+%281995%29&date1=&date2=&op=%3D&date3=&WAISqueryRule=%28%24WAISqueryString%29+AND+%28section%3D%24sect+OR+section%3D%24sect1+OR+section%3D%24sect2+OR+section%3D%24sect3+OR+section%3D%24sect4+OR+section%3D%24sect5+OR+section%3D%24sect6+OR+section%3D%24sect7%29+AND+%28%28date+%3E%3D+%24date1+AND+date+%3C%3D+%24date2%29+OR+%28date+%24op+%24date3%29%29&WAIStemplate=multidb_results.html&WrapperTemplate=fr_wrapper.html&WAISqueryString=Osama+Bin+Ladin&Submit.=Submit&WAISmaxHits=50");
		System.out.println("===========RECEIVING DATA==========");
		while ((tempData = readData.readLine()) != null) {
			System.out.println(tempData);
		}
		System.out.println("===========DONE WITH DATA==========");
		sock.close();
	}

Thank you Nate and bleb for your amazing help so far :slight_smile:

#7 
...
		sendData.writeBytes("Content-Length: 1523\r\n");
		sendData.writeBytes("POST /cgi-bin/multidb.cgi HTTP/1.0\r\n");
		sendData.writeBytes("A LOT OF DATA");

You are making a 2nd request there. It must be a blank line.

...
		sendData.writeBytes("Content-Length: 1523\r\n");
		sendData.writeBytes("\r\n");
		sendData.writeBytes("A LOT OF DATA");