by Steven J. Owens (unless otherwise attributed)
Note: This is a rough draft.
Doing an HTTP Post in Java isn't hard, but it's not nearly as easy and straightforward as it should be, especially in the context of Java. Here is a collection of tips and pointers to resources. Further down I have complete, working example code, including how to do the query parameter encoding, etc.
You should very strongly consider using multipart/form-data (RFC 2388) encoding for XML payloads, particularly for binary, compressed XML payloads. http://www.ietf.org/rfc/rfc2388.txt
There are several widely-used and useful packages for simulating an HTTP client. If you have extensive needs, I suggest you try one out:
I know for a fact that both HttpUnit (which is properly a JUnit style testing framework for web applications, but works well enough for our purposes) and the innovation.ch HTTPClient support multipart/form-data uploads. Martin Cooper informs me that Jakarta HttpClient supports multipart/form-data uploads. You have to dig for it in HttpUnit - it's in there, but the docs didn't exactly point it out when I looked into this about six or eight months ago (I ended up writing my own before I found that, but that's another story).
Personally, I like the looks of HttpUnit better, it appears to be structured and coded more cleanly than Tschalar's HTTPClient. I haven't looked at the Jakarta HttpClient at all, but it appears to be actively supported by the Jakarta developer community.
Using multipart/form-data upload will also require a little more coding on the server side, since servlet engines don't automatically support decoding multipart/form-data (I keep thinking this ought to be a standard feature in web servers or at least in servlet engines). This is not a big deal, there are libraries out there to support decoding multipart/form-data uploads. The one I'm most familiar with is (surprise surprise) Jason Hunter's, which is available at servlets.com for download.
Note that the license requires you to buy a copy of Jason's excellent book, Java Servlet Programming to use the multipart/form-data parser for commercial use. However, I tripped across another parser on the web, the other day, when looking for something else; I didn't bother to note it down, but rest assured that you'll be able to find one. Martin Cooper (who pointed out the Jakarta HttpClient to me) chimed in at this point to mention that Jakarta Commons FileUpload supports multipart/form-data and doesn't even require you to buy anything.
You may also want to look into how SOAP typically handles its payloads when using HTTP for the transport layer, though it's likely the SOAP client kit may be bulky or require additional support APIs.
To start, when posting, in general I suggest reading Michael Daconta's Java Traps article in JavaWorld:
Dodging the traps hiding in the URLConnection class
Actually, the Java Traps article talks about how to get the URLConnection class to do an HTTP POST instead of GET, but it still doesn't explain how to include POST parameters.
Again, it's not as easy as it should be. It's not hard, it just requires an annoying extra bit of work. You can code up the parameter yourself and get it going. Putting together a more generic solution would require a little work, but if you just want to do a one-off, no problem. It works this way.
An HTTP POST is pretty much like a GET. The two transactions look very much the same if you watch them with a packet sniffer (*). There's one big difference. A normal GET will request a URL with a question mark (?) and then the parameters appended to the URL in an encoded format. We've all seen GET URLs:
http://www.darksleep.com/notablog/format.cgi?article=Http_Post_In_Java.html
A POST will put the parameter stuff - in the same exact format, everything that you would normally see after the question mark on a GET URL in your browser's Location: or Address: window - at the end of the request, after the headers.
(* Using a packet sniffer to watch your browser or other client talk to your server is something which I EXTREMELY strongly recommend to anybody who is working with web protocols - not only is it an excellent education in how HTTP and browsers work, it's also an invaluable debugging tool.)Your problem is, you have to assemble that string by hand, instead of being able to just say "do a post and here are the parameters".
Fortunately, the only really obnoxious bit is done for you, by java.net.URLEncoder. URL encoding is the steps necessary to make the parameter data safe to be included in a URL, for example making sure that there are no spaces in the data (by replacing them with plus (+) signs).
From the javadocs for URLEncoder:
For example using UTF-8 as the encoding scheme the string "The string @foo-bar" would get converted to "The+string+%C3%BC%40foo-bar" because in UTF-8 the character is encoded as two bytes C3 (hex) and BC (hex), and the character @ is encoded as one byte 40 (hex).
The only part left for you is to concatenate the parameter name and parameter data with an equals sign in between them:
foo=bar
And then concatenate the parameters together with an & between them:
foo=bar&baz=xyzzy
Then you do the POST and write this string to the URL.
Here're some sample programs, SimplePost, NotSoSimplePost, HttpParameter and ParameterizedPost, and finally RunnablePost (which also uses HttpParameter). I'm actually going to include the source in this page (below), but for easy downloading you can get the source here:
I also just added a simpler version of HttpParameter, named QueryString.
And, if you're lazy (and trusting), you can get the compiled .class files here:
NOTE: Before you get too much further into this sample code, bear in mind that most of it was written with an earlier version of java. It all compiles and runs with current (1.4) java, but we now have the HttpURLConnection class available to us, which this example does not use. HttpURLConnection probably makes some of this easier.
Okay, now let's look at the source.
[If you're reading this and I haven't (yet) finished writing this part, well, it's fairly readable, take a crack at it. Email me with any questions you have, that'll motivate me to finish this section :-)]
Here's SimplePost. It just does a single HTTP Post, with a single parameter. Other than the fact that it's using POST, there's not much useful difference between this and a normal HTTP Get. The main point of this file is so you can see what it is that makes the URLConnection do a Post instead of a Get, and how you send parameters in the Post.
Notice that SimplePost just contains two static classes. There is no state, and I don't bother to make the class user's job any easier. I didn't handle any of the possible exceptions, I just declared the methods to throw all of them.
By the by, I could have declared most of these under simply "throws IOException", but I listed them specifically, so you'd know what might be going on. In general, I like to do this - I understand the idea behind being able to throw and/or catch classes of exceptions, but I like to specifically know what might go wrong. In the spirit of that, I'm also including comments in the code to point out to you where the exceptions might be thrown.
import java.net.* ;
import java.io.* ;
public class SimplePost {
// This is a very simple example of how to do an HTTP Post with a
single parameter.
public static String doPost(String baseURL, String path, String name, String value)
throws FileNotFoundException, java.io.UnsupportedEncodingException, MalformedURLException, IOException {
// new URL() might throw MalformedURLException
// new URL() might throw IOException
URL url = new URL(baseURL + path) ;
URLConnection urlc = url.openConnection() ;
urlc.setDoOutput(true) ;
urlc.setDoInput(true) ;
urlc.setAllowUserInteraction(false) ;
// URLConnection.getOutputStream might throw IOException
PrintWriter server = new PrintWriter(urlc.getOutputStream()) ;
// URLEncoder.encode might throw UnsupportedEncodingException
String parameterString = URLEncoder.encode(name, "UTF-8") + "=" + URLEncoder.encode(value, "UTF-8") ;
server.print(parameterString) ;
server.flush() ;
server.close() ;
InputStream in ;
// URLConnection.getInputStream might throw FileNotFoundException
// URLConnection.getInputStream might throw IOException
in = urlc.getInputStream();
// streamToString might throw IOException
String response = streamToString(in) ;
return response;
}
// This should really use a StringWriter or something similar and
// simpler to capture the response message. But for now,
// something quick and dirty.
public static String streamToString(InputStream in) throws IOException {
BufferedReader bis = new BufferedReader(new InputStreamReader(in));
StringBuffer out= new StringBuffer();
char[] inputLine = new char[2048];
int count = bis.read(inputLine);
while ( count > 0){
out.append(inputLine,0,count);
count = bis.read(inputLine);
}
return out.toString();
}
}
To add exception handling we're going to have to make it somewhat more stateful. Before we do that, we might as well make the baseURL, path and parameter stateful:
StatefulSimplePost
import java.net.* ;
import java.io.* ;
public class StatefulSimplePost {
public StatefulSimplePost(String baseURL, String path, String parameterName, String parameterValue) throws java.io.UnsupportedEncodingException {
setBaseURL(baseURL) ;
setPath(path) ;
setParameter(parameterName, parameterValue) ;
}
public String parameterEncoding = "UTF-8";
public String parameterEquals = "=" ;
public void setParameter(String parameterName, String parameterValue) throws java.io.UnsupportedEncodingException {
setParameter(URLEncoder.encode(parameterName, this.parameterEncoding)
+ this.parameterEquals
+ URLEncoder.encode(parameterValue, this.parameterEncoding)) ;
}
// A bunch of vanilla instance variables with get/set methods
public String parameter = "" ;
public void setParameter(String parameter) { this.parameter = parameter ; }
public String getParameter() { return this.parameter ; }
public String baseURL = "http://www.google.com";
public void setBaseURL(String baseURL) { this.baseURL = baseURL ; }
public String getBaseURL() { return this.baseURL ; }
public String path = "";
public void setPath(String path) { this.path = path ; }
public String getPath() { return this.path ; }
// This is a very simple example of how to do an HTTP Post with a single parameter.
public String doPost()
throws FileNotFoundException, java.io.UnsupportedEncodingException, MalformedURLException, IOException {
// new URL() might throw MalformedURLException
// new URL() might throw IOException
URL url = new URL(this.getBaseURL() + this.getPath()) ;
URLConnection urlc = url.openConnection() ;
urlc.setDoOutput(true) ;
urlc.setDoInput(true) ;
urlc.setAllowUserInteraction(false) ;
// URLConnection.getOutputStream might throw IOException
PrintWriter server = new PrintWriter(urlc.getOutputStream()) ;
// URLEncoder.encode might throw UnsupportedEncodingException
server.print(this.getParameter()) ;
server.flush() ;
server.close() ;
InputStream in ;
// URLConnection.getInputStream might throw FileNotFoundException
// URLConnection.getInputStream might throw IOException
in = urlc.getInputStream();
// streamToString might throw IOException
String response = streamToString(in) ;
return response;
}
// This should really use a StringWriter or something similar and
// simpler to capture the response message. But for now, something
// quick and dirty.
public static String streamToString(InputStream in) throws IOException {
BufferedReader bis = new BufferedReader(new InputStreamReader(in));
StringBuffer out= new StringBuffer();
char[] inputLine = new char[2048];
int count = bis.read(inputLine);
while ( count > 0){
out.append(inputLine,0,count);
count = bis.read(inputLine);
}
return out.toString();
}
}
Okay, now we can add the exception handling, in NotSoSimplePost. We need to add try/catch statements, and we need some instance variables to hold various possible conditions that might result - the state of the Post.
import java.net.* ;
import java.io.* ;
public class NotSoSimplePost {
public NotSoSimplePost(String baseURL, String path, String parameterName, String parameterValue) throws java.io.UnsupportedEncodingException {
setBaseURL(baseURL) ;
setPath(path) ;
setParameter(parameterName, parameterValue) ;
}
public void setParameter(String parameterName, String parameterValue) throws java.io.UnsupportedEncodingException {
setParameter(URLEncoder.encode(parameterName, this.parameterEncoding)
+ "="
+ URLEncoder.encode(parameterValue, this.parameterEncoding)) ;
}
// A bunch of vanilla instance variables with get/set methods
public String parameter = "" ;
public void setParameter(String parameter) { this.parameter = parameter ; }
public String getParameter() { return this.parameter ; }
public String baseURL = "http://www.google.com";
public void setBaseURL(String baseURL) { this.baseURL = baseURL ; }
public String getBaseURL() { return this.baseURL ; }
public String path = "";
public void setPath(String path) { this.path = path ; }
public String getPath() { return this.path ; }
public boolean fileNotFound = false ;
public void setFileNotFound(boolean fileNotFound) { this.fileNotFound = fileNotFound ; }
public boolean getFileNotFound() { return this.fileNotFound ; }
public boolean IOProblem = false ;
public void setIOProblem(boolean IOProblem) { this.IOProblem = IOProblem ; }
public boolean getIOProblem() { return this.IOProblem ; }
public String parameterEncoding = "UTF-8" ;
public String response = "No response yet";
public String getResponse() { return this.response; }
public void setResponse(String response) { this.response = response ; }
boolean postDone = false ;
// Now the meat of the object
public String doPost() {
String response = "";
try {
URLConnection urlc = doGutsOfPost() ;
InputStream in ;
try {
in = urlc.getInputStream();
} catch (FileNotFoundException e) {
System.out.println(e) ;
this.setFileNotFound(true) ;
in = getPostErrorReader(urlc, e) ;
in.close() ;
}
setResponse(getPostResponse(in)) ;
} catch (MalformedURLException e) {
e.printStackTrace() ;
} catch (IOException e) {
e.printStackTrace() ;
this.setIOProblem(true) ;
}
this.postDone = true ;
return response;
}
public URLConnection doGutsOfPost() throws MalformedURLException, IOException {
URL url = new URL(this.getBaseURL() + this.getPath()) ;
URLConnection urlc = url.openConnection() ;
urlc.setDoOutput(true) ;
urlc.setDoInput(true) ;
urlc.setAllowUserInteraction(false) ;
PrintWriter server = new PrintWriter(urlc.getOutputStream()) ;
server.print(this.getParameter()) ;
server.flush() ;
server.close() ;
return urlc ;
}
public String getPostResponse(InputStream in) throws IOException {
return streamToString(in);
}
public final static String streamToString(InputStream in) throws IOException {
BufferedReader bis = new BufferedReader(new InputStreamReader(in));
StringBuffer out= new StringBuffer();
char[] inputLine = new char[2048];
int count = bis.read(inputLine);
while ( count > 0){
out.append(inputLine,0,count);
count = bis.read(inputLine);
}
return out.toString();
}
public InputStream getPostErrorReader(URLConnection urlc, FileNotFoundException e) throws FileNotFoundException {
InputStream err = ((HttpURLConnection)urlc).getErrorStream() ;
if (err == null) {
System.out.println("Unable to getErrorStream from URL after FileNotFoundException") ;
throw(e);
}
return err ;
}
}
And now ParameterizedPost and HttpParameter. Most of the added complexity is off in HttpParameter, which is sort of the whole point of object-oriented design. The only change to the Post code itself is that it now takes (and stores in an instance variable) a parameter object of class HttpParameter:
import java.net.* ;
import java.io.* ;
public class ParameterizedPost {
public ParameterizedPost(String baseURL, String path, HttpParameter parameters) {
setBaseURL(baseURL) ;
setPath(path) ;
setParameters(parameters) ;
}
// A bunch of vanilla instance variables with get/set methods
public HttpParameter parameters ;
public void setParameters(HttpParameter parameters) {
this.parameters = parameters ;
}
public HttpParameter getParameters() {
return this.parameters ;
}
public String baseURL = "http://www.google.com";
public void setBaseURL(String baseURL) { this.baseURL = baseURL ; }
public String getBaseURL() { return this.baseURL ; }
public String path = "";
public void setPath(String path) { this.path = path ; }
public String getPath() { return this.path ; }
public boolean fileNotFound = false ;
public void setFileNotFound(boolean fileNotFound) { this.fileNotFound = fileNotFound ; }
public boolean getFileNotFound() { return this.fileNotFound ; }
public boolean IOProblem = false ;
public void setIOProblem(boolean IOProblem) { this.IOProblem = IOProblem ; }
public boolean getIOProblem() { return this.IOProblem ; }
public String parameterEncoding = "UTF-8" ;
public String response = "No response yet";
public String getResponse() { return this.response; }
public void setResponse(String response) { this.response = response ; }
boolean postDone = false ;
// Now the meat of the object
public String doPost() {
String response = "";
try {
// doGutsOfPost calls new URL(), which might throw MalformedURLConnection
URLConnection urlc = doGutsOfPost() ;
InputStream in ;
try {
// URLConnection.getInputStream might throw FileNotFoundException
// URLConnection.getInputStream might throw IOException
in = urlc.getInputStream();
} catch (FileNotFoundException e) {
System.out.println(e) ;
this.setFileNotFound(true) ;
in = getPostErrorReader(urlc, e) ;
in.close() ;
}
setResponse(getPostResponse(in)) ;
} catch (MalformedURLException e) {
e.printStackTrace() ;
} catch (IOException e) {
e.printStackTrace() ;
this.setIOProblem(true) ;
}
this.postDone = true ;
return response;
}
public URLConnection doGutsOfPost() throws MalformedURLException, UnsupportedEncodingException, IOException {
// new URL() might throw MalformedURLException
// new URL() might throw IOException
URL url = new URL(this.getBaseURL() + this.getPath()) ;
URLConnection urlc = url.openConnection() ;
urlc.setDoOutput(true) ;
urlc.setDoInput(true) ;
urlc.setAllowUserInteraction(false) ;
// URLConnection.getOutputStream might throw IOException
PrintWriter server = new PrintWriter(urlc.getOutputStream()) ;
// HttpParameter calls URLEncoder.encode, which might throw UnsupportedEncodingException
server.print(this.getParameters().getParameterString()) ;
server.flush() ;
server.close() ;
return urlc ;
}
public String getPostResponse(InputStream in) throws IOException {
return streamToString(in);
}
public final static String streamToString(InputStream in) throws IOException {
BufferedReader bis = new BufferedReader(new InputStreamReader(in));
StringBuffer out= new StringBuffer();
char[] inputLine = new char[2048];
int count = bis.read(inputLine);
while ( count > 0){
out.append(inputLine,0,count);
count = bis.read(inputLine);
}
return out.toString();
}
public InputStream getPostErrorReader(URLConnection urlc, FileNotFoundException e) throws FileNotFoundException {
InputStream err = ((HttpURLConnection)urlc).getErrorStream() ;
if (err == null) {
System.out.println("Unable to getErrorStream from URL after FileNotFoundException") ;
throw(e);
}
return err ;
}
}
And now, finally, RunnablePost:
import java.net.* ;
import java.io.* ;
public class RunnablePost implements Runnable {
public void run() {
doPost() ;
System.out.println(response) ;
}
public RunnablePost(String baseURL, String path, String parameterName, String parameterValue) throws java.io.UnsupportedEncodingException {
setBaseURL(baseURL) ;
setPath(path) ;
setParameter(parameterName, parameterValue) ;
}
public void setParameter(String parameterName, String parameterValue) throws java.io.UnsupportedEncodingException {
setParameter(URLEncoder.encode(parameterName, this.parameterEncoding)
+ "="
+ URLEncoder.encode(parameterValue, this.parameterEncoding)) ;
}
// A bunch of vanilla instance variables with get/set methods
public String parameter = "" ;
public void setParameter(String parameter) { this.parameter = parameter ; }
public String getParameter() { return this.parameter ; }
public String baseURL = "http://www.google.com";
public void setBaseURL(String baseURL) { this.baseURL = baseURL ; }
public String getBaseURL() { return this.baseURL ; }
public String path = "";
public void setPath(String path) { this.path = path ; }
public String getPath() { return this.path ; }
public boolean fileNotFound = false ;
public void setFileNotFound(boolean fileNotFound) { this.fileNotFound = fileNotFound ; }
public boolean getFileNotFound() { return this.fileNotFound ; }
public boolean IOProblem = false ;
public void setIOProblem(boolean IOProblem) { this.IOProblem = IOProblem ; }
public boolean getIOProblem() { return this.IOProblem ; }
public String parameterEncoding = "UTF-8" ;
public String response = "No response yet";
public String getResponse() { return this.response; }
public void setResponse(String response) { this.response = response ; }
boolean postDone = false ;
// Now the meat of the object
public String doPost() {
String response = "";
try {
URLConnection urlc = doGutsOfPost() ;
InputStream in ;
try {
in = urlc.getInputStream();
} catch (FileNotFoundException e) {
System.out.println(e) ;
this.setFileNotFound(true) ;
in = getPostErrorReader(urlc, e) ;
in.close() ;
}
setResponse(getPostResponse(in)) ;
} catch (MalformedURLException e) {
e.printStackTrace() ;
} catch (IOException e) {
e.printStackTrace() ;
this.setIOProblem(true) ;
}
this.postDone = true ;
return response;
}
public URLConnection doGutsOfPost() throws MalformedURLException, IOException {
URL url = new URL(this.getBaseURL() + this.getPath()) ;
URLConnection urlc = url.openConnection() ;
urlc.setDoOutput(true) ;
urlc.setDoInput(true) ;
urlc.setAllowUserInteraction(false) ;
PrintWriter server = new PrintWriter(urlc.getOutputStream()) ;
server.print(this.getParameter()) ;
server.flush() ;
server.close() ;
return urlc ;
}
public String getPostResponse(InputStream in) throws IOException {
return streamToString(in);
}
public final static String streamToString(InputStream in) throws IOException {
BufferedReader bis = new BufferedReader(new InputStreamReader(in));
StringBuffer out= new StringBuffer();
char[] inputLine = new char[2048];
int count = bis.read(inputLine);
while ( count > 0){
out.append(inputLine,0,count);
count = bis.read(inputLine);
}
return out.toString();
}
public InputStream getPostErrorReader(URLConnection urlc, FileNotFoundException e) throws FileNotFoundException {
InputStream err = ((HttpURLConnection)urlc).getErrorStream() ;
if (err == null) {
System.out.println("Unable to getErrorStream from URL after FileNotFoundException") ;
throw(e);
}
return err ;
}
}