Java String.replace() - replaces more than just the substring I specify?

As per this CodingBat problem I am trying to do the following:

Given a string, if the first or last chars are 'x', return the string without those 'x' chars, and otherwise return the string unchanged.

My code:

public String withoutX(String str) {
    if (str.startsWith("x")) {
        str = str.replace(str.substring(0, 1), "");
    }
    if (str.endsWith("x")) {
        str = str.replace(str.substring(str.length()-1), "");
    }
    return str;
}

This code replaces ALL the x characters in the string, rather than just the first and last. Why does this happen, and what would be a good way to solve it?