Remove all characters after a delimiter in a string

Question

I am building a web crawler application in golang.

After downloading the HTML of a page, I separate out the URLs. I am presented with URLs that have "#s" in them, such as "en.wikipedia.org/wiki/Race_condition#Computing". I would like to get rid of all characters following the "#", since these lead to the same page anyways. Any advice for how to do so?

score 2 · Answer 1 · answered Oct 28 '19 at 22:10

2

Use the url package:

u, _ := url.Parse("SOME_URL_HERE")
u.Fragment = ""
return u.String()

answered Oct 28 '19 at 22:10

Luke Joshua Park

9,527
5
27
44

score 1 · Accepted Answer · answered Oct 28 '19 at 23:37

An improvement on the answer by Luke Joshua Park is to parse the URL relative to the URL of the source page. This creates an absolute URL from what might be relative URL on the page (scheme not specified, host not specified, relative path). Another improvement is to check and handle errors.

func clean(pageURL, linkURL string) (string, error) {
    p, err := url.Parse(pageURL)
    if err != nil {
        return "", err
    }
    l, err := p.Parse(linkURL)
    if err != nil {
        return "", err
    }
    l.Fragment = ""   // chop off the fragment
    return l.String()
}

If you are not interested in getting an absolute URL, then chop off everything after the #. This works because the only valid use of # in a URL is the fragment separator.

 func clean(linkURL string) string {
    i := strings.LastIndexByte(linkURL, '#')
    if i < 0 {
        return linkURL
    }
    return linkURL[:i]
 }

Remove all characters after a delimiter in a string

2 Answers2