0

After installing and configuring HCL Component Pack 6.5, I navigated to https://<cnx-ihs-host>/social/home and saw parts of OrientMe with some errors:

enter image description here

The browser console shows multiple http 500 errors, for example https://cnx-ihs-host/community_suggestions/api/recommend/communities?count=30

{"errorMessage":"Bad XML response from Search Application: Error: Unexpected close tag\nLine: 11\nColumn: 7\nChar: >","error":"Error: Error: Unexpected close tag\nLine: 11\nColumn: 7\nChar: >\n    at getCommunitiesFromAtom (/home/ibm/app/lib/server/suggestedCommunitiesService.js:98:11)\n    at Request.request [as _callback] (/home/ibm/app/lib/server/suggestedCommunitiesService.js:163:19)\n    at Request.self.callback (/home/ibm/app/node_modules/request/request.js:185:22)\n    at emitTwo (events.js:126:13)\n    at Request.emit (events.js:214:7)\n    at Request.<anonymous> (/home/ibm/app/node_modules/request/request.js:1161:10)\n    at emitOne (events.js:116:13)\n    at Request.emit (events.js:211:7)\n    at IncomingMessage.<anonymous> (/home/ibm/app/node_modules/request/request.js:1083:12)\n    at Object.onceWrapper (events.js:313:30)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickCallback (internal/process/next_tick.js:181:9)"}

So I increment the IHS log to debug and found the following entry:

[debug] mod_proxy_http.c(63): proxy: HTTP: canonicalising URL //cnx-ihs-host/community_suggestions/api/recommend/communities
...
[debug] vhost.c(791): [client 1.2.3.4:44004] [strict] Invalid host name 'cnx65.ihs-host, cnx65.ihs-host', problem near: , cnx6
[debug] vhost.c(886): [client 172.19.34.4:44004] Client sent malformed Host header: cnx65-test-daniel.k8s02.company.internal, cnx65.ihs-host

For me this seems that the hostname was set multiple times like this:

Host: cnx65.ihs-host, cnx65.ihs-host

Why does this happend and how could it be fixed?

Analysis on k8s component pack

I searched for the source, which seems the community-suggestions deployment. For easier troubleshooting (just have to watch 1 pod instead of 3), the deployment was downgraded from 3 to 1 replicas:

kubectl scale deploy community-suggestions --replicas=1

Now there is just a single pod. To get more detailled output, I found the env variable LOG_LEVEL which was stick to info. So I used kubectl edit deploy community-suggestions to set it to debug. This gave me much more infos:

{"pid":23,"hostname":"community-suggestions-bdc8c6577-cxz5x","name":"community-suggestions/lib/server/suggestedCommunitiesService.js","level":30,"time":1583489495149,"msg":"Fetching Community Details using Blue API: https://cnx-ihs-host/search/atom/mysearch?returnType=json&constraint={'type':'field','id':'id','values':[\"ac16d726-118c-4876-a1fd-7f0574aa4cf3\",\"COMMUNITIES\"],'exactMatch':true}&scope=communities:entry","v":1}
2020-03-06T10:11:35.149Z - debug: [orient-web-client] ENTRY: RequestAuthUtils.sanitizeHeaders undefined
2020-03-06T10:11:35.149Z - debug: [orient-web-client] EXIT: RequestAuthUtils.sanitizeHeaders undefined
{"pid":23,"hostname":"community-suggestions-bdc8c6577-cxz5x","name":"community-suggestions/lib/server/suggestedCommunitiesService.js","level":20,"time":1583489495180,"msg":">> getCommunitiesFromAtom <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\n</p>\n<p>Additionally, a 400 Bad Request\nerror was encountered while trying to use an ErrorDocument to handle the request.</p>\n<hr>\n<address>IBM_HTTP_Server at cnx65.internal Port 443</address>\n</body></html> a","v":1}
{"pid":23,"hostname":"community-suggestions-bdc8c6577-cxz5x","name":"community-suggestions/lib/server/suggestedCommunitiesService.js","level":50,"time":1583489495180,"msg":"Error while calling resolve on getCommunitiesFromAtom {}","v":1}
{"pid":23,"hostname":"community-suggestions-bdc8c6577-cxz5x","name":"community-suggestions/lib/server/boot/root.js","level":50,"time":1583489495181,"msg":"Bad XML response from Search Application: Error: Error: Unexpected close tag\nLine: 11\nColumn: 7\nChar: >","v":1}

It shows that the problem is an API request to the search /search/atom/.... Opening those link in the browser with the same CNX user specified in component pack, I get a valid XML file without any error messages.

The mentioned /home/ibm/app/lib/server/suggestedCommunitiesService.js file from the exception generates it's target API address using this function:

function getSearchRoute(res, ids) {
  const restEndPoint = `/atom/mysearch?returnType=json&constraint={'type':'field','id':'id','values':${JSON.stringify(ids)},'exactMatch':true}&scope=communities:entry`;
  const connectionsSearchUrl = `${process.env.CONNECTIONS_SEARCH_URL}`;
  return connectionsSearchUrl + getAuthSchemeFragment(res.locals.isOauth) + restEndPoint;
}

Inside the pod, env variable CONNECTIONS_SEARCH_URL points correctly to https://cnx-ihs-host/search. Component Pack uses the request nodejs module to set a normal GET request:

request({
  method: 'GET',
  headers: scHeaders,
  uri,
  jar
}, ...

I see no reason why there should be an issue with the host header caused by this code.

Lion
  • 16,606
  • 23
  • 86
  • 148

2 Answers2

0

Is customizer deployed? Missing header could be customizer related.

Check nginx forwarding rules, check configmap (http and https).

stoeps
  • 56
  • 5
  • Currently no customizer deployed. Just a default kubernetes ingress that forwards to the IHS directly, but this should forward all headers I assume? – Lion Mar 06 '20 at 13:47
0

I had an ingress in front of IHS, which should act as router to IHS or the K8s part (having an reverse proxy is recommendede from the docs):

User <-----> Ingress <-----> IHS

Later, this proxy should also be used for k8s, so that it decides wherever to route to IHS (WebSphere backend) or the Component Pack (e.g. OrientMe).

The problem was that x-forwarded-host was set to cnx65.ihs-host, cnx65.ihs-host in combination with the default behavior of k8s ingress, which set x-forwarded-for headers as Host. I'm currently figuring out why those headers contains the hostname twice. But the main issue was the k8s ingress behavior, which results in Host: cnx65.ihs-host, cnx65.ihs-host being passed to IHS.

It seems not possible to fix this out of the box, but by adding a custom snippet in nginx.ingress.kubernetes.io/configuration-snippet that sets the host header to the real http host:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: external-service
  annotations: 
    nginx.ingress.kubernetes.io/secure-backends: "true"

    # https://github.com/kubernetes/ingress-nginx/issues/2463#issuecomment-423230548
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_clear_input_headers "Host" "X-Forwarded-Host";
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $http_x_forwarded_host;
Lion
  • 16,606
  • 23
  • 86
  • 148