0

Note: I already checked similar questions & tried their solutions but could not resolve this issue.
I have installed Solr 8.7.0 on Ubuntu 20.04 & have created some blank cores for testing.
I was able to access Admin UI at http://139.59.75.45:8983/solr/
But after I tried to upload a document from my desktop solr\example\films\films.csv it stopped responding.
I checked from terminal, solr service is active but when I tried

curl -X GET "http://139.59.75.45:8983/solr/"

I got following response:
curl: (7) Failed to connect to 139.59.75.45 port 8983: Connection refused.

Note I did not make any changes to core schema before uploading. Still after searching all logs, stackoverflow I could not find the reason. Please advise as to how to resolve this issue.
Following is solr service status output:

root@lemp-02:~# service solr status
● solr.service - LSB: Controls Apache Solr as a Service
     Loaded: loaded (/etc/init.d/solr; generated)
     Active: active (exited) since Sat 2020-12-26 18:22:53 UTC; 9h ago
       Docs: man:systemd-sysv-generator(8)
    Process: 4148 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Dec 26 18:22:45 lemp-02 systemd[1]: Starting LSB: Controls Apache Solr as a Service...
Dec 26 18:22:45 lemp-02 su[4160]: (to solr) root on none
Dec 26 18:22:45 lemp-02 su[4160]: pam_unix(su-l:session): session opened for user solr by (uid=0)
Dec 26 18:22:53 lemp-02 solr[4225]: Started Solr server on port 8983 (pid=4220). Happy searching!
Dec 26 18:22:53 lemp-02 systemd[1]: Started LSB: Controls Apache Solr as a Service.

Following is solr.log:

2020-12-26 18:22:47.606 INFO  (main) [   ] o.e.j.u.log Logging initialized @2098ms to org.eclipse.jetty.util.log.Slf4jLog
2020-12-26 18:22:48.045 INFO  (main) [   ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.9.1+1-Ubuntu-0ubuntu1.20.04
2020-12-26 18:22:48.080 INFO  (main) [   ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr-8.5.2/server/contexts/] at interval 0
2020-12-26 18:22:48.758 INFO  (main) [   ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2020-12-26 18:22:48.783 INFO  (main) [   ] o.e.j.s.session DefaultSessionIdManager workerName=node0
2020-12-26 18:22:48.783 INFO  (main) [   ] o.e.j.s.session No SessionScavenger set, using defaults
2020-12-26 18:22:48.791 INFO  (main) [   ] o.e.j.s.session node0 Scavenging every 600000ms
2020-12-26 18:22:48.990 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2020-12-26 18:22:48.999 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.5.2
2020-12-26 18:22:49.004 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port 8983
2020-12-26 18:22:49.004 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /opt/solr
2020-12-26 18:22:49.005 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-12-26T18:22:49.005242Z
2020-12-26 18:22:49.118 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using system property solr.solr.home: /var/solr/data
2020-12-26 18:22:49.128 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading container configuration from /var/solr/data/solr.xml
2020-12-26 18:22:49.281 INFO  (main) [   ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@33065d67, but no JMX reporters were configured - adding default JMX reporter.
2020-12-26 18:22:50.611 INFO  (main) [   ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
2020-12-26 18:22:50.981 WARN  (main) [   ] o.e.j.u.s.S.config Trusting all certificates configured for Client@1cb7936c[provider=null,keyStore=null,trustStore=null]
2020-12-26 18:22:50.982 WARN  (main) [   ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@1cb7936c[provider=null,keyStore=null,trustStore=null]
2020-12-26 18:22:51.458 WARN  (main) [   ] o.e.j.u.s.S.config Trusting all certificates configured for Client@35cd68d4[provider=null,keyStore=null,trustStore=null]
2020-12-26 18:22:51.458 WARN  (main) [   ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@35cd68d4[provider=null,keyStore=null,trustStore=null]
2020-12-26 18:22:51.504 WARN  (main) [   ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
2020-12-26 18:22:51.890 INFO  (main) [   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
2020-12-26 18:22:51.893 INFO  (main) [   ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
2020-12-26 18:22:52.080 INFO  (main) [   ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@33065d67
2020-12-26 18:22:52.086 INFO  (main) [   ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@33065d67
2020-12-26 18:22:52.101 INFO  (main) [   ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@33065d67
2020-12-26 18:22:52.175 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Found 2 core definitions underneath /var/solr/data
2020-12-26 18:22:52.176 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Cores are: [mycollection, test_core]
2020-12-26 18:22:52.386 INFO  (coreLoadExecutor-9-thread-2) [   x:test_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.5.2
2020-12-26 18:22:52.389 INFO  (coreLoadExecutor-9-thread-1) [   x:mycollection] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.5.2
2020-12-26 18:22:52.539 INFO  (main) [   ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@7894f09b{/solr,file:///opt/solr-8.5.2/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-8.5.2/server/solr-webapp/webapp}
2020-12-26 18:22:52.595 INFO  (main) [   ] o.e.j.s.AbstractConnector Started ServerConnector@66fdec9{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:8983}
2020-12-26 18:22:52.595 INFO  (main) [   ] o.e.j.s.Server Started @7094ms
2020-12-26 18:22:52.985 INFO  (coreLoadExecutor-9-thread-1) [   x:mycollection] o.a.s.s.IndexSchema [mycollection] Schema name=default-config
2020-12-26 18:22:52.986 INFO  (coreLoadExecutor-9-thread-2) [   x:test_core] o.a.s.s.IndexSchema [test_core] Schema name=default-config

EDIT - 1
Following are more logs & cli command results. It looks like its running out of memory.

Here are results of some cli commands:

root@lemp-02:~# nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2020-12-29 09:05 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000040s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
3306/tcp open  mysql

Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds
root@lemp-02:~# lsof -i -P -n
COMMAND    PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r  620 systemd-resolve   12u  IPv4  19121      0t0  UDP 127.0.0.53:53
systemd-r  620 systemd-resolve   13u  IPv4  19122      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd       800            root    3u  IPv4  22677      0t0  TCP *:22 (LISTEN)
sshd       800            root    4u  IPv6  22681      0t0  TCP *:22 (LISTEN)
nginx      804            root    6u  IPv4  22661      0t0  TCP *:80 (LISTEN)
nginx      804            root    7u  IPv6  22662      0t0  TCP *:80 (LISTEN)
nginx      805        www-data    6u  IPv4  22661      0t0  TCP *:80 (LISTEN)
nginx      805        www-data    7u  IPv6  22662      0t0  TCP *:80 (LISTEN)
mysqld     873           mysql   32u  IPv4  27590      0t0  TCP 127.0.0.1:33060 (LISTEN)
mysqld     873           mysql   34u  IPv4  28038      0t0  TCP 127.0.0.1:3306 (LISTEN)
master    1995            root   13u  IPv4  29598      0t0  TCP *:25 (LISTEN)
master    1995            root   14u  IPv6  29599      0t0  TCP *:25 (LISTEN)
sshd      2073            root    4u  IPv4  34342      0t0  TCP 139.59.75.45:22->39.42.164.21:55432 (ESTABLISHED)
root@lemp-02:~# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     LIMIT IN    Anywhere
80,443/tcp (Nginx Full)    ALLOW IN    Anywhere
8983                       ALLOW IN    Anywhere
23/tcp                     ALLOW IN    Anywhere
8983                       ALLOW IN    139.59.75.45
22/tcp (v6)                LIMIT IN    Anywhere (v6)
80,443/tcp (Nginx Full (v6)) ALLOW IN    Anywhere (v6)
8983 (v6)                  ALLOW IN    Anywhere (v6)
23/tcp (v6)                ALLOW IN    Anywhere (v6)

root@lemp-02:~# service solr status
● solr.service - LSB: Controls Apache Solr as a Service
     Loaded: loaded (/etc/init.d/solr; generated)
     Active: active (exited) since Tue 2020-12-29 09:03:45 UTC; 3min 6s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 770 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Dec 29 09:00:41 lemp-02 systemd[1]: Starting LSB: Controls Apache Solr as a Service...
Dec 29 09:00:41 lemp-02 su[776]: (to solr) root on none
Dec 29 09:00:41 lemp-02 su[776]: pam_unix(su-l:session): session opened for user solr by (uid=0)
Dec 29 09:03:45 lemp-02 systemd[1]: Started LSB: Controls Apache Solr as a Service.
root@lemp-02:~# netstat -tunelp | grep 8983
root@lemp-02:~# curl -X GET 'http://localhost:8983/solr'
curl: (7) Failed to connect to localhost port 8983: Connection refused
root@lemp-02:~#

Following is log data from /var/solr/logs/solr-8983-console.log
It seems solr-8983-console.log is more important & informative than solr.log

OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 1)
OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)
OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)
java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
    at org.eclipse.jetty.start.Main.start(Main.java:491)
    at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.net.SocketException: Unresolved address
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1873)
    ... 7 more
Caused by: java.net.SocketException: Unresolved address
    at java.base/sun.nio.ch.Net.translateToSocketException(Net.java:170)
    at java.base/sun.nio.ch.Net.translateException(Net.java:196)
    at java.base/sun.nio.ch.Net.translateException(Net.java:202)
    at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:82)
    at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.eclipse.jetty.server.Server.doStart(Server.java:385)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1924)
    ... 9 more
Caused by: java.nio.channels.UnresolvedAddressException
    at java.base/sun.nio.ch.Net.checkAddress(Net.java:130)
    at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
    at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
    ... 17 more
java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
    at org.eclipse.jetty.start.Main.start(Main.java:491)
    at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.net.SocketException: Unresolved address
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1873)
    ... 7 more
Caused by: java.net.SocketException: Unresolved address
    at java.base/sun.nio.ch.Net.translateToSocketException(Net.java:170)
    at java.base/sun.nio.ch.Net.translateException(Net.java:196)
    at java.base/sun.nio.ch.Net.translateException(Net.java:202)
    at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:82)
    at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.eclipse.jetty.server.Server.doStart(Server.java:385)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1924)
    ... 9 more
Caused by: java.nio.channels.UnresolvedAddressException
    at java.base/sun.nio.ch.Net.checkAddress(Net.java:130)
    at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
    at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
    ... 17 more

Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
       java -jar $JETTY_HOME/start.jar --help  # for more information

My questions:

  1. Solr service is running & Port 8983 is allowed when we check using UFW but why it does not show up when we lsof -i -P -n ?
  2. Which of the log configuration files present at following two locations should be changed to make any changes to log configuration:
    /opt/solr-8.7.0/server/resources/log4j2.xml and
    /var/solr/log4j2.xml ?
  3. Are log levels case sensitive, should we use finest or FINEST?
  4. From solr-8983-console.log it looks like the problem is memory. I am using USD 5 Droplet from Digitalocean RAM 1GB + 1vCPU. What is minimum requirement to run a solr instance without considering size of documents?

EDIT - 2
Just to confirm my doubts about memory, I created a new instance at Digitalocean with 16GB RAM + 2vCPUs. I installed solr on it & when checked its logs (solr-8983-console.log), I found same VM errors as given below which means its not the issue, there is something else wrong:

OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 1)
OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)
OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)
2020-12-29 18:21:50.206 INFO  (main) [   ] o.e.j.u.log Logging initialized @1178ms to org.eclipse.jetty.util.log.Slf4jLog
2020-12-29 18:21:50.305 WARN  (main) [   ] o.e.j.x.XmlConfiguration Ignored arg: <Arg name="threadpool">
    <New id="threadpool" class="com.codahale.metrics.jetty9.InstrumentedQueuedThreadPool"><Arg name="registry">
        <Call id="solrJettyMetricRegistry" name="getOrCreate" class="com.codahale.metrics.SharedMetricRegistries"><Arg>solr.jetty</Arg></Call>
      </Arg></New>
  </Arg>
2020-12-29 18:21:50.449 INFO  (main) [   ] o.e.j.s.Server jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 11.0.9.1+1-Ubuntu-0ubuntu1.20.04
2020-12-29 18:21:50.474 INFO  (main) [   ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr-8.7.0/server/contexts/] at interval 0
2020-12-29 18:21:50.849 INFO  (main) [   ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2020-12-29 18:21:50.863 INFO  (main) [   ] o.e.j.s.session DefaultSessionIdManager workerName=node0
2020-12-29 18:21:50.864 INFO  (main) [   ] o.e.j.s.session No SessionScavenger set, using defaults
2020-12-29 18:21:50.867 INFO  (main) [   ] o.e.j.s.session node0 Scavenging every 600000ms
2020-12-29 18:21:50.944 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2020-12-29 18:21:50.950 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.7.0
2020-12-29 18:21:50.951 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port 8983
2020-12-29 18:21:50.951 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /opt/solr
2020-12-29 18:21:50.953 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-12-29T18:21:50.953065Z
MyO
  • 413
  • 1
  • 8
  • 19
  • You should not expose Solr directly to the internet. If it works before attempting to upload the file, you should check the log after it has stopped responding - it should show the upload attempt at least. – MatsLindh Dec 27 '20 at 18:01
  • Thanks, Mats. Its only a test & learn instance & is open at same ip address. I have attached all the logs above but cant understand myself about any issues. – MyO Dec 28 '20 at 04:50
  • The log doesn't seem to show anything about content being submitted - I'd guess at least something would show up. If everything freezes after submitting the csv file and nothing happens in the log - and you can't access the admin interface after that - something weird is going on. Test an even more recent version of Solr (fetch the most recent one). – MatsLindh Dec 28 '20 at 08:58
  • @MatsLindh I have added some more logs, please have a look & advise me about the questions. – MyO Dec 29 '20 at 11:38
  • `Server VM warning: Failed to reserve shared memory. (error = 12)` - iirc error 12 is that the VM is out of memory. That would explain why everything is gone. The minimum memory requirement would depend on what you expect the server to able to do for you; in your case it's fairly certain that 1GB is far too low. For testing it's perfectly fine to just run the instance locally and play around with it - it works fine on Linux, OS X and Windows. – MatsLindh Dec 29 '20 at 11:42
  • But solr is not allowing me to play :( – MyO Dec 29 '20 at 11:44
  • solr-8983-console.log shows same error messages even on a machine with 16GB RAM + 2vCPUs which means its not a memory issue ! – MyO Dec 29 '20 at 18:37

0 Answers0