Inconsistent Search Results In Share

Description

[Description]

Searches in share are not consistent, sometimes searches return 0 results even though the search term should match files found.

[Steps to reproduce]

->Increase the memory to 4g for alfresco and solr6 services
->Set the alfresco version to 6.2.2.12 under the alfresco service section

->Enable dynamic sharding discovery in the alfresco service by adding the lines below under JAVA_OPTS

->Disable search default core creation, under the solr6 service delete the line below

  • Deploy ACS using docker compose and the edited docker-compose.yml file

  • Create two shards using the url below

  • Login to share

  • Create a site

  • Upload files with the name "test" (or upload the folder "my600" provided which contains 600 test files)

  • Let the indexing process complete (about 5 mins)

  • In a terminal run the attached shell script, this will simulate 350 calls to the share search looking for files with the string "test". Make sure the <HOST> field is changed to reflect the test environment

  • After about 5 mins check the output files created by the script, 350 files should be created

[Expected Behaviour]

All output files have the same content and match all 600 files in the repo

[Observed Behaviour]

The output files have different content, many of them do not find any files

while others find the correct number of results.

[Analysis to date]

The customer has not made any changes to the search layer and they are still using the same indices and instance as when they were using 5.2. When they moved to 6.2 that's when they noticed the issue. This intermittent behavior does not seem to affect the REST api (https://api-explorer.alfresco.com/api-explorer/#!/search/search).

If the logger org.alfresco.repo.search.impl.solr.SolrQueryHTTPClient is set to debug it can be seen that not all requests produce the statement similar to the one below:

DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-352] Sent :/solr/

NOTE: this issue is not reproducible when using a non sharded instance, only with sharding this issue is exposed.

Environment

tomcat/postgres

Testcase ID

None

Activity

Show:
Martin Stanford
March 8, 2021, 9:38 AM

These parameters are covered by the following docs ticket:

Jose Portillo
March 5, 2021, 5:08 PM

The customer has tested the changes in QA and all of their testing has passed. It will be good to know if this proposed change will be enough to take care of the issue.

Jose Portillo
March 3, 2021, 10:47 PM

After changing the following in alfresco-global.properties the tests have now passed without 0 results issues

 

 

I have passed these settings to the customer and they have tested in their DEV environment where things are no longer failing. They will put the changes in QA and re-test.

Jose Portillo
March 3, 2021, 4:46 PM
Edited

While debugging the remote java application I am able to see that the exceptions being thrown are these different ones:

org.apache.commons.httpclient.NoHttpResponseException: The server solr6 failed to respond

 

org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 500 ms

 

java.net.SocketTimeoutException: Read timed out

 

Doesn’t seem to be very consistent depending on the test run.

Jose Portillo
March 3, 2021, 3:22 PM

Enabled these extra loggers

org.apache.commons.httpclient.HttpConnection=trace
org.apache.commons.httpclient.HttpParser=trace

When we get 0 results I can see the request goes to solr but there’s an exception reading the response, not clear what the exception is or why it’s thrown

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpMethodDirector] [http-nio-8080-exec-377] Attempt number 1 to process request

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.open()

2021-03-03 15:15:15,615 DEBUG [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] Open connection to solr6:8983

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.print(String)

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[])

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[], int, int)

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.print(String)

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[])

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[], int, int)

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.print(String)

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[])

2021-03-03 15:15:15,615 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[], int, int)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.print(String)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[])

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[], int, int)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.print(String)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[])

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[], int, int)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.writeLine()

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[])

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.write(byte[], int, int)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.getRequestOutputStream()

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.flushRequestOutputStream()

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.readLine()

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpParser] [http-nio-8080-exec-377] enter HttpParser.readLine(InputStream, String)

2021-03-03 15:15:15,616 TRACE [org.apache.commons.httpclient.HttpParser] [http-nio-8080-exec-377] enter HttpParser.readRawLine()

2021-03-03 15:15:17,617 DEBUG [org.apache.commons.httpclient.HttpMethodDirector] [http-nio-8080-exec-377] Closing the connection.

2021-03-03 15:15:17,617 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.close()

2021-03-03 15:15:17,617 TRACE [org.apache.commons.httpclient.HttpConnection] [http-nio-8080-exec-377] enter HttpConnection.closeSockedAndStreams()

2021-03-03 15:15:17,617 DEBUG [org.apache.commons.httpclient.HttpMethodDirector] [http-nio-8080-exec-377] Method retry handler returned false. Automatic recovery will not be attempted

Done

Assignee

Unassigned

Reporter

Jose Portillo

Hot Fix Version

ACS 6.2.2, SS 1.4.3

ACT Numbers

00362668

Regression Since

5.2.7.8

Premier Customer

Yes

Delivery Team

Customer Excellence

Bug Priority

Category 1