Getting an IndexOutOfBoundException when IBigQueue.peek

[gimantha]

Hi @bulldog2011, @TobiasMende , @wjw465150 , @illya13

We use bigqueue implementation to buffer incoming data in our code base. We use a background thread to consume the queue (every 1.5 seconds). Recently we did some load test and published some data. We could observe the following exception occurred when we restart our server.

[2016-08-02 10:37:28,676] ERROR {org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsDataIndexer} - Error in processing index batch operations: null
java.lang.IndexOutOfBoundsException
at com.leansoft.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:458)
at com.leansoft.bigqueue.BigArrayImpl.get(BigArrayImpl.java:399)
at com.leansoft.bigqueue.BigQueueImpl.peek(BigQueueImpl.java:144)
at org.wso2.carbon.analytics.dataservice.core.indexing.LocalIndexDataStore$LocalIndexDataQueue.peekNext(LocalIndexDataStore.java:269)
I would like to know, under what conditions, the above exception can be thrown?
We use the bigqueue implementation here[1]

Appreciate your help!

[1] https://github.com/wso2/carbon-analytics/blob/master/components/analytics-core/org.wso2.carbon.analytics.dataservice.core/src/main/java/org/wso2/carbon/analytics/dataservice/core/indexing/LocalIndexDataStore.java

Thanks,

[jsabin]

Seeing the same problem. Anyone know the cause or how to prevent this? Seen this on version 0.7.0.

java.lang.IndexOutOfBoundsException
        at com.leansoft.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:458)
        at com.leansoft.bigqueue.BigArrayImpl.get(BigArrayImpl.java:399)
        at com.leansoft.bigqueue.BigQueueImpl.dequeue(BigQueueImpl.java:104)
        at com.proofpoint.queue.FileBackedQueue.dequeue(FileBackedQueue.java:183)

[MrWickedG]

Does anybody have any solution for this issue? Please let me know
@jsabin @gimantha

[gimantha]

How big is the content that you are storing in the queue?

…

On Mon, Nov 5, 2018 at 3:26 PM MrWickedG ***@***.***> wrote: Does anybody have any solution for this issue? Please let me know — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI7bX7Fz4muhjcUPDGB1BEkC2SYUm1ewks5usAtDgaJpZM4JaWEU> .

-- *Gimantha Bandara* | Associate Technical Lead | WSO2 Inc. (m) +94714961919 | (w) +94117435800 | (e) gimantha@wso2.com GET INTEGRATION AGILE Integration Agility for Digitally Driven Business

[MrWickedG]

@gimantha We create/delete 32 mb page files every several minutes, so i'd say its quite big while maintaining close to a hundred of queues. This error is quite occasional but we did not find a solution for it yet.

Did you solve it somehow?

[gimantha]

Unfortunately no. I was not able to reproduce the issue now.. :(

…

On Monday, November 5, 2018, MrWickedG ***@***.***> wrote: @gimantha <https://github.com/gimantha> We create/delete 32 mb page files every several minutes, so i'd say its quite big while maintaining close to a hundred of queues. This error is quite occasional but we did not find a solution for it yet. Did you solve it somehow? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI7bX5rBSLN_goor23uf4GHI0-9Cgu9hks5usB2EgaJpZM4JaWEU> .

-- *Gimantha Bandara* | Associate Technical Lead | WSO2 Inc. (m) +94714961919 | (w) +94117435800 | (e) gimantha@wso2.com GET INTEGRATION AGILE Integration Agility for Digitally Driven Business

[yoonforh]

i found the problem in BigQueueImpl.java.

in removeAll method of the BigQueueImpl.java
current code is

            queueFrontWriteLock.lock();
            this.innerArray.removeAll();
            this.queueFrontIndex.set(0L);

i think the code should be rearranged like this

            queueFrontWriteLock.lock();
            // if some thread access queueFrontIndex.get() without queueFrontWriteLock, 
            // then this could be a problem.
            // you should clear queueFrontIndex first (peek or dequeue)
            // before remove all the innerArray
            this.queueFrontIndex.set(0L);
            this.innerArray.removeAll();

this code can cause "java.lang.IndexOutOfBoundsException: null" in peek() method and in dequeue() method.
i found the dequeue() method scenario can happen when the gc() method is invoked at the time.

if some thread access queueFrontIndex.get() without queueFrontWriteLock, then this could be a problem.
if old version of removeAll() method is called and gc() is called after innerArray.removeAll() and before queueFrontIndex.set(0), then
queueFrontIndex will return some bigger value and removeBeforeIndex will be called on that index.
and later, dequeue will be called and queueFrontIndex will be zero but the innerArray.arrayTailIndex will be set to the some positive value.
yes, innerArray.validateIndex(0) will cause a problem!