We are writing a concurrent multimap to store multiple values for the same key in a multi threaded application. We have extended Guava ForwardingMultimap to do the same. Put and remove methods acquire a write lock on a key and release it at the end. Similarly get acquire a read lock on key and release it at the end. I have tested this by having multiple thread in local system and it works fine.

Now the problem is with the clear method, if one thread is doing put/remove and other thread does clear then the inconsistency happened. To handle this problem, in clear method we decided to put a writelock on public key and release it at the end. Similarly put and remove methods were improved to have a read lock on public key and release it at the end. It is something like below.

IResourceLockManager mLocks = new ResourceLockManager();
private final String mPublicKey = "$public";
private final int mLockTimeOut = 0;

Put/Remove Operation

public boolean put(Object aKey, Object aValue) {
   mLocks.acquireReadLock(mPublicKey, mLockTimeOut);
   mLocks.acquireWriteLock(aKey, mLockTimeOut);
   boolean result = false;
   try {
      result = delegate().put(aKey, aValue);
   } finally {
      mLocks.releaseWriteLock(aKey);
      mLocks.releaseReadLock(mPublicKey);
   }
   return result;
}

Clear Operation

public void clear() {
   mLocks.acquireWriteLock(mPublicKey, mLockTimeOut);
   try {
      delegate().clear();
   } finally {
      mLocks.releaseWriteLock(mPublicKey); 
   }
}

ResourceLockManager Code

Striped<ReadWriteLock> lockCache = Striped.lazyWeakReadWriteLock(DEFAULT_PARTITIONS);

acquireReadLock

lockCache.get(aKey).readLock().lock()

releaseReadLock

lockCache.get(aKey).readLock().unlock();

acquireWriteLock

lockCache.get(aKey).writeLock().lock()

releaseWriteLock

lockCache.get(aKey).writeLock().unlock();

After implementing read/write lock on public key to handle clear method case, I started facing below issue:

Exception in thread "pool-5-thread-10" java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
    at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
    at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
    at com.google.common.util.concurrent.ForwardingLock.unlock(ForwardingLock.java:48)
    at org.multimap.rt.locks.ResourceLockManager.releaseReadLock(ResourceLockManager.java:97)
    at org.multimap.rt.util.ConcurrentMultiMap.put(ConcurrentMultiMap.java:81)
    at org.multimap.rt.util.OneShotTask.run(ConcurrentMultiMapTest.java:233)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

The above issue is not just coming for put, it is sometimes coming during remove as well and it is not coming always. It is like it is coming once in every 3-4 runs.

Another Observation - To test concurrency, if we create 44 or less thread, then I am not seeing this issue in my local system. As soon as I make it 45 or more then I started seeing the exception.

Unit Test Code:

   public void TestConcurrentMultiMap() throws InterruptedException {
      ExecutorService executor = Executors.newFixedThreadPool(45);
      for (int i = 0; i < 45; i  ) {
         int num = (int) Math.round(Math.floor(Math.random() * 3));
         int oprn = (int) Math.round(Math.random());
         executor.execute(new OneShotTask(String.valueOf(num), myMap1, oprn));
      }
      executor.shutdown();
      executor.awaitTermination(1000L, TimeUnit.MILLISECONDS);
   }

class OneShotTask implements Runnable {
   String str;
   Multimap mMap;
   int mOprn;

   OneShotTask(String s, Multimap aMap, int oprn) {
      str = s;
      mOprn = oprn;
      mMap = aMap;
   }
   public void run() {
      try {
         Thread.sleep(100);
       } catch (InterruptedException e) {
         throw new RuntimeException(e);
       }
       if(mOprn == 0) {
          mMap.put("key", "value"   str );
       } else { 
          mMap.remove("key", "value"   str ); 
      }
   }
}

CodePudding user response：

Looks like there is a bug in Striped.lazyWeakReadWriteLock(). By changing the initialization from Striped.lazyWeakReadWriteLock to Striped.readWriteLock(), I am unable to reproduce it.

More details here - https://github.com/google/guava/issues/2477