Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are-CodePudding

The database in the /users folder has almost 50,000 user records and I am not able to read the keys because it is overflowing with memory.

This occurs in the java environment on the mobile device as well as in javaScript on the server with firebase functions.

The database has persistence enabled.

A simple datanapshot.getChildrenCount() in the /users folder is already running out of memory in both environments.

How can I fully read all unique keys of all users to store them in a local list?

Below is an example of the code I am using:

Java:

    public void getUsersCount() {
        ref.child("users").addListenerForSingleValueEvent(new ValueEventListener() {
            public void onDataChange(@NonNull DataSnapshot datasnapshot) {
                Log.w("tag", "TOTAL: "   datasnapshot.getChildrenCount()   " users");
//                for (DataSnapshot snap : datasnapshot.getChildren()) {
//                    list.add(snap.getKey()   "");
//                }
            }
            @Override
            public void onCancelled(@NonNull DatabaseError databaseError) { }
        });
    }

OutOfMemoryError:

2022-03-03 18:02:13.043 1512-1591/com.xxxx.xxxR W/om.xxxx.xxx: Throwing OutOfMemoryError "Failed to allocate a 69318664 byte allocation with 25149440 free bytes and 62MB until OOM, target footprint 496333008, growth limit 536870912" (VmSize 8600872 kB)
        2022-03-03 18:02:13.787 1512-1591/com.xxxx.xxxR W/om.xxxx.xxx: Throwing OutOfMemoryError "Failed to allocate a 69318664 byte allocation with 25165824 free bytes and 62MB until OOM, target footprint 496274696, growth limit 536870912" (VmSize 8600872 kB)
        2022-03-03 18:02:13.790 1512-1591/com.xxxx.xxxR E/RunLoop: Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are syncing to the client (e.g. by using queries or syncing a deeper path). See https://firebase.google.com/docs/database/ios/structure-data#best_practices_for_data_structure and https://firebase.google.com/docs/database/android/retrieve-data#filtering_data
java.lang.OutOfMemoryError: Failed to allocate a 69318664 byte allocation with 25165824 free bytes and 62MB until OOM, target footprint 496274696, growth limit 536870912
at java.lang.StringFactory.newStringFromChars(StringFactory.java:112)
at java.lang.StringBuilder.toString(StringBuilder.java:413)
at java.util.AbstractCollection.toString(AbstractCollection.java:464)
at java.lang.String.valueOf(String.java:2924)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at java.util.AbstractMap.toString(AbstractMap.java:559)
at java.lang.String.valueOf(String.java:2924)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at com.google.firebase.database.connection.PersistentConnectionImpl.onDataPush(PersistentConnectionImpl.java:950)
at com.google.firebase.database.connection.PersistentConnectionImpl.onDataMessage(PersistentConnectionImpl.java:522)
at com.google.firebase.database.connection.Connection.onDataMessage(Connection.java:167)
at com.google.firebase.database.connection.Connection.onMessage(Connection.java:131)
at com.google.firebase.database.connection.WebsocketConnection.appendFrame(WebsocketConnection.java:259)
at com.google.firebase.database.connection.WebsocketConnection.handleIncomingFrame(WebsocketConnection.java:302)
at com.google.firebase.database.connection.WebsocketConnection.access$500(WebsocketConnection.java:34)
at com.google.firebase.database.connection.WebsocketConnection$WSClientTubesock$2.run(WebsocketConnection.java:86)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:920)
        2022-03-03 18:02:13.795 1512-1512/com.xxxx.xxxR E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.xxxx.xxxR, PID: 1512
java.lang.RuntimeException: Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are syncing to the client (e.g. by using queries or syncing a deeper path). See https://firebase.google.com/docs/database/ios/structure-data#best_practices_for_data_structure and https://firebase.google.com/docs/database/android/retrieve-data#filtering_data
at com.google.firebase.database.android.AndroidPlatform$1$1.run(AndroidPlatform.java:96)
at android.os.Handler.handleCallback(Handler.java:938)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.app.ActivityThread.main(ActivityThread.java:8641)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1133)
Caused by: java.lang.OutOfMemoryError: Failed to allocate a 69318664 byte allocation with 25165824 free bytes and 62MB until OOM, target footprint 496274696, growth limit 536870912
at java.lang.StringFactory.newStringFromChars(StringFactory.java:112)
at java.lang.StringBuilder.toString(StringBuilder.java:413)
at java.util.AbstractCollection.toString(AbstractCollection.java:464)
at java.lang.String.valueOf(String.java:2924)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at java.util.AbstractMap.toString(AbstractMap.java:559)
at java.lang.String.valueOf(String.java:2924)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at com.google.firebase.database.connection.PersistentConnectionImpl.onDataPush(PersistentConnectionImpl.java:950)
at com.google.firebase.database.connection.PersistentConnectionImpl.onDataMessage(PersistentConnectionImpl.java:522)
at com.google.firebase.database.connection.Connection.onDataMessage(Connection.java:167)
at com.google.firebase.database.connection.Connection.onMessage(Connection.java:131)
at com.google.firebase.database.connection.WebsocketConnection.appendFrame(WebsocketConnection.java:259)
at com.google.firebase.database.connection.WebsocketConnection.handleIncomingFrame(WebsocketConnection.java:302)
at com.google.firebase.database.connection.WebsocketConnection.access$500(WebsocketConnection.java:34)
at com.google.firebase.database.connection.WebsocketConnection$WSClientTubesock$2.run(WebsocketConnection.java:86)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:920)

CodePudding user response：

A DataSnapshot object contains a snapshot of all data at the path or query that populated it. So what you call "a simple datanapshot.getChildrenCount()" is actually reading your entire users node, which apparently takes more memory than your device has.

The first step to fix this is to stop reading all nodes just to determine a node count. You should instead store the node count value in the database, and update that whenever you add/remove a node. A good example of how to do this, can be found in the child count example in the Cloud Functions repo.

If you need to process all user nodes, you can do that in batches/slices by using the limitToFirst and startAt/startAfter methods of the API.