I have a problem which has me stumped. Hoping someone can help me here.
So I'm sadly in the position of having a micro-service which for the moment needs to connect to 4 mongo databases all running in the cloud (Mongo Cloud Atlas) and on the same cluster. After setting everything up and deploying my service, I'm constantly getting errors for timeouts etc every couple of hours.
The various errors are as follows:
MongoSocketReadTimeoutException: Timeout while receiving message
MongoSocketReadException: Exception receiving message
MongoSocketOpenException: Exception opening socket
MongoSocketWriteException: Exception sending message
MongoSocketException: cluster0-shard-00-00-pri.xxxxxx.mongodb.net: Temporary failure in name resolution
** Its worth noting that all my other Spring services which connect to the same cluster but are only configured for a single database do not have any timeout issues
For further context, all my Java services are in in GCP (using cloud run) and all services are running on a VPC with peering to Mongo Atlas.
For the micro-service(s) which connect to a single Mongo DB I configure my Mongo Client like so:
@Configuration
public class MongoConfiguration {
@Value("${spring.data.mongodb.host}")
private String connectionString;
@Bean
public MongoClient mongoClient() {
CodecRegistry pojoCodecRegistry = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry codecRegistry = fromRegistries(MongoClientSettings.getDefaultCodecRegistry(), pojoCodecRegistry);
return MongoClients.create(MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(connectionString))
.uuidRepresentation(UuidRepresentation.STANDARD)
.codecRegistry(codecRegistry)
.build());
}
@Bean
public MongoClientSettings mongoClientSettings() {
return MongoClientSettings.builder()
.retryWrites(true)
.applyToClusterSettings(builder ->
builder.serverSelectionTimeout(5000, TimeUnit.MILLISECONDS))
.applyToConnectionPoolSettings((ConnectionPoolSettings.Builder builder) -> builder.maxSize(300) //connections count
.minSize(100)
.maxConnectionLifeTime(1000, TimeUnit.MILLISECONDS)
.maintenanceFrequency(5000, TimeUnit.MILLISECONDS)
.maxConnectionIdleTime(1000, TimeUnit.MILLISECONDS)
.maxWaitTime(150, TimeUnit.MILLISECONDS))
.applyToSocketSettings(builder -> builder.connectTimeout(10, TimeUnit.SECONDS)
.readTimeout(15, TimeUnit.SECONDS))
.applicationName("ProfileServiceApplication")
.retryWrites(true)
.build();
}
@Bean
public MongoCustomConversions mongoCustomConversions() {
return new MongoCustomConversions(Arrays.asList(
new BigDecimalDecimal128Converter(),
new Decimal128BigDecimalConverter()
));
}
@WritingConverter
private static class BigDecimalDecimal128Converter implements Converter<BigDecimal, Decimal128> {
@Override
public Decimal128 convert(@NonNull BigDecimal source) {
return new Decimal128(source);
}
}
@ReadingConverter
private static class Decimal128BigDecimalConverter implements Converter<Decimal128, BigDecimal> {
@Override
public BigDecimal convert(@NonNull Decimal128 source) {
return source.bigDecimalValue();
}
}
}
On the (problematic) service which connects to multiple Mongo DB's here's how I've configured my mongo clients:
Primary Mongo Client
@Configuration
public class TorchMongoConfig {
@Autowired
private Environment env;
@Primary
@Bean(name = "TorchMongoDBFactory")
public MongoDatabaseFactory torchFactory(@Qualifier("TORCH_CLIENT") MongoClient mongoClient) {
return new SimpleMongoClientDatabaseFactory(mongoClient, "Torch");
}
@Primary
@Bean
public MongoTemplate torchMongoTemplate(@Qualifier("TORCH_CLIENT") MongoClient client) {
return new MongoTemplate(torchFactory(client));
}
@Bean("TORCH_CLIENT")
@Primary
public MongoClient mongoClient() {
CodecRegistry pojoCodecRegistry = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry codecRegistry = fromRegistries(MongoClientSettings.getDefaultCodecRegistry(), pojoCodecRegistry);
return MongoClients.create(MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(env.getProperty("spring.data.mongo.torchdb.uri")))
.uuidRepresentation(UuidRepresentation.STANDARD)
.codecRegistry(codecRegistry)
.applyToConnectionPoolSettings(builder -> {
builder.maxConnectionLifeTime(1000, TimeUnit.MILLISECONDS)
.maintenanceFrequency(5000, TimeUnit.MILLISECONDS)
.maxConnectionIdleTime(1000, TimeUnit.MILLISECONDS)
.maxWaitTime(150, TimeUnit.MILLISECONDS);
})
.applyToClusterSettings(builder -> {
builder.serverSelectionTimeout(5000, TimeUnit.MILLISECONDS);
})
.build());
}
@Bean
public MongoClientSettingsBuilderCustomizer mongoDBDefaultSettings() {
return builder -> {
builder.applyToSocketSettings(builderSettings -> {
builderSettings.connectTimeout(10, TimeUnit.SECONDS)
.readTimeout(15, TimeUnit.SECONDS);
});
builder.applyToClusterSettings(builderSettings -> {
builderSettings
.serverSelectionTimeout(5000, TimeUnit.MILLISECONDS);
});
builder.applyToConnectionPoolSettings(builderSettings -> {
builderSettings.maxSize(300) //connections count
.minSize(100)
.maxConnectionLifeTime(1000, TimeUnit.MILLISECONDS)
.maintenanceFrequency(5000, TimeUnit.MILLISECONDS)
.maxConnectionIdleTime(1000, TimeUnit.MILLISECONDS)
.maxWaitTime(150, TimeUnit.MILLISECONDS);
});
// Apply other settings to the builder.
};
}
@Bean
public MongoCustomConversions mongoCustomConversions() {
return new MongoCustomConversions(Arrays.asList(
new BigDecimalDecimal128Converter(),
new Decimal128BigDecimalConverter()
));
}
@WritingConverter
private static class BigDecimalDecimal128Converter implements Converter<BigDecimal, Decimal128> {
@Override
public Decimal128 convert(@NonNull BigDecimal source) {
return new Decimal128(source);
}
}
@ReadingConverter
private static class Decimal128BigDecimalConverter implements Converter<Decimal128, BigDecimal> {
@Override
public BigDecimal convert(@NonNull Decimal128 source) {
return source.bigDecimalValue();
}
}
}
Secondary MongoClient
@Configuration
public class ProjectInformationMongoConfig {
@Autowired
private Environment env;
@Bean(name = "ProjectInformationMongoDBFactory")
public MongoDatabaseFactory userProjectMongoDatabaseFactory(@Qualifier("ProjectInformation") MongoClient mongoClient) {
return new SimpleMongoClientDatabaseFactory(mongoClient, "ProjectInformation");
}
@Bean(name = "projectInformationMongoTemplate")
public MongoTemplate userProjectMongoTemplate(@Qualifier("ProjectInformationMongoDBFactory") MongoDatabaseFactory mongoDatabaseFactory) {
return new MongoTemplate(mongoDatabaseFactory);
}
@Bean("ProjectInformation")
public MongoClient mongoClient() {
CodecRegistry pojoCodecRegistry = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry codecRegistry = fromRegistries(MongoClientSettings.getDefaultCodecRegistry(), pojoCodecRegistry);
return MongoClients.create(MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(env.getProperty("spring.data.mongo.projectinformation.uri")))
.uuidRepresentation(UuidRepresentation.STANDARD)
.codecRegistry(codecRegistry)
.applyToConnectionPoolSettings(builder -> {
builder.maxConnectionLifeTime(30, TimeUnit.MINUTES)
.maintenanceFrequency(5000, TimeUnit.MILLISECONDS)
.maxConnectionIdleTime(1000, TimeUnit.MILLISECONDS)
.maxWaitTime(150, TimeUnit.MILLISECONDS);
})
.applyToClusterSettings(builder -> {
builder.serverSelectionTimeout(5000, TimeUnit.MILLISECONDS);
})
.build());
}
}
The third and forth connections are identical to the above two so I will not repeat the code again.
Now when I want to access data on Mongo here's how I'm doing it via my DAL:
Repository
public class SavedUserProfilesDALImpl implements SavedUserProfilesDal {
private static final Logger logger = LoggerFactory.getLogger(SavedUserProfilesDal.class);
@Autowired
@Qualifier("torchMongoTemplate")
private final MongoTemplate mongoTemplate;
public SavedUserProfilesDALImpl(@Qualifier("torchMongoTemplate") MongoTemplate mongoTemplate) {
this.mongoTemplate = mongoTemplate;
}
@Override
public SavedUserProfiles save(SavedUserProfiles savedUserProfile) {
logger.info("MONGO: Executing query save / update UserSavedQuery with userId {}", savedUserProfile.getUserId());
try {
return mongoTemplate.save(savedUserProfile);
} catch (Exception ex) {
logger.error("Unable to perform Mongo Operation Save SavedUserProfiles {}", ex.getLocalizedMessage());
throw new ServiceUnavailableException("Unable to perform Mongo Operation {" ex.getLocalizedMessage() "}", "SavedUserProfiles");
}
}
And when I want to query something which is not using the primary bean I do like so:
@Repository
public class ThirdPartyDALImpl implements ThirdPartyContentDAL {
private final static Logger logger = LoggerFactory.getLogger(ThirdPartyDALImpl.class);
@Autowired
@Qualifier("projectInformationMongoTemplate")
private final MongoTemplate mongoTemplate;
public ThirdPartyDALImpl(@Qualifier("projectInformationMongoTemplate") MongoTemplate mongoTemplate) {
this.mongoTemplate = mongoTemplate;
}
@Override
public ThirdPartyContent fetchContentById(String thirdPartyContentId) {
Query query = new Query();
query.addCriteria(Criteria.where("_id").is(thirdPartyContentId));
try {
return mongoTemplate.findOne(query, ThirdPartyContent.class);
} catch (Exception ex) {
throw new ServiceUnavailableException("Unable to perform Mongo Operation {" ex.getLocalizedMessage() "}", "Project");
}
}
I'd like to stress, I believe the issues I'm having are related to something in my configuration above. I think given that all other services (connecting to a single Mongo DB) are all on the same network, firewall, VPC etc.
I guess this is a cry for help as I'm pulling my hair out with constant alerts :-D Any help would be amazing.
Thanks
CodePudding user response：
Ok so the Problem was on the GCP cloud run side. The CPU wasn't dedicated so the connections to Mongo were getting cut.