I have a microservice name order-creation-service(used Java) which creates food orders .Orders are stored in SQL database.Lets say DB is experiencing outage/down for 1 hour.But I dont want to stop my order creation process as this would result in financial loss.What is the way to handle this failure scenario ?
CodePudding user response：
Basically you are asking how to increase availability for a database.
There are several approaches to dial with such issues - most of them are some types of replication (sync, async or semi-sync). I wouldn't go with adding a cache or even switching your database technology. Adding a cache layer is another system to manage - and that will add consistency challenges to your system, not something anyone wants to deal with.
As a first step, I would look into a synchronous replication with manual failover. This is easiest way to increase availability.
After manual stuff is done, you should look into automatic failover. There are plenty of solutions (depending on what db you are using). As an example, I recommend to look into GitHub's infrastructure - which is based on Orchestrator: https://github.blog/2016-12-08-orchestrator-github/ This way you reduce the down time by a lot.
CodePudding user response：
If there's no avaiable DB in which to save incoming orders, then fundamentally the order service has to do one of two things:
- accept orders but don't save them in the DB
- reject orders
Note that if you take the "accept but don't save" approach, the DB can't guarantee to always have every order that's ever been accepted (even if there's a bulletproof process for getting the orders which were accepted during the DB's downtime into the DB, that process will have some latency).
You could for instance, save orders into a different (unlikely to become unavailable at the same time as the primary) DB. Note that this requires anything which queries the primary DB to be aware that it will either get an incomplete view or be able to query the failover DB as well as the primary (some queries may become massively more difficult and/or inefficient by introducing the failover DB as the primary DB).
It's far more often than not effectively impossible to take an application that hasn't been designed for high-availability and make it highly-available (if availability is defined as "being able to fulfill its functional requirements", not just "answers a health-check ping"). Choosing datastores which are themselves highly-available and being willing to design around their limitations (e.g. some queries may not be practical) and adopting other patterns throughout your architecture may make it easier to implement a highly-available application.