To synchronize data by timestamp, the data source in another business department, they provided the data interface, in the process of synchronization, found that data is always not consistent, after screening, issue to the data source:

1. Data changed the table structure, which leads to the synchronization failure;
2. Do physically deleted data source, data delete cannot be achieved synchronization,
the increase in the number of data to our side

At present only through communication with the data source, make sure they follow the rules of the synchronization, however, there is always such a risk: they suddenly changed, not inform us in time,

In the face of such synchronization problem, have a good way to technical?

CodePudding user response:

Using OGG synchronization, oracle goldengate

CodePudding user response:

This depends on what your synchronously,

Otherwise, there is no way to help you analyze problems,

CodePudding user response:

Synchronization method is to scan the timestamp, according to the time to touch the synchronization,
Now the main problem is there are always people change table structure data source, this is not unexpected for us, they didn't inform us in time,

CodePudding user response:

Scan time stamp, this method also is commonly used, and they were afraid they remove fields, that your query will be an error,

If they changed the table structure, after you get to the data, your system will change too, do not inform you, belong to management issue,

CodePudding user response:

It depends on the synchronous way, timestamp may delete data cannot be synchronized problems,


Delete records can consider to use the trigger, copied to the other table, and then use this form to do delete data synchronization,

CodePudding user response:

Do you have any other technical solutions, solve the problem of this due to management problems?

CodePudding user response:

The source side don't cooperate, how to get bad...

CodePudding user response:

Data source changed the table structure, the failure of synchronization problem can be addressed by etl tools, such as data services, the source table data synchronization to the target table, if the source table table structure change, in the process of etl will automatically convert the field type to match the field type of the target table;
Source to do physical deletion problem can by the CDC data services and database functions to solve together, specific as follows: in building an intermediate between source and target table table, data source table by table compares pumped into the middle table, following the middle table by the CDC, tracking results in the target table