If you want to perform content staging bi-directionally between two servers, i.e. you want to transfer changes made on Server 1 to Server 2 and also in the other direction from Server 2 to Server 1, you need to to configure both servers as a source and a target server at the same time, as described in the Content staging configuration topic.
On top of it, you also need to disable the Log staging changes option in Site Manager -> Settings -> Content staging on both servers.
If this option is enabled, changes made to the system via content staging synchronization (i.e. transferred from Server 1 to Server 2) are logged as new synchronization tasks on the target server (on Server 2). Now if you perform synchronization in the reversed direction, i.e. back from Server 2 to Server 1, these tasks are also performed and logged back on Server 1 as new staging tasks. This goes on and on in a never-ending loop, which results in the tasks remaining and not being deleted on both servers.
In order to prevent this behavior, you need to disable the Log staging changes option on both servers so that the staging changes are not logged as new synchronization tasks.