Yesterday, we added a development database to our EM12c system. About 15 minutes later, the incident management flooded us with alert mails about newly added database. Unfortunately its SID was same with our production DB, and when adding the database, my colleague didn’t change its name and accepted the defaults. I was out of office when I got those mails, after a short panic, I called my colleague to check the situation and she said it’s just the development database. We decided to remove development database from the Enterprise Manager, and add it again with another name.
I went to the target’s home page, clicked remove target under target setup menu. The Enterprise Manager warned me about it will delete the database and its related targets (a Database System). I clicked YES, and my colleague asked me why I removed the production database from Enterprise Manager! As a DBA, we are usually very careful when doing something, and I was sure that I removed the correct database. I checked the databases and saw that both PROD and DEV databases are gone. Of course, we added the production database again quickly, and then I started to investigate the situation. After examining PL/SQL codes of MGMT_ADMIN package, and doing more add-remove tests, I see that it’s all about the DB systems. Here are the records in MGMT_TARGETS_DELETE (it’s a table which is used by EM to store deleted targets):
As you can see, the Enterprise Manager removed the correct database and its DB system. When removing db system, it found the production database and its instances, and then it also removed those targets.
One of the problems here is, most users do not name database systems, they usually accept the one generated by Enterprise Manager. When adding a new Oracle database, Enterprise Manager generates a system name for it, adding “_sys” post-fix to database name. If this system name already exists in your Enterprise Manager environment, your new database will join to the existing system, and when you remove the database target, Enterprise Manager will remove all other targets belong to the db system.
Another problem is, before you remove the target, Enterprise Manager does not show the all related objects in confirmation page. So you should know if the DB systems have other databases/instances.
I’m still searching if there’s an option to remove a target database without removing its database system, but as I see, web interface do not let you do it. Even emcli doesn’t let you keep the DB system while removing the target database.