PostgresSQL server failing to start after updating keystore in server.xml with new certificate

PostgresSQL server failing to start after updating keystore in server.xml with new certificate

I installed PMP on a fresh Windows 2012 R2 server.  During configuration, I was confused by the "installed as a service" references in the documentation and was concerned that I had not installed it properly.  So I uninstalled it and tried to reinstall it.  The uninstall process had not deleted the original install folder. I tried to delete the folder but it was in use.  I suspected the postgressql database but didn't check for certain.  After rebooting I was able to install PMP again.  I'm not sure if that is causing the issue or not, but since the issue is with postgressql I thought I'd mention it.

I have configured the PMP windows service to run as a domain account (it's not clear in the documentation how to do this - but I need it running as a domain account so that it can access the share I want to put the key file that is supposed to be stored remotely).  I gave it local administrative rights (no indication that this is needed, but since it would need "write" rights to a number of folders and likely the postgressql database, I gave it local admin).  Again - not sure if that is related, but thought I would mention it.

I could successfully browse to the localhost:7272 port, and remotely. I was getting the certificate error so I decided to generate one with keytool, which I then signed with my enterprise CA.  I installed this certificate in the keystore and configured it to work in server.xml.  After restarting the PMP service, I was unable to browse to the PMP web page. I took a look in the wrapper.log file and found an error stating that "Trying to start PostgresSQL server failed".  After rebooting, PostgresSQL server came back up.  I suspected that this was okay as I will almost never need to change the keystore or XML file, so an occasional reboot is nothing to panic over.

After browsing to the web page via the netbios name, I got certificate errors, but it did work with the FQDN.  I then decided that i needed a new certificate that had both the FQDN and the netbios name; subject alternative DNS names.

I created this second certificate using the exact same process, with no problems at all.  After configuring the server.xml file again, I had to reboot the server again - postgresSQL was not starting.  However, NOW postgresSQL is no longer starting after the reboot.  Going into the various log files, I've noticed a few events that MIGHT be related:

pmp0.txt mentions a file that does not exist.  The error occurs MANY times, going back a day - prior even to updating the certificate for the first time).  Below are the last entries in the file, I rebooted it at 17:35 and got no more errors in this log (but the issue still remains):
[17:09:46:013]|[10-20-2015]|[com.adventnet.passtrix.client.util.ClientUtil]|[INFO]|[75]:  getSlaveHost - conf file does not exists ..\pgsql\bin\Primary.conf|
[17:12:24:940]|[10-20-2015]|[com.adventnet.passtrix.service.PassTrixService]|[INFO]|[18]: Entering PassTrixService StopService |

The serverout0.txt log file appears to have more information:
[17:34:54:645]|[10-20-2015]|[com.adventnet.mfw.Server]|[INFO]|[17]: Creating new Server instance|
[17:34:55:554]|[10-20-2015]|[com.adventnet.persistence.ConfigurationParser]|[INFO]|[17]: Processing D:\ManageEngine\PMP\bin\..\conf\product-config.xml|
[17:34:55:554]|[10-20-2015]|[com.adventnet.persistence.ConfigurationParser]|[INFO]|[17]: Processing D:\ManageEngine\PMP\bin\..\conf\customer-config.xml|
[17:34:56:071]|[10-20-2015]|[com.zoho.net.handshake.HandShakeServer]|[INFO]|[19]: Starting HandShakeServer...|
[17:34:56:086]|[10-20-2015]|[com.zoho.net.handshake.HandShakeServer]|[INFO]|[19]: Writing HandShakeServer port [49617] in .lock file|
[17:34:56:086]|[10-20-2015]|[com.zoho.net.handshake.HandShakeServer]|[INFO]|[19]: HandShakeServer listening port :: 49617|
[17:34:56:086]|[10-20-2015]|[com.zoho.net.handshake.HandShakeServer]|[INFO]|[19]: Waiting for client connection...|
[17:34:56:180]|[10-20-2015]|[com.zoho.net.handshake.HandShakeUtil]|[INFO]|[17]: Started HandShakeServer successfully.|
[17:34:56:305]|[10-20-2015]|[com.adventnet.persistence.ConfigurationParser]|[INFO]|[17]: Processing D:\ManageEngine\PMP\bin\..\conf\product-config.xml|
[17:34:56:305]|[10-20-2015]|[com.adventnet.persistence.ConfigurationParser]|[INFO]|[17]: Processing D:\ManageEngine\PMP\bin\..\conf\customer-config.xml|
[17:34:58:227]|[10-20-2015]|[com.zoho.framework.utils.crypto.EnDecrypt]|[SEVERE]|[17]: Encryption failed|
[17:34:58:383]|[10-20-2015]|[com.adventnet.persistence.PersistenceInitializer]|[INFO]|[17]: Reading ./../conf/database_params.conf |
[17:34:58:383]|[10-20-2015]|[com.zoho.framework.utils.crypto.EnDecrypt]|[SEVERE]|[17]: Encryption failed|
[17:34:58:602]|[10-20-2015]|[com.adventnet.ds.DefaultDataSourcePlugIn]|[INFO]|[17]: ConnectionPoolParams :: minSize :: [20], maxSize :: [1], idleTimeout :: [1,800 seconds], blockingTimeout :: [30 seconds]|
[17:34:58:899]|[10-20-2015]|[com.adventnet.ds.DefaultDataSourcePlugIn]|[INFO]|[17]: DataSourceorg.jboss.resource.adapter.jdbc.WrapperDataSource@1ba3a5a|
[17:34:58:993]|[10-20-2015]|[com.adventnet.persistence.PersistenceInitializer]|[INFO]|[17]: Archive Adapter class ::: com.adventnet.db.archive.DefaultArchiveAdapter|
[17:34:58:993]|[10-20-2015]|[com.adventnet.persistence.PersistenceInitializer]|[INFO]|[17]: Storage Adapter class ::: null|
[17:34:59:024]|[10-20-2015]|[com.adventnet.db.api.RelationalAPI]|[SEVERE]|[17]: haltjvm.on.dbcrash is set to [true]|
[17:34:59:024]|[10-20-2015]|[com.adventnet.db.adapter.Jdbc20DBAdapter]|[INFO]|[17]: shutDownStrings :: []|
[17:34:59:024]|[10-20-2015]|[com.adventnet.persistence.PersistenceInitializer]|[SEVERE]|[17]: DBCrashHandler [com.adventnet.passtrix.PMPDBCrashHandler] has been set to RelationalAPI.|



I also get this appearing constantly (appears to be a heartbeat check or something):

[17:38:50:869]|[10-20-2015]|[com.adventnet.db.adapter.postgres.DefaultPostgresDBInitializer]|[INFO]|[17]: pg_isready returning status :: 2|
[17:38:50:869]|[10-20-2015]|[com.adventnet.db.adapter.postgres.DefaultPostgresDBInitializer]|[INFO]|[17]: isServerRunning :: false|
[17:38:51:874]|[10-20-2015]|[com.adventnet.db.adapter.postgres.DefaultPostgresDBInitializer]|[INFO]|[17]: Command to be executed [D:\ManageEngine\PMP\bin\..\pgsql\bin\pg_isready.exe, --host, localhost, --port, 2345, --username, postgres] |
[17:38:52:748]|[10-20-2015]|[SYSOUT]|[INFO]|[274]: localhost:2345 - no response|



I'm not really sure where to go from here? One thing that I have noticed is that the GUI start / stop service options do not behave well with User Account Control.  I cannot start the service from the tray icon, as I get an "Access Denied error", and this also happens if I try to run from the start menu.  If I run it from the start menu as administrator then the service will start, but the tray icon will incorrectly state that it has not started.  Trying to start the service again I get a message stating "Unable to start to PMP server.  REASON: An instance of the service is already running". 


The critical issue is that postgresSQL isn't starting and I'm not sure why?  What can I try, or have I done something wrong?  Many thanks!

              New to ADManager Plus?

                New to ADSelfService Plus?