{"id":2125,"date":"2017-10-16T18:42:25","date_gmt":"2017-10-16T23:42:25","guid":{"rendered":"http:\/\/osric.com\/chris\/accidental-developer\/?p=2125"},"modified":"2017-10-16T18:44:35","modified_gmt":"2017-10-16T23:44:35","slug":"freeipa-failed-to-start-pki-tomcatd-service","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2017\/10\/freeipa-failed-to-start-pki-tomcatd-service\/","title":{"rendered":"FreeIPA: Failed to start pki-tomcatd Service"},"content":{"rendered":"<p>After a recent CentOS update, FreeIPA 4.5 failed to start with the following error message:<br \/>\n<code>Failed to start pki-tomcatd Service<\/code><\/p>\n<p>What changed? The following were the 3 packages updated:<\/p>\n<ul>\n<li>httpd.x86_64<\/li>\n<li>httpd-tools.x86_64<\/li>\n<li>mod_session.x86_64<\/li>\n<\/ul>\n<p>I successfully restarted FreeIPA without the pki-tomcatd service:<br \/>\n<code>$ sudo ipactl start --ignore-service-failure<\/code><\/p>\n<p>But it&#8217;s not ideal to run it without the PKI service. What is going on? According to the log at <code>\/var\/log\/pki\/pki-tomcat\/ca\/debug<\/code>:<\/p>\n<p><code>java.lang.Exception: Certificate auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101) Certificate type not approved for application.<\/code><\/p>\n<p>Which cert is that? Where is it? How did it get created? Didn&#8217;t FreeIPA create it? Why isn&#8217;t it valid? Why doesn&#8217;t it give me any additional info?<\/p>\n<p>Eventually I found the certificate location (although I don&#8217;t recall how, likely a post on the FreeIPA mailing list):<br \/>\n<code>\/var\/lib\/pki\/pki-tomcat\/alias -&gt; \/etc\/pki\/pki-tomcat\/alias<\/code><\/p>\n<p>I ran certutil to find out more about the certificate:<br \/>\n<code>$ certutil -L -d \/etc\/pki\/pki-tomcat\/alias<br \/>\ncertutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate\/key database is in an old, unsupported format.<\/code><\/p>\n<p>That uninformative and misleading error message looked familiar to me. Indeed, I wrote a post about it 7 months ago:<br \/>\n<a href=\"https:\/\/osric.com\/chris\/accidental-developer\/2017\/03\/certutil-function-failed-sec_error_legacy_database-the-certificatekey-database-is-in-an-old-unsupported-format\/\">certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate\/key database is in an old, unsupported format<\/a> <\/p>\n<p><code>$ sudo certutil -L -d \/etc\/pki\/pki-tomcat\/alias -n 'auditSigningCert cert-pki-ca'<\/code><\/p>\n<p>The expiration date looked fine, which was the first thing I suspected.<\/p>\n<p>I did note the following, which looked interesting:<br \/>\n<code>Mozilla-CA-Policy: false (attribute missing)<\/code><\/p>\n<p>But after reading about that at <a href=\"http:\/\/mozilla.github.io\/ca-policy\/\">http:\/\/mozilla.github.io\/ca-policy\/<\/a> it looked like it shouldn&#8217;t be needed.<\/p>\n<p>Fortunately, I have another working FreeIPA replica that I had not yet upgraded, so I compared the certificates on both systems:<\/p>\n<p>On the IPA replica with errors:<\/p>\n<pre><code>$ sudo certutil -L -d \/etc\/pki\/pki-tomcat\/alias\r\n\r\nCertificate Nickname                                         Trust Attributes\r\n                                                             SSL,S\/MIME,JAR\/XPI\r\n\r\ncaSigningCert cert-pki-ca                                    CTu,Cu,Cu\r\nauditSigningCert cert-pki-ca                                 u,u,u\r\nocspSigningCert cert-pki-ca                                  u,u,u\r\nServer-Cert cert-pki-ca                                      u,u,u\r\nsubsystemCert cert-pki-ca                                    u,u,u<\/code><\/pre>\n<p>On the working IPA replica:<\/p>\n<pre><code>$ sudo certutil -L -d \/etc\/pki\/pki-tomcat\/alias\r\n\r\nCertificate Nickname                                         Trust Attributes\r\n                                                             SSL,S\/MIME,JAR\/XPI\r\n\r\ncaSigningCert cert-pki-ca                                    CTu,Cu,Cu\r\nServer-Cert cert-pki-ca                                      u,u,u\r\nauditSigningCert cert-pki-ca                                 u,u,Pu\r\nocspSigningCert cert-pki-ca                                  u,u,u\r\nsubsystemCert cert-pki-ca                                    u,u,u<\/code><\/pre>\n<p>Note the P trust attribute in the latter. What does it mean? From <code>man certutil<\/code>:<\/p>\n<pre><code>-t trustargs\r\n           Specify the trust attributes to modify in an existing certificate\r\n           or to apply to a certificate when creating it or adding it to a\r\n           database. There are three available trust categories for each\r\n           certificate, expressed in the order SSL, email, object signing for\r\n           each trust setting. In each category position, use none, any, or\r\n           all of the attribute codes:\r\n\r\n           \u00b7   p - Valid peer\r\n\r\n           \u00b7   P - Trusted peer (implies p)\r\n\r\n           \u00b7   c - Valid CA\r\n\r\n           \u00b7   C - Trusted CA (implies c)\r\n\r\n           \u00b7   T - trusted CA for client authentication (ssl server only)<\/code><\/pre>\n<p>I modified the trust attributes of the certificate accordingly:<\/p>\n<p><code>$ sudo certutil -M -t ',,P' -d \/etc\/pki\/pki-tomcat\/alias -n 'auditSigningCert cert-pki-ca'<\/code><\/p>\n<p>I tried restarting FreeIPA again:<\/p>\n<pre><code>$ sudo ipactl restart\r\nStopping pki-tomcatd Service\r\nRestarting Directory Service\r\nRestarting krb5kdc Service\r\nRestarting kadmin Service\r\nRestarting httpd Service\r\nRestarting ipa-custodia Service\r\nRestarting ntpd Service\r\nRestarting pki-tomcatd Service\r\nRestarting ipa-otpd Service\r\nipa: INFO: The ipactl command was successful<\/code><\/pre>\n<p>It worked!<\/p>\n<p>But why? What does the trust attribute for JAR\/XPI mean? I don&#8217;t really know &#8212; I suppose it means that that the Java code we&#8217;re running should trust the certificate. Since I didn&#8217;t have this problem when I upgraded the working replica, I&#8217;m guessing that I must have done something to change it (and break it) along the way. It likely had nothing to do with the CentOS updates I applied, but I just happened to run into the problem after restarting FreeIPA post-updates.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>ipactl restart failed with the error &#8220;Failed to start pki-tomcatd Service&#8221; &#8212; I investigated and discovered one of my certificates did not have the appropriate trust settings.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[422],"tags":[417],"class_list":["post-2125","post","type-post","status-publish","format-standard","hentry","category-sysadmin","tag-freeipa"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2125","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=2125"}],"version-history":[{"count":12,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2125\/revisions"}],"predecessor-version":[{"id":2142,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2125\/revisions\/2142"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=2125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=2125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=2125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}