Parabuild 4.0.61 GUI unresponsive and process "run wild

[Peter Kristiansen]

Hi,

We have experienced a new issue after upgrading to Parabuild 4.0.61. The symptoms are that the GUI is unresponsive and the parabuild process takes more than 90% CPU time. I can see that our integration to ClearCase is started and run (not at the regular intervals though) and even if they return changes a build is not started.

There are indications that if the process is left to itself, it might "slow down" to ~70% CPU load after some hours at which point it is possible to communicate with the GUI. Albeit it responds quite slowly. In all cases a restart of the Parabuild service solves the problem.

Nothing is printed in the log files regarding this issue. Is there some way to enable debugging, either when the problem has occured or even just have it running from startup so we can see what the root cause of this issue is?

br
Peter

[support]

[Peter Kristiansen]

[support]

[Peter Kristiansen]

It isn't that easy to disable virusscanning due to internal policies.

We had another incident today with the machine experiencing the same problems, this time running with debug information.

I can't really see much from them, it appears to be the same output. The only thing that stands out is this:
[AutomaticScheduler:27580] 15:20:31,769 DEBUG: autobuild.build.ic ( 91) - getting changes since: 30074
[AutomaticScheduler:27580] 15:20:31,769 DEBUG: autobuild.f.yb ( 9) - begin getChangesSince changeListID: 30074
[AutomaticScheduler:19238] 15:20:31,769 DEBUG: autobuild.e.l (105) - e: viewtier.autobuild.common.CommandStoppedException
[AutomaticScheduler:19238] 15:20:31,769 DEBUG: autobuild.common.w (242) - result of delete for "\\.\c:\pb\bin\..\etc\temp\.auto1371367815510860126.err"false
[AutomaticScheduler:27580] 15:20:31,801 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0
[HttpProcessor[80][23]] 15:20:32,051 DEBUG: autobuild.services.f(162) - using hard stop
[AutomaticScheduler:19238] 15:20:32,066 WARN: autobuild.common.w (257) - Could not delete file "\\.\c:\pb\bin\..\etc\temp\.auto1371367815510860126.err"
java.lang.Throwable: Stack trace at call
at viewtier.autobuild.common.w.c(w.java:257)
at viewtier.autobuild.e.l.j(l.java:7)
at viewtier.autobuild.f.vb.a(vb.java:38)
at viewtier.autobuild.f.yb.b(yb.java:42)
at viewtier.autobuild.build.ic.i(ic.java:157)
at viewtier.autobuild.build.ic.run(ic.java:58)
[AutomaticScheduler:19238] 15:20:32,066 DEBUG: autobuild.common.w (242) - result of delete for "\\.\c:\pb\bin\..\etc\temp\.auto5526044803818908027.out"false
[AutomaticScheduler:19238] 15:20:32,066 WARN: autobuild.common.w (257) - Could not delete file "\\.\c:\pb\bin\..\etc\temp\.auto5526044803818908027.out"
java.lang.Throwable: Stack trace at call
at viewtier.autobuild.common.w.c(w.java:257)
at viewtier.autobuild.e.l.j(l.java:38)
at viewtier.autobuild.f.vb.a(vb.java:38)
at viewtier.autobuild.f.yb.b(yb.java:42)
at viewtier.autobuild.build.ic.i(ic.java:157)
at viewtier.autobuild.build.ic.run(ic.java:58)
[AutomaticScheduler:19238] 15:20:32,066 DEBUG: error.a.c ( 7) - Incremented error counter: 5
[AutomaticScheduler:19238] 15:20:32,410 ERROR: error.a.c ( 1) - Error while accessing file system-based VCS: viewtier.autobuild.common.CommandStoppedException: viewtier.autobuild.common.BuildException: Error while accessing file system-based VCS: viewtier.autobuild.common.CommandStoppedException
at viewtier.autobuild.f.wb.a(wb.java:62)
at viewtier.autobuild.f.yb.b(yb.java:6)
at viewtier.autobuild.build.ic.i(ic.java:157)
at viewtier.autobuild.build.ic.run(ic.java:58)
Caused by: viewtier.autobuild.common.CommandStoppedException
at viewtier.autobuild.e.l.d(l.java:47)
at viewtier.autobuild.e.l.c(l.java:152)
at viewtier.autobuild.f.yb.b(yb.java:29)
... 2 more
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at viewtier.autobuild.e.l.d(l.java:57)
... 4 more

[AutomaticScheduler:25091] 15:20:33,019 DEBUG: autobuild.remote.g ( 5) - Hosts.size(): 1


The rest of the debug files appear to look similar to this:
[HttpProcessor[80][23]] 15:20:06,816 DEBUG: autobuild.services.f(192) - Stop build
[HttpProcessor[80][23]] 15:20:06,816 DEBUG: autobuild.build.ic (124) - scheduler.toString(): Thread[AutomaticScheduler:19233,5,main]
[HttpProcessor[80][23]] 15:20:06,926 DEBUG: autobuild.services.f(162) - using hard stop
[AutomaticScheduler:27760] 15:20:07,035 DEBUG: autobuild.f.yb ( 7) - result: []
[AutomaticScheduler:19237] 15:20:11,035 DEBUG: autobuild.f.yb ( 7) - result: []
[AutomaticScheduler:22676] 15:20:11,051 DEBUG: autobuild.remote.g ( 5) - Hosts.size(): 1
[AutomaticScheduler:22676] 15:20:11,051 DEBUG: autobuild.remote.g (257) - Selected next agent <Build Manager> for build OLA-11Q4
[AutomaticScheduler:22676] 15:20:11,051 DEBUG: autobuild.build.ic ( 91) - getting changes since: 30037
[AutomaticScheduler:22676] 15:20:11,051 DEBUG: autobuild.f.ac (242) - begin getChangesSince changeListID: 30037
[AutomaticScheduler:22676] 15:20:11,066 DEBUG: autobuild.f.ac (376) - begin syncToLatest
[AutomaticScheduler:22676] 15:20:11,066 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 3600000.0
[AutomaticScheduler:22676] 15:20:11,566 DEBUG: autobuild.f.ac (134) - isCVSNT: true
[AutomaticScheduler:22676] 15:20:11,566 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0
[AutomaticScheduler:22676] 15:20:12,566 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0
[AutomaticScheduler:19324] 15:20:18,769 DEBUG: autobuild.remote.g ( 5) - Hosts.size(): 1
[AutomaticScheduler:19324] 15:20:18,769 DEBUG: autobuild.remote.g (257) - Selected next agent <Build Manager> for build STIK-11Q2-groovy
[AutomaticScheduler:19324] 15:20:18,769 DEBUG: autobuild.build.ic ( 91) - getting changes since: 29766
[AutomaticScheduler:19324] 15:20:18,769 DEBUG: autobuild.f.ac (242) - begin getChangesSince changeListID: 29766
[AutomaticScheduler:19324] 15:20:18,769 DEBUG: autobuild.f.ac (376) - begin syncToLatest
[AutomaticScheduler:19324] 15:20:18,785 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 3600000.0
[HttpProcessor[80][23]] 15:20:18,879 DEBUG: autobuild.services.f(192) - Stop build
[HttpProcessor[80][23]] 15:20:18,879 DEBUG: autobuild.build.ic (124) - scheduler.toString(): Thread[AutomaticScheduler:19234,5,main]
[HttpProcessor[80][23]] 15:20:18,988 DEBUG: autobuild.services.f(162) - using hard stop
[AutomaticScheduler:19324] 15:20:19,285 DEBUG: autobuild.f.ac (134) - isCVSNT: true
[AutomaticScheduler:19324] 15:20:19,316 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0
[AutomaticScheduler:26815] 15:20:19,441 DEBUG: autobuild.remote.g ( 5) - Hosts.size(): 1
[AutomaticScheduler:26815] 15:20:19,441 DEBUG: autobuild.remote.g (257) - Selected next agent <Build Manager> for build OA-Splittter-11Q3R
[AutomaticScheduler:26815] 15:20:19,457 DEBUG: autobuild.build.ic ( 91) - getting changes since: 27755
[AutomaticScheduler:26815] 15:20:19,457 DEBUG: autobuild.f.ac (242) - begin getChangesSince changeListID: 27755
[AutomaticScheduler:26815] 15:20:19,488 DEBUG: autobuild.f.ac (376) - begin syncToLatest
[AutomaticScheduler:26815] 15:20:19,551 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 3600000.0
[AutomaticScheduler:19324] 15:20:19,816 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0
[AutomaticScheduler:26815] 15:20:20,051 DEBUG: autobuild.f.ac (134) - isCVSNT: true
[AutomaticScheduler:26815] 15:20:20,051 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0
[AutomaticScheduler:26815] 15:20:20,551 DEBUG: autobuild.e.l ( 70) - Deadline timeout, ms: 9360000.0


Any hints as to what to do now?

br
Peter

[Peter Kristiansen]

We had another incident today - it is getting quite serious that this happens so often.

I have the debug log-files but couldn't really see anything useful in them, should I send them to you?

Peter

[support]

It looks like an AV issue. Those 'cannot delete file' messages are very suspicious. This may happenn when AV cannot keep up with I/O.

You can ask your IT to at leasttemporary deisable it.

Slava

[Peter Kristiansen]

I have made a request for a temporary dispensation for disabling virusscanner for the parabuild installation library.

Is there anything else we can do or look at?

Peter

[Peter Kristiansen]

Hi,

We finally got the virusscanner deactivated, but to no avail. We saw the same behaviour again.

We have been paying attention to it and it seems that the process ends up in this state when we deactivate multiple builds. We are not as of yet sure that it is completely reproducable, but the last times this happend it was immediately following several deactivations of builds.

We'll continue to try to see if we can find any relations between the "crash" and deactivation of builds.

I've noticed that after the process starts to run wild it will still start checking for changes but only for a while. My guess is that all builds that has already been scheduled for getting changes will do so, but no new ones are scheduled.

Should there be any changes detected during one of these runs, no build is started. We have had at least one such case.

I hope you have some further ideas as to what we can look into.

br
Peter

[Peter Kristiansen]

Hi,

We have made an observation that whenever we deactivate, or stop, a build the CPU load for the parabuild.exe increases dramatically. I noticed that the process was running with ~38% and after a deactivate it increased to ~50%. After another deactivate it reached ~55%.

It has seemed as if we deactivate too many builds in too quick succession we can get it to choke.

I've set up a test installation on another server to see if we can reproduce it, and it seems as if we can. After restarting the Parabuild proces takes ~0% CPU, but stopping or deactivating a build immediately increases it to 50% (it is a server not doing anything other than test for this purpose and 2 cores). If we later activate and deactivate the same build we don't see any increase in activity, but a second build has the same effect as to increase the load.

Do you know of any other with such, or similar, problems with 4.0.61?
Any ideas what we can try out to see if we can come up with more information?

Kind regards
Peter

[Peter Kristiansen]

Sorry that I keep spamming, but I just had a thought. Is it possible that you issue us a 4.1 upgrade license for our test server? It is the same as the testserver you issued the 4.0 upgrade test license to.

I'm currently preparing that we can try to do a few upgrades to see if that has any positive effect on the issue.

br
Peter

[support]

[support]

[support]

[cbk]