Builds not going into getting changes - start > Pending b

[Peter Kristiansen]

We are using Parabuild 4.1.3 and have started seeing a lot of problems with builds which doesn't start when changes are made to the VCS. Specifically we see that the builds are in idle state but never go into Getting Changes. This can be seen when deactivating and re-activating a build where they instantaneous goes back to idle.

It is not all builds and we haven't found any pattern as to which builds it happens to.

We've managed to track down the following in the debug log file for the builds which stops working.
[AutomaticScheduler:47624] 14:45:06,617 DEBUG: autobuild.build.cc (191) - Error at run cycle
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(Unknown Source)
at viewtier.autobuild.d.l.d(l.java:104)
at viewtier.autobuild.d.l.c(l.java:100)
at viewtier.autobuild.h.a.f.a(f.java:91)
at viewtier.autobuild.h.a.f.b(f.java:38)
at viewtier.autobuild.build.cc.i(cc.java:6)
at viewtier.autobuild.build.cc.run(cc.java:4)

The only way we have found to get these builds out of Pending Build is to restart the Parabuild Service. Note, that builds which are in Idle state (but never progressing to Getting Changes) seems to be unaffected by the restart of the service - they still do not progress to getting changes. This last bit I've only encountered once so I cannot be 100% sure it is reproducible.

We see several of these errors (Error at run cycle as above) after restarting Parabuild. It seems to be when all the builds are going through their first getting changes cycle. It also happens that builds "deteriorate" i.e. we see the Error at run cycle for builds which ran perfectly half an hour before.

Given that the error includes a java.lang.OutOfMemoryError we will be trying to increase -xms and -xmx.

Any other advice?

br
Peter

[hallm4]

We've had similar issues.

Usually the problem for us is that there a machine or number of machines in the build pool that are offline in some way. Parabuild _slowly_ polls all the machines before picking one to use for the build.

Let's say you have 10 build machines, and 2 of them are offline / down, but not 'disabled' in Parabuild. Parabuild will still try to talk to them, until some timeout limit is reached. So instead of getting a quick build start, you might wait ~5mins or so or more before the build starts until Parabuild gets the status of all the machines in the pool.

I would double check to make sure that Parabuild can
a) talk to all machines
b) talk to the source control mechanism

If you're getting out of memory issues, you can increase the memory settings to parabuild in the startup script

Our JAVA_OPTS are

JAVA_OPTS="-Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=192m -XX:+UseParallelGC -Djava.awt.headless=true $GC_LOGGING"

[Peter Kristiansen]

Hi,

We actually solved the problem by giving less memory... We changed Xmx from 600 to 450 and the Out of memory problem went away. Now, I have absolutely no idea how that could be, but decided not to argue with results :o)

br
Peter