Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Max File Handles reached #24

Open
jbsouthe opened this issue Mar 16, 2022 · 2 comments
Open

Max File Handles reached #24

jbsouthe opened this issue Mar 16, 2022 · 2 comments

Comments

@jbsouthe
Copy link
Member

I have a customer using this plugin, and they are seeing file handles exceed their ulimit(10500) after this has been running for some time. I'm asking them to capture an lsof on the machine agent when this happens again, but i took a look to make sure that the plugin is disconnecting between runs and it is, so i thought, maybe the underlying library can't handle that and isn't releasing something on each run, so i googled a bit and found this: https://stackoverflow.com/questions/69358137/ibm-mq-how-to-handle-open-close-connections-on-long-running-services-workers

I'm going to add to this, when i get that lsof output as a comment, and i will try changing the connection to be long running as a test as well, in a local code base.

Has this issue been reported or has anyone seen something like this already?

I recommended just changing the max file handles to unlimited but they can't because of NFS issues they are also dealing within their environment.
@jbsouthe
Copy link
Member Author

Here is the error:
[Monitor-Task-Thread38] 17 Mar 2022 17:39:52,459 ERROR WMQMonitorTask-WMQMonitor - MQJE001: Completion Code '2', Reason '2538'.
com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2538'.
at com.ibm.mq.MQManagedConnectionJ11.(MQManagedConnectionJ11.java:255) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQClientManagedConnectionFactoryJ11._createManagedConnection(MQClientManagedConnectionFactoryJ11.java:450) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQClientManagedConnectionFactoryJ11.createManagedConnection(MQClientManagedConnectionFactoryJ11.java:487) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.StoredManagedConnection.(StoredManagedConnection.java:97) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java:194) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueueManagerFactory.java:874) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQQueueManagerFactory.procure(MQQueueManagerFactory.java:822) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQQueueManagerFactory.constructQueueManager(MQQueueManagerFactory.java:764) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQQueueManagerFactory.createQueueManager(MQQueueManagerFactory.java:200) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQQueueManager.(MQQueueManager.java:979) ~[?:9.2.0.0 - p920-L200709]
at com.appdynamics.extensions.webspheremq.WMQMonitorTask.initMQQueueManager(WMQMonitorTask.java:91) [?:?]
at com.appdynamics.extensions.webspheremq.WMQMonitorTask.run(WMQMonitorTask.java:64) [?:?]
at com.appdynamics.extensions.TasksExecutionServiceProvider$1.run(TasksExecutionServiceProvider.java:48) [?:?]
at com.appdynamics.extensions.executorservice.MonitorThreadPoolExecutor$TaskRunnable.run(MonitorThreadPoolExecutor.java:113) [?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '10.97.2.71(6601)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host '/10.97.2.71:6601' rejected. [1=java.net.SocketExcep
tion[Too many open files],3=/10.97.2.71:6601,4=TCP,5=Socket.bind]],3=10.97.2.71(6601),5=RemoteTCPConnection.bindAndConnectSocket]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13588) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100(RemoteFAP.java:13125) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1430) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQSESSION.MQCONNX_j(MQSESSION.java:916) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQManagedConnectionJ11.(MQManagedConnectionJ11.java:240) ~[?:9.2.0.0 - p920-L200709]
... 18 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '/10.97.2.71:6601' rejected. [1=java.net.SocketException[Too many open files],3=/10.97.2.71:6601,4=TCP,5=Socket.bind]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:905) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1385) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:978) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:567) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:246) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:154) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13328) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100(RemoteFAP.java:13125) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1430) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1389) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQSESSION.MQCONNX_j(MQSESSION.java:916) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQManagedConnectionJ11.(MQManagedConnectionJ11.java:240) ~[?:9.2.0.0 - p920-L200709]
... 18 more
Caused by: java.net.SocketException: Too many open files
at java.net.Socket.createImpl(Socket.java:478) ~[?:1.8.0_322]
at java.net.Socket.getImpl(Socket.java:538) ~[?:1.8.0_322]
at java.net.Socket.bind(Socket.java:662) ~[?:1.8.0_322]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:805) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1385) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:978) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:567) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:246) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:154) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13328) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100(RemoteFAP.java:13125) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1430) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1389) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQSESSION.MQCONNX_j(MQSESSION.java:916) ~[?:9.2.0.0 - p920-L200709]
at com.ibm.mq.MQManagedConnectionJ11.(MQManagedConnectionJ11.java:240) ~[?:9.2.0.0 - p920-L200709]
... 18 more

@jbsouthe
Copy link
Member Author

jbsouthe commented Apr 5, 2022

Customer has been testing this for about 3 weeks and so far they have not had any negative issues and have not run out of file handles. Attached is the update, I just made private attributes for agent and ibmQueueManager then set them if they are null, rather than creating new on each iteration. Please let me know if I can help in any way, and hopefully this can be accepted into the repo, attached. Feel free to ping me on Slack or Teams if you want to discuss this.
Thanks, John

WMQMonitorTask.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jbsouthe