Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AzureMonitorLinuxAgent eventually cannot obtain gig token and MCS error [SSL Handshake error] #1950

Open
TheKrisSodroski opened this issue Aug 8, 2024 · 5 comments
Open

AzureMonitorLinuxAgent eventually cannot obtain gig token and MCS error [SSL Handshake error] #1950

TheKrisSodroski opened this issue Aug 8, 2024 · 5 comments

Comments

@TheKrisSodroski
Copy link

TheKrisSodroski commented Aug 8, 2024 •
edited
Loading

I have a ubuntu 22_04 that I deploy that uses AzureMonitorLinuxAgent to send it's syslogs to log analytics.

On first deployment, syslogs begin to be sent without issue.

2024-08-06T15:54:31.6687790Z: [PERSISTENCE] Local Persistency is enabled
2024-08-06T15:54:31.6829040Z: [DAEMON] START mdsd daemon ver(1.31.1) pid(1990) uid(104) gid (111)

2024-08-06T15:54:31.7098930Z: [PERSISTENCE] Local Persistency is enabled
2024-08-06T15:54:31.7100470Z: (wstr)"Msgpack array size is not set. Will use default 10485760 items.
Msgpack map size is not set. Will use default 10485760 items.
Msgpack string size is not set. Will use default 10485760 bytes.
Msgpack bin size is not set. Will use default 10485760 bytes.
Msgpack ext size is not set. Will use default 10485760 bytes.
Msgpack nesting is not set. Will use default 10 levels.
"
2024-08-06T15:54:31.7103480Z: Event ingestion rate limiting (EPS) is set to 20000 events per second.
2024-08-06T15:54:31.7104900Z: Trying to lock '/run/azuremonitoragent/default.lock', fd=11
2024-08-06T15:54:31.7130750Z: Lock '/run/azuremonitoragent/default.lock'  was taken successfully.
2024-08-06T15:54:32.1683230Z: Detected cloud region "eastus" via IMDS
2024-08-06T15:54:32.1683760Z: Detected cloud environment "azurepubliccloud" via IMDS; the domain ".com" will be used
2024-08-06T15:54:32.1801120Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/FetchIMDSMetadata.cpp:140,FetchMetadataFromIMDS]Setting resource id from IMDS: /subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM
2024-08-06T15:54:32.1802360Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:428,Initialize]McsManager successfully initialized
2024-08-06T15:54:32.5882700Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshMSIToken.cpp:94,RefreshMsiTokenThreadProc]Next refresh of MSI token for MCS in 86400 seconds
2024-08-06T15:54:37.2385090Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:1032,CallMcsWithRedirect]MCS redirected to endpoint https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com for path locations/eastus/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/agentConfigurations
2024-08-06T15:54:37.3073130Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:580,ReconcileConfigurationsTable]Configuration [dcr-8951f5701ca74c6e8a0c56da6d9ab774] added
2024-08-06T15:54:37.3602910Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:238,RefreshGigToken]Retrieved gig token for configuration id [dcr-8951f5701ca74c6e8a0c56da6d9ab774] channel id [ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe]: [eyJhbGciOi...]
2024-08-06T15:54:42.1816450Z: Parsing Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
2024-08-06T15:54:42.1817000Z: Loaded Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
2024-08-06T15:54:42.1817410Z: Loading Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:42.1817630Z: Parsing content for configuration with id dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:42.1818410Z: Using disk quota specified in AgentSettings: 10240
2024-08-06T15:54:42.1818820Z: Loaded Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:47.2802900Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:238,RefreshGigToken]Retrieved gig token for configuration id [dcr-8951f5701ca74c6e8a0c56da6d9ab774] channel id [ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe]: [eyJhbGciOi...]
2024-08-06T15:54:51.7835520Z: Loaded data sources dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:51.7981310Z: TcMalloc reading tcmalloc.max_total_thread_cache_bytes=33554432 Bytes
2024-08-06T15:54:51.7981680Z: TcMalloc release frequency is set to 1.
2024-08-06T15:54:51.7981850Z: TcMalloc release rate is set to 10.
2024-08-06T15:54:51.7982060Z: [BackPressure] periodic timer was set to 1000 milliseconds.
2024-08-06T15:54:51.7982220Z: [BackPressure] memory threshold was set to 3584 MB.
2024-08-06T15:54:51.7982550Z: [BackPressure] throttling interval period is 1000ms, quota is 0.95 (95%), throttle time is 950ms
2024-08-06T15:54:51.7997950Z: Parsing Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.lkg.json
2024-08-06T15:54:51.7998270Z: Loaded Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.lkg.json
2024-08-06T15:54:51.7998800Z: Loading Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:51.7999010Z: Parsing content for configuration with id dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:51.7999390Z: Using disk quota specified in AgentSettings: 10240
2024-08-06T15:54:51.7999580Z: Loaded Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:51.8010310Z: Loaded data sources dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-06T15:54:54.3613120Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-06T15:56:10.1992250Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-06T15:57:10.1800430Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe

But once the token refresh cycle happens, I'm unable to obtain tokens, and thus, logs stop flowing to log analytics.

mdsd.err

2024-08-08T01:27:38.8104310Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:318,GetAgentConfigurations]Could not obtain configuration from https://global.handler.control.monitor.azure.com after first round of tries. Will try again with a fallback endpoint. ErrorCode:1310977
2024-08-08T01:27:48.6237060Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:174,RefreshGigToken]Failed to get gig token from https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/agentConfigurations/dcr-8951f5701ca74c6e8a0c56da6d9ab774/channels/ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe/issueIngestionToken?operatingLocation=eastus&platform=linux&includeMeConfig=true&api-version=2022-06-02 after first round of tries, will try again with a fallback endpoint. Error:  ErrorCode:-2146171645
2024-08-08T01:29:58.9342860Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:354,GetAgentConfigurations]Could not obtain configuration from https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com after attempting an immediate retry. ErrorCode:1310977
2024-08-08T01:29:58.9500790Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:384,GetAgentConfigurations]Response Code: 0; Response: . ErrorCode:-2146172665
2024-08-08T01:30:08.7089620Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:203,RefreshGigToken]Failed to get gig token from https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com after attempting immediate retry. Error:  ErrorCode:-2146171645

mdsd.info

2024-08-07T13:27:37.3373390Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:963,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com][locations/eastus/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/agentConfigurations][platform=linux&includeMeConfig=true&api-version=2022-06-02], request id [e2038aad-7349-4c40-b8f7-95f8074eecec], responseCode [500], error from Mcs [{"error":{"code":"ProcessingError","message":"An internal server error occured."}}]
2024-08-07T13:27:37.3374100Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 10 seconds
2024-08-07T13:27:43.6279320Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:27:58.7400970Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:580,ReconcileConfigurationsTable]Configuration [dcr-8951f5701ca74c6e8a0c56da6d9ab774] updated
2024-08-07T13:27:59.3510570Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:963,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/agentConfigurations/dcr-8951f5701ca74c6e8a0c56da6d9ab774/channels/ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe/issueIngestionToken?operatingLocation=eastus&platform=linux&includeMeConfig=true&api-version=2022-06-02][][], request id [bee4562f-3205-46af-b449-3350bcf81651], responseCode [500], error from Mcs [{"error":{"code":"ProcessingError","message":"An internal server error occured."}}]
2024-08-07T13:27:59.3511000Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 10 seconds
2024-08-07T13:28:20.6487340Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:963,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/agentConfigurations/dcr-8951f5701ca74c6e8a0c56da6d9ab774/channels/ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe/issueIngestionToken?operatingLocation=eastus&platform=linux&includeMeConfig=true&api-version=2022-06-02][][], request id [664856c5-582a-4182-94c0-f6c64a7d5a16], responseCode [500], error from Mcs [{"error":{"code":"ProcessingError","message":"An internal server error occured."}}]
2024-08-07T13:28:20.6487880Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 20 seconds
2024-08-07T13:29:02.3595000Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:238,RefreshGigToken]Retrieved gig token for configuration id [dcr-8951f5701ca74c6e8a0c56da6d9ab774] channel id [ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe]: [eyJhbGciOi...]
2024-08-07T13:29:02.3664950Z: Parsing Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
2024-08-07T13:29:02.3665450Z: Loaded Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
2024-08-07T13:29:02.3666560Z: Loading Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:29:02.3670730Z: Parsing content for configuration with id dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:29:02.3673110Z: Using disk quota specified in AgentSettings: 10240
2024-08-07T13:29:02.3673600Z: Loaded Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:29:02.3700580Z: Loaded data sources dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:29:58.4274710Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:30:58.3490730Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:31:58.3533610Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:32:58.2896600Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:33:58.4695480Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:34:58.6313710Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:35:58.3263870Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:36:58.5140640Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:37:58.3722880Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:38:58.4075780Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:39:58.3411320Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:40:58.3547950Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:41:58.3671750Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:42:58.3143470Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:43:58.3616830Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:44:58.3634480Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:45:58.3172860Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:46:58.3763090Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:47:38.5982810Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:580,ReconcileConfigurationsTable]Configuration [dcr-8951f5701ca74c6e8a0c56da6d9ab774] updated
2024-08-07T13:47:38.6473480Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:238,RefreshGigToken]Retrieved gig token for configuration id [dcr-8951f5701ca74c6e8a0c56da6d9ab774] channel id [ods-367ce7c4-bfaf-44a0-919f-56828e39a3fe]: [eyJhbGciOi...]
2024-08-07T13:47:38.6502210Z: Parsing Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
2024-08-07T13:47:38.6502780Z: Loaded Mcs document /etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
2024-08-07T13:47:38.6503080Z: Loading Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:47:38.6503330Z: Parsing content for configuration with id dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:47:38.6503740Z: Using disk quota specified in AgentSettings: 10240
2024-08-07T13:47:38.6503940Z: Loaded Azure Monitor configuration dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:47:38.6519630Z: Loaded data sources dcr-8951f5701ca74c6e8a0c56da6d9ab774
2024-08-07T13:47:57.6828900Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:48:57.7077660Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:49:57.6738440Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:50:57.7691640Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:51:57.6974410Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:52:18.3703970Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:980,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com][locations/eastus/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/reportHealth][platform=linux&api-version=2022-06-02], request id [019a72c3-1235-4e9c-afee-6eae27cde1ea], error from Mcs [Error in SSL handshake]
2024-08-07T13:52:18.3704430Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 10 seconds
2024-08-07T13:52:38.3867740Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:980,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com][locations/eastus/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/reportHealth][platform=linux&api-version=2022-06-02], request id [85b50e35-c650-410b-8a36-7a6221174d0a], error from Mcs [Error in SSL handshake]
2024-08-07T13:52:38.3879470Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 20 seconds
2024-08-07T13:52:57.7139150Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe
2024-08-07T13:53:18.4055420Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:980,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com][locations/eastus/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/reportHealth][platform=linux&api-version=2022-06-02], request id [09855c8e-92f1-4e97-996f-92026ba79bef], error from Mcs [Error in SSL handshake]
2024-08-07T13:53:18.4055850Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 40 seconds
2024-08-07T13:53:42.7162900Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:980,CallMcs]Failed from MCS path [https://MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com][locations/eastus/subscriptions/MySubscriptionID/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/virtualMachines/MyVM/agentConfigurations][platform=linux&includeMeConfig=true&api-version=2022-06-02], request id [5c1645ff-3deb-44ef-8023-f12aa53519f3], error from Mcs [Error in SSL handshake]
2024-08-07T13:53:42.7163400Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/McsManager.cpp:993,CallMcs]Will retry calling MCS in 10 seconds
2024-08-07T13:53:57.7553990Z: Heartbeating for ODSUploader https://367ce7c4-bfaf-44a0-919f-56828e39a3fe

As you can see, over time, the agent is unable to obtain a token at all.

Restarting the VM does not help, nor does restarting the azure monitor service. It is simply unable to obtain the token.

I've also given the VM full access to the internet and can verify that I can reach the private link and get the SSL cert for the private link MyVM-monitor-pe-kdzp.eastus-1.handler.control.monitor.azure.com but I cannot access global.handler.control.monitor.azure.com (no hostname exists) despite there being a DNS record

Deleting the VM and redeploying allows it to work again, but eventually, this issue happens again.

Any help is much appreciated.

/var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-1.31.1/ama_tst# ls
AMA-Troubleshooting-Tool.md  ama_troubleshooter.sh  ama_tst.tgz  modules
root@Rsyslog-Dev:/var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-1.31.1/ama_tst# sh ama_troubleshooter.sh -A
Python version being used is:
Python 3.10.12

Starting AMA Troubleshooting Tool v.1.3...

CHECKING INSTALLATION...
Checking if running a supported OS version...
Checking if enough disk space is available...
Checking if machine has a supported package manager...
Checking if packages and subcomponents are installed correctly...
Checking if running a supported version of AMA...
Checking if rsyslog or syslog-ng exists...
Checking if syslog user exists...
============================================
================================================================================
CHECKING CONNECTION...
Checking AMA parameters in /etc/default/azuremonitoragent...
Checking DCR...
Checking if machine is connected to the internet...
Checking if machine can connect to Azure Monitor control-plane and data ingestion endpoints...
ERROR(S) FOUND.
================================================================================
================================================================================
ALL ERRORS/WARNINGS ENCOUNTERED:
  ERROR FOUND: Machine couldn't connect to global.handler.control.monitor.azure.com: curl/openssl command failed. 

Error Details:
 $ echo | openssl s_client -connect global.handler.control.monitor.azure.com:443 -brief -CApath /etc/ssl/certs 

Command 'echo | openssl s_client -connect global.handler.control.monitor.azure.com:443 -brief -CApath /etc/ssl/certs' returned non-zero exit status 1.
--------------------------------------------------------------------------------
Please review the errors found above.
================================================================================
If you still have an issue, please run the troubleshooter again and collect the logs for AMA.
In addition, please include the following information:
  - Azure Subscription ID where the Log Analytics Workspace is located
  - Workspace ID the agent has been onboarded to
  - Workspace Name
  - Region Workspace is located
  - Pricing Tier assigned to the Workspace
  - Linux Distribution on the VM
  - Azure Monitor Agent Version
================================================================================
Restarting AMA can solve some of the problems. If you need to restart Azure Monitor Agent on this machine, please execute the following commands as the root user:
  $ cd /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-<agent version number>/
  $ ./shim.sh -disable
  $ ./shim.sh -enecho | openssl s_client -connect global.handler.control.monitor.azure.com:443 -brief -CApath /etc/ssl/certs
400783CD52720000:error:10080002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/bio_addr.c:738:No address associated with hostnameal.handler.control.monitor.azure.com:443 -brief -CApath /etc/ssl/certs
connect:errno=2
root@Rsyslog-Dev:/var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-1.31.1/ama_tst#
@TheKrisSodroski TheKrisSodroski changed the title AzureMonitorLinuxAgent only able to obtain gig/MSI token once AzureMonitorLinuxAgent eventually cannot obtain gig token and MCS error [SSL Handshake error] Aug 8, 2024
@rmdoliveira
Copy link

Same problem Here.

@TheKrisSodroski
Copy link
Author

TheKrisSodroski commented Aug 9, 2024 •
edited
Loading

@rmdoliveira

I've found, through much trial and error, that the VMs cannot share a Azure Monitor private link scope. As soon as you attempt to share the private link scope, the dns records get messed up and the VM can no longer access the token endpoints.

If you run the troubleshooter like I did at the bottom of my post and post it here, maybe I can help you with your issue.

@rapsomanikis-nviso
Copy link

Hey @TheKrisSodroski

Could you share any insights on how you resolved the issue?

@TheKrisSodroski
Copy link
Author

Hey @TheKrisSodroski

Could you share any insights on how you resolved the issue?

Each Vnet that needs access to Azure Monitor should have it's own Nic/private endpoint.

@rabi-sahu
Copy link

The Solution which i Found is by Updating Host file of My Syslog Collector - /etc/host .
and updated my DCR DNS record which i used on my Privet Endpoint .

This solves my issue on reporting to LAW

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@TheKrisSodroski @rabi-sahu @rapsomanikis-nviso @rmdoliveira