azure-data-explorer - commands failures caused by Admin node change

Question

I have noticed that many times we face transient issue that anything that is running on an ADX cluster abruptly fails due to the following error:-

ADX async command has completed with a 'Abandoned' state. Status: 'Admin node has changed'

This is mostly noticed in case of commands that run for long time , obviously because the probability that admin node will change during the lifespan of the command execution increases. Is this standard behavior of an ADX cluster and that we have to take into account the fact that this may happen now and then. Is there any guidance on frequency of this happening or any hint about which circumstances cause this admin node to be changed. As for when the admin node actually changes , is there anything we can do to avoid command failures caused by it ?

标签： azure-data-explorer

Answer 1

An admin node may change occasionally (e.g. once in a ~week), but it is not expected to occur frequently.

If it is happening frequently on your cluster, it may indicate something bad in the usage pattern that is overloading the admin node, poor choice of SKU (not enough CPU/RAM to handle the workload), or an issue with the service or underlying platform.

• If you're unsure of which it is - you can consider opening a support ticket.

As a side note, commands that are long-running (e.g. ~5-10 mins, or longer) are discouraged. You should follow the notes in the documentation recommending splitting a single command into multiple (shorter/lighter) ones, each handling a subset of the work to be done.