I'm not a MySQL guru and I can't diagnose the problem with this information, but I tried searching for the formula in the source code. Here it is:
server_buffers + total_per_thread_buffers * max_connections
Where:
server_buffers = key_buffer_size + innodb_buffer_pool_size + innodb_additional_mem_pool_size + innodb_log_buffer_size + query_cache_size
and:
total_per_thread_buffers = read_buffer_size + read_rnd_buffer_size + sort_buffer_size + thread_stack + max_allowed_packet + join_buffer_size
Now you have to check each of these values and figure out which one is responsible for this huge number. And don't trust this script unreservedly - I tried running it on one of my DB servers and it calculated that the maximum memory is 140% of the total amount of physical memory, but the system has been running for years without any stability issues.
Good luck!