Troubleshooting¶
Log Files¶
The default location of standard node log files for various systems:
| OS | Location |
|---|---|
| Windows | C:\Users\<user>\AppData\Local\Nano\log -or- %LOCALAPPDATA%\Nano\log |
| macOS | /Users/<user>/Library/Nano/log |
| Linux | /home/<user>/Nano/log |
To get a static file name for the currently active log file, see the stable_log_filename configuration option
What to do if the node crashes (Linux)¶
Do not restart the node
If your node crashes, please follow this guide before attempting to run it again.
If the node crashes, the most commonly seen message is "Segmentation fault (core dumped)". When using docker, this message will only show up in the docker logs. In any case, this is often not enough to go on in terms of figuring out what went wrong. The next steps detail what you should do to provide us with as much information as possible about the problem.
When you are done gathering all information, please create a new Github issue, or reach us on Discord in the #support channel, detailing your issue as much as possible.
Getting the latest node log
The following command will order the log files such that the first one in the output is the most recent. If you restarted the node since the crash, then the relevant log file is not the latest one. Please be careful to give us the relevant log file.
# Nano -> NanoBeta if debugging a beta node
ls -dlt ~/Nano/log/* | head
Please provide the complete log file.
Please follow the steps below for the corresponding node version you are using. Should there be an error obtaining the information in a newer version, the older version steps should then be attempted.
v21+ nodes¶
Step 1: Make sure addr2line is installed
It is likely installed already, consult documentation for your linux distribution if it is not mentioned below:
Ubuntu/Debian
apt-get install binutils
Fedora 22+
dnf install binutils
(Optional) Step 2: Save crash dump files
The next step will clean up the dump files generated during the crash, if you wish to keep these then save nano_node_backtrace.dump, and all nano_node_crash_load_address_dump_*.txt files.
Step 3: Generate crash report
Run:
./nano_node --debug_generate_crash_report
This will generate a text file nano_node_crash_report.txt please send us the contents of this file.
v20 nodes¶
Step 1: Getting dmesg information
Depending on the error, it is possible you do not find any useful information in this step, in which case please move on to Step 3.
Run the following command and look for nano_node at the end. If you see a relevant message, gather all messages with a similar timestamp - the number within brackets on the left.
dmesg
Example output:
[ 6.336071] IPv6: ADDRCONF(NETDEV_CHANGE): wlp2s0: link becomes ready
[ 6.375123] wlp2s0: Limiting TX power to 23 (23 - 0) dBm as advertised by **:**:**:**:**:**
[ 6141.711993] show_signal_msg: 23 callbacks suppressed
[ 6141.711995] I/O[14487]: segfault at 1 ip 000055c69d3a1634 sp 00007f6e9332df10 error 6 in nano_node[55c69d25f000+70b000]
[ 6141.711999] Code: 24 70 48 83 c5 10 48 89 c3 48 39 ef 74 b6 e8 e3 b8 39 00 eb af 90 41 57 41 56 41 55 41 54 49 89 fc 55 53 48 81 ec a8 00 00 00 <c6> 04 25 01 00 00 00 31 64 48 8b 04 25 28 00 00 00 48 89 84 24 98
From this output, only the last 3 lines are relevant.
Step 2: Getting syslog information
More information might be available in syslog. Run the following command and look for the time the crash ocurred.
cat /var/log/syslog
Example output:
Aug 15 11:56:07 ubuntu-server kernel: [6141.711993] show_signal_msg: 23 callbacks suppressed
Aug 15 11:56:07 ubuntu-server kernel: [6141.711995] I/O[25975]: segfault at 1 ip 000055b2960e2d24 sp 00007fcff50f6fc0 error 6 in nano_node[55b295f9b000+6d8000]
Aug 15 11:56:07 ubuntu-server kernel: [6141.711999] Code: 24 70 48 83 c5 10 48 89 c3 48 39 ef 74 b6 e8 e3 b8 39 00 eb af 90 41 57 41 56 41 55 41 54 49 89 fc 55 53 48 81 ec a8 00 00 00 <c6> 04 25 01 00 00 00 31 64 48 8b 04 25 28 00 00 00 48 89 84 24 98
Include the relevant lines from the output. In this example, the log is similar to the one from Step 2.
Step 3: Getting a backtrace dump
This command will produce some basic information about the error.
Not using docker:
./nano_node --debug_output_last_backtrace_dump > nano_node_backtrace_output.txt
Using docker:
mkdir -p /tmp/nano_node_crash && cd $_
docker exec ${NANO_NAME} nano_node --debug_output_last_backtrace_dump > nano_node_backtrace_output.txt
docker exec ${NANO_NAME} sh -c 'mkdir -p crash_files; mv nano_node_crash*.txt crash_files/'
docker cp ${NANO_NAME}:/crash_files/ . && mv crash_files/* .
Step 4: Producing the archive file
See the output of this command for the name of the file you should include in your report.
FILE="nano_node_crash_$(date +"%Y-%m-%d_%H-%M-%S.tar.gz")" && tar czf $FILE --exclude=*.tar.gz nano_node_* && echo "Created archive $FILE"
Statistics from RPC¶
The "stats" RPC command can be used by external processes to query statistics, such as traffic counters. This is useful for diagnostics, monitoring and display in admin consoles.
Statistics are optionally logged to separate text files.
For implementations details, please see Statistics API
Duplicate observer stats
Under certain conditions, confirmations seen through the observer type stat can be duplicates. In order to get accurate data, block hashes must be tracked and validated against previously seen hashes.
Configuration¶
All configuration nodes and values are optional, with the default values shown in comments below:
"node": {
...
"statistics": {
// Sampling configuration (optional)
// Only activate if you need sampling information, as there's some overhead associated with this feature.
"sampling": {
"enabled": "true", // If sampling is enabled. Default false.
"capacity": "5", // How many samples to keep. Must be set if sampling is enabled.
"interval": "1000" // Sample interval in milliseconds. Must be set if sampling is enabled.
},
// File logging (optional)
"log": {
"interval_counters": "5000", // How often to write counters to file in milliseconds. Default 0 (off)
"interval_samples": "5000", // How often to write samples to file, milliseconds. Default 0 (off)
"rotation_count": "5", // Rotate file after writing statistics this many times. Default 100.
"headers": "true", // Write header containing log
"filename_counters": "counters.stat",
"filename_samples": "samples.stat"
}
}
}
Available type, detail and direction values¶
version 25.0+
type:
traffic_tcp
error
message
block
ledger
rollback
bootstrap
tcp_server
vote
election
http_callback
ipc
tcp
confirmation_height
confirmation_observer
drop
aggregator
requests
filter
telemetry
vote_generator
vote_cache
hinting
blockprocessor
bootstrap_server
active
active_started
active_confirmed
active_dropped
active_timeout
backlog
unchecked
election_scheduler
optimistic_scheduler
handshake
bootstrap_ascending
bootstrap_ascending_accounts
details:
all
// common
ok
loop
total
process
update
request
broadcast
// processing queue
queue
overfill
batch
// error specific
insufficient_work
http_callback
unreachable_host
invalid_network
// confirmation_observer specific
active_quorum
active_conf_height
inactive_conf_height
// ledger, block, bootstrap
send
receive
open
change
state_block
epoch_block
fork
old
gap_previous
gap_source
rollback_failed
progress
bad_signature
negative_spend
unreceivable
gap_epoch_open_pending
opened_burn_account
balance_mismatch
representative_mismatch
block_position
// message specific
not_a_type
invalid
keepalive
publish
confirm_req
confirm_ack
node_id_handshake
telemetry_req
telemetry_ack
asc_pull_req
asc_pull_ack
// bootstrap, callback
initiate
initiate_legacy_age
initiate_lazy
initiate_wallet_lazy
// bootstrap specific
bulk_pull
bulk_pull_account
bulk_pull_error_starting_request
bulk_pull_failed_account
bulk_pull_request_failure
bulk_push
frontier_req
error_socket_close
// vote specific
vote_valid
vote_replay
vote_indeterminate
vote_invalid
vote_overflow
// election specific
vote_new
vote_processed
vote_cached
election_block_conflict
generate_vote
generate_vote_normal
generate_vote_final
// election types
normal
hinted
optimistic
// received messages
invalid_header
invalid_message_type
invalid_keepalive_message
invalid_publish_message
invalid_confirm_req_message
invalid_confirm_ack_message
invalid_node_id_handshake_message
invalid_telemetry_req_message
invalid_telemetry_ack_message
invalid_bulk_pull_message
invalid_bulk_pull_account_message
invalid_frontier_req_message
invalid_asc_pull_req_message
invalid_asc_pull_ack_message
message_too_big
outdated_version
// tcp
tcp_accept_success
tcp_accept_failure
tcp_write_drop
tcp_write_no_socket_drop
tcp_excluded
tcp_max_per_ip
tcp_max_per_subnetwork
tcp_silent_connection_drop
tcp_io_timeout_drop
tcp_connect_error
tcp_read_error
tcp_write_error
// ipc
invocations
// peering
handshake
// confirmation height
blocks_confirmed
blocks_confirmed_unbounded
blocks_confirmed_bounded
// [request] aggregator
aggregator_accepted
aggregator_dropped
// requests
requests_cached_hashes
requests_generated_hashes
requests_cached_votes
requests_generated_votes
requests_cannot_vote
requests_unknown
// duplicate
duplicate_publish
// telemetry
invalid_signature
node_id_mismatch
genesis_mismatch
request_within_protection_cache_zone
no_response_received
unsolicited_telemetry_ack
failed_send_telemetry_req
empty_payload
cleanup_outdated
// vote generator
generator_broadcasts
generator_replies
generator_replies_discarded
generator_spacing
// hinting
missing_block
// bootstrap server
response
write_error
blocks
drop
response_blocks
response_account_info
channel_full
// backlog
activated
// active
insert
insert_failed
// unchecked
put
satisfied
trigger
// election scheduler
insert_manual
insert_priority
insert_priority_success
erase_oldest
// handshake
invalid_node_id
missing_cookie
invalid_genesis
// bootstrap ascending
missing_tag
reply
throttled
track
timeout
nothing_new
// bootstrap ascending connections
connect
reuse
// bootstrap ascending accounts
prioritize
prioritize_failed
block
unblock
unblock_failed
next_priority
next_database
next_none
blocking_insert
blocking_erase_overflow
priority_insert
priority_erase_threshold
priority_erase_block
priority_erase_overflow
deprioritize
deprioritize_failed
dir (direction) :
in
out
previous versions up to 24.0
type:
traffic_udp
traffic_tcp
error
message
block
ledger
rollback
bootstrap
vote
election
http_callback
peering
ipc
tcp
udp
confirmation_height
confirmation_observer
drop
aggregator
requests
filter
telemetry
details:
all
// error specific
bad_sender
insufficient_work
http_callback
unreachable_host
// confirmation_observer specific
active_quorum
active_conf_height
inactive_conf_height
// ledger block bootstrap
send
receive
open
change
state_block
epoch_block
fork
old
gap_previous
gap_source
// message specific
keepalive
publish
republish_vote
confirm_req
confirm_ack
node_id_handshake
telemetry_req
telemetry_ack
// bootstrap callback
initiate
initiate_lazy
initiate_wallet_lazy
// bootstrap specific
bulk_pull
bulk_pull_account
bulk_pull_deserialize_receive_block
bulk_pull_error_starting_request
bulk_pull_failed_account
bulk_pull_receive_block_failure
bulk_pull_request_failure
bulk_push
frontier_req
frontier_confirmation_failed
frontier_confirmation_successful
error_socket_close
// vote specific
vote_valid
vote_replay
vote_indeterminate
vote_invalid
vote_overflow
// election specific
vote_new
vote_cached
late_block
late_block_seconds
election_non_priority
election_priority
election_block_conflict
election_difficulty_update
election_drop
election_restart
// udp
blocking
overflow
invalid_magic
invalid_network
invalid_header
invalid_message_type
invalid_keepalive_message
invalid_publish_message
invalid_confirm_req_message
invalid_confirm_ack_message
invalid_node_id_handshake_message
invalid_telemetry_req_message
invalid_telemetry_ack_message
outdated_version
// tcp
tcp_accept_success
tcp_accept_failure
tcp_write_drop
tcp_write_no_socket_drop
tcp_excluded
// ipc
invocations
// peering
handshake
// confirmation height
blocks_confirmed
blocks_confirmed_unbounded
blocks_confirmed_bounded
invalid_block
// [request] aggregator
aggregator_accepted
aggregator_dropped
// requests
requests_cached_hashes
requests_generated_hashes
requests_cached_votes
requests_generated_votes
requests_cannot_vote
requests_unknown
// duplicate
duplicate_publish
// telemetry
invalid_signature
different_genesis_hash
node_id_mismatch
request_within_protection_cache_zone
no_response_received
unsolicited_telemetry_ack
failed_send_telemetry_req
dir (direction) :
in
out
RPC Command¶
Counters query:¶
{
"action": "stats",
"type": "counters"
}
Counters response¶
{
"type": "counters",
"created": "2018.03.29 01:46:36",
"entries": [
{
"time": "01:46:36",
"type": "traffic",
"detail": "all",
"dir": "in",
"value": "3122792"
},
{
"time": "01:46:36",
"type": "traffic",
"detail": "all",
"dir": "out",
"value": "203184"
},
{
"time": "01:46:36",
"type": "message",
"detail": "all",
"dir": "in",
"value": "12494"
},
{
"time": "01:46:36",
"type": "message",
"detail": "all",
"dir": "out",
"value": "1380"
},
{
"time": "01:46:36",
"type": "message",
"detail": "keepalive",
"dir": "in",
"value": "172"
},
...
]
}
Samples query:¶
{
"action": "stats",
"type": "samples"
}
Samples response¶
{
"type": "samples",
"created": "2018.03.29 01:47:08",
"entries": [
{
"time": "01:47:04",
"type": "traffic",
"detail": "all",
"dir": "in",
"value": "59480"
},
{
"time": "01:47:05",
"type": "traffic",
"detail": "all",
"dir": "in",
"value": "44496"
},
{
"time": "01:47:06",
"type": "traffic",
"detail": "all",
"dir": "in",
"value": "44136"
},
{
"time": "01:47:07",
"type": "traffic",
"detail": "all",
"dir": "in",
"value": "18784"
},
{
"time": "01:47:08",
"type": "traffic",
"detail": "all",
"dir": "in",
"value": "22680"
},
{
"time": "01:47:03",
"type": "traffic",
"detail": "all",
"dir": "out",
"value": "4128"
},
{
"time": "01:47:04",
"type": "message",
"detail": "all",
"dir": "out",
"value": "17"
},
{
"time": "01:47:05",
"type": "message",
"detail": "all",
"dir": "out",
"value": "10"
},
...
]
}
Log file example¶
counters.stat
As specified in the example config, sampling interval is 1 second, stats are logged every 5 seconds, and the file rotates after 5 log cycles.
counters,2018.03.29 01:45:36
01:44:56,bootstrap,all,out,1
01:45:36,bootstrap,initiate,out,2
counters,2018.03.29 01:45:41
01:45:41,traffic,all,in,456344
01:45:41,traffic,all,out,189520
01:45:41,message,all,in,1925
01:45:41,message,all,out,1289
01:45:38,message,keepalive,in,165
01:45:41,message,keepalive,out,1027
01:45:41,message,publish,in,34
01:45:38,message,confirm_req,in,164
01:45:41,message,confirm_req,out,262
01:45:41,message,confirm_ack,in,1562
01:45:36,bootstrap,all,out,2
01:45:41,bootstrap,initiate,out,3
samples.stat
As specified in the example config, logging is done every 5 seconds and the sampling capacity is 5 (how many samplings are kept)
samples,2018.03.29 01:45:36
01:45:36,bootstrap,initiate,out,2
samples,2018.03.29 01:45:41
01:45:37,traffic,all,in,322608
01:45:38,traffic,all,in,37064
01:45:39,traffic,all,in,38752
01:45:40,traffic,all,in,25632
01:45:38,traffic,all,out,185072
01:45:39,traffic,all,out,3072
01:45:41,traffic,all,out,920
01:45:37,message,all,in,1387
01:45:38,message,all,in,126
01:45:39,message,all,in,179
01:45:40,message,all,in,101
01:45:37,message,all,out,1254
01:45:38,message,all,out,10
01:45:39,message,all,out,16
01:45:41,message,all,out,6
01:45:38,message,keepalive,in,165
01:45:38,message,keepalive,out,1011
01:45:39,message,keepalive,out,12
01:45:41,message,keepalive,out,3
01:45:37,message,publish,in,19
01:45:38,message,publish,in,8
01:45:40,message,publish,in,3
01:45:41,message,publish,in,4
01:45:38,message,confirm_req,in,164
01:45:37,message,confirm_req,out,249
01:45:38,message,confirm_req,out,3
01:45:39,message,confirm_req,out,6
01:45:41,message,confirm_req,out,3
01:45:37,message,confirm_ack,in,1046
01:45:38,message,confirm_ack,in,141
01:45:39,message,confirm_ack,in,150
01:45:40,message,confirm_ack,in,100
01:45:36,bootstrap,all,out,2
01:45:36,bootstrap,initiate,out,2
01:45:41,bootstrap,initiate,out,1
Troubleshooting UPnP¶
Ensure UPnP is enabled¶
UPnP will be enabled unless the external port is set in either the config
[node]
...
# The external address of this node (NAT). If not set, the node will request this information via UPnP.
# type:string,ip
external_address = "::ffff:<some_public_ipv4>"
--config node.external_address="::ffff:<some_public_ipv4>"
Enable UPnP logging¶
Appending this to your launch command will enable upnp logging.
--config node.logging.upnp_details=true
Error UPnP Messages¶
Check the beginning of the logs for UPNP_* messages
Port Mapping Conflict
Check for static routes
[2019-Oct-29 11:06:56.641389]: UPnP failed 718: ConflictInMappingEntry
[2019-Oct-29 11:06:56.644387]: UPnP failed 718: ConflictInMappingEntry
Normal UPnP Messages¶
[2019-Oct-29 11:06:56.641389]: UPNP_GetSpecificPortMappingEntry failed 714: NoSuchEntryInArray
[2019-Oct-29 11:06:56.644387]: UPNP_GetSpecificPortMappingEntry failed 714: NoSuchEntryInArray