Strange Issue

Scrutinizer is an enterprise/business class NetFlow and sFlow analysis tool. Scrutinizer provides historical trends of the company's critical network interfaces as well as the details on:

Who: The end system causing the traffic
What: The application/protocol that is being used
When: The time frame it has been occurring for
Where: The network connection that is affected

Moderators: scottr, Moderator Team

Forum rules
Post Reply
yozh
Posts: 8
Joined: Mon Sep 28, 2015 10:02 am

Strange Issue

Post by yozh » Mon Sep 28, 2015 10:06 am

Hello,

I have a copy of scrutinizer running for a while now and last time I checked all was working fine. I needed to verify some IP acctivity yesterday, went to the web gui, and not able to display any reports. I keep getting errors that there is no data, or it cant communicate with the collector.

I see in apache logs this:

Code: Select all

[Sun Sep 27 10:18:47.438827 2015] [fcgid:warn] [pid 4128:tid 860] [client 192.168.1.209:61713] mod_fcgid: stderr: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Sun Sep 27 10:18:47.438827 2015] [fcgid:warn] [pid 4128:tid 860] [client 192.168.1.209:61713] mod_fcgid: stderr: Run Died of: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Sun Sep 27 10:18:47.438827 2015] [core:error] [pid 4128:tid 860] [client 192.168.1.209:61713] End of script output before headers: scrut_fcgi.fcgi
[Sun Sep 27 10:18:47.501344 2015] [fcgid:warn] [pid 4128:tid 872] [client 192.168.1.104:37396] mod_fcgid: stderr: HTTP Asyn Response non-200 from 192.168.1.209: 500, referer: http://192.168.1.209/
[Sun Sep 27 10:18:52.798212 2015] [fcgid:warn] [pid 4128:tid 860] [client 192.168.1.209:61715] mod_fcgid: stderr: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Sun Sep 27 10:18:52.798212 2015] [fcgid:warn] [pid 4128:tid 860] [client 192.168.1.209:61715] mod_fcgid: stderr: Run Died of: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Sun Sep 27 10:18:52.798212 2015] [core:error] [pid 4128:tid 860] [client 192.168.1.209:61715] End of script output before headers: scrut_fcgi.fcgi
[Sun Sep 27 10:18:52.813873 2015] [fcgid:warn] [pid 4128:tid 844] [client 192.168.1.104:37401] mod_fcgid: stderr: HTTP Asyn Response non-200 from 192.168.1.209: 500, referer: http://192.168.1.209/
[Sun Sep 27 10:19:18.454509 2015] [fcgid:warn] [pid 4128:tid 860] [client 192.168.1.209:61722] mod_fcgid: stderr: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Sun Sep 27 10:19:18.454509 2015] [fcgid:warn] [pid 4128:tid 860] [client 192.168.1.209:61722] mod_fcgid: stderr: Run Died of: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Sun Sep 27 10:19:18.454509 2015] [core:error] [pid 4128:tid 860] [client 192.168.1.209:61722] End of script output before headers: scrut_fcgi.fcgi
[Sun Sep 27 10:19:18.470159 2015] [fcgid:warn] [pid 4128:tid 872] [client 192.168.1.104:37396] mod_fcgid: stderr: HTTP Asyn Response non-200 from 192.168.1.209: 500, referer: http://192.168.1.209/
[Sun Sep 27 17:17:56.460010 2015] [cgi:error] [pid 4128:tid 844] [client 192.168.1.104:40392] AH01215: readline() on closed filehandle UA at /<C:\\Program Files\\Scrutinizer\\cgi-bin\\admin.cgi>Plixer/Scrutinizer/legacy/recordkeeping.pm line 317.: C:/Program Files/Scrutinizer/cgi-bin/admin.cgi, referer: http://192.168.1.209/index.html?
[Mon Sep 28 09:51:58.082319 2015] [fcgid:warn] [pid 4128:tid 876] [client 192.168.1.209:50867] mod_fcgid: stderr: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Mon Sep 28 09:51:58.113582 2015] [fcgid:warn] [pid 4128:tid 824] [client 192.168.1.209:50868] mod_fcgid: stderr: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Mon Sep 28 09:51:58.129207 2015] [fcgid:warn] [pid 4128:tid 876] [client 192.168.1.209:50867] mod_fcgid: stderr: Run Died of: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Mon Sep 28 09:51:58.129207 2015] [fcgid:warn] [pid 4128:tid 824] [client 192.168.1.209:50868] mod_fcgid: stderr: Run Died of: DBD::mysql::db selectrow_array failed: Incorrect file format 'rollcall' at /<C:\\Program Files\\Scrutinizer\\html\\fcgi\\scrut_fcgi.fcgi>Plixer/Scrutinizer/reporting/tempTable.pm line 506.
[Mon Sep 28 09:51:58.129207 2015] [core:error] [pid 4128:tid 876] [client 192.168.1.209:50867] End of script output before headers: scrut_fcgi.fcgi
[Mon Sep 28 09:51:58.129207 2015] [core:error] [pid 4128:tid 824] [client 192.168.1.209:50868] End of script output before headers: scrut_fcgi.fcgi
[Mon Sep 28 09:51:58.222953 2015] [fcgid:warn] [pid 4128:tid 828] [client 192.168.1.104:41177] mod_fcgid: stderr: HTTP Asyn Response non-200 from 192.168.1.209: 500, referer: http://192.168.1.209/
[Mon Sep 28 09:51:58.222953 2015] [fcgid:warn] [pid 4128:tid 828] [client 192.168.1.104:41177] mod_fcgid: stderr: HTTP Asyn Response non-200 from 192.168.1.209: 500, referer: http://192.168.1.209/
When this issues started occurring, I upgraded to the latest release available, but still getting this errors. Anyone can help please ?
Top
ryans
Posts: 25
Joined: Mon Aug 04, 2014 10:47 am

Re: Strange Issue

Post by ryans » Mon Sep 28, 2015 10:19 am

Hello,

It looks the be an issue with crashed mysql tables. Try running these commands:

'mysqlcheck -uroot -p --auto-repair --all-databases'

After that one, log into mysql with 'mysql -uroot -p' and run this command:

'repair table plixer.rollcall use_frm;'

Let me know what kind of output you get from this.

~Ryan
Top
yozh
Posts: 8
Joined: Mon Sep 28, 2015 10:02 am

Re: Strange Issue

Post by yozh » Mon Sep 28, 2015 10:22 am

ryans wrote:Hello,

It looks the be an issue with crashed mysql tables. Try running these commands:

'mysqlcheck -uroot -p --auto-repair --all-databases'

After that one, log into mysql with 'mysql -uroot -p' and run this command:

'repair table plixer.rollcall use_frm;'

Let me know what kind of output you get from this.

~Ryan

Thank you for an extremely fast reply. After running first command this is the results I got:

Code: Select all

Repairing tables
mysql.innodb_index_stats
Error    : Unknown storage engine 'InnoDB'
error    : Corrupt
mysql.innodb_table_stats
Error    : Unknown storage engine 'InnoDB'
error    : Corrupt
mysql.slave_master_info
Error    : Unknown storage engine 'InnoDB'
error    : Corrupt
mysql.slave_relay_log_info
Error    : Unknown storage engine 'InnoDB'
error    : Corrupt
mysql.slave_worker_info
Error    : Unknown storage engine 'InnoDB'
error    : Corrupt
plixer.rollcall
Error    : Incorrect file format 'rollcall'
error    : Corrupt
Top
ryans
Posts: 25
Joined: Mon Aug 04, 2014 10:47 am

Re: Strange Issue

Post by ryans » Mon Sep 28, 2015 10:33 am

That looks ok. Now run the next and i think you should be up and running again.
Top
yozh
Posts: 8
Joined: Mon Sep 28, 2015 10:02 am

Re: Strange Issue

Post by yozh » Mon Sep 28, 2015 10:49 am

ryans wrote:Hello,

It looks the be an issue with crashed mysql tables. Try running these commands:

'mysqlcheck -uroot -p --auto-repair --all-databases'

After that one, log into mysql with 'mysql -uroot -p' and run this command:

'repair table plixer.rollcall use_frm;'

Let me know what kind of output you get from this.

~Ryan
Looking good !
Top
yozh
Posts: 8
Joined: Mon Sep 28, 2015 10:02 am

Re: Strange Issue

Post by yozh » Mon Sep 28, 2015 11:00 am

ryans wrote:That looks ok. Now run the next and i think you should be up and running again.
Actually maybe spoke a little too soon. I dont think I`m getting any historical data now.... I can view last few minutes, but no historical data is coming up.
Top
ryans
Posts: 25
Joined: Mon Aug 04, 2014 10:47 am

Re: Strange Issue

Post by ryans » Mon Sep 28, 2015 11:05 am

The 'rollcall' table is where we take data that needs to be rolled from 1 -> 5 minute averages, 5 -> 30 minute and so on up the chain. It will take some time for it to catch up .I would let it run for about an hour or so and see if you have historical data.
Top
yozh
Posts: 8
Joined: Mon Sep 28, 2015 10:02 am

Re: Strange Issue

Post by yozh » Mon Sep 28, 2015 11:07 am

ryans wrote:The 'rollcall' table is where we take data that needs to be rolled from 1 -> 5 minute averages, 5 -> 30 minute and so on up the chain. It will take some time for it to catch up .I would let it run for about an hour or so and see if you have historical data.
Ah great... I`ll let it run... I need some data from this saturday, trying to find 1 flow... I hope its there....

Anyway to monitor for this type of corruption ?
Top
Post Reply

Return to “Scrutinizer”

Who is online

Users browsing this forum: No registered users and 5 guests