Category: Linux

Zenoss: Performance issues with too many events

When you have too many events in your zenoss environment the zenoss webinterface will be very sloooooooow. And you get all kind of errors:

  • Script don’t respond
  • Connection refused. Check zeneventserver status on deamons
  • A zenoss error has occurred

When you start top you see alot of java executables when you click on the Infrastructure zenoss button. Java sometimes take 350% CPU.

java -Djetty.host=localhost -server -XX:+HeapDumpOnOutOfMemoryError -DZENOSS_COMMAND=zeneventserver -DZENHOME=/opt/zenoss -Djetty.home=/opt/zenoss -Djetty.logs=/opt/zenoss/log -Dlogback.configurationFile=/opt/zenoss/etc/zeneventserver/logback.xml -Xmx1024m -DZENOSS_DAEMON=y -jar /opt/zenoss/lib/jetty-start-7.5.3.v20111011.jar --config=/opt/zenoss/etc/zeneventserver/jetty/start.config --ini=/opt/zenoss/etc/zeneventserver/jetty/jetty.ini --pre=etc/zeneventserver/jetty/jetty-logging.xml

I’ve read a lot of zenoss documentation on the internet but didn’t   found a nice article to get rid of all the events. So here is an article how I fixed it.

Basic steps:

  1. Backup Zenoss
  2. Stop zenoss
  3. Create a new zeneventserver database
  4. Remove zeneventserver content
  5. Restore the zenoss backup
  6. Start zenoss
  7. Enjoy your fast zenoss 🙂

Detailed steps:

  • ssh zenoss host
  • Switch zenoss user

# su zenoss

  • Create backup:

$ /opt/zenoss/bin/zenbackup -v10

  • Stop Zenoss service

$ zenoss stop

  • edit  the zeneventserver script

nano /opt/zenoss/bin/zeneventserver-create-db

  • Search for root and add the root password

root-password

  • Run the script

$ zeneventserver-create-db --force --dbtype=mysql

  • Clear the zeneventserver folder

rm -rf $ZENHOME/var/zeneventserver/*

Now some tricky part. Zenoss change some MySQL passwords when you do a restore.  This result in a access denied for user [email protected] during a restore. There is a fix for this problem. Reset the [email protected]’localhost’ and [email protected]’%’ MySQL passwords before you do a restore.

First get the current mysql from the global.conf file (yellow). This password is the password you need for the restore.

$nano /opt/zenoss/etc/global.conf

global.conf

Tip: Too check the password (encrypted). You can do the same after you change the password:

$ mysql -uroot -p
mysql> select * from mysql.user;

mysql-encrypted

Now reset the password

SET PASSWORD FOR 'zenoss'@'localhost' = PASSWORD('BEagPxxxxxxxxxxxxxxx');
SET PASSWORD FOR 'zenoss'@'%' = PASSWORD('BEagPxxxxxxxxxxxxxxx');

When you check the permissions now you see another encryption:

$ mysql -uroot -p
mysql> select * from mysql.user;

(I don’t have an image example because this is an production enviroment)

Optional: To check the zenoss user permissions:

mysql> SELECT user, host, db, select_priv, insert_priv, grant_priv FROM mysql.db;

mysql-permissions

Optional: When you still have errors or the above rights ain’t good try these two MySQL scripts:

mysql> CREATE USER 'zenoss'@'%' IDENTIFIED BY 'some_pass';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zenoss'@'localhost'
-> WITH GRANT OPTION;

mysql> CREATE USER 'zenoss'@'%' IDENTIFIED BY 'some_pass';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zenoss'@'localhost'
-> WITH GRANT OPTION;

Ok, now everting is set do a restore. The -v stands for verbose and with the no-eventsdb you don’t restore all the events. That’s exactly what we want

zenrestore --file=/opt/zenoss/backups/zenbackup_2014013 -v --no-eventsdb

Now start zenoss

$zenoss start

That’s it.  Enjoy the performance and set some parameters that your events ain’t that big any more in the future.

Source: http://www.nickyeates.com/technology/zenoss/useful_commands_zenoss4
Source: http://community.zenoss.org/docs/DOC-3048

IPtables howto

I copy/paste this howto from the debian wiki @ https://wiki.debian.org/iptables and add this to my blog to for fast finding the best iptables article on the net.

Iptables provides packet filtering, network address translation (NAT) and other packet mangling.

Two of the most common uses of iptables is to provide firewall support and NAT.

Configuring iptables manually is challenging for the uninitiated. Fortunately, there are many configuration tools (wizards) available to assist: e.g., fwbuilder, bastille, ferm (wiki page), ufw (Uncomplicated Firewall, from Ubuntu).

Viewing current configuration

See what rules are already configured. Issue this command:

 iptables -L

The output will be similar to this:

 Chain INPUT (policy ACCEPT)
 target     prot opt source               destination

 Chain FORWARD (policy ACCEPT)
 target     prot opt source               destination

 Chain OUTPUT (policy ACCEPT)
 target     prot opt source               destination

This allows anyone access to anything from anywhere.

Storing iptables rules in a file

Note: there is a package designed to help with this: iptables-persistent

Let’s tighten that up a bit by creating a test iptables file:

 nano /etc/iptables.test.rules

In this file enter some basic rules:

*filter

# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allows all outbound traffic
# You could modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

# Allows SSH connections 
# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
-A INPUT -p tcp -m state --state NEW --dport 30000 -j ACCEPT

# Now you should read up on iptables rules and consider whether ssh access 
# for everyone is really desired. Most likely you will only allow access from certain IPs.

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls (access via 'dmesg' command)
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy:
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

That may look complicated, but look at each section at a time. You will see that it simply shuts all ports except the ones we have allowed – which in this case are ports 80 and 443 (the standard web browser ports) and the SSH port defined earlier.

Activate these new rules:

 iptables-restore < /etc/iptables.test.rules

And see the difference:

 iptables -L

Now the output tells us that only the ports defined above are open. All the others are closed.

Once you are happy, save the new rules to the master iptables file:

 iptables-save > /etc/iptables.up.rules

To make sure the iptables rules are started on a reboot we’ll create a new file:

 nano /etc/network/if-pre-up.d/iptables

or in the the if-up.d if the if-pre-up.d not work

nano /etc/network/if-up.d/iptables

Add these lines to it:

 #!/bin/bash
 /sbin/iptables-restore < /etc/iptables.up.rules

The file needs to be executable so change the permissions:

 chmod +x /etc/network/if-pre-up.d/iptables

Show spam script on linux webserver

Sometimes a website on a shared server is compromised by a hacker. Most of the time they use the website to spam the WWW around. It can be hard to find the specific script who is responsible for the spam. Here I will describe a method you can use to find and eliminate the script.

First stop postfix to stop the mail flood

# service stop postfix

Now get the mailque and write down the message id (yellow)

# mailq

mailq.png

Get the message header

# postcat -vq A9CC9182699 |more

message header

Now see the yellow text above. You have find the culprit. Now find the file location:

# find /var/www/web/ -name *addwp.php*

Eliminate the process and patch the website.

Delete the queue (USE THIS WITH CAUTION!!!!)

# postsuper -d ALL

And start postfix

# service start postfix

To find and kill a spamming perl script try this:

Find the process:

lsof -i :25
readlink -f /proc/{PID of process}/exe
kill PID

Check and delete the (sendmail) QUEUE:

cd /var/spool/mqueue/
ls 
nano queue file
rm -f *

It can be very difficult to find a specific perl spam script because the path is not always visible. To find the script do a locate and isolate/rename the files.

locate *.pl |grep /var/www/

For more commands see my blog article: http://blog.wapnet.nl/2014/04/linux-find-out-what-a-process-is-doing/

I found another nice tool to find suspicious scripts: http://cbl.abuseat.org/findbot.pl

Zenoss Backups

Create a zenoss user cron

sudo crontab -u zenoss -e

Use this lines to backup every day 10 AM.

HISTFILE=/home/zenoss/.bash_history
HOME=/home/zenoss
INSTANCE_HOME=/opt/zenoss
LD_LIBRARY_PATH=/opt/zenoss/lib
LOGNAME=zenoss
MAIL=/var/spool/mail/zenoss
PATH=/opt/zenoss/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/zenoss/bin
PWD=/home/zenoss
PYTHONPATH=/opt/zenoss/lib/python
USER=zenoss
ZENHOME=/opt/zenoss
30 10 * * * /opt/zenoss/bin/zenbackup -v10

Note: When I not put the variables to the cron I get this error:
ERROR: $ZENHOME is not set.
This is usually caused by executing this command as root rather than as the zenoss user. Either define $ZENHOME or run this command as a different user.

Because I can’t find a nice solution on the internet I fix it to add the SET lines to the cron.

Create a cleanup job

sudo crontab -e

Use this line to cleanup the backups every sunday 12 AM

00 12 * * 7 root /usr/bin/find /opt/zenoss/backups -mtime +30 -type f -exec rm \{\} \;

Crunch – wordlist / password generator for brute force

Crunch is a wordlist generator where you can specify a standard character set or a character set you specify. crunch can generate all possible combinations and permutations.

Features

  • crunch generates wordlists in both combination and permutation ways
  • it can breakup output by number of lines or file size
  • now has resume support
  • pattern now supports number and symbols
  • pattern now supports upper and lower case characters separately
  • adds a status report when generating multiple files
  • new -l option for literal support of @,%^
  • new -d option to limit duplicate characters see man file for details
  • now has unicode support

See the sourceforge project: crunch – wordlist generator